When designing our next-gen airborne LIDAR system, we intended to store our data on storage drives that were formatted with NTFS. This is the way our previous-gen systems work. Our systems run Linux (CentOS) but using NTFS hasn’t caused us any issues. It also allows our customers to remove the storage drives and connect them directly to a Windows PC for processing.
As our system collects data, it is divided up and written to multiple SATA drives (we do plan to move to removeable nvme drives once the motherboard manufactures catch up to the times). Two thirds of the way through development we encountered a massive problem. Although storing the data to NTFS worked great for our last-gen systems, CentOS couldn’t write to NTFS fast enough to handle the increased data flow of our new system. The only solution we could come up with at the time was to reformat the storage drives to a native Linux file system. We chose ext4 and got the write speeds we needed.
Problem 1: Our customer now has several terabytes of data on ext4 drives while the processing software runs on a Windows PC.
Our Current Solution: We have the customers install “Linux Reader” software that allows them to copy the data from the ext4 drives.
Other Solutions We’ve Tried: Having the customers use ext2fs software. Also, having them boot the processing PCs into Ubuntu, transferring the data, then rebooting back into Windows.
Problem 2: The solutions for problem 1 feel unprofessional. They also each have their own set of problems. For example, the Linux Reader software sometimes doesn’t see all of the files on the drives.
I’ve had a few additional ideas that I haven’t tried yet (such as running Ubuntu on a VM on the processing PCs to read the drives and copy the data). Its hard to justify continuing down this rabbit hole when we already have a solution that “works” (mostly). Its just that when the customer pays such a large sum for professional equipment, they shouldn’t have to deal with shoddy “workarounds” just to process their own data.
Does anyone here have any ideas or suggestions for a better solution?
