I download lots of media files. So far I have been storing these files after I am done with them on a 2TB hard disk. I have been copying over the files with rsync. This has so far worked fairly well. However the hard disk I am using is starting to get close to full. Now I am needing to find a solution so I can span my files over multiple disks. If I were to continue to do it as I do now, I would end up copying over files that would already be on the other disk. Does the datahoading community have any solutions to this?
For more information, my system is using Linux. The 2TB drive is formatted with ext4. When I make the backup to the drive I use ’rsync -rutp’. I don’t use multiple disks at the same time due to having only one usb sata enclosure for 3 1/2 inch disks. I don’t keep the drive connected all the time due to not needing it all the time. I keep local copies until I am done with the files (and they are backed up).
Welcome! Without buying more enclosures and increasing the number of drives you can access at one time, you will need to partition your files based on your own use case and maintain an index so that you easily can retrieve the right drive when you need to access data. Perhaps you get a drive for each year. Perhaps images go to one and video to another. perhaps you split on the file name. For an index, this can be as simple as labeling the drives and putting them on your shelf. As mentioned by others, there are software solutions for indexing file metadata as well.
If you buy more enclosures you can use MergerFS or another union file system to bring both disks together and provide a single view while using ext4 for each drive. This allows you to easily remove a single drive and plug it into another basic linux distro, but you will not get any data striping or other protections. So if 1 drive dies, you will loose whatever data was stored on that disk. Because of that, I advise you to still think about partitioning your files even if you union them so that you understand your failure scope.