mjhjmtu

I'm copying terabytes worth of data from an OLD OLD OLD *nix system. I could spend the money to buy new hardware to plug the drives into but I want to avoid spending money.

The system is crappy. It freezes and crashes a lot. So I have to restart the copy process every time. And it takes a long time for it to "catch up" because it has to "skip" the files that it already copied.

I could use rsync but the system is so slow that every time I start an rsync that initial inventory scan/compare takes a while.

So then I thought I could generate a one-time-list of all the files that need to be copied and save them to a file. Then I could have a script that goes through the file and copies that file then deletes that line. If the script/copy fails in the middle I could just continue from where it left off. I don't know how to handle/process the part that would delete the line from the queue but while trying to figure that out I realized there must be some publicly available tool or script that already does what I need.

So the goal is that it would go through some input queue of files to copy and copy them and, as it finishes a copy, it would remove it from the queue. I don't need it for this particular use-case but it should be able to accommodate new entries being added to the queue. And, of course, the queue should be file based so it can be continued from in case of a system error/issue.

Anyone know of one?