Equally divide contents of a directory between disks - rounded up to the nearest top level directory
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
I have a Music directory, and an MP3 player with two SD card slots. The contents of the directory is too big for one card, so I'd like to split it over two.
I like to have stuff in alphabetical order. If I filled up SD1 and left SD2 partly empty, whenever I add new music I'd either have to break the alphabetical order by putting new stuff on SD2, or re-jig everything around.
The other thing is that my directories are grouped by Artist, so if I got a new album by an artist on SD1 it wouldn't fit.
In short, I want the two cards filled up roughly equally.
Right now my solution is actually OK:
- List the directories and output the list to a file
- Open the file and find the halfway point (I'm sure there's a clever way to do this but it's quick to do manually in vim)
- Cut the bottom half of the list and paste it to a second file
- Run
rsync --files-from=using one file on SD1 and the next on SD2. Job done!
This got me thinking though, is there a more accurate way to do this so that the two SD cards get filled more evenly? How would you go about this?
I guess I would have to maybe
- Run
duon the Music dir to find thetotalSize. - Then loop through every Artist dir to get their sizes and keep a running total.
- Once that total surpasses the
totalSize * 0.5, output a list of all the dirs you just counted to get the first half and put the rest in a second file to get the other half? - Do the same rsync from-file
Anyone got any better ideas?
shell-script rsync
asked Apr 18 at 16:04
benoliver999
32
add a comment |Â
up vote
0
down vote
favorite
I have a Music directory, and an MP3 player with two SD card slots. The contents of the directory is too big for one card, so I'd like to split it over two.
I like to have stuff in alphabetical order. If I filled up SD1 and left SD2 partly empty, whenever I add new music I'd either have to break the alphabetical order by putting new stuff on SD2, or re-jig everything around.
The other thing is that my directories are grouped by Artist, so if I got a new album by an artist on SD1 it wouldn't fit.
In short, I want the two cards filled up roughly equally.
Right now my solution is actually OK:
- List the directories and output the list to a file
- Open the file and find the halfway point (I'm sure there's a clever way to do this but it's quick to do manually in vim)
- Cut the bottom half of the list and paste it to a second file
- Run
rsync --files-from=using one file on SD1 and the next on SD2. Job done!
This got me thinking though, is there a more accurate way to do this so that the two SD cards get filled more evenly? How would you go about this?
I guess I would have to maybe
- Run
duon the Music dir to find thetotalSize. - Then loop through every Artist dir to get their sizes and keep a running total.
- Once that total surpasses the
totalSize * 0.5, output a list of all the dirs you just counted to get the first half and put the rest in a second file to get the other half? - Do the same rsync from-file
Anyone got any better ideas?
shell-script rsync
asked Apr 18 at 16:04
benoliver999
32
1
Is there anything preventing you from getting larger SD cards?
– Mioriin
Apr 18 at 16:21
@Mioriin They are 400GB each, my collection is quite large.
– benoliver999
Apr 18 at 16:26
1
Perhaps you should start sorting through what music you need readily available and keep the rest on larger storage somewhere (2-4TiB HDDs are fairly cheap these days).
– Mioriin
Apr 18 at 16:31
@Mioriin I have it all stored on a server at home, that's not the issue. My SD cards are plenty big enough to take the whole lot so I don't see why I wouldn't, I was just wondering if anyone had any bright ideas about splitting it up in a smart way.
– benoliver999
Apr 18 at 16:38
add a comment |Â
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have a Music directory, and an MP3 player with two SD card slots. The contents of the directory is too big for one card, so I'd like to split it over two.
I like to have stuff in alphabetical order. If I filled up SD1 and left SD2 partly empty, whenever I add new music I'd either have to break the alphabetical order by putting new stuff on SD2, or re-jig everything around.
The other thing is that my directories are grouped by Artist, so if I got a new album by an artist on SD1 it wouldn't fit.
In short, I want the two cards filled up roughly equally.
Right now my solution is actually OK:
- List the directories and output the list to a file
- Open the file and find the halfway point (I'm sure there's a clever way to do this but it's quick to do manually in vim)
- Cut the bottom half of the list and paste it to a second file
- Run
rsync --files-from=using one file on SD1 and the next on SD2. Job done!
This got me thinking though, is there a more accurate way to do this so that the two SD cards get filled more evenly? How would you go about this?
I guess I would have to maybe
- Run
duon the Music dir to find thetotalSize. - Then loop through every Artist dir to get their sizes and keep a running total.
- Once that total surpasses the
totalSize * 0.5, output a list of all the dirs you just counted to get the first half and put the rest in a second file to get the other half? - Do the same rsync from-file
Anyone got any better ideas?
shell-script rsync
asked Apr 18 at 16:04
benoliver999
32
I have a Music directory, and an MP3 player with two SD card slots. The contents of the directory is too big for one card, so I'd like to split it over two.
I like to have stuff in alphabetical order. If I filled up SD1 and left SD2 partly empty, whenever I add new music I'd either have to break the alphabetical order by putting new stuff on SD2, or re-jig everything around.
The other thing is that my directories are grouped by Artist, so if I got a new album by an artist on SD1 it wouldn't fit.
In short, I want the two cards filled up roughly equally.
Right now my solution is actually OK:
- List the directories and output the list to a file
- Open the file and find the halfway point (I'm sure there's a clever way to do this but it's quick to do manually in vim)
- Cut the bottom half of the list and paste it to a second file
- Run
rsync --files-from=using one file on SD1 and the next on SD2. Job done!
This got me thinking though, is there a more accurate way to do this so that the two SD cards get filled more evenly? How would you go about this?
I guess I would have to maybe
- Run
duon the Music dir to find thetotalSize. - Then loop through every Artist dir to get their sizes and keep a running total.
- Once that total surpasses the
totalSize * 0.5, output a list of all the dirs you just counted to get the first half and put the rest in a second file to get the other half? - Do the same rsync from-file
Anyone got any better ideas?
shell-script rsync
asked Apr 18 at 16:04
benoliver999
32
asked Apr 18 at 16:04
benoliver999
32
asked Apr 18 at 16:04
benoliver999
32
asked Apr 18 at 16:04
benoliver999
32
32
1
Is there anything preventing you from getting larger SD cards?
– Mioriin
Apr 18 at 16:21
@Mioriin They are 400GB each, my collection is quite large.
– benoliver999
Apr 18 at 16:26
1
Perhaps you should start sorting through what music you need readily available and keep the rest on larger storage somewhere (2-4TiB HDDs are fairly cheap these days).
– Mioriin
Apr 18 at 16:31
@Mioriin I have it all stored on a server at home, that's not the issue. My SD cards are plenty big enough to take the whole lot so I don't see why I wouldn't, I was just wondering if anyone had any bright ideas about splitting it up in a smart way.
– benoliver999
Apr 18 at 16:38
add a comment |Â
1
Is there anything preventing you from getting larger SD cards?
– Mioriin
Apr 18 at 16:21
@Mioriin They are 400GB each, my collection is quite large.
– benoliver999
Apr 18 at 16:26
1
Perhaps you should start sorting through what music you need readily available and keep the rest on larger storage somewhere (2-4TiB HDDs are fairly cheap these days).
– Mioriin
Apr 18 at 16:31
@Mioriin I have it all stored on a server at home, that's not the issue. My SD cards are plenty big enough to take the whole lot so I don't see why I wouldn't, I was just wondering if anyone had any bright ideas about splitting it up in a smart way.
– benoliver999
Apr 18 at 16:38
1
1
Is there anything preventing you from getting larger SD cards?
– Mioriin
Apr 18 at 16:21
Is there anything preventing you from getting larger SD cards?
– Mioriin
Apr 18 at 16:21
@Mioriin They are 400GB each, my collection is quite large.
– benoliver999
Apr 18 at 16:26
@Mioriin They are 400GB each, my collection is quite large.
– benoliver999
Apr 18 at 16:26
1
1
Perhaps you should start sorting through what music you need readily available and keep the rest on larger storage somewhere (2-4TiB HDDs are fairly cheap these days).
– Mioriin
Apr 18 at 16:31
Perhaps you should start sorting through what music you need readily available and keep the rest on larger storage somewhere (2-4TiB HDDs are fairly cheap these days).
– Mioriin
Apr 18 at 16:31
@Mioriin I have it all stored on a server at home, that's not the issue. My SD cards are plenty big enough to take the whole lot so I don't see why I wouldn't, I was just wondering if anyone had any bright ideas about splitting it up in a smart way.
– benoliver999
Apr 18 at 16:38
@Mioriin I have it all stored on a server at home, that's not the issue. My SD cards are plenty big enough to take the whole lot so I don't see why I wouldn't, I was just wondering if anyone had any bright ideas about splitting it up in a smart way.
– benoliver999
Apr 18 at 16:38
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
0
down vote
accepted
So, somewhat ironically, this would be really easy to do if those two SD cards where two separate computers on the same network, as there are all kinds of cluster storage solutions that do exactly what you're asking.
Given that that's not a very realistic option however (using a clustered filesystem is overkill for this), your best bet is probably a program called mhddfs. The official homepage is in Russian, but there's a decent post here about it, and it's packaged a number of different distributions. In essence, it does file-level RAID. The big advantage here is that once you get the mhddfs mount set up, you can literally just call rsync once. Each file will end up entirely on exactly one filesystem, and it will intelligently manage distribution across the filesystems. Additionally, it stores no state on the underlying devices (it just looks at the data distribution across the devices to figure out where to put any newly created files, and properly checks all of them for existing files), so it won't waste any space either.
answered Apr 18 at 19:45
Austin Hemmelgarn
5,104915
Wow this is exactly what I need. Very clever stuff! Sort of like a RAID array but one that lets you use the drives individually afterwards - I like it! The only thing it doesn't do is keep directories together (from what I can tell?) so at the split between drive 1 and 2 there might be some duplication in directory names? Doesn't matter though because if this works it would be so easy to use that it's really not an issue anymore.
– benoliver999
Apr 19 at 6:17
Ah the only catch, which to be fair I didn't mention in my post, is that I would need both SDs connected at once! I think I have a second SD reader somewhere but for people in the future reading, bear this in mind.
– benoliver999
Apr 19 at 6:22
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
So, somewhat ironically, this would be really easy to do if those two SD cards where two separate computers on the same network, as there are all kinds of cluster storage solutions that do exactly what you're asking.
Given that that's not a very realistic option however (using a clustered filesystem is overkill for this), your best bet is probably a program called mhddfs. The official homepage is in Russian, but there's a decent post here about it, and it's packaged a number of different distributions. In essence, it does file-level RAID. The big advantage here is that once you get the mhddfs mount set up, you can literally just call rsync once. Each file will end up entirely on exactly one filesystem, and it will intelligently manage distribution across the filesystems. Additionally, it stores no state on the underlying devices (it just looks at the data distribution across the devices to figure out where to put any newly created files, and properly checks all of them for existing files), so it won't waste any space either.
answered Apr 18 at 19:45
Austin Hemmelgarn
5,104915
Wow this is exactly what I need. Very clever stuff! Sort of like a RAID array but one that lets you use the drives individually afterwards - I like it! The only thing it doesn't do is keep directories together (from what I can tell?) so at the split between drive 1 and 2 there might be some duplication in directory names? Doesn't matter though because if this works it would be so easy to use that it's really not an issue anymore.
– benoliver999
Apr 19 at 6:17
Ah the only catch, which to be fair I didn't mention in my post, is that I would need both SDs connected at once! I think I have a second SD reader somewhere but for people in the future reading, bear this in mind.
– benoliver999
Apr 19 at 6:22
add a comment |Â
up vote
0
down vote
accepted
So, somewhat ironically, this would be really easy to do if those two SD cards where two separate computers on the same network, as there are all kinds of cluster storage solutions that do exactly what you're asking.
Given that that's not a very realistic option however (using a clustered filesystem is overkill for this), your best bet is probably a program called mhddfs. The official homepage is in Russian, but there's a decent post here about it, and it's packaged a number of different distributions. In essence, it does file-level RAID. The big advantage here is that once you get the mhddfs mount set up, you can literally just call rsync once. Each file will end up entirely on exactly one filesystem, and it will intelligently manage distribution across the filesystems. Additionally, it stores no state on the underlying devices (it just looks at the data distribution across the devices to figure out where to put any newly created files, and properly checks all of them for existing files), so it won't waste any space either.
answered Apr 18 at 19:45
Austin Hemmelgarn
5,104915
Wow this is exactly what I need. Very clever stuff! Sort of like a RAID array but one that lets you use the drives individually afterwards - I like it! The only thing it doesn't do is keep directories together (from what I can tell?) so at the split between drive 1 and 2 there might be some duplication in directory names? Doesn't matter though because if this works it would be so easy to use that it's really not an issue anymore.
– benoliver999
Apr 19 at 6:17
Ah the only catch, which to be fair I didn't mention in my post, is that I would need both SDs connected at once! I think I have a second SD reader somewhere but for people in the future reading, bear this in mind.
– benoliver999
Apr 19 at 6:22
add a comment |Â
up vote
0
down vote
accepted
up vote
0
down vote
accepted
So, somewhat ironically, this would be really easy to do if those two SD cards where two separate computers on the same network, as there are all kinds of cluster storage solutions that do exactly what you're asking.
Given that that's not a very realistic option however (using a clustered filesystem is overkill for this), your best bet is probably a program called mhddfs. The official homepage is in Russian, but there's a decent post here about it, and it's packaged a number of different distributions. In essence, it does file-level RAID. The big advantage here is that once you get the mhddfs mount set up, you can literally just call rsync once. Each file will end up entirely on exactly one filesystem, and it will intelligently manage distribution across the filesystems. Additionally, it stores no state on the underlying devices (it just looks at the data distribution across the devices to figure out where to put any newly created files, and properly checks all of them for existing files), so it won't waste any space either.
answered Apr 18 at 19:45
Austin Hemmelgarn
5,104915
So, somewhat ironically, this would be really easy to do if those two SD cards where two separate computers on the same network, as there are all kinds of cluster storage solutions that do exactly what you're asking.
Given that that's not a very realistic option however (using a clustered filesystem is overkill for this), your best bet is probably a program called mhddfs. The official homepage is in Russian, but there's a decent post here about it, and it's packaged a number of different distributions. In essence, it does file-level RAID. The big advantage here is that once you get the mhddfs mount set up, you can literally just call rsync once. Each file will end up entirely on exactly one filesystem, and it will intelligently manage distribution across the filesystems. Additionally, it stores no state on the underlying devices (it just looks at the data distribution across the devices to figure out where to put any newly created files, and properly checks all of them for existing files), so it won't waste any space either.
answered Apr 18 at 19:45
Austin Hemmelgarn
5,104915
answered Apr 18 at 19:45
Austin Hemmelgarn
5,104915
answered Apr 18 at 19:45
Austin Hemmelgarn
5,104915
answered Apr 18 at 19:45
Austin Hemmelgarn
5,104915
5,104915
Wow this is exactly what I need. Very clever stuff! Sort of like a RAID array but one that lets you use the drives individually afterwards - I like it! The only thing it doesn't do is keep directories together (from what I can tell?) so at the split between drive 1 and 2 there might be some duplication in directory names? Doesn't matter though because if this works it would be so easy to use that it's really not an issue anymore.
– benoliver999
Apr 19 at 6:17
Ah the only catch, which to be fair I didn't mention in my post, is that I would need both SDs connected at once! I think I have a second SD reader somewhere but for people in the future reading, bear this in mind.
– benoliver999
Apr 19 at 6:22
add a comment |Â
Wow this is exactly what I need. Very clever stuff! Sort of like a RAID array but one that lets you use the drives individually afterwards - I like it! The only thing it doesn't do is keep directories together (from what I can tell?) so at the split between drive 1 and 2 there might be some duplication in directory names? Doesn't matter though because if this works it would be so easy to use that it's really not an issue anymore.
– benoliver999
Apr 19 at 6:17
Ah the only catch, which to be fair I didn't mention in my post, is that I would need both SDs connected at once! I think I have a second SD reader somewhere but for people in the future reading, bear this in mind.
– benoliver999
Apr 19 at 6:22
Wow this is exactly what I need. Very clever stuff! Sort of like a RAID array but one that lets you use the drives individually afterwards - I like it! The only thing it doesn't do is keep directories together (from what I can tell?) so at the split between drive 1 and 2 there might be some duplication in directory names? Doesn't matter though because if this works it would be so easy to use that it's really not an issue anymore.
– benoliver999
Apr 19 at 6:17
Wow this is exactly what I need. Very clever stuff! Sort of like a RAID array but one that lets you use the drives individually afterwards - I like it! The only thing it doesn't do is keep directories together (from what I can tell?) so at the split between drive 1 and 2 there might be some duplication in directory names? Doesn't matter though because if this works it would be so easy to use that it's really not an issue anymore.
– benoliver999
Apr 19 at 6:17
Ah the only catch, which to be fair I didn't mention in my post, is that I would need both SDs connected at once! I think I have a second SD reader somewhere but for people in the future reading, bear this in mind.
– benoliver999
Apr 19 at 6:22
Ah the only catch, which to be fair I didn't mention in my post, is that I would need both SDs connected at once! I think I have a second SD reader somewhere but for people in the future reading, bear this in mind.
– benoliver999
Apr 19 at 6:22
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f438547%2fequally-divide-contents-of-a-directory-between-disks-rounded-up-to-the-nearest%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
1
Is there anything preventing you from getting larger SD cards?
– Mioriin
Apr 18 at 16:21
@Mioriin They are 400GB each, my collection is quite large.
– benoliver999
Apr 18 at 16:26
1
Perhaps you should start sorting through what music you need readily available and keep the rest on larger storage somewhere (2-4TiB HDDs are fairly cheap these days).
– Mioriin
Apr 18 at 16:31
@Mioriin I have it all stored on a server at home, that's not the issue. My SD cards are plenty big enough to take the whole lot so I don't see why I wouldn't, I was just wondering if anyone had any bright ideas about splitting it up in a smart way.
– benoliver999
Apr 18 at 16:38