How to archive into multiple zip a directory containing several millions sub directory?

I have large directory that contains several millions+ sub-dir, each sub-dir contain 3 or less image files. What is the most efficient way for me to zip them so I can move it to a new server?

The sub-dir are 5 letter words compose of random letters + numbers.

For ex:

/foo/re63d/image.jpg, /foo/re63d/image2.jpg

/foo/4sf3g/image.jpg, /foo/4sf3g/image2.jpg

/foo/vj33s/image.jpg, /foo/vj33s/image2.jpg

and so on several millions times

I was thinking of using shell script to zip/tar them alphabetically. For ex every folders starting with

00* zip them into 0_0.zip

01* zip them into 0_1.zip

a0* into a_0.zip

a1* into a_1.zip

aa* into a_a.zip

This way it's easier for me to manage them. I want to be able to choose what sub section I want to zip at a time like aa* to af* as I want to be able control how long each process run. How would I go about doing this?

edited Dec 22 '18 at 11:18

Rui F Ribeiro

39.1k1479130

asked Dec 21 '18 at 23:30

Perz

4

Not a real answer, but if what you want is to reliably move a lot of files to another server, I'd say rsync would be a better tool.
– nxnev
Dec 21 '18 at 23:34

I like the ability to see what I transfer. How much should I be worry about the integrity of the transfers if there's a broken connection?
– Perz
Dec 21 '18 at 23:43

1

@nxnev I would say it constitutes an answer because it's better for OPs use-case (It's not what he/she wants, but what he/she needs.) Perz: rsync is rock solid and will continue where it left off when there is a broken connection and has a compression algorithm built-in
– Fabby
Dec 21 '18 at 23:48

Thank you. I will have a look at rsync then.
– Perz
Dec 21 '18 at 23:52

What do you mean by efficient: Smallest size, least cpu time, least effort my you?
– ctrl-alt-delor
Dec 22 '18 at 12:45

|
show 3 more comments

I have large directory that contains several millions+ sub-dir, each sub-dir contain 3 or less image files. What is the most efficient way for me to zip them so I can move it to a new server?

The sub-dir are 5 letter words compose of random letters + numbers.

For ex:

/foo/re63d/image.jpg, /foo/re63d/image2.jpg

/foo/4sf3g/image.jpg, /foo/4sf3g/image2.jpg

/foo/vj33s/image.jpg, /foo/vj33s/image2.jpg

and so on several millions times

edited Dec 22 '18 at 11:18

Rui F Ribeiro

39.1k1479130

asked Dec 21 '18 at 23:30

Perz

4

Not a real answer, but if what you want is to reliably move a lot of files to another server, I'd say rsync would be a better tool.
– nxnev
Dec 21 '18 at 23:34

I like the ability to see what I transfer. How much should I be worry about the integrity of the transfers if there's a broken connection?
– Perz
Dec 21 '18 at 23:43

1

@nxnev I would say it constitutes an answer because it's better for OPs use-case (It's not what he/she wants, but what he/she needs.) Perz: rsync is rock solid and will continue where it left off when there is a broken connection and has a compression algorithm built-in
– Fabby
Dec 21 '18 at 23:48

Thank you. I will have a look at rsync then.
– Perz
Dec 21 '18 at 23:52

What do you mean by efficient: Smallest size, least cpu time, least effort my you?
– ctrl-alt-delor
Dec 22 '18 at 12:45

|
show 3 more comments

I have large directory that contains several millions+ sub-dir, each sub-dir contain 3 or less image files. What is the most efficient way for me to zip them so I can move it to a new server?

The sub-dir are 5 letter words compose of random letters + numbers.

For ex:

/foo/re63d/image.jpg, /foo/re63d/image2.jpg

/foo/4sf3g/image.jpg, /foo/4sf3g/image2.jpg

/foo/vj33s/image.jpg, /foo/vj33s/image2.jpg

and so on several millions times

edited Dec 22 '18 at 11:18

Rui F Ribeiro

39.1k1479130

asked Dec 21 '18 at 23:30

Perz

I have large directory that contains several millions+ sub-dir, each sub-dir contain 3 or less image files. What is the most efficient way for me to zip them so I can move it to a new server?

The sub-dir are 5 letter words compose of random letters + numbers.

For ex:

/foo/re63d/image.jpg, /foo/re63d/image2.jpg

/foo/4sf3g/image.jpg, /foo/4sf3g/image2.jpg

/foo/vj33s/image.jpg, /foo/vj33s/image2.jpg

and so on several millions times

bash shell-script tar zip

edited Dec 22 '18 at 11:18

Rui F Ribeiro

39.1k1479130

asked Dec 21 '18 at 23:30

Perz

edited Dec 22 '18 at 11:18

Rui F Ribeiro

39.1k1479130

asked Dec 21 '18 at 23:30

Perz

edited Dec 22 '18 at 11:18

Rui F Ribeiro

39.1k1479130

edited Dec 22 '18 at 11:18

Rui F Ribeiro

39.1k1479130

edited Dec 22 '18 at 11:18

Rui F Ribeiro

39.1k1479130

asked Dec 21 '18 at 23:30

Perz

asked Dec 21 '18 at 23:30

Perz

asked Dec 21 '18 at 23:30

Perz

4

Not a real answer, but if what you want is to reliably move a lot of files to another server, I'd say rsync would be a better tool.
– nxnev
Dec 21 '18 at 23:34

I like the ability to see what I transfer. How much should I be worry about the integrity of the transfers if there's a broken connection?
– Perz
Dec 21 '18 at 23:43

1

@nxnev I would say it constitutes an answer because it's better for OPs use-case (It's not what he/she wants, but what he/she needs.) Perz: rsync is rock solid and will continue where it left off when there is a broken connection and has a compression algorithm built-in
– Fabby
Dec 21 '18 at 23:48

Thank you. I will have a look at rsync then.
– Perz
Dec 21 '18 at 23:52

What do you mean by efficient: Smallest size, least cpu time, least effort my you?
– ctrl-alt-delor
Dec 22 '18 at 12:45

|
show 3 more comments

4

Not a real answer, but if what you want is to reliably move a lot of files to another server, I'd say rsync would be a better tool.
– nxnev
Dec 21 '18 at 23:34

I like the ability to see what I transfer. How much should I be worry about the integrity of the transfers if there's a broken connection?
– Perz
Dec 21 '18 at 23:43

1

@nxnev I would say it constitutes an answer because it's better for OPs use-case (It's not what he/she wants, but what he/she needs.) Perz: rsync is rock solid and will continue where it left off when there is a broken connection and has a compression algorithm built-in
– Fabby
Dec 21 '18 at 23:48

Thank you. I will have a look at rsync then.
– Perz
Dec 21 '18 at 23:52

What do you mean by efficient: Smallest size, least cpu time, least effort my you?
– ctrl-alt-delor
Dec 22 '18 at 12:45

Not a real answer, but if what you want is to reliably move a lot of files to another server, I'd say rsync would be a better tool.
– nxnev
Dec 21 '18 at 23:34

I like the ability to see what I transfer. How much should I be worry about the integrity of the transfers if there's a broken connection?
– Perz
Dec 21 '18 at 23:43

@nxnev I would say it constitutes an answer because it's better for OPs use-case (It's not what he/she wants, but what he/she needs.) Perz: rsync is rock solid and will continue where it left off when there is a broken connection and has a compression algorithm built-in
– Fabby
Dec 21 '18 at 23:48

Thank you. I will have a look at rsync then.
– Perz
Dec 21 '18 at 23:52

What do you mean by efficient: Smallest size, least cpu time, least effort my you?
– ctrl-alt-delor
Dec 22 '18 at 12:45

|
show 3 more comments

0

active

oldest

votes

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f490420%2fhow-to-archive-into-multiple-zip-a-directory-containing-several-millions-sub-dir%23new-answer', 'question_page');

);

Post as a guest

Name

Required, but never shown

0

active

oldest

votes

0

active

oldest

votes

draft saved

draft discarded

Thanks for contributing an answer to Unix & Linux Stack Exchange!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

搜尋此網誌

mjhjmtu