How to recursively copy only the files from folders and subfolders?
Clash Royale CLAN TAG#URR8PPP
Under /home
folder we have many subfolders, as the following:
/home/user1
/home/user2/user_sub_2
/home/user3/user_sub_3/info_sub
/home/user4/INFO_FOLDER
We want to copy all the files under /home
, recursively, to the /tmp/calculation
folder. All files should be placed directly into the target folder (no subdirectories should be created). If two or more files have the same name, then the most recently modified file should be copied to /tmp/calculation
.
What is the right approach to do this action?
linux bash shell-script find rsync
add a comment |
Under /home
folder we have many subfolders, as the following:
/home/user1
/home/user2/user_sub_2
/home/user3/user_sub_3/info_sub
/home/user4/INFO_FOLDER
We want to copy all the files under /home
, recursively, to the /tmp/calculation
folder. All files should be placed directly into the target folder (no subdirectories should be created). If two or more files have the same name, then the most recently modified file should be copied to /tmp/calculation
.
What is the right approach to do this action?
linux bash shell-script find rsync
I tag rsync because I think rsync is one of the options
– yael
Feb 25 at 20:27
add a comment |
Under /home
folder we have many subfolders, as the following:
/home/user1
/home/user2/user_sub_2
/home/user3/user_sub_3/info_sub
/home/user4/INFO_FOLDER
We want to copy all the files under /home
, recursively, to the /tmp/calculation
folder. All files should be placed directly into the target folder (no subdirectories should be created). If two or more files have the same name, then the most recently modified file should be copied to /tmp/calculation
.
What is the right approach to do this action?
linux bash shell-script find rsync
Under /home
folder we have many subfolders, as the following:
/home/user1
/home/user2/user_sub_2
/home/user3/user_sub_3/info_sub
/home/user4/INFO_FOLDER
We want to copy all the files under /home
, recursively, to the /tmp/calculation
folder. All files should be placed directly into the target folder (no subdirectories should be created). If two or more files have the same name, then the most recently modified file should be copied to /tmp/calculation
.
What is the right approach to do this action?
linux bash shell-script find rsync
linux bash shell-script find rsync
edited Feb 26 at 8:05
Kusalananda
137k17258426
137k17258426
asked Feb 25 at 20:04
yaelyael
2,74222576
2,74222576
I tag rsync because I think rsync is one of the options
– yael
Feb 25 at 20:27
add a comment |
I tag rsync because I think rsync is one of the options
– yael
Feb 25 at 20:27
I tag rsync because I think rsync is one of the options
– yael
Feb 25 at 20:27
I tag rsync because I think rsync is one of the options
– yael
Feb 25 at 20:27
add a comment |
2 Answers
2
active
oldest
votes
find /home ! -type d -exec bash -c '
for pathname do
if [ "$pathname" -nt "/tmp/calculation/$pathname##*/" ]
then
cp "$pathname" /tmp/calculation
fi
done' bash +
This would find all non-directory files under /home
, and for batches of these it would call a short bash
script.
The short bash
script would loop over the current batch of pathnames, and for each would test with the -nt
test whether current file is newer than the copy in the target directory (or whether a copy does not exist there). If the file in the target directory is older or if it does not exist, cp
is used to copy the current file to the target directory.
The parameter expansion $pathname##*/
would remove any directory path before the actual filename, leaving only the filename portion of the pathname. It could be replaced by $(basename "$pathname")
.
Related:
- Understanding the -exec option of `find`
Mostly unrelated:
The -nt
test is a non-standard test. This is why I chose to use bash
for the internal script that find
calls. Using sh -c
instead of bash -c
would probably have worked, but the semantics of the test may differ slightly between shell that may masquerade as sh
.
For example, in the bash
, zsh
and ksh
shells, the -nt
test is true if the first operand has a modification timestamp that in newer than that of the second operand, or if the second operand does not exist.
In the dash
shell, however, both files most exist and the first file has to be newer than the second for the test to be true (according to the documentation). This difference would not have been an issue in this case.
In the yash
shell it's not specified in the manual what happens if either file does not exist.
It is therefore safest to use a specific shell when using a non-standard facility, even if it, in this specific case, would probably have worked with sh -c
anyway.
(The downside with using bash
in this instance is that it only has a one second resolution in the timestamps that it compares, but that's another story)
why you use - -type d , this searched only folders
– yael
Feb 25 at 20:58
@yael I used! -type d
not-type d
. I did this because you never said anything about what types of files you wanted to copy, so instead of-type f
I used! -type d
to copy any type of non-directory file.
– Kusalananda
Feb 25 at 21:00
ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?
– yael
Feb 25 at 21:02
@yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?
– Kusalananda
Feb 25 at 21:04
add a comment |
I tried with below command and it worked fine
for file in `find /home -type f | awk -F "/" 'print $NF'`
do
for dir in `find . -type d| sed "s/.//g"| sed 's/$///g'`
do
sudo find $dir -type f -iname $file | awk 'print "ls -ltrh" " " $0'
done | sh | sed -n '$p'
done | awk 'print "cp -rvfp" " " $NF " " "/destinationlocationtocopy/"' | sh
Could you describe your code?
– Kusalananda
Feb 26 at 9:37
@Kusalananda i missed the copy statement same has been added now
– Praveen Kumar BS
Feb 26 at 10:23
Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?
– Kusalananda
Feb 26 at 10:29
i want to find the recent file . if there is any duplicate files i would like to use only recent file only
– Praveen Kumar BS
Feb 26 at 10:48
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f502979%2fhow-to-recursively-copy-only-the-files-from-folders-and-subfolders%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
find /home ! -type d -exec bash -c '
for pathname do
if [ "$pathname" -nt "/tmp/calculation/$pathname##*/" ]
then
cp "$pathname" /tmp/calculation
fi
done' bash +
This would find all non-directory files under /home
, and for batches of these it would call a short bash
script.
The short bash
script would loop over the current batch of pathnames, and for each would test with the -nt
test whether current file is newer than the copy in the target directory (or whether a copy does not exist there). If the file in the target directory is older or if it does not exist, cp
is used to copy the current file to the target directory.
The parameter expansion $pathname##*/
would remove any directory path before the actual filename, leaving only the filename portion of the pathname. It could be replaced by $(basename "$pathname")
.
Related:
- Understanding the -exec option of `find`
Mostly unrelated:
The -nt
test is a non-standard test. This is why I chose to use bash
for the internal script that find
calls. Using sh -c
instead of bash -c
would probably have worked, but the semantics of the test may differ slightly between shell that may masquerade as sh
.
For example, in the bash
, zsh
and ksh
shells, the -nt
test is true if the first operand has a modification timestamp that in newer than that of the second operand, or if the second operand does not exist.
In the dash
shell, however, both files most exist and the first file has to be newer than the second for the test to be true (according to the documentation). This difference would not have been an issue in this case.
In the yash
shell it's not specified in the manual what happens if either file does not exist.
It is therefore safest to use a specific shell when using a non-standard facility, even if it, in this specific case, would probably have worked with sh -c
anyway.
(The downside with using bash
in this instance is that it only has a one second resolution in the timestamps that it compares, but that's another story)
why you use - -type d , this searched only folders
– yael
Feb 25 at 20:58
@yael I used! -type d
not-type d
. I did this because you never said anything about what types of files you wanted to copy, so instead of-type f
I used! -type d
to copy any type of non-directory file.
– Kusalananda
Feb 25 at 21:00
ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?
– yael
Feb 25 at 21:02
@yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?
– Kusalananda
Feb 25 at 21:04
add a comment |
find /home ! -type d -exec bash -c '
for pathname do
if [ "$pathname" -nt "/tmp/calculation/$pathname##*/" ]
then
cp "$pathname" /tmp/calculation
fi
done' bash +
This would find all non-directory files under /home
, and for batches of these it would call a short bash
script.
The short bash
script would loop over the current batch of pathnames, and for each would test with the -nt
test whether current file is newer than the copy in the target directory (or whether a copy does not exist there). If the file in the target directory is older or if it does not exist, cp
is used to copy the current file to the target directory.
The parameter expansion $pathname##*/
would remove any directory path before the actual filename, leaving only the filename portion of the pathname. It could be replaced by $(basename "$pathname")
.
Related:
- Understanding the -exec option of `find`
Mostly unrelated:
The -nt
test is a non-standard test. This is why I chose to use bash
for the internal script that find
calls. Using sh -c
instead of bash -c
would probably have worked, but the semantics of the test may differ slightly between shell that may masquerade as sh
.
For example, in the bash
, zsh
and ksh
shells, the -nt
test is true if the first operand has a modification timestamp that in newer than that of the second operand, or if the second operand does not exist.
In the dash
shell, however, both files most exist and the first file has to be newer than the second for the test to be true (according to the documentation). This difference would not have been an issue in this case.
In the yash
shell it's not specified in the manual what happens if either file does not exist.
It is therefore safest to use a specific shell when using a non-standard facility, even if it, in this specific case, would probably have worked with sh -c
anyway.
(The downside with using bash
in this instance is that it only has a one second resolution in the timestamps that it compares, but that's another story)
why you use - -type d , this searched only folders
– yael
Feb 25 at 20:58
@yael I used! -type d
not-type d
. I did this because you never said anything about what types of files you wanted to copy, so instead of-type f
I used! -type d
to copy any type of non-directory file.
– Kusalananda
Feb 25 at 21:00
ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?
– yael
Feb 25 at 21:02
@yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?
– Kusalananda
Feb 25 at 21:04
add a comment |
find /home ! -type d -exec bash -c '
for pathname do
if [ "$pathname" -nt "/tmp/calculation/$pathname##*/" ]
then
cp "$pathname" /tmp/calculation
fi
done' bash +
This would find all non-directory files under /home
, and for batches of these it would call a short bash
script.
The short bash
script would loop over the current batch of pathnames, and for each would test with the -nt
test whether current file is newer than the copy in the target directory (or whether a copy does not exist there). If the file in the target directory is older or if it does not exist, cp
is used to copy the current file to the target directory.
The parameter expansion $pathname##*/
would remove any directory path before the actual filename, leaving only the filename portion of the pathname. It could be replaced by $(basename "$pathname")
.
Related:
- Understanding the -exec option of `find`
Mostly unrelated:
The -nt
test is a non-standard test. This is why I chose to use bash
for the internal script that find
calls. Using sh -c
instead of bash -c
would probably have worked, but the semantics of the test may differ slightly between shell that may masquerade as sh
.
For example, in the bash
, zsh
and ksh
shells, the -nt
test is true if the first operand has a modification timestamp that in newer than that of the second operand, or if the second operand does not exist.
In the dash
shell, however, both files most exist and the first file has to be newer than the second for the test to be true (according to the documentation). This difference would not have been an issue in this case.
In the yash
shell it's not specified in the manual what happens if either file does not exist.
It is therefore safest to use a specific shell when using a non-standard facility, even if it, in this specific case, would probably have worked with sh -c
anyway.
(The downside with using bash
in this instance is that it only has a one second resolution in the timestamps that it compares, but that's another story)
find /home ! -type d -exec bash -c '
for pathname do
if [ "$pathname" -nt "/tmp/calculation/$pathname##*/" ]
then
cp "$pathname" /tmp/calculation
fi
done' bash +
This would find all non-directory files under /home
, and for batches of these it would call a short bash
script.
The short bash
script would loop over the current batch of pathnames, and for each would test with the -nt
test whether current file is newer than the copy in the target directory (or whether a copy does not exist there). If the file in the target directory is older or if it does not exist, cp
is used to copy the current file to the target directory.
The parameter expansion $pathname##*/
would remove any directory path before the actual filename, leaving only the filename portion of the pathname. It could be replaced by $(basename "$pathname")
.
Related:
- Understanding the -exec option of `find`
Mostly unrelated:
The -nt
test is a non-standard test. This is why I chose to use bash
for the internal script that find
calls. Using sh -c
instead of bash -c
would probably have worked, but the semantics of the test may differ slightly between shell that may masquerade as sh
.
For example, in the bash
, zsh
and ksh
shells, the -nt
test is true if the first operand has a modification timestamp that in newer than that of the second operand, or if the second operand does not exist.
In the dash
shell, however, both files most exist and the first file has to be newer than the second for the test to be true (according to the documentation). This difference would not have been an issue in this case.
In the yash
shell it's not specified in the manual what happens if either file does not exist.
It is therefore safest to use a specific shell when using a non-standard facility, even if it, in this specific case, would probably have worked with sh -c
anyway.
(The downside with using bash
in this instance is that it only has a one second resolution in the timestamps that it compares, but that's another story)
edited Feb 26 at 8:12
answered Feb 25 at 20:54
KusalanandaKusalananda
137k17258426
137k17258426
why you use - -type d , this searched only folders
– yael
Feb 25 at 20:58
@yael I used! -type d
not-type d
. I did this because you never said anything about what types of files you wanted to copy, so instead of-type f
I used! -type d
to copy any type of non-directory file.
– Kusalananda
Feb 25 at 21:00
ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?
– yael
Feb 25 at 21:02
@yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?
– Kusalananda
Feb 25 at 21:04
add a comment |
why you use - -type d , this searched only folders
– yael
Feb 25 at 20:58
@yael I used! -type d
not-type d
. I did this because you never said anything about what types of files you wanted to copy, so instead of-type f
I used! -type d
to copy any type of non-directory file.
– Kusalananda
Feb 25 at 21:00
ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?
– yael
Feb 25 at 21:02
@yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?
– Kusalananda
Feb 25 at 21:04
why you use - -type d , this searched only folders
– yael
Feb 25 at 20:58
why you use - -type d , this searched only folders
– yael
Feb 25 at 20:58
@yael I used
! -type d
not -type d
. I did this because you never said anything about what types of files you wanted to copy, so instead of -type f
I used ! -type d
to copy any type of non-directory file.– Kusalananda
Feb 25 at 21:00
@yael I used
! -type d
not -type d
. I did this because you never said anything about what types of files you wanted to copy, so instead of -type f
I used ! -type d
to copy any type of non-directory file.– Kusalananda
Feb 25 at 21:00
ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?
– yael
Feb 25 at 21:02
ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?
– yael
Feb 25 at 21:02
@yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?
– Kusalananda
Feb 25 at 21:04
@yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?
– Kusalananda
Feb 25 at 21:04
add a comment |
I tried with below command and it worked fine
for file in `find /home -type f | awk -F "/" 'print $NF'`
do
for dir in `find . -type d| sed "s/.//g"| sed 's/$///g'`
do
sudo find $dir -type f -iname $file | awk 'print "ls -ltrh" " " $0'
done | sh | sed -n '$p'
done | awk 'print "cp -rvfp" " " $NF " " "/destinationlocationtocopy/"' | sh
Could you describe your code?
– Kusalananda
Feb 26 at 9:37
@Kusalananda i missed the copy statement same has been added now
– Praveen Kumar BS
Feb 26 at 10:23
Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?
– Kusalananda
Feb 26 at 10:29
i want to find the recent file . if there is any duplicate files i would like to use only recent file only
– Praveen Kumar BS
Feb 26 at 10:48
add a comment |
I tried with below command and it worked fine
for file in `find /home -type f | awk -F "/" 'print $NF'`
do
for dir in `find . -type d| sed "s/.//g"| sed 's/$///g'`
do
sudo find $dir -type f -iname $file | awk 'print "ls -ltrh" " " $0'
done | sh | sed -n '$p'
done | awk 'print "cp -rvfp" " " $NF " " "/destinationlocationtocopy/"' | sh
Could you describe your code?
– Kusalananda
Feb 26 at 9:37
@Kusalananda i missed the copy statement same has been added now
– Praveen Kumar BS
Feb 26 at 10:23
Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?
– Kusalananda
Feb 26 at 10:29
i want to find the recent file . if there is any duplicate files i would like to use only recent file only
– Praveen Kumar BS
Feb 26 at 10:48
add a comment |
I tried with below command and it worked fine
for file in `find /home -type f | awk -F "/" 'print $NF'`
do
for dir in `find . -type d| sed "s/.//g"| sed 's/$///g'`
do
sudo find $dir -type f -iname $file | awk 'print "ls -ltrh" " " $0'
done | sh | sed -n '$p'
done | awk 'print "cp -rvfp" " " $NF " " "/destinationlocationtocopy/"' | sh
I tried with below command and it worked fine
for file in `find /home -type f | awk -F "/" 'print $NF'`
do
for dir in `find . -type d| sed "s/.//g"| sed 's/$///g'`
do
sudo find $dir -type f -iname $file | awk 'print "ls -ltrh" " " $0'
done | sh | sed -n '$p'
done | awk 'print "cp -rvfp" " " $NF " " "/destinationlocationtocopy/"' | sh
edited Feb 26 at 10:27
Kusalananda
137k17258426
137k17258426
answered Feb 26 at 9:33
Praveen Kumar BSPraveen Kumar BS
1,6471311
1,6471311
Could you describe your code?
– Kusalananda
Feb 26 at 9:37
@Kusalananda i missed the copy statement same has been added now
– Praveen Kumar BS
Feb 26 at 10:23
Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?
– Kusalananda
Feb 26 at 10:29
i want to find the recent file . if there is any duplicate files i would like to use only recent file only
– Praveen Kumar BS
Feb 26 at 10:48
add a comment |
Could you describe your code?
– Kusalananda
Feb 26 at 9:37
@Kusalananda i missed the copy statement same has been added now
– Praveen Kumar BS
Feb 26 at 10:23
Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?
– Kusalananda
Feb 26 at 10:29
i want to find the recent file . if there is any duplicate files i would like to use only recent file only
– Praveen Kumar BS
Feb 26 at 10:48
Could you describe your code?
– Kusalananda
Feb 26 at 9:37
Could you describe your code?
– Kusalananda
Feb 26 at 9:37
@Kusalananda i missed the copy statement same has been added now
– Praveen Kumar BS
Feb 26 at 10:23
@Kusalananda i missed the copy statement same has been added now
– Praveen Kumar BS
Feb 26 at 10:23
Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?
– Kusalananda
Feb 26 at 10:29
Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?
– Kusalananda
Feb 26 at 10:29
i want to find the recent file . if there is any duplicate files i would like to use only recent file only
– Praveen Kumar BS
Feb 26 at 10:48
i want to find the recent file . if there is any duplicate files i would like to use only recent file only
– Praveen Kumar BS
Feb 26 at 10:48
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f502979%2fhow-to-recursively-copy-only-the-files-from-folders-and-subfolders%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I tag rsync because I think rsync is one of the options
– yael
Feb 25 at 20:27