How to recursively copy only the files from folders and subfolders?

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP












1















Under /home folder we have many subfolders, as the following:



/home/user1
/home/user2/user_sub_2
/home/user3/user_sub_3/info_sub
/home/user4/INFO_FOLDER


We want to copy all the files under /home, recursively, to the /tmp/calculation folder. All files should be placed directly into the target folder (no subdirectories should be created). If two or more files have the same name, then the most recently modified file should be copied to /tmp/calculation.



What is the right approach to do this action?










share|improve this question
























  • I tag rsync because I think rsync is one of the options

    – yael
    Feb 25 at 20:27















1















Under /home folder we have many subfolders, as the following:



/home/user1
/home/user2/user_sub_2
/home/user3/user_sub_3/info_sub
/home/user4/INFO_FOLDER


We want to copy all the files under /home, recursively, to the /tmp/calculation folder. All files should be placed directly into the target folder (no subdirectories should be created). If two or more files have the same name, then the most recently modified file should be copied to /tmp/calculation.



What is the right approach to do this action?










share|improve this question
























  • I tag rsync because I think rsync is one of the options

    – yael
    Feb 25 at 20:27













1












1








1








Under /home folder we have many subfolders, as the following:



/home/user1
/home/user2/user_sub_2
/home/user3/user_sub_3/info_sub
/home/user4/INFO_FOLDER


We want to copy all the files under /home, recursively, to the /tmp/calculation folder. All files should be placed directly into the target folder (no subdirectories should be created). If two or more files have the same name, then the most recently modified file should be copied to /tmp/calculation.



What is the right approach to do this action?










share|improve this question
















Under /home folder we have many subfolders, as the following:



/home/user1
/home/user2/user_sub_2
/home/user3/user_sub_3/info_sub
/home/user4/INFO_FOLDER


We want to copy all the files under /home, recursively, to the /tmp/calculation folder. All files should be placed directly into the target folder (no subdirectories should be created). If two or more files have the same name, then the most recently modified file should be copied to /tmp/calculation.



What is the right approach to do this action?







linux bash shell-script find rsync






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Feb 26 at 8:05









Kusalananda

137k17258426




137k17258426










asked Feb 25 at 20:04









yaelyael

2,74222576




2,74222576












  • I tag rsync because I think rsync is one of the options

    – yael
    Feb 25 at 20:27

















  • I tag rsync because I think rsync is one of the options

    – yael
    Feb 25 at 20:27
















I tag rsync because I think rsync is one of the options

– yael
Feb 25 at 20:27





I tag rsync because I think rsync is one of the options

– yael
Feb 25 at 20:27










2 Answers
2






active

oldest

votes


















4














find /home ! -type d -exec bash -c '
for pathname do
if [ "$pathname" -nt "/tmp/calculation/$pathname##*/" ]
then
cp "$pathname" /tmp/calculation
fi
done' bash +


This would find all non-directory files under /home, and for batches of these it would call a short bash script.



The short bash script would loop over the current batch of pathnames, and for each would test with the -nt test whether current file is newer than the copy in the target directory (or whether a copy does not exist there). If the file in the target directory is older or if it does not exist, cp is used to copy the current file to the target directory.



The parameter expansion $pathname##*/ would remove any directory path before the actual filename, leaving only the filename portion of the pathname. It could be replaced by $(basename "$pathname").



Related:



  • Understanding the -exec option of `find`


Mostly unrelated:



The -nt test is a non-standard test. This is why I chose to use bash for the internal script that find calls. Using sh -c instead of bash -c would probably have worked, but the semantics of the test may differ slightly between shell that may masquerade as sh.



For example, in the bash, zsh and ksh shells, the -nt test is true if the first operand has a modification timestamp that in newer than that of the second operand, or if the second operand does not exist.



In the dash shell, however, both files most exist and the first file has to be newer than the second for the test to be true (according to the documentation). This difference would not have been an issue in this case.



In the yash shell it's not specified in the manual what happens if either file does not exist.



It is therefore safest to use a specific shell when using a non-standard facility, even if it, in this specific case, would probably have worked with sh -c anyway.



(The downside with using bash in this instance is that it only has a one second resolution in the timestamps that it compares, but that's another story)






share|improve this answer

























  • why you use - -type d , this searched only folders

    – yael
    Feb 25 at 20:58











  • @yael I used ! -type d not -type d. I did this because you never said anything about what types of files you wanted to copy, so instead of -type f I used ! -type d to copy any type of non-directory file.

    – Kusalananda
    Feb 25 at 21:00












  • ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?

    – yael
    Feb 25 at 21:02











  • @yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?

    – Kusalananda
    Feb 25 at 21:04


















0














I tried with below command and it worked fine



for file in `find /home -type f | awk -F "/" 'print $NF'`
do
for dir in `find . -type d| sed "s/.//g"| sed 's/$///g'`
do
sudo find $dir -type f -iname $file | awk 'print "ls -ltrh" " " $0'
done | sh | sed -n '$p'
done | awk 'print "cp -rvfp" " " $NF " " "/destinationlocationtocopy/"' | sh





share|improve this answer

























  • Could you describe your code?

    – Kusalananda
    Feb 26 at 9:37











  • @Kusalananda i missed the copy statement same has been added now

    – Praveen Kumar BS
    Feb 26 at 10:23











  • Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?

    – Kusalananda
    Feb 26 at 10:29












  • i want to find the recent file . if there is any duplicate files i would like to use only recent file only

    – Praveen Kumar BS
    Feb 26 at 10:48










Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f502979%2fhow-to-recursively-copy-only-the-files-from-folders-and-subfolders%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























2 Answers
2






active

oldest

votes








2 Answers
2






active

oldest

votes









active

oldest

votes






active

oldest

votes









4














find /home ! -type d -exec bash -c '
for pathname do
if [ "$pathname" -nt "/tmp/calculation/$pathname##*/" ]
then
cp "$pathname" /tmp/calculation
fi
done' bash +


This would find all non-directory files under /home, and for batches of these it would call a short bash script.



The short bash script would loop over the current batch of pathnames, and for each would test with the -nt test whether current file is newer than the copy in the target directory (or whether a copy does not exist there). If the file in the target directory is older or if it does not exist, cp is used to copy the current file to the target directory.



The parameter expansion $pathname##*/ would remove any directory path before the actual filename, leaving only the filename portion of the pathname. It could be replaced by $(basename "$pathname").



Related:



  • Understanding the -exec option of `find`


Mostly unrelated:



The -nt test is a non-standard test. This is why I chose to use bash for the internal script that find calls. Using sh -c instead of bash -c would probably have worked, but the semantics of the test may differ slightly between shell that may masquerade as sh.



For example, in the bash, zsh and ksh shells, the -nt test is true if the first operand has a modification timestamp that in newer than that of the second operand, or if the second operand does not exist.



In the dash shell, however, both files most exist and the first file has to be newer than the second for the test to be true (according to the documentation). This difference would not have been an issue in this case.



In the yash shell it's not specified in the manual what happens if either file does not exist.



It is therefore safest to use a specific shell when using a non-standard facility, even if it, in this specific case, would probably have worked with sh -c anyway.



(The downside with using bash in this instance is that it only has a one second resolution in the timestamps that it compares, but that's another story)






share|improve this answer

























  • why you use - -type d , this searched only folders

    – yael
    Feb 25 at 20:58











  • @yael I used ! -type d not -type d. I did this because you never said anything about what types of files you wanted to copy, so instead of -type f I used ! -type d to copy any type of non-directory file.

    – Kusalananda
    Feb 25 at 21:00












  • ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?

    – yael
    Feb 25 at 21:02











  • @yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?

    – Kusalananda
    Feb 25 at 21:04















4














find /home ! -type d -exec bash -c '
for pathname do
if [ "$pathname" -nt "/tmp/calculation/$pathname##*/" ]
then
cp "$pathname" /tmp/calculation
fi
done' bash +


This would find all non-directory files under /home, and for batches of these it would call a short bash script.



The short bash script would loop over the current batch of pathnames, and for each would test with the -nt test whether current file is newer than the copy in the target directory (or whether a copy does not exist there). If the file in the target directory is older or if it does not exist, cp is used to copy the current file to the target directory.



The parameter expansion $pathname##*/ would remove any directory path before the actual filename, leaving only the filename portion of the pathname. It could be replaced by $(basename "$pathname").



Related:



  • Understanding the -exec option of `find`


Mostly unrelated:



The -nt test is a non-standard test. This is why I chose to use bash for the internal script that find calls. Using sh -c instead of bash -c would probably have worked, but the semantics of the test may differ slightly between shell that may masquerade as sh.



For example, in the bash, zsh and ksh shells, the -nt test is true if the first operand has a modification timestamp that in newer than that of the second operand, or if the second operand does not exist.



In the dash shell, however, both files most exist and the first file has to be newer than the second for the test to be true (according to the documentation). This difference would not have been an issue in this case.



In the yash shell it's not specified in the manual what happens if either file does not exist.



It is therefore safest to use a specific shell when using a non-standard facility, even if it, in this specific case, would probably have worked with sh -c anyway.



(The downside with using bash in this instance is that it only has a one second resolution in the timestamps that it compares, but that's another story)






share|improve this answer

























  • why you use - -type d , this searched only folders

    – yael
    Feb 25 at 20:58











  • @yael I used ! -type d not -type d. I did this because you never said anything about what types of files you wanted to copy, so instead of -type f I used ! -type d to copy any type of non-directory file.

    – Kusalananda
    Feb 25 at 21:00












  • ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?

    – yael
    Feb 25 at 21:02











  • @yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?

    – Kusalananda
    Feb 25 at 21:04













4












4








4







find /home ! -type d -exec bash -c '
for pathname do
if [ "$pathname" -nt "/tmp/calculation/$pathname##*/" ]
then
cp "$pathname" /tmp/calculation
fi
done' bash +


This would find all non-directory files under /home, and for batches of these it would call a short bash script.



The short bash script would loop over the current batch of pathnames, and for each would test with the -nt test whether current file is newer than the copy in the target directory (or whether a copy does not exist there). If the file in the target directory is older or if it does not exist, cp is used to copy the current file to the target directory.



The parameter expansion $pathname##*/ would remove any directory path before the actual filename, leaving only the filename portion of the pathname. It could be replaced by $(basename "$pathname").



Related:



  • Understanding the -exec option of `find`


Mostly unrelated:



The -nt test is a non-standard test. This is why I chose to use bash for the internal script that find calls. Using sh -c instead of bash -c would probably have worked, but the semantics of the test may differ slightly between shell that may masquerade as sh.



For example, in the bash, zsh and ksh shells, the -nt test is true if the first operand has a modification timestamp that in newer than that of the second operand, or if the second operand does not exist.



In the dash shell, however, both files most exist and the first file has to be newer than the second for the test to be true (according to the documentation). This difference would not have been an issue in this case.



In the yash shell it's not specified in the manual what happens if either file does not exist.



It is therefore safest to use a specific shell when using a non-standard facility, even if it, in this specific case, would probably have worked with sh -c anyway.



(The downside with using bash in this instance is that it only has a one second resolution in the timestamps that it compares, but that's another story)






share|improve this answer















find /home ! -type d -exec bash -c '
for pathname do
if [ "$pathname" -nt "/tmp/calculation/$pathname##*/" ]
then
cp "$pathname" /tmp/calculation
fi
done' bash +


This would find all non-directory files under /home, and for batches of these it would call a short bash script.



The short bash script would loop over the current batch of pathnames, and for each would test with the -nt test whether current file is newer than the copy in the target directory (or whether a copy does not exist there). If the file in the target directory is older or if it does not exist, cp is used to copy the current file to the target directory.



The parameter expansion $pathname##*/ would remove any directory path before the actual filename, leaving only the filename portion of the pathname. It could be replaced by $(basename "$pathname").



Related:



  • Understanding the -exec option of `find`


Mostly unrelated:



The -nt test is a non-standard test. This is why I chose to use bash for the internal script that find calls. Using sh -c instead of bash -c would probably have worked, but the semantics of the test may differ slightly between shell that may masquerade as sh.



For example, in the bash, zsh and ksh shells, the -nt test is true if the first operand has a modification timestamp that in newer than that of the second operand, or if the second operand does not exist.



In the dash shell, however, both files most exist and the first file has to be newer than the second for the test to be true (according to the documentation). This difference would not have been an issue in this case.



In the yash shell it's not specified in the manual what happens if either file does not exist.



It is therefore safest to use a specific shell when using a non-standard facility, even if it, in this specific case, would probably have worked with sh -c anyway.



(The downside with using bash in this instance is that it only has a one second resolution in the timestamps that it compares, but that's another story)







share|improve this answer














share|improve this answer



share|improve this answer








edited Feb 26 at 8:12

























answered Feb 25 at 20:54









KusalanandaKusalananda

137k17258426




137k17258426












  • why you use - -type d , this searched only folders

    – yael
    Feb 25 at 20:58











  • @yael I used ! -type d not -type d. I did this because you never said anything about what types of files you wanted to copy, so instead of -type f I used ! -type d to copy any type of non-directory file.

    – Kusalananda
    Feb 25 at 21:00












  • ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?

    – yael
    Feb 25 at 21:02











  • @yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?

    – Kusalananda
    Feb 25 at 21:04

















  • why you use - -type d , this searched only folders

    – yael
    Feb 25 at 20:58











  • @yael I used ! -type d not -type d. I did this because you never said anything about what types of files you wanted to copy, so instead of -type f I used ! -type d to copy any type of non-directory file.

    – Kusalananda
    Feb 25 at 21:00












  • ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?

    – yael
    Feb 25 at 21:02











  • @yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?

    – Kusalananda
    Feb 25 at 21:04
















why you use - -type d , this searched only folders

– yael
Feb 25 at 20:58





why you use - -type d , this searched only folders

– yael
Feb 25 at 20:58













@yael I used ! -type d not -type d. I did this because you never said anything about what types of files you wanted to copy, so instead of -type f I used ! -type d to copy any type of non-directory file.

– Kusalananda
Feb 25 at 21:00






@yael I used ! -type d not -type d. I did this because you never said anything about what types of files you wanted to copy, so instead of -type f I used ! -type d to copy any type of non-directory file.

– Kusalananda
Feb 25 at 21:00














ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?

– yael
Feb 25 at 21:02





ok , lets say - /tmp/calculation folder is empty folder , in that case can you add the right syntax for this case?

– yael
Feb 25 at 21:02













@yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?

– Kusalananda
Feb 25 at 21:04





@yael I don't understand what you mean by your last comment. If the folder is empty, it should copy files there. Does it not do that for you?

– Kusalananda
Feb 25 at 21:04













0














I tried with below command and it worked fine



for file in `find /home -type f | awk -F "/" 'print $NF'`
do
for dir in `find . -type d| sed "s/.//g"| sed 's/$///g'`
do
sudo find $dir -type f -iname $file | awk 'print "ls -ltrh" " " $0'
done | sh | sed -n '$p'
done | awk 'print "cp -rvfp" " " $NF " " "/destinationlocationtocopy/"' | sh





share|improve this answer

























  • Could you describe your code?

    – Kusalananda
    Feb 26 at 9:37











  • @Kusalananda i missed the copy statement same has been added now

    – Praveen Kumar BS
    Feb 26 at 10:23











  • Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?

    – Kusalananda
    Feb 26 at 10:29












  • i want to find the recent file . if there is any duplicate files i would like to use only recent file only

    – Praveen Kumar BS
    Feb 26 at 10:48















0














I tried with below command and it worked fine



for file in `find /home -type f | awk -F "/" 'print $NF'`
do
for dir in `find . -type d| sed "s/.//g"| sed 's/$///g'`
do
sudo find $dir -type f -iname $file | awk 'print "ls -ltrh" " " $0'
done | sh | sed -n '$p'
done | awk 'print "cp -rvfp" " " $NF " " "/destinationlocationtocopy/"' | sh





share|improve this answer

























  • Could you describe your code?

    – Kusalananda
    Feb 26 at 9:37











  • @Kusalananda i missed the copy statement same has been added now

    – Praveen Kumar BS
    Feb 26 at 10:23











  • Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?

    – Kusalananda
    Feb 26 at 10:29












  • i want to find the recent file . if there is any duplicate files i would like to use only recent file only

    – Praveen Kumar BS
    Feb 26 at 10:48













0












0








0







I tried with below command and it worked fine



for file in `find /home -type f | awk -F "/" 'print $NF'`
do
for dir in `find . -type d| sed "s/.//g"| sed 's/$///g'`
do
sudo find $dir -type f -iname $file | awk 'print "ls -ltrh" " " $0'
done | sh | sed -n '$p'
done | awk 'print "cp -rvfp" " " $NF " " "/destinationlocationtocopy/"' | sh





share|improve this answer















I tried with below command and it worked fine



for file in `find /home -type f | awk -F "/" 'print $NF'`
do
for dir in `find . -type d| sed "s/.//g"| sed 's/$///g'`
do
sudo find $dir -type f -iname $file | awk 'print "ls -ltrh" " " $0'
done | sh | sed -n '$p'
done | awk 'print "cp -rvfp" " " $NF " " "/destinationlocationtocopy/"' | sh






share|improve this answer














share|improve this answer



share|improve this answer








edited Feb 26 at 10:27









Kusalananda

137k17258426




137k17258426










answered Feb 26 at 9:33









Praveen Kumar BSPraveen Kumar BS

1,6471311




1,6471311












  • Could you describe your code?

    – Kusalananda
    Feb 26 at 9:37











  • @Kusalananda i missed the copy statement same has been added now

    – Praveen Kumar BS
    Feb 26 at 10:23











  • Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?

    – Kusalananda
    Feb 26 at 10:29












  • i want to find the recent file . if there is any duplicate files i would like to use only recent file only

    – Praveen Kumar BS
    Feb 26 at 10:48

















  • Could you describe your code?

    – Kusalananda
    Feb 26 at 9:37











  • @Kusalananda i missed the copy statement same has been added now

    – Praveen Kumar BS
    Feb 26 at 10:23











  • Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?

    – Kusalananda
    Feb 26 at 10:29












  • i want to find the recent file . if there is any duplicate files i would like to use only recent file only

    – Praveen Kumar BS
    Feb 26 at 10:48
















Could you describe your code?

– Kusalananda
Feb 26 at 9:37





Could you describe your code?

– Kusalananda
Feb 26 at 9:37













@Kusalananda i missed the copy statement same has been added now

– Praveen Kumar BS
Feb 26 at 10:23





@Kusalananda i missed the copy statement same has been added now

– Praveen Kumar BS
Feb 26 at 10:23













Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?

– Kusalananda
Feb 26 at 10:29






Hmm... you find a file, and then you go looking for it in all directories under the current directory? I don't really see the point of the inner loop. If you've found a file, why not copy it?

– Kusalananda
Feb 26 at 10:29














i want to find the recent file . if there is any duplicate files i would like to use only recent file only

– Praveen Kumar BS
Feb 26 at 10:48





i want to find the recent file . if there is any duplicate files i would like to use only recent file only

– Praveen Kumar BS
Feb 26 at 10:48

















draft saved

draft discarded
















































Thanks for contributing an answer to Unix & Linux Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f502979%2fhow-to-recursively-copy-only-the-files-from-folders-and-subfolders%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown






Popular posts from this blog

How to check contact read email or not when send email to Individual?

Displaying single band from multi-band raster using QGIS

How many registers does an x86_64 CPU actually have?