How to Copy Millions of Files from Remote Server to Local One [duplicate]

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP












0
















This question already has an answer here:



  • find and rsync?

    1 answer



I have this problem, I have an Ubuntu server (SSH enabled on port 4444). Inside this server there's a directory called /folder/a/, this directory literally contains millions of small size text files.



What i want to do is to find several thousands of files from those millions of files and copy or transfer those files to my local pc, i.e. using find command. When using scp or cpio it won't work properly.



Do you have any suggestion how to do this from my local pc, i.e not from running ssh terminal on my remote server.






linux shell ubuntu command-line cpio







share|improve this question















marked as duplicate by user1133275, Rui F Ribeiro linux
Users with the  linux badge can single-handedly close linux questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
 Feb 10 at 8:50


This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.


















  • It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

    – Emmanuel Rosa
    Feb 10 at 5:20











  • i know but, i want to do something like this (from local pc) ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000" but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

    – xcode
    Feb 10 at 5:26















0
















This question already has an answer here:



  • find and rsync?

    1 answer



I have this problem, I have an Ubuntu server (SSH enabled on port 4444). Inside this server there's a directory called /folder/a/, this directory literally contains millions of small size text files.



What i want to do is to find several thousands of files from those millions of files and copy or transfer those files to my local pc, i.e. using find command. When using scp or cpio it won't work properly.



Do you have any suggestion how to do this from my local pc, i.e not from running ssh terminal on my remote server.






linux shell ubuntu command-line cpio







share|improve this question















marked as duplicate by user1133275, Rui F Ribeiro linux
Users with the  linux badge can single-handedly close linux questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
 Feb 10 at 8:50


This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.


















  • It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

    – Emmanuel Rosa
    Feb 10 at 5:20











  • i know but, i want to do something like this (from local pc) ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000" but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

    – xcode
    Feb 10 at 5:26













0












0








0









This question already has an answer here:



  • find and rsync?

    1 answer



I have this problem, I have an Ubuntu server (SSH enabled on port 4444). Inside this server there's a directory called /folder/a/, this directory literally contains millions of small size text files.



What i want to do is to find several thousands of files from those millions of files and copy or transfer those files to my local pc, i.e. using find command. When using scp or cpio it won't work properly.



Do you have any suggestion how to do this from my local pc, i.e not from running ssh terminal on my remote server.






linux shell ubuntu command-line cpio







share|improve this question

















This question already has an answer here:



  • find and rsync?

    1 answer



I have this problem, I have an Ubuntu server (SSH enabled on port 4444). Inside this server there's a directory called /folder/a/, this directory literally contains millions of small size text files.



What i want to do is to find several thousands of files from those millions of files and copy or transfer those files to my local pc, i.e. using find command. When using scp or cpio it won't work properly.



Do you have any suggestion how to do this from my local pc, i.e not from running ssh terminal on my remote server.





This question already has an answer here:



  • find and rsync?

    1 answer






linux shell ubuntu command-line cpio




linux shell ubuntu command-line cpio






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Feb 10 at 5:08









Emmanuel Rosa

3,2351612




3,2351612










asked Feb 10 at 5:04









xcodexcode

1113




1113




marked as duplicate by user1133275, Rui F Ribeiro linux
Users with the  linux badge can single-handedly close linux questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
 Feb 10 at 8:50


This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.









marked as duplicate by user1133275, Rui F Ribeiro linux
Users with the  linux badge can single-handedly close linux questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
 Feb 10 at 8:50


This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.














  • It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

    – Emmanuel Rosa
    Feb 10 at 5:20











  • i know but, i want to do something like this (from local pc) ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000" but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

    – xcode
    Feb 10 at 5:26

















  • It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

    – Emmanuel Rosa
    Feb 10 at 5:20











  • i know but, i want to do something like this (from local pc) ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000" but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

    – xcode
    Feb 10 at 5:26
















It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

– Emmanuel Rosa
Feb 10 at 5:20





It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

– Emmanuel Rosa
Feb 10 at 5:20













i know but, i want to do something like this (from local pc) ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000" but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

– xcode
Feb 10 at 5:26





i know but, i want to do something like this (from local pc) ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000" but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

– xcode
Feb 10 at 5:26










1 Answer
1






active

oldest

votes


















1














rsync -e 'ssh -p 4444' -a --include="pattern you want" --exclude="*" $S:/folder/a ~/copy


Or



ssh -p 4444 $S "find /folder/a -name 'pattern you want' | tar -cO -T -" > ~/copy.tar


Or



rsync -e 'ssh -p 4444' -a --files-from=<(ssh -p 444 $S find /folder/a -name 'pattern you want') $S:/folder/a ~/copy





share|improve this answer

























  • thanks, how to combine rsync command with find command, since i want to use find with many of its options?

    – xcode
    Feb 10 at 5:42











  • @xcode added to answer

    – user1133275
    Feb 10 at 5:53

















1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














rsync -e 'ssh -p 4444' -a --include="pattern you want" --exclude="*" $S:/folder/a ~/copy


Or



ssh -p 4444 $S "find /folder/a -name 'pattern you want' | tar -cO -T -" > ~/copy.tar


Or



rsync -e 'ssh -p 4444' -a --files-from=<(ssh -p 444 $S find /folder/a -name 'pattern you want') $S:/folder/a ~/copy





share|improve this answer

























  • thanks, how to combine rsync command with find command, since i want to use find with many of its options?

    – xcode
    Feb 10 at 5:42











  • @xcode added to answer

    – user1133275
    Feb 10 at 5:53















1














rsync -e 'ssh -p 4444' -a --include="pattern you want" --exclude="*" $S:/folder/a ~/copy


Or



ssh -p 4444 $S "find /folder/a -name 'pattern you want' | tar -cO -T -" > ~/copy.tar


Or



rsync -e 'ssh -p 4444' -a --files-from=<(ssh -p 444 $S find /folder/a -name 'pattern you want') $S:/folder/a ~/copy





share|improve this answer

























  • thanks, how to combine rsync command with find command, since i want to use find with many of its options?

    – xcode
    Feb 10 at 5:42











  • @xcode added to answer

    – user1133275
    Feb 10 at 5:53













1












1








1







rsync -e 'ssh -p 4444' -a --include="pattern you want" --exclude="*" $S:/folder/a ~/copy


Or



ssh -p 4444 $S "find /folder/a -name 'pattern you want' | tar -cO -T -" > ~/copy.tar


Or



rsync -e 'ssh -p 4444' -a --files-from=<(ssh -p 444 $S find /folder/a -name 'pattern you want') $S:/folder/a ~/copy





share|improve this answer















rsync -e 'ssh -p 4444' -a --include="pattern you want" --exclude="*" $S:/folder/a ~/copy


Or



ssh -p 4444 $S "find /folder/a -name 'pattern you want' | tar -cO -T -" > ~/copy.tar


Or



rsync -e 'ssh -p 4444' -a --files-from=<(ssh -p 444 $S find /folder/a -name 'pattern you want') $S:/folder/a ~/copy






share|improve this answer














share|improve this answer



share|improve this answer








edited Feb 10 at 5:53

























answered Feb 10 at 5:24









user1133275user1133275

3,525823




3,525823












  • thanks, how to combine rsync command with find command, since i want to use find with many of its options?

    – xcode
    Feb 10 at 5:42











  • @xcode added to answer

    – user1133275
    Feb 10 at 5:53

















  • thanks, how to combine rsync command with find command, since i want to use find with many of its options?

    – xcode
    Feb 10 at 5:42











  • @xcode added to answer

    – user1133275
    Feb 10 at 5:53
















thanks, how to combine rsync command with find command, since i want to use find with many of its options?

– xcode
Feb 10 at 5:42





thanks, how to combine rsync command with find command, since i want to use find with many of its options?

– xcode
Feb 10 at 5:42













@xcode added to answer

– user1133275
Feb 10 at 5:53





@xcode added to answer

– user1133275
Feb 10 at 5:53


-

Popular posts from this blog

How to check contact read email or not when send email to Individual?

Image
Clash Royale CLAN TAG #URR8PPP up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Sep 26 at 0:12 ToanLuong 49 9 add a comment  |  up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this questi...
Read more

Running qemu-guest-agent on windows server 2008

Image
Clash Royale CLAN TAG #URR8PPP up vote 0 down vote favorite I know how I need to configure any linux guestvm in order to make qemu-guest-agent work. This is because there is a lot of documentation on it available, this is not the same with a windows guest. I wanted to know what steps I need to take to make qemu-GA work with a windows guest. I know I need to specify a channel type in the xml file of my windows vm, but further than that?? Does somebody has some more experience with this? Thanks. Running system: Debian 9 (on VMware) 3GB ram & 100GB DISK Running libvirt Daemon Several other guest vms running in background Windows guest: Windows server 2012 512MB RAM & 32GB DISK Virtio drivers working windows virtual-machine kvm qemu libvirtd share | improve this question asked Jan 13 at 0:02 codenerd123 1 1 add a comment  |  up vote 0 down vote favorite I know how I need to configure any linux guestvm in...
Read more

Christian Cage

Image
Christian Cage Christian in March 2015 Birth name William Jason Reso Born ( 1973-11-30 ) November 30, 1973 (age 45) Kitchener, Ontario, Canada Residence Tampa, Florida, United States Spouse(s) Denise Hartmann ( m.   2001 ) Children 1 Professional wrestling career Ring name(s) Christian [1] Christian Cage [2] Conquistador Dos [3] Male Nurse [4] Billed height 6 ft 1 in (1.85 m) [1] Billed weight 212 lb (96 kg) [1] Billed from Tampa, Florida (WM18, TNA) Toronto, Ontario, Canada [1] Trained by Dory Funk Jr. [2] Ron Hutchison [5] Debut 1995 [5] Retired 2014 [6] William Jason Reso (born November 30, 1973) is a Canadian actor, podcaster and retired professional wrestler best known for his time in WWE under the ring name Christian , a shortened version of his original ring name Christian Cage that was also used during his tenure in Total Nonstop Action Wrestling (TNA). Reso was trained by former professional wrestlers Ron Hutchison and Dory Funk Jr. and m...
Read more