How to Copy Millions of Files from Remote Server to Local One [duplicate]

This question already has an answer here:

find and rsync?

1 answer

I have this problem, I have an Ubuntu server (SSH enabled on port 4444). Inside this server there's a directory called /folder/a/, this directory literally contains millions of small size text files.

What i want to do is to find several thousands of files from those millions of files and copy or transfer those files to my local pc, i.e. using find command. When using scp or cpio it won't work properly.

Do you have any suggestion how to do this from my local pc, i.e not from running ssh terminal on my remote server.

edited Feb 10 at 5:08

Emmanuel Rosa

3,2351612

asked Feb 10 at 5:04

xcode

1113

marked as duplicate by user1133275, Rui F Ribeiro linux
Users with the linux badge can single-handedly close linux questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
Feb 10 at 8:50

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

– Emmanuel Rosa
Feb 10 at 5:20

i know but, i want to do something like this (from local pc) ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000" but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

– xcode
Feb 10 at 5:26

add a comment |

This question already has an answer here:

find and rsync?

1 answer

Do you have any suggestion how to do this from my local pc, i.e not from running ssh terminal on my remote server.

edited Feb 10 at 5:08

Emmanuel Rosa

3,2351612

asked Feb 10 at 5:04

xcode

1113

marked as duplicate by user1133275, Rui F Ribeiro linux
Users with the linux badge can single-handedly close linux questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
Feb 10 at 8:50

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

– Emmanuel Rosa
Feb 10 at 5:20

i know but, i want to do something like this (from local pc) ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000" but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

– xcode
Feb 10 at 5:26

add a comment |

This question already has an answer here:

find and rsync?

1 answer

Do you have any suggestion how to do this from my local pc, i.e not from running ssh terminal on my remote server.

edited Feb 10 at 5:08

Emmanuel Rosa

3,2351612

asked Feb 10 at 5:04

xcode

1113

This question already has an answer here:

find and rsync?

1 answer

Do you have any suggestion how to do this from my local pc, i.e not from running ssh terminal on my remote server.

This question already has an answer here:

find and rsync?

1 answer

linux shell ubuntu command-line cpio

edited Feb 10 at 5:08

Emmanuel Rosa

3,2351612

asked Feb 10 at 5:04

xcode

1113

edited Feb 10 at 5:08

Emmanuel Rosa

3,2351612

asked Feb 10 at 5:04

xcode

1113

edited Feb 10 at 5:08

Emmanuel Rosa

3,2351612

edited Feb 10 at 5:08

Emmanuel Rosa

3,2351612

edited Feb 10 at 5:08

Emmanuel Rosa

3,2351612

asked Feb 10 at 5:04

xcode

1113

asked Feb 10 at 5:04

xcode

1113

asked Feb 10 at 5:04

xcode

1113

marked as duplicate by user1133275, Rui F Ribeiro linux
Users with the linux badge can single-handedly close linux questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
Feb 10 at 8:50

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

marked as duplicate by user1133275, Rui F Ribeiro linux
Users with the linux badge can single-handedly close linux questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
Feb 10 at 8:50

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

– Emmanuel Rosa
Feb 10 at 5:20

i know but, i want to do something like this (from local pc) ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000" but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

– xcode
Feb 10 at 5:26

add a comment |

It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

– Emmanuel Rosa
Feb 10 at 5:20

i know but, i want to do something like this (from local pc) ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000" but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

– xcode
Feb 10 at 5:26

It's possible to run ssh on the local PC with a command which is executed on the remote PC, and whose output is redirected back to the local PC. You can use this technique to execute find to build a list of files which is fed into tar. The output of tar is directed to the standard output of the ssh command. Note that the finding and archiving happens on the remove (server) PC. If that's not an option, you'd have to mount the remote directory into your local PC, which you can do with a network filesystem such as NFS, or even with SSH itself.

– Emmanuel Rosa
Feb 10 at 5:20

i know but, i want to do something like this (from local pc)

ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000"

but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

– xcode
Feb 10 at 5:26

i know but, i want to do something like this (from local pc)

ssh -p 4444 username@11.11.11.11 "cd /folder && find ./a/ -xdev -name 'X*.somename.*' -mtime +0 -mtime -7 -print | head -15000"

but the problem is i dont know how to pipe or combine those find command with scp or rsync or even tar then transfer it to my local pc directory

– xcode
Feb 10 at 5:26

add a comment |

1 Answer
1

active

oldest

votes

rsync -e 'ssh -p 4444' -a --include="pattern you want" --exclude="*" $S:/folder/a ~/copy

ssh -p 4444 $S "find /folder/a -name 'pattern you want' | tar -cO -T -" > ~/copy.tar

rsync -e 'ssh -p 4444' -a --files-from=<(ssh -p 444 $S find /folder/a -name 'pattern you want') $S:/folder/a ~/copy

edited Feb 10 at 5:53

answered Feb 10 at 5:24

user1133275

3,525823

thanks, how to combine rsync command with find command, since i want to use find with many of its options?

– xcode
Feb 10 at 5:42

@xcode added to answer

– user1133275
Feb 10 at 5:53

add a comment |

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

rsync -e 'ssh -p 4444' -a --include="pattern you want" --exclude="*" $S:/folder/a ~/copy

ssh -p 4444 $S "find /folder/a -name 'pattern you want' | tar -cO -T -" > ~/copy.tar

rsync -e 'ssh -p 4444' -a --files-from=<(ssh -p 444 $S find /folder/a -name 'pattern you want') $S:/folder/a ~/copy

edited Feb 10 at 5:53

answered Feb 10 at 5:24

user1133275

3,525823

thanks, how to combine rsync command with find command, since i want to use find with many of its options?

– xcode
Feb 10 at 5:42

@xcode added to answer

– user1133275
Feb 10 at 5:53

add a comment |

rsync -e 'ssh -p 4444' -a --include="pattern you want" --exclude="*" $S:/folder/a ~/copy

ssh -p 4444 $S "find /folder/a -name 'pattern you want' | tar -cO -T -" > ~/copy.tar

rsync -e 'ssh -p 4444' -a --files-from=<(ssh -p 444 $S find /folder/a -name 'pattern you want') $S:/folder/a ~/copy

edited Feb 10 at 5:53

answered Feb 10 at 5:24

user1133275

3,525823

thanks, how to combine rsync command with find command, since i want to use find with many of its options?

– xcode
Feb 10 at 5:42

@xcode added to answer

– user1133275
Feb 10 at 5:53

add a comment |

rsync -e 'ssh -p 4444' -a --include="pattern you want" --exclude="*" $S:/folder/a ~/copy

ssh -p 4444 $S "find /folder/a -name 'pattern you want' | tar -cO -T -" > ~/copy.tar

rsync -e 'ssh -p 4444' -a --files-from=<(ssh -p 444 $S find /folder/a -name 'pattern you want') $S:/folder/a ~/copy

edited Feb 10 at 5:53

answered Feb 10 at 5:24

user1133275

3,525823

rsync -e 'ssh -p 4444' -a --include="pattern you want" --exclude="*" $S:/folder/a ~/copy

ssh -p 4444 $S "find /folder/a -name 'pattern you want' | tar -cO -T -" > ~/copy.tar

rsync -e 'ssh -p 4444' -a --files-from=<(ssh -p 444 $S find /folder/a -name 'pattern you want') $S:/folder/a ~/copy

edited Feb 10 at 5:53

answered Feb 10 at 5:24

user1133275

3,525823

edited Feb 10 at 5:53

answered Feb 10 at 5:24

user1133275

3,525823

answered Feb 10 at 5:24

user1133275

3,525823

answered Feb 10 at 5:24

user1133275

3,525823

thanks, how to combine rsync command with find command, since i want to use find with many of its options?

– xcode
Feb 10 at 5:42

@xcode added to answer

– user1133275
Feb 10 at 5:53

add a comment |

thanks, how to combine rsync command with find command, since i want to use find with many of its options?

– xcode
Feb 10 at 5:42

@xcode added to answer

– user1133275
Feb 10 at 5:53

thanks, how to combine rsync command with find command, since i want to use find with many of its options?

– xcode
Feb 10 at 5:42

@xcode added to answer

– user1133275
Feb 10 at 5:53

add a comment |

搜尋此網誌

mjhjmtu