scp remote file and append to local file

up vote
2
down vote

favorite

I have multiple files to be pulled from remote server. For further processing of the files in the local server, I need to merge (concatenate) them into single file, which can't be done in the remote file though.

I am not sure how scp work internally, but for the best performance I believe instead of writing those files into local directory and then merge, I feel, I should merge them on the fly and then write into single file. Can you please let me know if merging (appending) the files on fly during scp from remote to local files possible?

If not any better idea?

edited Dec 15 '17 at 12:03

Marco

768516

asked Dec 15 '17 at 11:51

Betta

133

add a commentÂ |Â

up vote
2
down vote

favorite

If not any better idea?

edited Dec 15 '17 at 12:03

Marco

768516

asked Dec 15 '17 at 11:51

Betta

133

add a commentÂ |Â

up vote
2
down vote

favorite

If not any better idea?

edited Dec 15 '17 at 12:03

Marco

768516

asked Dec 15 '17 at 11:51

Betta

133

If not any better idea?

edited Dec 15 '17 at 12:03

Marco

768516

asked Dec 15 '17 at 11:51

Betta

133

edited Dec 15 '17 at 12:03

Marco

768516

edited Dec 15 '17 at 12:03

Marco

768516

edited Dec 15 '17 at 12:03

Marco

768516

asked Dec 15 '17 at 11:51

Betta

133

asked Dec 15 '17 at 11:51

Betta

133

asked Dec 15 '17 at 11:51

Betta

133

add a commentÂ |Â

3 Answers
3

active

oldest

votes

up vote
4
down vote

accepted

Use SSH directly instead of scp and run cat. Where you would do:

scp remote:file1,file2... local-dir

Instead do:

ssh remote cat file1 file2 ... > locale-file

answered Dec 15 '17 at 12:04

muru

33.5k577144

Thanks Muru, How does this work internally? Does it first create files on local and then executes cat or directly cat before writing in local?
â€“Â Betta
Dec 15 '17 at 12:11

The local-file will be created before SSH is started, after that the output of cat should be dumped directly into it.
â€“Â muru
Dec 15 '17 at 12:13

so we are doing ssh and cat. I feel this is equivalent to ssh and cp, not as secure as scp. Am I wrong?
â€“Â Betta
Dec 16 '17 at 6:12

@Betta why do you imagine ssh and cat is not as secure as scp?
â€“Â muru
Dec 16 '17 at 6:16

i know that both scp and ssh>cat uses secure channel supported by ssh, but in that case what is the significance of scp? shh>cp should have been sufficient? Pardon my ignorance!
â€“Â Betta
Dec 16 '17 at 6:30

Â |Â
show 1 more comment

up vote
0
down vote

Below steps to performed being in local server only

I tested both steps it worked fine

First step

ssh username@remoteserverip "cat file1 file2 file3 >> /remoteserverpath/Mergedfile"

Second step You are copying merged file from remote server to local server You can do this by rsync or scp

I prefer rsync

rsync -avzh username@remoteserverip:/remoteserverpath/Mergedfile localserverpath_where_you_want_to_save

answered Dec 15 '17 at 13:06

Praveen Kumar BS

1,010128

I said, I can't merge on the remote server! :)
â€“Â Betta
Dec 16 '17 at 6:32

I achieved to copy the single file content over rysnc in localserver. i am trying to do for multiple files too below is command used to copy the single file content of remote server in local server rsync -avzh username@remoteserverip:/path/remoteserverfile /localserverpath/Mergedfile Merged file contains remoteserverfile content
â€“Â Praveen Kumar BS
Dec 18 '17 at 18:22

add a commentÂ |Â

up vote
0
down vote

This is silly, but it seems you can actually do this with just scp, by copying the remote files to a local fifo and piping them out of it:

$Ã‚Â mkfifo p
$Ã‚Â while :; do cat p >> output ; done &
$ scp somehost:test/* p
bar 100% 4 10.9KB/s 00:00 
doo 100% 4 8.6KB/s 00:00 
foo 100% 4 13.6KB/s 00:00 
$Ã‚Â kill %1
# output contains the files concatenated

(tested with OpenSSH 7.4p1-10+deb9u2 on Debian)

answered Dec 15 '17 at 13:17

ilkkachu

49.9k674137

add a commentÂ |Â

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f411043%2fscp-remote-file-and-append-to-local-file%23new-answer', 'question_page');

);

Post as a guest

Name

3 Answers
3

active

oldest

votes

3 Answers
3

active

oldest

votes

up vote
4
down vote

accepted

Use SSH directly instead of scp and run cat. Where you would do:

scp remote:file1,file2... local-dir

Instead do:

ssh remote cat file1 file2 ... > locale-file

answered Dec 15 '17 at 12:04

muru

33.5k577144

Thanks Muru, How does this work internally? Does it first create files on local and then executes cat or directly cat before writing in local?
â€“Â Betta
Dec 15 '17 at 12:11

The local-file will be created before SSH is started, after that the output of cat should be dumped directly into it.
â€“Â muru
Dec 15 '17 at 12:13

so we are doing ssh and cat. I feel this is equivalent to ssh and cp, not as secure as scp. Am I wrong?
â€“Â Betta
Dec 16 '17 at 6:12

@Betta why do you imagine ssh and cat is not as secure as scp?
â€“Â muru
Dec 16 '17 at 6:16

i know that both scp and ssh>cat uses secure channel supported by ssh, but in that case what is the significance of scp? shh>cp should have been sufficient? Pardon my ignorance!
â€“Â Betta
Dec 16 '17 at 6:30

Â |Â
show 1 more comment

up vote
4
down vote

accepted

Use SSH directly instead of scp and run cat. Where you would do:

scp remote:file1,file2... local-dir

Instead do:

ssh remote cat file1 file2 ... > locale-file

answered Dec 15 '17 at 12:04

muru

33.5k577144

Thanks Muru, How does this work internally? Does it first create files on local and then executes cat or directly cat before writing in local?
â€“Â Betta
Dec 15 '17 at 12:11

The local-file will be created before SSH is started, after that the output of cat should be dumped directly into it.
â€“Â muru
Dec 15 '17 at 12:13

so we are doing ssh and cat. I feel this is equivalent to ssh and cp, not as secure as scp. Am I wrong?
â€“Â Betta
Dec 16 '17 at 6:12

@Betta why do you imagine ssh and cat is not as secure as scp?
â€“Â muru
Dec 16 '17 at 6:16

i know that both scp and ssh>cat uses secure channel supported by ssh, but in that case what is the significance of scp? shh>cp should have been sufficient? Pardon my ignorance!
â€“Â Betta
Dec 16 '17 at 6:30

Â |Â
show 1 more comment

up vote
4
down vote

accepted

Use SSH directly instead of scp and run cat. Where you would do:

scp remote:file1,file2... local-dir

Instead do:

ssh remote cat file1 file2 ... > locale-file

answered Dec 15 '17 at 12:04

muru

33.5k577144

Use SSH directly instead of scp and run cat. Where you would do:

scp remote:file1,file2... local-dir

Instead do:

ssh remote cat file1 file2 ... > locale-file

answered Dec 15 '17 at 12:04

muru

33.5k577144

answered Dec 15 '17 at 12:04

muru

33.5k577144

answered Dec 15 '17 at 12:04

muru

33.5k577144

answered Dec 15 '17 at 12:04

muru

33.5k577144

Thanks Muru, How does this work internally? Does it first create files on local and then executes cat or directly cat before writing in local?
â€“Â Betta
Dec 15 '17 at 12:11

The local-file will be created before SSH is started, after that the output of cat should be dumped directly into it.
â€“Â muru
Dec 15 '17 at 12:13

so we are doing ssh and cat. I feel this is equivalent to ssh and cp, not as secure as scp. Am I wrong?
â€“Â Betta
Dec 16 '17 at 6:12

@Betta why do you imagine ssh and cat is not as secure as scp?
â€“Â muru
Dec 16 '17 at 6:16

i know that both scp and ssh>cat uses secure channel supported by ssh, but in that case what is the significance of scp? shh>cp should have been sufficient? Pardon my ignorance!
â€“Â Betta
Dec 16 '17 at 6:30

Â |Â
show 1 more comment

Thanks Muru, How does this work internally? Does it first create files on local and then executes cat or directly cat before writing in local?
â€“Â Betta
Dec 15 '17 at 12:11

The local-file will be created before SSH is started, after that the output of cat should be dumped directly into it.
â€“Â muru
Dec 15 '17 at 12:13

so we are doing ssh and cat. I feel this is equivalent to ssh and cp, not as secure as scp. Am I wrong?
â€“Â Betta
Dec 16 '17 at 6:12

@Betta why do you imagine ssh and cat is not as secure as scp?
â€“Â muru
Dec 16 '17 at 6:16

i know that both scp and ssh>cat uses secure channel supported by ssh, but in that case what is the significance of scp? shh>cp should have been sufficient? Pardon my ignorance!
â€“Â Betta
Dec 16 '17 at 6:30

Thanks Muru, How does this work internally? Does it first create files on local and then executes cat or directly cat before writing in local?
â€“Â Betta
Dec 15 '17 at 12:11

The local-file will be created before SSH is started, after that the output of cat should be dumped directly into it.
â€“Â muru
Dec 15 '17 at 12:13

so we are doing ssh and cat. I feel this is equivalent to ssh and cp, not as secure as scp. Am I wrong?
â€“Â Betta
Dec 16 '17 at 6:12

@Betta why do you imagine ssh and cat is not as secure as scp?
â€“Â muru
Dec 16 '17 at 6:16

i know that both scp and ssh>cat uses secure channel supported by ssh, but in that case what is the significance of scp? shh>cp should have been sufficient? Pardon my ignorance!
â€“Â Betta
Dec 16 '17 at 6:30

Â |Â
show 1 more comment

up vote
0
down vote

Below steps to performed being in local server only

I tested both steps it worked fine

First step

ssh username@remoteserverip "cat file1 file2 file3 >> /remoteserverpath/Mergedfile"

Second step You are copying merged file from remote server to local server You can do this by rsync or scp

I prefer rsync

rsync -avzh username@remoteserverip:/remoteserverpath/Mergedfile localserverpath_where_you_want_to_save

answered Dec 15 '17 at 13:06

Praveen Kumar BS

1,010128

I said, I can't merge on the remote server! :)
â€“Â Betta
Dec 16 '17 at 6:32

I achieved to copy the single file content over rysnc in localserver. i am trying to do for multiple files too below is command used to copy the single file content of remote server in local server rsync -avzh username@remoteserverip:/path/remoteserverfile /localserverpath/Mergedfile Merged file contains remoteserverfile content
â€“Â Praveen Kumar BS
Dec 18 '17 at 18:22

add a commentÂ |Â

up vote
0
down vote

Below steps to performed being in local server only

I tested both steps it worked fine

First step

ssh username@remoteserverip "cat file1 file2 file3 >> /remoteserverpath/Mergedfile"

Second step You are copying merged file from remote server to local server You can do this by rsync or scp

I prefer rsync

rsync -avzh username@remoteserverip:/remoteserverpath/Mergedfile localserverpath_where_you_want_to_save

answered Dec 15 '17 at 13:06

Praveen Kumar BS

1,010128

I said, I can't merge on the remote server! :)
â€“Â Betta
Dec 16 '17 at 6:32

I achieved to copy the single file content over rysnc in localserver. i am trying to do for multiple files too below is command used to copy the single file content of remote server in local server rsync -avzh username@remoteserverip:/path/remoteserverfile /localserverpath/Mergedfile Merged file contains remoteserverfile content
â€“Â Praveen Kumar BS
Dec 18 '17 at 18:22

add a commentÂ |Â

up vote
0
down vote

Below steps to performed being in local server only

I tested both steps it worked fine

First step

ssh username@remoteserverip "cat file1 file2 file3 >> /remoteserverpath/Mergedfile"

Second step You are copying merged file from remote server to local server You can do this by rsync or scp

I prefer rsync

rsync -avzh username@remoteserverip:/remoteserverpath/Mergedfile localserverpath_where_you_want_to_save

answered Dec 15 '17 at 13:06

Praveen Kumar BS

1,010128

Below steps to performed being in local server only

I tested both steps it worked fine

First step

ssh username@remoteserverip "cat file1 file2 file3 >> /remoteserverpath/Mergedfile"

Second step You are copying merged file from remote server to local server You can do this by rsync or scp

I prefer rsync

rsync -avzh username@remoteserverip:/remoteserverpath/Mergedfile localserverpath_where_you_want_to_save

answered Dec 15 '17 at 13:06

Praveen Kumar BS

1,010128

answered Dec 15 '17 at 13:06

Praveen Kumar BS

1,010128

answered Dec 15 '17 at 13:06

Praveen Kumar BS

1,010128

answered Dec 15 '17 at 13:06

Praveen Kumar BS

1,010128

I said, I can't merge on the remote server! :)
â€“Â Betta
Dec 16 '17 at 6:32

I achieved to copy the single file content over rysnc in localserver. i am trying to do for multiple files too below is command used to copy the single file content of remote server in local server rsync -avzh username@remoteserverip:/path/remoteserverfile /localserverpath/Mergedfile Merged file contains remoteserverfile content
â€“Â Praveen Kumar BS
Dec 18 '17 at 18:22

add a commentÂ |Â

I said, I can't merge on the remote server! :)
â€“Â Betta
Dec 16 '17 at 6:32

I achieved to copy the single file content over rysnc in localserver. i am trying to do for multiple files too below is command used to copy the single file content of remote server in local server rsync -avzh username@remoteserverip:/path/remoteserverfile /localserverpath/Mergedfile Merged file contains remoteserverfile content
â€“Â Praveen Kumar BS
Dec 18 '17 at 18:22

I said, I can't merge on the remote server! :)
â€“Â Betta
Dec 16 '17 at 6:32

I achieved to copy the single file content over rysnc in localserver. i am trying to do for multiple files too below is command used to copy the single file content of remote server in local server rsync -avzh username@remoteserverip:/path/remoteserverfile /localserverpath/Mergedfile Merged file contains remoteserverfile content
â€“Â Praveen Kumar BS
Dec 18 '17 at 18:22

add a commentÂ |Â

up vote
0
down vote

This is silly, but it seems you can actually do this with just scp, by copying the remote files to a local fifo and piping them out of it:

$Ã‚Â mkfifo p
$Ã‚Â while :; do cat p >> output ; done &
$ scp somehost:test/* p
bar 100% 4 10.9KB/s 00:00 
doo 100% 4 8.6KB/s 00:00 
foo 100% 4 13.6KB/s 00:00 
$Ã‚Â kill %1
# output contains the files concatenated

(tested with OpenSSH 7.4p1-10+deb9u2 on Debian)

answered Dec 15 '17 at 13:17

ilkkachu

49.9k674137

add a commentÂ |Â

up vote
0
down vote

This is silly, but it seems you can actually do this with just scp, by copying the remote files to a local fifo and piping them out of it:

$Ã‚Â mkfifo p
$Ã‚Â while :; do cat p >> output ; done &
$ scp somehost:test/* p
bar 100% 4 10.9KB/s 00:00 
doo 100% 4 8.6KB/s 00:00 
foo 100% 4 13.6KB/s 00:00 
$Ã‚Â kill %1
# output contains the files concatenated

(tested with OpenSSH 7.4p1-10+deb9u2 on Debian)

answered Dec 15 '17 at 13:17

ilkkachu

49.9k674137

add a commentÂ |Â

up vote
0
down vote

This is silly, but it seems you can actually do this with just scp, by copying the remote files to a local fifo and piping them out of it:

$Ã‚Â mkfifo p
$Ã‚Â while :; do cat p >> output ; done &
$ scp somehost:test/* p
bar 100% 4 10.9KB/s 00:00 
doo 100% 4 8.6KB/s 00:00 
foo 100% 4 13.6KB/s 00:00 
$Ã‚Â kill %1
# output contains the files concatenated

(tested with OpenSSH 7.4p1-10+deb9u2 on Debian)

answered Dec 15 '17 at 13:17

ilkkachu

49.9k674137

This is silly, but it seems you can actually do this with just scp, by copying the remote files to a local fifo and piping them out of it:

$Ã‚Â mkfifo p
$Ã‚Â while :; do cat p >> output ; done &
$ scp somehost:test/* p
bar 100% 4 10.9KB/s 00:00 
doo 100% 4 8.6KB/s 00:00 
foo 100% 4 13.6KB/s 00:00 
$Ã‚Â kill %1
# output contains the files concatenated

(tested with OpenSSH 7.4p1-10+deb9u2 on Debian)

answered Dec 15 '17 at 13:17

ilkkachu

49.9k674137

answered Dec 15 '17 at 13:17

ilkkachu

49.9k674137

answered Dec 15 '17 at 13:17

ilkkachu

49.9k674137

answered Dec 15 '17 at 13:17

ilkkachu

49.9k674137

add a commentÂ |Â

draft saved

draft discarded

draft saved

draft discarded

Post as a guest

Name

搜尋此網誌

mjhjmtu