tar with ssh: what happens if creation speed is higher than network speed?
Clash Royale CLAN TAG#URR8PPP
I'm concerned what will happen if speed of source read + stream create operations gets higher than underlying network protocol speed?
Is tar going to create and cache data locally before it is transported over network? If so, is there any sane way of preventing this ?
The scenario is like:
tar cf '/path/to/dir' | pigz -p 3 | ssh user@remotemachine.tld "cat > /backup/data.tar.gz"
and the host machine (the one that issues tar cf command) should not get additional data on it's local storage.
networking tar io
asked Jan 14 at 13:27
Miloš ĐakonovićMiloš Đakonović
378317
add a comment |
I'm concerned what will happen if speed of source read + stream create operations gets higher than underlying network protocol speed?
Is tar going to create and cache data locally before it is transported over network? If so, is there any sane way of preventing this ?
The scenario is like:
tar cf '/path/to/dir' | pigz -p 3 | ssh user@remotemachine.tld "cat > /backup/data.tar.gz"
and the host machine (the one that issues tar cf command) should not get additional data on it's local storage.
networking tar io
asked Jan 14 at 13:27
Miloš ĐakonovićMiloš Đakonović
378317
Do you expect something harmful to happen in this scenario? Why do you want to prevent any possible caching? Is there an actual problem you're trying to solve here?
– ilkkachu
Jan 14 at 21:19
@ilkkachu yes. Running out of available space. The machine cannot afford using of local disk space for this operation.
– Miloš Đakonović
Jan 15 at 9:41
@ilkkachu so, is there any possible caching that should be prevented? Thanks in advance.
– Miloš Đakonović
Jan 15 at 9:43
add a comment |
I'm concerned what will happen if speed of source read + stream create operations gets higher than underlying network protocol speed?
Is tar going to create and cache data locally before it is transported over network? If so, is there any sane way of preventing this ?
The scenario is like:
tar cf '/path/to/dir' | pigz -p 3 | ssh user@remotemachine.tld "cat > /backup/data.tar.gz"
and the host machine (the one that issues tar cf command) should not get additional data on it's local storage.
networking tar io
asked Jan 14 at 13:27
Miloš ĐakonovićMiloš Đakonović
378317
I'm concerned what will happen if speed of source read + stream create operations gets higher than underlying network protocol speed?
Is tar going to create and cache data locally before it is transported over network? If so, is there any sane way of preventing this ?
The scenario is like:
tar cf '/path/to/dir' | pigz -p 3 | ssh user@remotemachine.tld "cat > /backup/data.tar.gz"
and the host machine (the one that issues tar cf command) should not get additional data on it's local storage.
networking tar io
networking tar io
asked Jan 14 at 13:27
Miloš ĐakonovićMiloš Đakonović
378317
asked Jan 14 at 13:27
Miloš ĐakonovićMiloš Đakonović
378317
asked Jan 14 at 13:27
Miloš ĐakonovićMiloš Đakonović
378317
asked Jan 14 at 13:27
Miloš ĐakonovićMiloš Đakonović
378317
asked Jan 14 at 13:27
Miloš ĐakonovićMiloš Đakonović
378317
378317
Do you expect something harmful to happen in this scenario? Why do you want to prevent any possible caching? Is there an actual problem you're trying to solve here?
– ilkkachu
Jan 14 at 21:19
@ilkkachu yes. Running out of available space. The machine cannot afford using of local disk space for this operation.
– Miloš Đakonović
Jan 15 at 9:41
@ilkkachu so, is there any possible caching that should be prevented? Thanks in advance.
– Miloš Đakonović
Jan 15 at 9:43
add a comment |
Do you expect something harmful to happen in this scenario? Why do you want to prevent any possible caching? Is there an actual problem you're trying to solve here?
– ilkkachu
Jan 14 at 21:19
@ilkkachu yes. Running out of available space. The machine cannot afford using of local disk space for this operation.
– Miloš Đakonović
Jan 15 at 9:41
@ilkkachu so, is there any possible caching that should be prevented? Thanks in advance.
– Miloš Đakonović
Jan 15 at 9:43
Do you expect something harmful to happen in this scenario? Why do you want to prevent any possible caching? Is there an actual problem you're trying to solve here?
– ilkkachu
Jan 14 at 21:19
Do you expect something harmful to happen in this scenario? Why do you want to prevent any possible caching? Is there an actual problem you're trying to solve here?
– ilkkachu
Jan 14 at 21:19
@ilkkachu yes. Running out of available space. The machine cannot afford using of local disk space for this operation.
– Miloš Đakonović
Jan 15 at 9:41
@ilkkachu yes. Running out of available space. The machine cannot afford using of local disk space for this operation.
– Miloš Đakonović
Jan 15 at 9:41
@ilkkachu so, is there any possible caching that should be prevented? Thanks in advance.
– Miloš Đakonović
Jan 15 at 9:43
@ilkkachu so, is there any possible caching that should be prevented? Thanks in advance.
– Miloš Đakonović
Jan 15 at 9:43
add a comment |
1 Answer
1
active
oldest
votes
First over ssh you have compression with will mitigate from one side your concerns.
Second the pipe mechanism will "hold" the source until you have transferred all the information in the pipe.
answered Jan 14 at 13:46
Romeo NinovRomeo Ninov
5,92332028
to hold the source... to hold the source created (thus, cached somewhere on local disk) or to hold reading + creating operation until previous is transported? The big difference to me. Thanks for answer.
– Miloš Đakonović
Jan 14 at 13:51
Hold writing (to the pipe) operation. Pipes do not use caching on disk
– Romeo Ninov
Jan 14 at 13:52
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f494421%2ftar-with-ssh-what-happens-if-creation-speed-is-higher-than-network-speed%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
First over ssh you have compression with will mitigate from one side your concerns.
Second the pipe mechanism will "hold" the source until you have transferred all the information in the pipe.
answered Jan 14 at 13:46
Romeo NinovRomeo Ninov
5,92332028
to hold the source... to hold the source created (thus, cached somewhere on local disk) or to hold reading + creating operation until previous is transported? The big difference to me. Thanks for answer.
– Miloš Đakonović
Jan 14 at 13:51
Hold writing (to the pipe) operation. Pipes do not use caching on disk
– Romeo Ninov
Jan 14 at 13:52
add a comment |
First over ssh you have compression with will mitigate from one side your concerns.
Second the pipe mechanism will "hold" the source until you have transferred all the information in the pipe.
answered Jan 14 at 13:46
Romeo NinovRomeo Ninov
5,92332028
to hold the source... to hold the source created (thus, cached somewhere on local disk) or to hold reading + creating operation until previous is transported? The big difference to me. Thanks for answer.
– Miloš Đakonović
Jan 14 at 13:51
Hold writing (to the pipe) operation. Pipes do not use caching on disk
– Romeo Ninov
Jan 14 at 13:52
add a comment |
First over ssh you have compression with will mitigate from one side your concerns.
Second the pipe mechanism will "hold" the source until you have transferred all the information in the pipe.
answered Jan 14 at 13:46
Romeo NinovRomeo Ninov
5,92332028
First over ssh you have compression with will mitigate from one side your concerns.
Second the pipe mechanism will "hold" the source until you have transferred all the information in the pipe.
answered Jan 14 at 13:46
Romeo NinovRomeo Ninov
5,92332028
answered Jan 14 at 13:46
Romeo NinovRomeo Ninov
5,92332028
answered Jan 14 at 13:46
Romeo NinovRomeo Ninov
5,92332028
answered Jan 14 at 13:46
Romeo NinovRomeo Ninov
5,92332028
5,92332028
to hold the source... to hold the source created (thus, cached somewhere on local disk) or to hold reading + creating operation until previous is transported? The big difference to me. Thanks for answer.
– Miloš Đakonović
Jan 14 at 13:51
Hold writing (to the pipe) operation. Pipes do not use caching on disk
– Romeo Ninov
Jan 14 at 13:52
add a comment |
to hold the source... to hold the source created (thus, cached somewhere on local disk) or to hold reading + creating operation until previous is transported? The big difference to me. Thanks for answer.
– Miloš Đakonović
Jan 14 at 13:51
Hold writing (to the pipe) operation. Pipes do not use caching on disk
– Romeo Ninov
Jan 14 at 13:52
to hold the source... to hold the source created (thus, cached somewhere on local disk) or to hold reading + creating operation until previous is transported? The big difference to me. Thanks for answer.
– Miloš Đakonović
Jan 14 at 13:51
to hold the source... to hold the source created (thus, cached somewhere on local disk) or to hold reading + creating operation until previous is transported? The big difference to me. Thanks for answer.
– Miloš Đakonović
Jan 14 at 13:51
Hold writing (to the pipe) operation. Pipes do not use caching on disk
– Romeo Ninov
Jan 14 at 13:52
Hold writing (to the pipe) operation. Pipes do not use caching on disk
– Romeo Ninov
Jan 14 at 13:52
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f494421%2ftar-with-ssh-what-happens-if-creation-speed-is-higher-than-network-speed%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Do you expect something harmful to happen in this scenario? Why do you want to prevent any possible caching? Is there an actual problem you're trying to solve here?
– ilkkachu
Jan 14 at 21:19
@ilkkachu yes. Running out of available space. The machine cannot afford using of local disk space for this operation.
– Miloš Đakonović
Jan 15 at 9:41
@ilkkachu so, is there any possible caching that should be prevented? Thanks in advance.
– Miloš Đakonović
Jan 15 at 9:43