Getting size of 40k paths from remote network

-1

I've around 40k lines of file with paths which I need to take size from remote site (using rsh & du -scL command). I tried with while read line but due to remote connection, it exits after 100+ lines. So I tried to copy all the lines in to a file with du -scL and input the file in to one rsh but again it's crashed saying 'command too long'. I need to do a script which calc the size of all these paths from remote site using rsh and du.

#!bin/bash
for line in `cat $destbang1`
do
 rsh vnc.<remotesite> du -sL $line | awk 'print $1' >> /tmp/size1.txt
done
total=`gawk ' sum += $1 ; END print sum' /tmp/size1.txt`
echo $total

edited Jan 6 at 21:34

Rui F Ribeiro

39.6k1479132

asked Sep 11 '17 at 9:24

Ratheesh P

You would be better served using find with the exec flag and so something like find -type d -exec du '' ;

– Raman Sailopal
Sep 11 '17 at 10:17

Thanks Raman for the suggestion.But does that can process 40k lines in du via rsh? .I tried while read with -exec and crashed in 100-200 lines.I also tried something like rsh <remotemachine> du -scL cat /file/which/contain/40kpath/lists.txt but it exited with "command too long"

– Ratheesh P
Sep 11 '17 at 12:36

1

Is there any reason you're not using ssh.

– Raman Sailopal
Sep 11 '17 at 12:39

1

Don't do 100 rsh or ssh commands without any delay between them; each one creates a socket that remains unavailable for a few minutes after it's closed, and if enough of these build up you won't be able to make any more connections for awhile. Instead, get the script working on your local system, show us the code, and we'll help you copy it and execute it on the remote system.

– Mark Plotnick
Sep 11 '17 at 20:54

1

If you are asked for code then do not show it in a comment; add it to your question instead.

– Hauke Laging
Sep 17 '17 at 15:03

|
show 3 more comments

-1

#!bin/bash
for line in `cat $destbang1`
do
 rsh vnc.<remotesite> du -sL $line | awk 'print $1' >> /tmp/size1.txt
done
total=`gawk ' sum += $1 ; END print sum' /tmp/size1.txt`
echo $total

edited Jan 6 at 21:34

Rui F Ribeiro

39.6k1479132

asked Sep 11 '17 at 9:24

Ratheesh P

You would be better served using find with the exec flag and so something like find -type d -exec du '' ;

– Raman Sailopal
Sep 11 '17 at 10:17

Thanks Raman for the suggestion.But does that can process 40k lines in du via rsh? .I tried while read with -exec and crashed in 100-200 lines.I also tried something like rsh <remotemachine> du -scL cat /file/which/contain/40kpath/lists.txt but it exited with "command too long"

– Ratheesh P
Sep 11 '17 at 12:36

1

Is there any reason you're not using ssh.

– Raman Sailopal
Sep 11 '17 at 12:39

1

Don't do 100 rsh or ssh commands without any delay between them; each one creates a socket that remains unavailable for a few minutes after it's closed, and if enough of these build up you won't be able to make any more connections for awhile. Instead, get the script working on your local system, show us the code, and we'll help you copy it and execute it on the remote system.

– Mark Plotnick
Sep 11 '17 at 20:54

1

If you are asked for code then do not show it in a comment; add it to your question instead.

– Hauke Laging
Sep 17 '17 at 15:03

|
show 3 more comments

-1

#!bin/bash
for line in `cat $destbang1`
do
 rsh vnc.<remotesite> du -sL $line | awk 'print $1' >> /tmp/size1.txt
done
total=`gawk ' sum += $1 ; END print sum' /tmp/size1.txt`
echo $total

edited Jan 6 at 21:34

Rui F Ribeiro

39.6k1479132

asked Sep 11 '17 at 9:24

Ratheesh P

#!bin/bash
for line in `cat $destbang1`
do
 rsh vnc.<remotesite> du -sL $line | awk 'print $1' >> /tmp/size1.txt
done
total=`gawk ' sum += $1 ; END print sum' /tmp/size1.txt`
echo $total

linux shell scripting disk-usage rsh

edited Jan 6 at 21:34

Rui F Ribeiro

39.6k1479132

asked Sep 11 '17 at 9:24

Ratheesh P

edited Jan 6 at 21:34

Rui F Ribeiro

39.6k1479132

asked Sep 11 '17 at 9:24

Ratheesh P

edited Jan 6 at 21:34

Rui F Ribeiro

39.6k1479132

edited Jan 6 at 21:34

Rui F Ribeiro

39.6k1479132

edited Jan 6 at 21:34

Rui F Ribeiro

39.6k1479132

asked Sep 11 '17 at 9:24

Ratheesh P

asked Sep 11 '17 at 9:24

Ratheesh P

asked Sep 11 '17 at 9:24

Ratheesh P

You would be better served using find with the exec flag and so something like find -type d -exec du '' ;

– Raman Sailopal
Sep 11 '17 at 10:17

Thanks Raman for the suggestion.But does that can process 40k lines in du via rsh? .I tried while read with -exec and crashed in 100-200 lines.I also tried something like rsh <remotemachine> du -scL cat /file/which/contain/40kpath/lists.txt but it exited with "command too long"

– Ratheesh P
Sep 11 '17 at 12:36

1

Is there any reason you're not using ssh.

– Raman Sailopal
Sep 11 '17 at 12:39

1

Don't do 100 rsh or ssh commands without any delay between them; each one creates a socket that remains unavailable for a few minutes after it's closed, and if enough of these build up you won't be able to make any more connections for awhile. Instead, get the script working on your local system, show us the code, and we'll help you copy it and execute it on the remote system.

– Mark Plotnick
Sep 11 '17 at 20:54

1

If you are asked for code then do not show it in a comment; add it to your question instead.

– Hauke Laging
Sep 17 '17 at 15:03

|
show 3 more comments

You would be better served using find with the exec flag and so something like find -type d -exec du '' ;

– Raman Sailopal
Sep 11 '17 at 10:17

Thanks Raman for the suggestion.But does that can process 40k lines in du via rsh? .I tried while read with -exec and crashed in 100-200 lines.I also tried something like rsh <remotemachine> du -scL cat /file/which/contain/40kpath/lists.txt but it exited with "command too long"

– Ratheesh P
Sep 11 '17 at 12:36

1

Is there any reason you're not using ssh.

– Raman Sailopal
Sep 11 '17 at 12:39

1

Don't do 100 rsh or ssh commands without any delay between them; each one creates a socket that remains unavailable for a few minutes after it's closed, and if enough of these build up you won't be able to make any more connections for awhile. Instead, get the script working on your local system, show us the code, and we'll help you copy it and execute it on the remote system.

– Mark Plotnick
Sep 11 '17 at 20:54

1

If you are asked for code then do not show it in a comment; add it to your question instead.

– Hauke Laging
Sep 17 '17 at 15:03

You would be better served using find with the exec flag and so something like find -type d -exec du '' ;

– Raman Sailopal
Sep 11 '17 at 10:17

Thanks Raman for the suggestion.But does that can process 40k lines in du via rsh? .I tried while read with -exec and crashed in 100-200 lines.I also tried something like rsh <remotemachine> du -scL cat /file/which/contain/40kpath/lists.txt but it exited with "command too long"

– Ratheesh P
Sep 11 '17 at 12:36

Is there any reason you're not using ssh.

– Raman Sailopal
Sep 11 '17 at 12:39

Don't do 100 rsh or ssh commands without any delay between them; each one creates a socket that remains unavailable for a few minutes after it's closed, and if enough of these build up you won't be able to make any more connections for awhile. Instead, get the script working on your local system, show us the code, and we'll help you copy it and execute it on the remote system.

– Mark Plotnick
Sep 11 '17 at 20:54

If you are asked for code then do not show it in a comment; add it to your question instead.

– Hauke Laging
Sep 17 '17 at 15:03

|
show 3 more comments

2 Answers
2

active

oldest

votes

You can use xargs for creating several rsh calls with a low enough number or arguments.

Assuming the remot and local limit are the same:

xargs --delimiter='n' rsh /path/to/script.sh <lists.txt

answered Sep 17 '17 at 15:04

Hauke Laging

56.2k1285135

add a comment |

You can run the entire process with a single rsh remote shell:

tr '12' '' < "$destbang1" |
 rsh vnc.remotesite du -sL --files0-from=- |
 awk 'sum += $1 END print sum'

This requires that your du command understands the --files0-from option and that none of the file names in your $deskbang1 file contains an embedded newline.

answered Sep 18 '17 at 10:06

roaima

43.5k553116

add a comment |

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f391566%2fgetting-size-of-40k-paths-from-remote-network%23new-answer', 'question_page');

);

Post as a guest

Name

Required, but never shown

2 Answers
2

active

oldest

votes

2 Answers
2

active

oldest

votes

You can use xargs for creating several rsh calls with a low enough number or arguments.

Assuming the remot and local limit are the same:

xargs --delimiter='n' rsh /path/to/script.sh <lists.txt

answered Sep 17 '17 at 15:04

Hauke Laging

56.2k1285135

add a comment |

You can use xargs for creating several rsh calls with a low enough number or arguments.

Assuming the remot and local limit are the same:

xargs --delimiter='n' rsh /path/to/script.sh <lists.txt

answered Sep 17 '17 at 15:04

Hauke Laging

56.2k1285135

add a comment |

You can use xargs for creating several rsh calls with a low enough number or arguments.

Assuming the remot and local limit are the same:

xargs --delimiter='n' rsh /path/to/script.sh <lists.txt

answered Sep 17 '17 at 15:04

Hauke Laging

56.2k1285135

You can use xargs for creating several rsh calls with a low enough number or arguments.

Assuming the remot and local limit are the same:

xargs --delimiter='n' rsh /path/to/script.sh <lists.txt

answered Sep 17 '17 at 15:04

Hauke Laging

56.2k1285135

answered Sep 17 '17 at 15:04

Hauke Laging

56.2k1285135

answered Sep 17 '17 at 15:04

Hauke Laging

56.2k1285135

answered Sep 17 '17 at 15:04

Hauke Laging

56.2k1285135

add a comment |

You can run the entire process with a single rsh remote shell:

tr '12' '' < "$destbang1" |
 rsh vnc.remotesite du -sL --files0-from=- |
 awk 'sum += $1 END print sum'

This requires that your du command understands the --files0-from option and that none of the file names in your $deskbang1 file contains an embedded newline.

answered Sep 18 '17 at 10:06

roaima

43.5k553116

add a comment |

You can run the entire process with a single rsh remote shell:

tr '12' '' < "$destbang1" |
 rsh vnc.remotesite du -sL --files0-from=- |
 awk 'sum += $1 END print sum'

This requires that your du command understands the --files0-from option and that none of the file names in your $deskbang1 file contains an embedded newline.

answered Sep 18 '17 at 10:06

roaima

43.5k553116

add a comment |

You can run the entire process with a single rsh remote shell:

tr '12' '' < "$destbang1" |
 rsh vnc.remotesite du -sL --files0-from=- |
 awk 'sum += $1 END print sum'

This requires that your du command understands the --files0-from option and that none of the file names in your $deskbang1 file contains an embedded newline.

answered Sep 18 '17 at 10:06

roaima

43.5k553116

You can run the entire process with a single rsh remote shell:

tr '12' '' < "$destbang1" |
 rsh vnc.remotesite du -sL --files0-from=- |
 awk 'sum += $1 END print sum'

This requires that your du command understands the --files0-from option and that none of the file names in your $deskbang1 file contains an embedded newline.

answered Sep 18 '17 at 10:06

roaima

43.5k553116

answered Sep 18 '17 at 10:06

roaima

43.5k553116

answered Sep 18 '17 at 10:06

roaima

43.5k553116

answered Sep 18 '17 at 10:06

roaima

43.5k553116

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Unix & Linux Stack Exchange!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

搜尋此網誌

mjhjmtu