Gnu parallel rsync hanging indefinitely
Multi tool use
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
To be clear, I am running this exact command(variables changed) in a gnu parallel setting:
/usr/bin/parallel -q -j20
rsync --recursive -av -e ssh -oCompression=no -x -T
-oUserKnownHostsFile=/dev/null -oConnectTimeout=15
-i ssh_key -l some_username --chmod=Dugo+rwx,Fugo+rw
--files-from /some/base/dir somehostame:/some/other/dir
This consumes about 50 files(containing list of file/directory names for rsync), across 20 workers through gnu parallel. I ran the above yesterday night, and I came back to work to see that there is still 1 rsync job, hanging:
rsync --recursive -av -e ssh -oCompression=no -x -T -
oUserKnownHostsFile=/dev/null -oConnectTimeout=15
-i ssh-beta-secdata-20180515t213639 -l diskmaker
--chmod=Dugo+rwx,Fugo+rw
--files-from ./some_filelist /some/base/dir somehostname:/some/other/dir
So to be clear, some_filelist is a list of directories I want to do rsync in.
I'll take the files from the base(as if chroot'd) from /some/base/dir to a remote directory somehostname:/some/other/dir.
So there are like, 500 jobs that parallel pulls from. I analyzed each file list, and there's nothing special about this ./some_filelist; standard 200 files from an rsync dryrun.
Does anyone have diagnosis for this problem? I am completely lost to how this could happen. I can post more snippets if required.
bash ssh rsync gnu-parallel
asked May 16 at 15:39
OneRaynyDay
1063
add a comment |Â
up vote
0
down vote
favorite
To be clear, I am running this exact command(variables changed) in a gnu parallel setting:
/usr/bin/parallel -q -j20
rsync --recursive -av -e ssh -oCompression=no -x -T
-oUserKnownHostsFile=/dev/null -oConnectTimeout=15
-i ssh_key -l some_username --chmod=Dugo+rwx,Fugo+rw
--files-from /some/base/dir somehostame:/some/other/dir
This consumes about 50 files(containing list of file/directory names for rsync), across 20 workers through gnu parallel. I ran the above yesterday night, and I came back to work to see that there is still 1 rsync job, hanging:
rsync --recursive -av -e ssh -oCompression=no -x -T -
oUserKnownHostsFile=/dev/null -oConnectTimeout=15
-i ssh-beta-secdata-20180515t213639 -l diskmaker
--chmod=Dugo+rwx,Fugo+rw
--files-from ./some_filelist /some/base/dir somehostname:/some/other/dir
So to be clear, some_filelist is a list of directories I want to do rsync in.
I'll take the files from the base(as if chroot'd) from /some/base/dir to a remote directory somehostname:/some/other/dir.
So there are like, 500 jobs that parallel pulls from. I analyzed each file list, and there's nothing special about this ./some_filelist; standard 200 files from an rsync dryrun.
Does anyone have diagnosis for this problem? I am completely lost to how this could happen. I can post more snippets if required.
bash ssh rsync gnu-parallel
asked May 16 at 15:39
OneRaynyDay
1063
can you list what files the hanging process has open (lsof -p pid), and what syscall it is hanging on (strace -p pid).
– meuh
May 16 at 19:38
Does it work, if you run the hanging command on the command line? Can you try splitting./some_filelistand see if some parts work find? Is the the same part that is hanging if you useparallel --shuf?
– Ole Tange
May 17 at 11:49
Hope you make sure differentrsyncprocesses do not read/write to the same dirs.
– akostadinov
May 17 at 21:45
add a comment |Â
up vote
0
down vote
favorite
up vote
0
down vote
favorite
To be clear, I am running this exact command(variables changed) in a gnu parallel setting:
/usr/bin/parallel -q -j20
rsync --recursive -av -e ssh -oCompression=no -x -T
-oUserKnownHostsFile=/dev/null -oConnectTimeout=15
-i ssh_key -l some_username --chmod=Dugo+rwx,Fugo+rw
--files-from /some/base/dir somehostame:/some/other/dir
This consumes about 50 files(containing list of file/directory names for rsync), across 20 workers through gnu parallel. I ran the above yesterday night, and I came back to work to see that there is still 1 rsync job, hanging:
rsync --recursive -av -e ssh -oCompression=no -x -T -
oUserKnownHostsFile=/dev/null -oConnectTimeout=15
-i ssh-beta-secdata-20180515t213639 -l diskmaker
--chmod=Dugo+rwx,Fugo+rw
--files-from ./some_filelist /some/base/dir somehostname:/some/other/dir
So to be clear, some_filelist is a list of directories I want to do rsync in.
I'll take the files from the base(as if chroot'd) from /some/base/dir to a remote directory somehostname:/some/other/dir.
So there are like, 500 jobs that parallel pulls from. I analyzed each file list, and there's nothing special about this ./some_filelist; standard 200 files from an rsync dryrun.
Does anyone have diagnosis for this problem? I am completely lost to how this could happen. I can post more snippets if required.
bash ssh rsync gnu-parallel
asked May 16 at 15:39
OneRaynyDay
1063
To be clear, I am running this exact command(variables changed) in a gnu parallel setting:
/usr/bin/parallel -q -j20
rsync --recursive -av -e ssh -oCompression=no -x -T
-oUserKnownHostsFile=/dev/null -oConnectTimeout=15
-i ssh_key -l some_username --chmod=Dugo+rwx,Fugo+rw
--files-from /some/base/dir somehostame:/some/other/dir
This consumes about 50 files(containing list of file/directory names for rsync), across 20 workers through gnu parallel. I ran the above yesterday night, and I came back to work to see that there is still 1 rsync job, hanging:
rsync --recursive -av -e ssh -oCompression=no -x -T -
oUserKnownHostsFile=/dev/null -oConnectTimeout=15
-i ssh-beta-secdata-20180515t213639 -l diskmaker
--chmod=Dugo+rwx,Fugo+rw
--files-from ./some_filelist /some/base/dir somehostname:/some/other/dir
So to be clear, some_filelist is a list of directories I want to do rsync in.
I'll take the files from the base(as if chroot'd) from /some/base/dir to a remote directory somehostname:/some/other/dir.
So there are like, 500 jobs that parallel pulls from. I analyzed each file list, and there's nothing special about this ./some_filelist; standard 200 files from an rsync dryrun.
Does anyone have diagnosis for this problem? I am completely lost to how this could happen. I can post more snippets if required.
bash ssh rsync gnu-parallel
asked May 16 at 15:39
OneRaynyDay
1063
asked May 16 at 15:39
OneRaynyDay
1063
asked May 16 at 15:39
OneRaynyDay
1063
asked May 16 at 15:39
OneRaynyDay
1063
1063
can you list what files the hanging process has open (lsof -p pid), and what syscall it is hanging on (strace -p pid).
– meuh
May 16 at 19:38
Does it work, if you run the hanging command on the command line? Can you try splitting./some_filelistand see if some parts work find? Is the the same part that is hanging if you useparallel --shuf?
– Ole Tange
May 17 at 11:49
Hope you make sure differentrsyncprocesses do not read/write to the same dirs.
– akostadinov
May 17 at 21:45
add a comment |Â
can you list what files the hanging process has open (lsof -p pid), and what syscall it is hanging on (strace -p pid).
– meuh
May 16 at 19:38
Does it work, if you run the hanging command on the command line? Can you try splitting./some_filelistand see if some parts work find? Is the the same part that is hanging if you useparallel --shuf?
– Ole Tange
May 17 at 11:49
Hope you make sure differentrsyncprocesses do not read/write to the same dirs.
– akostadinov
May 17 at 21:45
can you list what files the hanging process has open (
lsof -p pid), and what syscall it is hanging on (strace -p pid).– meuh
May 16 at 19:38
can you list what files the hanging process has open (
lsof -p pid), and what syscall it is hanging on (strace -p pid).– meuh
May 16 at 19:38
Does it work, if you run the hanging command on the command line? Can you try splitting
./some_filelist and see if some parts work find? Is the the same part that is hanging if you use parallel --shuf?– Ole Tange
May 17 at 11:49
Does it work, if you run the hanging command on the command line? Can you try splitting
./some_filelist and see if some parts work find? Is the the same part that is hanging if you use parallel --shuf?– Ole Tange
May 17 at 11:49
Hope you make sure different
rsync processes do not read/write to the same dirs.– akostadinov
May 17 at 21:45
Hope you make sure different
rsync processes do not read/write to the same dirs.– akostadinov
May 17 at 21:45
add a comment |Â
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f444180%2fgnu-parallel-rsync-hanging-indefinitely%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
can you list what files the hanging process has open (
lsof -p pid), and what syscall it is hanging on (strace -p pid).– meuh
May 16 at 19:38
Does it work, if you run the hanging command on the command line? Can you try splitting
./some_filelistand see if some parts work find? Is the the same part that is hanging if you useparallel --shuf?– Ole Tange
May 17 at 11:49
Hope you make sure different
rsyncprocesses do not read/write to the same dirs.– akostadinov
May 17 at 21:45