Using xargs instead of GNU Parallel

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












GNU Parallel can generate all combinations of input, export a function and run it on 3 remote servers without mixing the output, retrying the command if it fails:



#!/bin/bash

env_parallel --record-env
process_output() =1; @a=split//;
 while(@a)
 print ".",shift @a;
 select($a,$a,$a,0.100);
 ';

env_parallel --env _ --retries 4 -Sserver1..3,: 
 'echo X=1 Y=2 Z=3 | process_output' 
 ::: "a' "b"1..3 ::: "c" 'd"3..4 ::: "e$ f'"7..8


Output:



.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.


xargs -P$(nproc) can run one job per processor.



Given



  • the function
    process_output,

  • the servers to run on (server1, server2, server3),

  • the argument groups (a' "b1, a' "b2, a' "b3;c" 'd1, c" 'd2, c" 'd3;e$ f'7, e$ f'8)

how can you emulate GNU Parallel using xargs to get the same done using process_output and the servers?




xargs




share|improve this question






















  • you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
    – RomanPerekhrest
    Nov 19 '17 at 8:08










  • Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
    – Fedor Dikarev
    Nov 19 '17 at 12:21










  • I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
    – David Favor
    Nov 19 '17 at 13:05










  • @ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
    – Hannu
    Nov 19 '17 at 16:14














up vote
0
down vote

favorite












GNU Parallel can generate all combinations of input, export a function and run it on 3 remote servers without mixing the output, retrying the command if it fails:



#!/bin/bash

env_parallel --record-env
process_output() =1; @a=split//;
 while(@a)
 print ".",shift @a;
 select($a,$a,$a,0.100);
 ';

env_parallel --env _ --retries 4 -Sserver1..3,: 
 'echo X=1 Y=2 Z=3 | process_output' 
 ::: "a' "b"1..3 ::: "c" 'd"3..4 ::: "e$ f'"7..8


Output:



.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.


xargs -P$(nproc) can run one job per processor.



Given



  • the function
    process_output,

  • the servers to run on (server1, server2, server3),

  • the argument groups (a' "b1, a' "b2, a' "b3;c" 'd1, c" 'd2, c" 'd3;e$ f'7, e$ f'8)

how can you emulate GNU Parallel using xargs to get the same done using process_output and the servers?




xargs




share|improve this question






















  • you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
    – RomanPerekhrest
    Nov 19 '17 at 8:08










  • Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
    – Fedor Dikarev
    Nov 19 '17 at 12:21










  • I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
    – David Favor
    Nov 19 '17 at 13:05










  • @ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
    – Hannu
    Nov 19 '17 at 16:14












up vote
0
down vote

favorite









up vote
0
down vote

favorite











GNU Parallel can generate all combinations of input, export a function and run it on 3 remote servers without mixing the output, retrying the command if it fails:



#!/bin/bash

env_parallel --record-env
process_output() =1; @a=split//;
 while(@a)
 print ".",shift @a;
 select($a,$a,$a,0.100);
 ';

env_parallel --env _ --retries 4 -Sserver1..3,: 
 'echo X=1 Y=2 Z=3 | process_output' 
 ::: "a' "b"1..3 ::: "c" 'd"3..4 ::: "e$ f'"7..8


Output:



.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.


xargs -P$(nproc) can run one job per processor.



Given



  • the function
    process_output,

  • the servers to run on (server1, server2, server3),

  • the argument groups (a' "b1, a' "b2, a' "b3;c" 'd1, c" 'd2, c" 'd3;e$ f'7, e$ f'8)

how can you emulate GNU Parallel using xargs to get the same done using process_output and the servers?




xargs




share|improve this question














GNU Parallel can generate all combinations of input, export a function and run it on 3 remote servers without mixing the output, retrying the command if it fails:



#!/bin/bash

env_parallel --record-env
process_output() =1; @a=split//;
 while(@a)
 print ".",shift @a;
 select($a,$a,$a,0.100);
 ';

env_parallel --env _ --retries 4 -Sserver1..3,: 
 'echo X=1 Y=2 Z=3 | process_output' 
 ::: "a' "b"1..3 ::: "c" 'd"3..4 ::: "e$ f'"7..8


Output:



.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.


xargs -P$(nproc) can run one job per processor.



Given



  • the function
    process_output,

  • the servers to run on (server1, server2, server3),

  • the argument groups (a' "b1, a' "b2, a' "b3;c" 'd1, c" 'd2, c" 'd3;e$ f'7, e$ f'8)

how can you emulate GNU Parallel using xargs to get the same done using process_output and the servers?





xargs





share|improve this question













share|improve this question




share|improve this question








edited Nov 19 '17 at 12:13

























asked Nov 19 '17 at 6:44









Ole Tange

11.4k1344102




11.4k1344102











  • you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
    – RomanPerekhrest
    Nov 19 '17 at 8:08










  • Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
    – Fedor Dikarev
    Nov 19 '17 at 12:21










  • I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
    – David Favor
    Nov 19 '17 at 13:05










  • @ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
    – Hannu
    Nov 19 '17 at 16:14
















  • you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
    – RomanPerekhrest
    Nov 19 '17 at 8:08










  • Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
    – Fedor Dikarev
    Nov 19 '17 at 12:21










  • I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
    – David Favor
    Nov 19 '17 at 13:05










  • @ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
    – Hannu
    Nov 19 '17 at 16:14















you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
– RomanPerekhrest
Nov 19 '17 at 8:08




you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
– RomanPerekhrest
Nov 19 '17 at 8:08












Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
– Fedor Dikarev
Nov 19 '17 at 12:21




Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
– Fedor Dikarev
Nov 19 '17 at 12:21












I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
– David Favor
Nov 19 '17 at 13:05




I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
– David Favor
Nov 19 '17 at 13:05












@ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
– Hannu
Nov 19 '17 at 16:14




@ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
– Hannu
Nov 19 '17 at 16:14















active

oldest

votes











Your Answer







StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f405552%2fusing-xargs-instead-of-gnu-parallel%23new-answer', 'question_page');

);

Post as a guest






















active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes















 

draft saved


draft discarded















































 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f405552%2fusing-xargs-instead-of-gnu-parallel%23new-answer', 'question_page');

);

Post as a guest
































































-

Popular posts from this blog

How to check contact read email or not when send email to Individual?

Image
Clash Royale CLAN TAG #URR8PPP up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Sep 26 at 0:12 ToanLuong 49 9 add a comment  |  up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this questi...
Read more

Displaying single band from multi-band raster using QGIS

Image
Clash Royale CLAN TAG #URR8PPP 1 How can I extract a single band from multi-band raster in QGIS? I have an remote sensed image which has 6 bands (including NDVI band), I want to display each band separately, but have no idea how to do. I have seen some questions similar here but none worked for me. The original image (has 6 bands) is: I want to display the band 6 which should be like this: But I tried gdal_translate, and couldn't get the correct result. What I have got is: qgis raster multi-band share | improve this question edited Mar 5 at 0:53 Summer asked Mar 4 at 6:42 Summer Summer 23 6 Is this any help gis.stackexchange.com/questions/220658/… ? if not gis.stackexchange.com/questions/62133/… might help. – Michael Stimson Mar 4 at 6:46 Thanks for answering but when I used gdal_translate, qgis showed that 'Error 4: Kayena.tif: No such file or directory". Would you know how to fi...
Read more

How many registers does an x86_64 CPU actually have?

Image
Clash Royale CLAN TAG #URR8PPP up vote 2 down vote favorite I am currently learning reverse engineering and am studying the flags register. I had in my mind that rflags was just another name for one of the 16 general purpose registers, for example rax or rbx . But it looks like rflags is actually an additional register. So that makes 17 registers in total... how many more could there be? I have spent at least an hour on this and found numerous different answers. The best answer so far is this, which says that there are 40 registers in total. 16 General Purpose Registers 2 Status Registers 6 Code Segment Registers 16 SSE Registers 8 FPU/MMX Registers But if I add that up, I get 48. Could anybody provide an official answer on how many registers an x86_64 CPU has (e.g. an Intel i7). Additionally, I have seen references to 'hardware' and 'architectural' registers. What are those registers and how many are there? register x86-64 share | improve this...
Read more