Using xargs instead of GNU Parallel

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












GNU Parallel can generate all combinations of input, export a function and run it on 3 remote servers without mixing the output, retrying the command if it fails:



#!/bin/bash

env_parallel --record-env
process_output() =1; @a=split//;
 while(@a)
 print ".",shift @a;
 select($a,$a,$a,0.100);
 ';

env_parallel --env _ --retries 4 -Sserver1..3,: 
 'echo X=1 Y=2 Z=3 | process_output' 
 ::: "a' "b"1..3 ::: "c" 'd"3..4 ::: "e$ f'"7..8


Output:



.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.


xargs -P$(nproc) can run one job per processor.



Given



  • the function
    process_output,

  • the servers to run on (server1, server2, server3),

  • the argument groups (a' "b1, a' "b2, a' "b3;c" 'd1, c" 'd2, c" 'd3;e$ f'7, e$ f'8)

how can you emulate GNU Parallel using xargs to get the same done using process_output and the servers?




xargs




share|improve this question






















  • you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
    – RomanPerekhrest
    Nov 19 '17 at 8:08










  • Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
    – Fedor Dikarev
    Nov 19 '17 at 12:21










  • I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
    – David Favor
    Nov 19 '17 at 13:05










  • @ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
    – Hannu
    Nov 19 '17 at 16:14














up vote
0
down vote

favorite












GNU Parallel can generate all combinations of input, export a function and run it on 3 remote servers without mixing the output, retrying the command if it fails:



#!/bin/bash

env_parallel --record-env
process_output() =1; @a=split//;
 while(@a)
 print ".",shift @a;
 select($a,$a,$a,0.100);
 ';

env_parallel --env _ --retries 4 -Sserver1..3,: 
 'echo X=1 Y=2 Z=3 | process_output' 
 ::: "a' "b"1..3 ::: "c" 'd"3..4 ::: "e$ f'"7..8


Output:



.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.


xargs -P$(nproc) can run one job per processor.



Given



  • the function
    process_output,

  • the servers to run on (server1, server2, server3),

  • the argument groups (a' "b1, a' "b2, a' "b3;c" 'd1, c" 'd2, c" 'd3;e$ f'7, e$ f'8)

how can you emulate GNU Parallel using xargs to get the same done using process_output and the servers?




xargs




share|improve this question






















  • you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
    – RomanPerekhrest
    Nov 19 '17 at 8:08










  • Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
    – Fedor Dikarev
    Nov 19 '17 at 12:21










  • I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
    – David Favor
    Nov 19 '17 at 13:05










  • @ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
    – Hannu
    Nov 19 '17 at 16:14












up vote
0
down vote

favorite









up vote
0
down vote

favorite











GNU Parallel can generate all combinations of input, export a function and run it on 3 remote servers without mixing the output, retrying the command if it fails:



#!/bin/bash

env_parallel --record-env
process_output() =1; @a=split//;
 while(@a)
 print ".",shift @a;
 select($a,$a,$a,0.100);
 ';

env_parallel --env _ --retries 4 -Sserver1..3,: 
 'echo X=1 Y=2 Z=3 | process_output' 
 ::: "a' "b"1..3 ::: "c" 'd"3..4 ::: "e$ f'"7..8


Output:



.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.


xargs -P$(nproc) can run one job per processor.



Given



  • the function
    process_output,

  • the servers to run on (server1, server2, server3),

  • the argument groups (a' "b1, a' "b2, a' "b3;c" 'd1, c" 'd2, c" 'd3;e$ f'7, e$ f'8)

how can you emulate GNU Parallel using xargs to get the same done using process_output and the servers?




xargs




share|improve this question














GNU Parallel can generate all combinations of input, export a function and run it on 3 remote servers without mixing the output, retrying the command if it fails:



#!/bin/bash

env_parallel --record-env
process_output() =1; @a=split//;
 while(@a)
 print ".",shift @a;
 select($a,$a,$a,0.100);
 ';

env_parallel --env _ --retries 4 -Sserver1..3,: 
 'echo X=1 Y=2 Z=3 | process_output' 
 ::: "a' "b"1..3 ::: "c" 'd"3..4 ::: "e$ f'"7..8


Output:



.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.


xargs -P$(nproc) can run one job per processor.



Given



  • the function
    process_output,

  • the servers to run on (server1, server2, server3),

  • the argument groups (a' "b1, a' "b2, a' "b3;c" 'd1, c" 'd2, c" 'd3;e$ f'7, e$ f'8)

how can you emulate GNU Parallel using xargs to get the same done using process_output and the servers?





xargs





share|improve this question













share|improve this question




share|improve this question








edited Nov 19 '17 at 12:13

























asked Nov 19 '17 at 6:44









Ole Tange

11.4k1344102




11.4k1344102











  • you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
    – RomanPerekhrest
    Nov 19 '17 at 8:08










  • Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
    – Fedor Dikarev
    Nov 19 '17 at 12:21










  • I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
    – David Favor
    Nov 19 '17 at 13:05










  • @ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
    – Hannu
    Nov 19 '17 at 16:14
















  • you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
    – RomanPerekhrest
    Nov 19 '17 at 8:08










  • Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
    – Fedor Dikarev
    Nov 19 '17 at 12:21










  • I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
    – David Favor
    Nov 19 '17 at 13:05










  • @ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
    – Hannu
    Nov 19 '17 at 16:14















you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
– RomanPerekhrest
Nov 19 '17 at 8:08




you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers?
– RomanPerekhrest
Nov 19 '17 at 8:08












Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
– Fedor Dikarev
Nov 19 '17 at 12:21




Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ?
– Fedor Dikarev
Nov 19 '17 at 12:21












I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
– David Favor
Nov 19 '17 at 13:05




I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers.
– David Favor
Nov 19 '17 at 13:05












@ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
– Hannu
Nov 19 '17 at 16:14




@ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
– Hannu
Nov 19 '17 at 16:14















active

oldest

votes











Your Answer







StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f405552%2fusing-xargs-instead-of-gnu-parallel%23new-answer', 'question_page');

);

Post as a guest






















active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes















 

draft saved


draft discarded















































 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f405552%2fusing-xargs-instead-of-gnu-parallel%23new-answer', 'question_page');

);

Post as a guest
































































-

Popular posts from this blog

How to check contact read email or not when send email to Individual?

Image
Clash Royale CLAN TAG #URR8PPP up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Sep 26 at 0:12 ToanLuong 49 9 add a comment  |  up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Se
Read more

Bahrain

Image
For other uses, see Bahrain (disambiguation). Sovereign island state in the Persian Gulf Kingdom of Bahrain مملكة البحرين   (Arabic) Mamlakat al-Baḥrayn Flag Coat of arms Anthem:  نشيد البحرين الوطني Bahrainona Our Bahrain Location of   Bahrain    (circled in red) Capital and largest city Manama 26°13′N 50°35′E  /  26.217°N 50.583°E  / 26.217; 50.583 Official languages Arabic Ethnic groups (2010 [1] ) 50.7% Arab (46% Bahrani, 4.7% other Arabs) 45.5% Asian 1% European 1.2% Others Religion Islam Demonym(s) Bahraini Government Unitary constitutional monarchy • King Hamad bin Isa Al Khalifa • Crown Prince Salman bin Hamad bin Isa Al Khalifa • Prime Minister Khalifa bin Salman Al Khalifa Legislature National Assembly • Upper house Consultative Council • Lower house Council of Representatives Independence • Declared Independence [2] 14 August 1971 • from United Kingdom [1] 15 August 1971 • Admitted to the United Nations 21 September 1971 • Kingdom of Bahrain 14 February 2002
Read more

Postfix configuration issue with fips on centos 7; mailgun relay

Image
Clash Royale CLAN TAG #URR8PPP up vote 0 down vote favorite I am trying to setup postfix to relay all mail generated on the local machine via SMTP to a mailgun relay. I have used the mailgun relay before with success on an ubuntu server, but I am migrating to a Centos 7 server which I will be running in FIPS mode. There error log is below, slightly sanitized. I have a small enough network that I choose to have each machine reach out to mailgun individually (this the loopback-only, 127.0.0.0/8 restrictions) and no firewall open port allowing smtp in to the machine. I assume the FIPS mode (and with it disabling of MD5) is causing problems, but I don't know how to overcome it or if it is even possible for tls_fprint to use some supported hash such as sha256 or sha512. However, the relay=none is slightly concerning since I have relayhost set, but perhaps that is because the smtp process is failing? Any help would be appreciated! postconf -n: alias_database = hash:/etc/alia
Read more