Run curl commands in parallel using xargs with inputs from csv

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












I have a csv file with several columns including a sequence number and an image URL. The file is 450+ lines.



1,text1,text2,http://image_a.jpg
2,text1,text2,http://image_b.jpg
3,text1,text2,http://image_c.jpg
.
.
.


The goal is to download the image at the URL with a filename being the sequence. I accomplished this successfully with the following script



while IFS=, read col1 col2 col3 col4
do
 curl $col4 > $col1.jpg
done < myfile.csv


Now I want to run the curl commands in parallel using xargs -P. (I don't want to just background using & as I want to limit the number of processes.) I see examples using -I and -R but cannot figure out how to use multiple fields from a line of the csv file in the xargs command.



I realize gnu parallel and wget may be better options but they are not available in my unix environment.



Thanks.






curl xargs







share|improve this question









New contributor




Scott Nelson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.



















  • maybe this will help .... stackoverflow.com/questions/38160/…
    – jsotola
    6 mins ago















up vote
0
down vote

favorite












I have a csv file with several columns including a sequence number and an image URL. The file is 450+ lines.



1,text1,text2,http://image_a.jpg
2,text1,text2,http://image_b.jpg
3,text1,text2,http://image_c.jpg
.
.
.


The goal is to download the image at the URL with a filename being the sequence. I accomplished this successfully with the following script



while IFS=, read col1 col2 col3 col4
do
 curl $col4 > $col1.jpg
done < myfile.csv


Now I want to run the curl commands in parallel using xargs -P. (I don't want to just background using & as I want to limit the number of processes.) I see examples using -I and -R but cannot figure out how to use multiple fields from a line of the csv file in the xargs command.



I realize gnu parallel and wget may be better options but they are not available in my unix environment.



Thanks.






curl xargs







share|improve this question









New contributor




Scott Nelson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.



















  • maybe this will help .... stackoverflow.com/questions/38160/…
    – jsotola
    6 mins ago













up vote
0
down vote

favorite









up vote
0
down vote

favorite











I have a csv file with several columns including a sequence number and an image URL. The file is 450+ lines.



1,text1,text2,http://image_a.jpg
2,text1,text2,http://image_b.jpg
3,text1,text2,http://image_c.jpg
.
.
.


The goal is to download the image at the URL with a filename being the sequence. I accomplished this successfully with the following script



while IFS=, read col1 col2 col3 col4
do
 curl $col4 > $col1.jpg
done < myfile.csv


Now I want to run the curl commands in parallel using xargs -P. (I don't want to just background using & as I want to limit the number of processes.) I see examples using -I and -R but cannot figure out how to use multiple fields from a line of the csv file in the xargs command.



I realize gnu parallel and wget may be better options but they are not available in my unix environment.



Thanks.






curl xargs







share|improve this question









New contributor




Scott Nelson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











I have a csv file with several columns including a sequence number and an image URL. The file is 450+ lines.



1,text1,text2,http://image_a.jpg
2,text1,text2,http://image_b.jpg
3,text1,text2,http://image_c.jpg
.
.
.


The goal is to download the image at the URL with a filename being the sequence. I accomplished this successfully with the following script



while IFS=, read col1 col2 col3 col4
do
 curl $col4 > $col1.jpg
done < myfile.csv


Now I want to run the curl commands in parallel using xargs -P. (I don't want to just background using & as I want to limit the number of processes.) I see examples using -I and -R but cannot figure out how to use multiple fields from a line of the csv file in the xargs command.



I realize gnu parallel and wget may be better options but they are not available in my unix environment.



Thanks.






curl xargs




curl xargs






share|improve this question









New contributor




Scott Nelson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question









New contributor




Scott Nelson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited 5 mins ago





















New contributor




Scott Nelson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked 25 mins ago









Scott Nelson

1011




1011




New contributor




Scott Nelson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





Scott Nelson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






Scott Nelson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











  • maybe this will help .... stackoverflow.com/questions/38160/…
    – jsotola
    6 mins ago

















  • maybe this will help .... stackoverflow.com/questions/38160/…
    – jsotola
    6 mins ago
















maybe this will help .... stackoverflow.com/questions/38160/…
– jsotola
6 mins ago





maybe this will help .... stackoverflow.com/questions/38160/…
– jsotola
6 mins ago
















active

oldest

votes











Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);






Scott Nelson is a new contributor. Be nice, and check out our Code of Conduct.









 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f479233%2frun-curl-commands-in-parallel-using-xargs-with-inputs-from-csv%23new-answer', 'question_page');

);

Post as a guest






















active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes








Scott Nelson is a new contributor. Be nice, and check out our Code of Conduct.









 

draft saved


draft discarded


















Scott Nelson is a new contributor. Be nice, and check out our Code of Conduct.












Scott Nelson is a new contributor. Be nice, and check out our Code of Conduct.











Scott Nelson is a new contributor. Be nice, and check out our Code of Conduct.













 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f479233%2frun-curl-commands-in-parallel-using-xargs-with-inputs-from-csv%23new-answer', 'question_page');

);

Post as a guest
































































-

Popular posts from this blog

How to check contact read email or not when send email to Individual?

Image
Clash Royale CLAN TAG #URR8PPP up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Sep 26 at 0:12 ToanLuong 49 9 add a comment  |  up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Se
Read more

Bahrain

Image
For other uses, see Bahrain (disambiguation). Sovereign island state in the Persian Gulf Kingdom of Bahrain مملكة البحرين   (Arabic) Mamlakat al-Baḥrayn Flag Coat of arms Anthem:  نشيد البحرين الوطني Bahrainona Our Bahrain Location of   Bahrain    (circled in red) Capital and largest city Manama 26°13′N 50°35′E  /  26.217°N 50.583°E  / 26.217; 50.583 Official languages Arabic Ethnic groups (2010 [1] ) 50.7% Arab (46% Bahrani, 4.7% other Arabs) 45.5% Asian 1% European 1.2% Others Religion Islam Demonym(s) Bahraini Government Unitary constitutional monarchy • King Hamad bin Isa Al Khalifa • Crown Prince Salman bin Hamad bin Isa Al Khalifa • Prime Minister Khalifa bin Salman Al Khalifa Legislature National Assembly • Upper house Consultative Council • Lower house Council of Representatives Independence • Declared Independence [2] 14 August 1971 • from United Kingdom [1] 15 August 1971 • Admitted to the United Nations 21 September 1971 • Kingdom of Bahrain 14 February 2002
Read more

Postfix configuration issue with fips on centos 7; mailgun relay

Image
Clash Royale CLAN TAG #URR8PPP up vote 0 down vote favorite I am trying to setup postfix to relay all mail generated on the local machine via SMTP to a mailgun relay. I have used the mailgun relay before with success on an ubuntu server, but I am migrating to a Centos 7 server which I will be running in FIPS mode. There error log is below, slightly sanitized. I have a small enough network that I choose to have each machine reach out to mailgun individually (this the loopback-only, 127.0.0.0/8 restrictions) and no firewall open port allowing smtp in to the machine. I assume the FIPS mode (and with it disabling of MD5) is causing problems, but I don't know how to overcome it or if it is even possible for tls_fprint to use some supported hash such as sha256 or sha512. However, the relay=none is slightly concerning since I have relayhost set, but perhaps that is because the smtp process is failing? Any help would be appreciated! postconf -n: alias_database = hash:/etc/alia
Read more