How to download multiple files from website with wget and perform functions on downloaded file before next file is downloaded

I am using wget to download a number of json files from a website. However, I want to perform some sort of function on these files to reduce their size before downloading the next file.

Is there a way to specify a wget command and then perform a series of functions on the downloaded file before the next file is downloaded?

An example of my script is below:

for FILE in wget -A json -np -nd -r **website URL**
 do **somefunctions** 
 rm $FILE
done

but this doesn't seem to work.

edited Jan 2 at 18:01

nohillside

2,382919

asked Jan 2 at 17:16

lstbl

213

1

Your -A option needs to be something along the lines of -A "*.json" if you wish to download all the .json files. However since wget is non-interactive you will simply download all the files of that type first before it will move on to the next line... You could have a file listing each individual json file url, assigning them to a variable that loops, downloading each individually then completing a function before moving on to the next line in the file. However you would most likely need to manually create such a file...

– kemotep
Jan 2 at 18:30

that's what I figured... is there a way to list all of the file names using wget without actually downloading them? Can't seem to find anything in the help menu or by googling...

– lstbl
Jan 2 at 20:15

1

You would have to use curl and pipe the results to a file. But the website you are "scrapping" for these results could block directory listing for security reasons so you will have to copy and paste urls manually if they prevent this.

– kemotep
Jan 2 at 20:21

add a comment |

I am using wget to download a number of json files from a website. However, I want to perform some sort of function on these files to reduce their size before downloading the next file.

Is there a way to specify a wget command and then perform a series of functions on the downloaded file before the next file is downloaded?

An example of my script is below:

for FILE in wget -A json -np -nd -r **website URL**
 do **somefunctions** 
 rm $FILE
done

but this doesn't seem to work.

edited Jan 2 at 18:01

nohillside

2,382919

asked Jan 2 at 17:16

lstbl

213

1

Your -A option needs to be something along the lines of -A "*.json" if you wish to download all the .json files. However since wget is non-interactive you will simply download all the files of that type first before it will move on to the next line... You could have a file listing each individual json file url, assigning them to a variable that loops, downloading each individually then completing a function before moving on to the next line in the file. However you would most likely need to manually create such a file...

– kemotep
Jan 2 at 18:30

that's what I figured... is there a way to list all of the file names using wget without actually downloading them? Can't seem to find anything in the help menu or by googling...

– lstbl
Jan 2 at 20:15

1

You would have to use curl and pipe the results to a file. But the website you are "scrapping" for these results could block directory listing for security reasons so you will have to copy and paste urls manually if they prevent this.

– kemotep
Jan 2 at 20:21

add a comment |

I am using wget to download a number of json files from a website. However, I want to perform some sort of function on these files to reduce their size before downloading the next file.

Is there a way to specify a wget command and then perform a series of functions on the downloaded file before the next file is downloaded?

An example of my script is below:

for FILE in wget -A json -np -nd -r **website URL**
 do **somefunctions** 
 rm $FILE
done

but this doesn't seem to work.

edited Jan 2 at 18:01

nohillside

2,382919

asked Jan 2 at 17:16

lstbl

213

I am using wget to download a number of json files from a website. However, I want to perform some sort of function on these files to reduce their size before downloading the next file.

Is there a way to specify a wget command and then perform a series of functions on the downloaded file before the next file is downloaded?

An example of my script is below:

for FILE in wget -A json -np -nd -r **website URL**
 do **somefunctions** 
 rm $FILE
done

but this doesn't seem to work.

shell-script wget download

edited Jan 2 at 18:01

nohillside

2,382919

asked Jan 2 at 17:16

lstbl

213

edited Jan 2 at 18:01

nohillside

2,382919

asked Jan 2 at 17:16

lstbl

213

edited Jan 2 at 18:01

nohillside

2,382919

edited Jan 2 at 18:01

nohillside

2,382919

edited Jan 2 at 18:01

nohillside

2,382919

asked Jan 2 at 17:16

lstbl

213

asked Jan 2 at 17:16

lstbl

213

asked Jan 2 at 17:16

lstbl

213

1

Your -A option needs to be something along the lines of -A "*.json" if you wish to download all the .json files. However since wget is non-interactive you will simply download all the files of that type first before it will move on to the next line... You could have a file listing each individual json file url, assigning them to a variable that loops, downloading each individually then completing a function before moving on to the next line in the file. However you would most likely need to manually create such a file...

– kemotep
Jan 2 at 18:30

that's what I figured... is there a way to list all of the file names using wget without actually downloading them? Can't seem to find anything in the help menu or by googling...

– lstbl
Jan 2 at 20:15

1

You would have to use curl and pipe the results to a file. But the website you are "scrapping" for these results could block directory listing for security reasons so you will have to copy and paste urls manually if they prevent this.

– kemotep
Jan 2 at 20:21

add a comment |

1

Your -A option needs to be something along the lines of -A "*.json" if you wish to download all the .json files. However since wget is non-interactive you will simply download all the files of that type first before it will move on to the next line... You could have a file listing each individual json file url, assigning them to a variable that loops, downloading each individually then completing a function before moving on to the next line in the file. However you would most likely need to manually create such a file...

– kemotep
Jan 2 at 18:30

that's what I figured... is there a way to list all of the file names using wget without actually downloading them? Can't seem to find anything in the help menu or by googling...

– lstbl
Jan 2 at 20:15

1

You would have to use curl and pipe the results to a file. But the website you are "scrapping" for these results could block directory listing for security reasons so you will have to copy and paste urls manually if they prevent this.

– kemotep
Jan 2 at 20:21

Your -A option needs to be something along the lines of -A "*.json" if you wish to download all the .json files. However since wget is non-interactive you will simply download all the files of that type first before it will move on to the next line... You could have a file listing each individual json file url, assigning them to a variable that loops, downloading each individually then completing a function before moving on to the next line in the file. However you would most likely need to manually create such a file...

– kemotep
Jan 2 at 18:30

that's what I figured... is there a way to list all of the file names using wget without actually downloading them? Can't seem to find anything in the help menu or by googling...

– lstbl
Jan 2 at 20:15

You would have to use curl and pipe the results to a file. But the website you are "scrapping" for these results could block directory listing for security reasons so you will have to copy and paste urls manually if they prevent this.

– kemotep
Jan 2 at 20:21

add a comment |

1 Answer
1

active

oldest

votes

Ended up using the command

wget -O **URL** | awk **file suffix** | cut **delimiter1** | cut **delimiter2** > filenames.txt

then I just iterated through that file to download the files one at a time:

while read FILE; do **commands**; done <filenames.txt

Not extremely elegant, but it did the trick!

answered Jan 2 at 20:39

lstbl

213

add a comment |

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f492052%2fhow-to-download-multiple-files-from-website-with-wget-and-perform-functions-on-d%23new-answer', 'question_page');

);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

Ended up using the command

wget -O **URL** | awk **file suffix** | cut **delimiter1** | cut **delimiter2** > filenames.txt

then I just iterated through that file to download the files one at a time:

while read FILE; do **commands**; done <filenames.txt

Not extremely elegant, but it did the trick!

answered Jan 2 at 20:39

lstbl

213

add a comment |

Ended up using the command

wget -O **URL** | awk **file suffix** | cut **delimiter1** | cut **delimiter2** > filenames.txt

then I just iterated through that file to download the files one at a time:

while read FILE; do **commands**; done <filenames.txt

Not extremely elegant, but it did the trick!

answered Jan 2 at 20:39

lstbl

213

add a comment |

Ended up using the command

wget -O **URL** | awk **file suffix** | cut **delimiter1** | cut **delimiter2** > filenames.txt

then I just iterated through that file to download the files one at a time:

while read FILE; do **commands**; done <filenames.txt

Not extremely elegant, but it did the trick!

answered Jan 2 at 20:39

lstbl

213

Ended up using the command

wget -O **URL** | awk **file suffix** | cut **delimiter1** | cut **delimiter2** > filenames.txt

then I just iterated through that file to download the files one at a time:

while read FILE; do **commands**; done <filenames.txt

Not extremely elegant, but it did the trick!

answered Jan 2 at 20:39

lstbl

213

answered Jan 2 at 20:39

lstbl

213

answered Jan 2 at 20:39

lstbl

213

answered Jan 2 at 20:39

lstbl

213

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Unix & Linux Stack Exchange!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

搜尋此網誌

mjhjmtu