Oneliner for outputting to file only if string is matched (bash)?

I am making a script where I curl a URL and output to a file like this:

curl http://example.com/$1 > $1

Is there any way to make it dismiss anything that doesn't include, say <head>? I could of course do something like this:

web="$(curl http://example.com/$1)"
if $(echo "$web" | grep -q "<head>"); then
printf "$web" > "$1"
fi

But this script may potentially be ran very many times, so I would like to save time.

asked Jan 12 at 20:33

DisplayName

4,43894580

Are you trying to catch cases where the URL does not return a valid HTML page, or are you trying to catch HTML error responses?

– datUser
Jan 12 at 20:50

@datUser It always returns a valid page, but I only want to output some valid pages to a file (like if it includes "<head"> in my given example). The 4 line script I made works for this purpose, but I am looking for a shorter, faster alternative.

– DisplayName
Jan 12 at 20:52

1

shorter does not mean faster. Any script can be one line by using ; excessively.

– Jesse_b
Jan 12 at 21:06

@Jesse_b True, but I was thinking that something involving maybe a double pipe (||) or something instead of if would be faster

– DisplayName
Jan 12 at 22:04

add a comment |

I am making a script where I curl a URL and output to a file like this:

curl http://example.com/$1 > $1

Is there any way to make it dismiss anything that doesn't include, say <head>? I could of course do something like this:

web="$(curl http://example.com/$1)"
if $(echo "$web" | grep -q "<head>"); then
printf "$web" > "$1"
fi

But this script may potentially be ran very many times, so I would like to save time.

asked Jan 12 at 20:33

DisplayName

4,43894580

Are you trying to catch cases where the URL does not return a valid HTML page, or are you trying to catch HTML error responses?

– datUser
Jan 12 at 20:50

@datUser It always returns a valid page, but I only want to output some valid pages to a file (like if it includes "<head"> in my given example). The 4 line script I made works for this purpose, but I am looking for a shorter, faster alternative.

– DisplayName
Jan 12 at 20:52

1

shorter does not mean faster. Any script can be one line by using ; excessively.

– Jesse_b
Jan 12 at 21:06

@Jesse_b True, but I was thinking that something involving maybe a double pipe (||) or something instead of if would be faster

– DisplayName
Jan 12 at 22:04

add a comment |

I am making a script where I curl a URL and output to a file like this:

curl http://example.com/$1 > $1

Is there any way to make it dismiss anything that doesn't include, say <head>? I could of course do something like this:

web="$(curl http://example.com/$1)"
if $(echo "$web" | grep -q "<head>"); then
printf "$web" > "$1"
fi

But this script may potentially be ran very many times, so I would like to save time.

asked Jan 12 at 20:33

DisplayName

4,43894580

I am making a script where I curl a URL and output to a file like this:

curl http://example.com/$1 > $1

Is there any way to make it dismiss anything that doesn't include, say <head>? I could of course do something like this:

web="$(curl http://example.com/$1)"
if $(echo "$web" | grep -q "<head>"); then
printf "$web" > "$1"
fi

But this script may potentially be ran very many times, so I would like to save time.

bash text-processing io-redirection io

asked Jan 12 at 20:33

DisplayName

4,43894580

asked Jan 12 at 20:33

DisplayName

4,43894580

asked Jan 12 at 20:33

DisplayName

4,43894580

asked Jan 12 at 20:33

DisplayName

4,43894580

asked Jan 12 at 20:33

DisplayName

4,43894580

Are you trying to catch cases where the URL does not return a valid HTML page, or are you trying to catch HTML error responses?

– datUser
Jan 12 at 20:50

@datUser It always returns a valid page, but I only want to output some valid pages to a file (like if it includes "<head"> in my given example). The 4 line script I made works for this purpose, but I am looking for a shorter, faster alternative.

– DisplayName
Jan 12 at 20:52

1

shorter does not mean faster. Any script can be one line by using ; excessively.

– Jesse_b
Jan 12 at 21:06

@Jesse_b True, but I was thinking that something involving maybe a double pipe (||) or something instead of if would be faster

– DisplayName
Jan 12 at 22:04

add a comment |

Are you trying to catch cases where the URL does not return a valid HTML page, or are you trying to catch HTML error responses?

– datUser
Jan 12 at 20:50

@datUser It always returns a valid page, but I only want to output some valid pages to a file (like if it includes "<head"> in my given example). The 4 line script I made works for this purpose, but I am looking for a shorter, faster alternative.

– DisplayName
Jan 12 at 20:52

1

shorter does not mean faster. Any script can be one line by using ; excessively.

– Jesse_b
Jan 12 at 21:06

@Jesse_b True, but I was thinking that something involving maybe a double pipe (||) or something instead of if would be faster

– DisplayName
Jan 12 at 22:04

Are you trying to catch cases where the URL does not return a valid HTML page, or are you trying to catch HTML error responses?

– datUser
Jan 12 at 20:50

@datUser It always returns a valid page, but I only want to output some valid pages to a file (like if it includes "<head"> in my given example). The 4 line script I made works for this purpose, but I am looking for a shorter, faster alternative.

– DisplayName
Jan 12 at 20:52

shorter does not mean faster. Any script can be one line by using ; excessively.

– Jesse_b
Jan 12 at 21:06

@Jesse_b True, but I was thinking that something involving maybe a double pipe (||) or something instead of if would be faster

– DisplayName
Jan 12 at 22:04

add a comment |

3 Answers
3

active

oldest

votes

Just save it to a file and delete it if it does not include that tag:

curl "http://example.com/$1" >"$1"
! grep -qF '<head>' "$1" && rm "$1"

Or, if you want to complicate it slightly,

if curl "http://example.com/$1" | tee "$1" | ! grep -qF '<head>'
then
 rm "$1"
fi

Or do all the fetching first, then go through the resulting files and delete them,

mkdir outdir # assuming this is not pre-existing

# iterates over all positional parameters
for path do
 curl "http://example.com/$path" >outdir/"$path"
done

find outdir -type f ! -exec grep -qF '<head>' ; -delete

But this would only work if you want to get e.g. all the known files under a fixed path.

edited Jan 12 at 21:25

answered Jan 12 at 21:16

Kusalananda

127k16239393

add a comment |

You could do something like this:

grep -hIz '<head>' < <(curl -s "http://example.com/$1") > "$1" || rm "$1"

This will supress all output unless it contains <head> in which case it will print all output. If no match is made an empty file will be created anyway so it will need to be removed.

answered Jan 12 at 21:29

Jesse_b

12.4k23066

add a comment |

Just for fun:

curl "http://example.com/$1" | tee temp | grep -q derp && cat temp; rm temp

Saves curl output to file and pipes the output to grep which then searches for derp in the output and if found outputs the content of the curl command to stdout, if derp is not found, there is no script output. The temp file is then removed.

Not sure you can do this without 'buffering' to a file, since bash does not have a way to conditionally buffer output from a command.

edited Jan 12 at 21:30

answered Jan 12 at 21:22

datUser

2,5061133

add a comment |

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f494154%2foneliner-for-outputting-to-file-only-if-string-is-matched-bash%23new-answer', 'question_page');

);

Post as a guest

Name

Required, but never shown

3 Answers
3

active

oldest

votes

3 Answers
3

active

oldest

votes

Just save it to a file and delete it if it does not include that tag:

curl "http://example.com/$1" >"$1"
! grep -qF '<head>' "$1" && rm "$1"

Or, if you want to complicate it slightly,

if curl "http://example.com/$1" | tee "$1" | ! grep -qF '<head>'
then
 rm "$1"
fi

Or do all the fetching first, then go through the resulting files and delete them,

mkdir outdir # assuming this is not pre-existing

# iterates over all positional parameters
for path do
 curl "http://example.com/$path" >outdir/"$path"
done

find outdir -type f ! -exec grep -qF '<head>' ; -delete

But this would only work if you want to get e.g. all the known files under a fixed path.

edited Jan 12 at 21:25

answered Jan 12 at 21:16

Kusalananda

127k16239393

add a comment |

Just save it to a file and delete it if it does not include that tag:

curl "http://example.com/$1" >"$1"
! grep -qF '<head>' "$1" && rm "$1"

Or, if you want to complicate it slightly,

if curl "http://example.com/$1" | tee "$1" | ! grep -qF '<head>'
then
 rm "$1"
fi

Or do all the fetching first, then go through the resulting files and delete them,

mkdir outdir # assuming this is not pre-existing

# iterates over all positional parameters
for path do
 curl "http://example.com/$path" >outdir/"$path"
done

find outdir -type f ! -exec grep -qF '<head>' ; -delete

But this would only work if you want to get e.g. all the known files under a fixed path.

edited Jan 12 at 21:25

answered Jan 12 at 21:16

Kusalananda

127k16239393

add a comment |

Just save it to a file and delete it if it does not include that tag:

curl "http://example.com/$1" >"$1"
! grep -qF '<head>' "$1" && rm "$1"

Or, if you want to complicate it slightly,

if curl "http://example.com/$1" | tee "$1" | ! grep -qF '<head>'
then
 rm "$1"
fi

Or do all the fetching first, then go through the resulting files and delete them,

mkdir outdir # assuming this is not pre-existing

# iterates over all positional parameters
for path do
 curl "http://example.com/$path" >outdir/"$path"
done

find outdir -type f ! -exec grep -qF '<head>' ; -delete

But this would only work if you want to get e.g. all the known files under a fixed path.

edited Jan 12 at 21:25

answered Jan 12 at 21:16

Kusalananda

127k16239393

Just save it to a file and delete it if it does not include that tag:

curl "http://example.com/$1" >"$1"
! grep -qF '<head>' "$1" && rm "$1"

Or, if you want to complicate it slightly,

if curl "http://example.com/$1" | tee "$1" | ! grep -qF '<head>'
then
 rm "$1"
fi

Or do all the fetching first, then go through the resulting files and delete them,

mkdir outdir # assuming this is not pre-existing

# iterates over all positional parameters
for path do
 curl "http://example.com/$path" >outdir/"$path"
done

find outdir -type f ! -exec grep -qF '<head>' ; -delete

But this would only work if you want to get e.g. all the known files under a fixed path.

edited Jan 12 at 21:25

answered Jan 12 at 21:16

Kusalananda

127k16239393

edited Jan 12 at 21:25

answered Jan 12 at 21:16

Kusalananda

127k16239393

answered Jan 12 at 21:16

Kusalananda

127k16239393

answered Jan 12 at 21:16

Kusalananda

127k16239393

add a comment |

You could do something like this:

grep -hIz '<head>' < <(curl -s "http://example.com/$1") > "$1" || rm "$1"

This will supress all output unless it contains <head> in which case it will print all output. If no match is made an empty file will be created anyway so it will need to be removed.

answered Jan 12 at 21:29

Jesse_b

12.4k23066

add a comment |

You could do something like this:

grep -hIz '<head>' < <(curl -s "http://example.com/$1") > "$1" || rm "$1"

This will supress all output unless it contains <head> in which case it will print all output. If no match is made an empty file will be created anyway so it will need to be removed.

answered Jan 12 at 21:29

Jesse_b

12.4k23066

add a comment |

You could do something like this:

grep -hIz '<head>' < <(curl -s "http://example.com/$1") > "$1" || rm "$1"

This will supress all output unless it contains <head> in which case it will print all output. If no match is made an empty file will be created anyway so it will need to be removed.

answered Jan 12 at 21:29

Jesse_b

12.4k23066

You could do something like this:

grep -hIz '<head>' < <(curl -s "http://example.com/$1") > "$1" || rm "$1"

This will supress all output unless it contains <head> in which case it will print all output. If no match is made an empty file will be created anyway so it will need to be removed.

answered Jan 12 at 21:29

Jesse_b

12.4k23066

answered Jan 12 at 21:29

Jesse_b

12.4k23066

answered Jan 12 at 21:29

Jesse_b

12.4k23066

answered Jan 12 at 21:29

Jesse_b

12.4k23066

add a comment |

Just for fun:

curl "http://example.com/$1" | tee temp | grep -q derp && cat temp; rm temp

Not sure you can do this without 'buffering' to a file, since bash does not have a way to conditionally buffer output from a command.

edited Jan 12 at 21:30

answered Jan 12 at 21:22

datUser

2,5061133

add a comment |

Just for fun:

curl "http://example.com/$1" | tee temp | grep -q derp && cat temp; rm temp

Not sure you can do this without 'buffering' to a file, since bash does not have a way to conditionally buffer output from a command.

edited Jan 12 at 21:30

answered Jan 12 at 21:22

datUser

2,5061133

add a comment |

Just for fun:

curl "http://example.com/$1" | tee temp | grep -q derp && cat temp; rm temp

Not sure you can do this without 'buffering' to a file, since bash does not have a way to conditionally buffer output from a command.

edited Jan 12 at 21:30

answered Jan 12 at 21:22

datUser

2,5061133

Just for fun:

curl "http://example.com/$1" | tee temp | grep -q derp && cat temp; rm temp

Not sure you can do this without 'buffering' to a file, since bash does not have a way to conditionally buffer output from a command.

edited Jan 12 at 21:30

answered Jan 12 at 21:22

datUser

2,5061133

edited Jan 12 at 21:30

answered Jan 12 at 21:22

datUser

2,5061133

answered Jan 12 at 21:22

datUser

2,5061133

answered Jan 12 at 21:22

datUser

2,5061133

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Unix & Linux Stack Exchange!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

搜尋此網誌

mjhjmtu