How can I harden bash scripts against causing harm when changed in the future?
Clash Royale CLAN TAG#URR8PPP
up vote
43
down vote
favorite
So, I deleted my home folder (or, more precisely, all files I had write access to). What happened is that I had
build="build"
...
rm -rf "$build/"*
...
<do other things with $build>
in a bash script and, after no longer needing $build
, removing the declaration and all its usages -- but the rm
. Bash happily expands to rm -rf /*
. Yea.
I felt stupid, installed the backup, redid the work I lost. Trying to move past the shame.
Now, I wonder: what are techniques to write bash scripts so that such mistakes can't happen, or are at least less likely? For instance, had I written
FileUtils.rm_rf("#build/*")
in a Ruby script, the interpreter would have complained about build
not being declared, so there the language protects me.
What I have considered in bash, besides corraling rm
(which, as many answers in related questions mention, is not unproblematic):
rm -rf "./$build/"*
That would have killed my current work (a Git repo) but nothing else.- A variant/parameterization of
rm
that requires interaction when acting outside of the current directory. (Could not find any.)
Similar effect.
Is that it, or are there other ways to write bash scripts that are "robust" in this sense?
bash shell-script rm
add a comment |Â
up vote
43
down vote
favorite
So, I deleted my home folder (or, more precisely, all files I had write access to). What happened is that I had
build="build"
...
rm -rf "$build/"*
...
<do other things with $build>
in a bash script and, after no longer needing $build
, removing the declaration and all its usages -- but the rm
. Bash happily expands to rm -rf /*
. Yea.
I felt stupid, installed the backup, redid the work I lost. Trying to move past the shame.
Now, I wonder: what are techniques to write bash scripts so that such mistakes can't happen, or are at least less likely? For instance, had I written
FileUtils.rm_rf("#build/*")
in a Ruby script, the interpreter would have complained about build
not being declared, so there the language protects me.
What I have considered in bash, besides corraling rm
(which, as many answers in related questions mention, is not unproblematic):
rm -rf "./$build/"*
That would have killed my current work (a Git repo) but nothing else.- A variant/parameterization of
rm
that requires interaction when acting outside of the current directory. (Could not find any.)
Similar effect.
Is that it, or are there other ways to write bash scripts that are "robust" in this sense?
bash shell-script rm
3
There is no reason torm -rf "$build/*
no matter where the quote marks go.rm -rf "$build
will do the same thing because of thef
.
â Monty Harder
Jul 11 at 14:36
1
Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
â Charles Duffy
Jul 11 at 15:15
@CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
â Raphael
Jul 11 at 15:18
2
Gotcha. Be sure to note the caveats in BashFAQ #112.set -u
isn't nearly as frowned on asset -e
is, but it still does have its gotchas.
â Charles Duffy
Jul 11 at 15:20
2
joke answer: Just put#! /usr/bin/env ruby
at the top of every shell script and forget about bash ;)
â Pod
Jul 12 at 15:25
add a comment |Â
up vote
43
down vote
favorite
up vote
43
down vote
favorite
So, I deleted my home folder (or, more precisely, all files I had write access to). What happened is that I had
build="build"
...
rm -rf "$build/"*
...
<do other things with $build>
in a bash script and, after no longer needing $build
, removing the declaration and all its usages -- but the rm
. Bash happily expands to rm -rf /*
. Yea.
I felt stupid, installed the backup, redid the work I lost. Trying to move past the shame.
Now, I wonder: what are techniques to write bash scripts so that such mistakes can't happen, or are at least less likely? For instance, had I written
FileUtils.rm_rf("#build/*")
in a Ruby script, the interpreter would have complained about build
not being declared, so there the language protects me.
What I have considered in bash, besides corraling rm
(which, as many answers in related questions mention, is not unproblematic):
rm -rf "./$build/"*
That would have killed my current work (a Git repo) but nothing else.- A variant/parameterization of
rm
that requires interaction when acting outside of the current directory. (Could not find any.)
Similar effect.
Is that it, or are there other ways to write bash scripts that are "robust" in this sense?
bash shell-script rm
So, I deleted my home folder (or, more precisely, all files I had write access to). What happened is that I had
build="build"
...
rm -rf "$build/"*
...
<do other things with $build>
in a bash script and, after no longer needing $build
, removing the declaration and all its usages -- but the rm
. Bash happily expands to rm -rf /*
. Yea.
I felt stupid, installed the backup, redid the work I lost. Trying to move past the shame.
Now, I wonder: what are techniques to write bash scripts so that such mistakes can't happen, or are at least less likely? For instance, had I written
FileUtils.rm_rf("#build/*")
in a Ruby script, the interpreter would have complained about build
not being declared, so there the language protects me.
What I have considered in bash, besides corraling rm
(which, as many answers in related questions mention, is not unproblematic):
rm -rf "./$build/"*
That would have killed my current work (a Git repo) but nothing else.- A variant/parameterization of
rm
that requires interaction when acting outside of the current directory. (Could not find any.)
Similar effect.
Is that it, or are there other ways to write bash scripts that are "robust" in this sense?
bash shell-script rm
edited Jul 11 at 14:22
asked Jul 11 at 12:57
Raphael
776820
776820
3
There is no reason torm -rf "$build/*
no matter where the quote marks go.rm -rf "$build
will do the same thing because of thef
.
â Monty Harder
Jul 11 at 14:36
1
Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
â Charles Duffy
Jul 11 at 15:15
@CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
â Raphael
Jul 11 at 15:18
2
Gotcha. Be sure to note the caveats in BashFAQ #112.set -u
isn't nearly as frowned on asset -e
is, but it still does have its gotchas.
â Charles Duffy
Jul 11 at 15:20
2
joke answer: Just put#! /usr/bin/env ruby
at the top of every shell script and forget about bash ;)
â Pod
Jul 12 at 15:25
add a comment |Â
3
There is no reason torm -rf "$build/*
no matter where the quote marks go.rm -rf "$build
will do the same thing because of thef
.
â Monty Harder
Jul 11 at 14:36
1
Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
â Charles Duffy
Jul 11 at 15:15
@CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
â Raphael
Jul 11 at 15:18
2
Gotcha. Be sure to note the caveats in BashFAQ #112.set -u
isn't nearly as frowned on asset -e
is, but it still does have its gotchas.
â Charles Duffy
Jul 11 at 15:20
2
joke answer: Just put#! /usr/bin/env ruby
at the top of every shell script and forget about bash ;)
â Pod
Jul 12 at 15:25
3
3
There is no reason to
rm -rf "$build/*
no matter where the quote marks go. rm -rf "$build
will do the same thing because of the f
.â Monty Harder
Jul 11 at 14:36
There is no reason to
rm -rf "$build/*
no matter where the quote marks go. rm -rf "$build
will do the same thing because of the f
.â Monty Harder
Jul 11 at 14:36
1
1
Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
â Charles Duffy
Jul 11 at 15:15
Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
â Charles Duffy
Jul 11 at 15:15
@CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
â Raphael
Jul 11 at 15:18
@CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
â Raphael
Jul 11 at 15:18
2
2
Gotcha. Be sure to note the caveats in BashFAQ #112.
set -u
isn't nearly as frowned on as set -e
is, but it still does have its gotchas.â Charles Duffy
Jul 11 at 15:20
Gotcha. Be sure to note the caveats in BashFAQ #112.
set -u
isn't nearly as frowned on as set -e
is, but it still does have its gotchas.â Charles Duffy
Jul 11 at 15:20
2
2
joke answer: Just put
#! /usr/bin/env ruby
at the top of every shell script and forget about bash ;)â Pod
Jul 12 at 15:25
joke answer: Just put
#! /usr/bin/env ruby
at the top of every shell script and forget about bash ;)â Pod
Jul 12 at 15:25
add a comment |Â
7 Answers
7
active
oldest
votes
up vote
70
down vote
accepted
set -u
or
set -o nounset
This would make the current shell treat expansions of unset variables as an error:
$ unset build
$ set -u
$ rm -rf "$build"/*
bash: build: unbound variable
set -u
and set -o nounset
are POSIX shell options.
An empty value would not trigger an error though.
For that, use
$ rm -rf "$build:?Error, variable is empty or unset"/*
bash: build: Error, variable is empty or unset
The expansion of $variable:?word
would expand to the value of variable
unless it's empty or unset. If it's empty or unset, the word
would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the :
out would trigger the error only for an unset value, just like under set -u
.
$variable:?word
is a POSIX parameter expansion.
Neither of these would cause an interactive shell to terminate unless set -e
(or set -o errexit
) was also in effect. $variable:?word
causes scripts to exit if the variable is empty or unset. set -u
would cause a script to exit if used together with set -e
.
As for your second question. There is no way to limit rm
to not work outside of the current directory.
The GNU implementation of rm
has a --one-file-system
option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm
call in a function that actually checks the arguments.
As a side note: $build
is exactly equivalent to $build
unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "$buildx"
.
Very good, thanks! 1) Is it$build:?msg
or$build?msg
? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
â Raphael
Jul 11 at 13:31
1
@Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
â Kusalananda
Jul 11 at 13:34
2
@Raphael Ok, I've added a short sentence about what happens without the:
(it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
â Kusalananda
Jul 11 at 14:17
1
Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
â Raphael
Jul 11 at 14:25
2
@MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
â Kusalananda
Jul 13 at 6:52
 |Â
show 5 more comments
up vote
11
down vote
I'm going to suggest normal validation checks using test
/[ ]
You would had been safe if you'd written your script as such:
build="build"
...
[ -n "$build" ] || exit 1
rm -rf "$build/"*
...
The [ -n "$build" ]
checks that "$build"
is a non-zero length string.
The ||
is the logical OR operator in bash. It causes another command to be run if the first one failed.
In this way, had $build
been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).
This also would have protected you in case you removed all uses $build
because [ -n "" ]
will always be false.
The advantage of using test
/[ ]
is there are many other more meaningful checks that it can also use.
For example:
[ -f FILE ] True if FILE exists and is a regular file.
[ -d FILE ] True if FILE exists and is a directory.
[ -O FILE ] True if FILE exists and is owned by the effective user ID.
Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as$variable:?word
@Kusalananda proposes?
â Raphael
Jul 11 at 18:38
@Raphael it works similarly in the case of "does the string have a value" buttest
(i.e.[
) has many other checks that are relevant, such as-d
(the argument is a directory),-f
(the argument is a file),-O
(the file is owned by the current user) and so-on.
â Centimane
Jul 11 at 18:43
1
@Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed$build:?word
along with it? The$variable:?word
format doesn't protect you if you remove all instances of the variable.
â Centimane
Jul 11 at 18:50
1
1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blownif
isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using$v:?w
would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
â Raphael
Jul 12 at 7:29
That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
â Raphael
Jul 12 at 7:31
add a comment |Â
up vote
3
down vote
In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):
# mktemp -d is also a good, reliable choice
trashdir=/tmp/.trash-$USER/trash-`date`
mkdir -p "$trashdir"
...
mv "$build"/* "$trashdir"
...
Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir
destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).
Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.
Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.
2
"This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if/tmp
is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of/tmp
).
â Raphael
Jul 12 at 10:02
1
You could usemktemp -d
to get that temporary directory without treading on another process's toes (and to correctly honour$TMPDIR
).
â Toby Speight
Jul 12 at 15:03
Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
â user117529
Jul 12 at 18:54
add a comment |Â
up vote
2
down vote
Use bash paramater substitution to assign defaults when variable are uninitialized, eg:
rm -rf $variable:-"/nonexistent"
add a comment |Â
up vote
1
down vote
The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.
There is most likely no need to glob the contents of the $build
directory in order to delete them but not the actual $build
directory itself. So if you skipped the extraneous *
then an unset value would turn into rm -rf /
which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root
option).
Skipping the trailing /
as well would result in rm ''
which would result in the error message:
rm: can't remove '': No such file or directory
This works even if your rm command does not implement protection for /
.
add a comment |Â
up vote
0
down vote
I always try to start my Bash scripts with a line #!/bin/bash -ue
.
-e
means "fail on first uncathed error";
-u
means "fail on first usage of undeclared variable".
Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'nt'
but for my purposes this is overkill.
add a comment |Â
up vote
-2
down vote
You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.
In this case:
rmdir $build
Since rmdir
will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:
rm -rf $build/$parameter
rmdir $build
Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir
will throw an error. Handle it properly, then:
rmdir $build || build_dir_not_empty "$build"
This way of thinking has served me well because... yep, been there, done that.
6
Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output.make clean
will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that havingrm -rf $build/$parameter
only moves the issue downward a bit.
â Raphael
Jul 11 at 22:13
Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
â Rich
Jul 17 at 23:29
"Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
â Raphael
Jul 18 at 12:44
Hm. I made no assumptions, and wrote a general answer. There's nothing at all aboutmake
in the question. Writing an answer that's only applicable tomake
would be a very specific and surprising assumption. My answer works for cases other thanmake
, other than software development, other than your specific novice problem that you were having by deleting your stuff.
â Rich
Jul 23 at 15:25
1
Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
â Raphael
Jul 25 at 5:17
 |Â
show 3 more comments
7 Answers
7
active
oldest
votes
7 Answers
7
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
70
down vote
accepted
set -u
or
set -o nounset
This would make the current shell treat expansions of unset variables as an error:
$ unset build
$ set -u
$ rm -rf "$build"/*
bash: build: unbound variable
set -u
and set -o nounset
are POSIX shell options.
An empty value would not trigger an error though.
For that, use
$ rm -rf "$build:?Error, variable is empty or unset"/*
bash: build: Error, variable is empty or unset
The expansion of $variable:?word
would expand to the value of variable
unless it's empty or unset. If it's empty or unset, the word
would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the :
out would trigger the error only for an unset value, just like under set -u
.
$variable:?word
is a POSIX parameter expansion.
Neither of these would cause an interactive shell to terminate unless set -e
(or set -o errexit
) was also in effect. $variable:?word
causes scripts to exit if the variable is empty or unset. set -u
would cause a script to exit if used together with set -e
.
As for your second question. There is no way to limit rm
to not work outside of the current directory.
The GNU implementation of rm
has a --one-file-system
option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm
call in a function that actually checks the arguments.
As a side note: $build
is exactly equivalent to $build
unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "$buildx"
.
Very good, thanks! 1) Is it$build:?msg
or$build?msg
? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
â Raphael
Jul 11 at 13:31
1
@Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
â Kusalananda
Jul 11 at 13:34
2
@Raphael Ok, I've added a short sentence about what happens without the:
(it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
â Kusalananda
Jul 11 at 14:17
1
Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
â Raphael
Jul 11 at 14:25
2
@MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
â Kusalananda
Jul 13 at 6:52
 |Â
show 5 more comments
up vote
70
down vote
accepted
set -u
or
set -o nounset
This would make the current shell treat expansions of unset variables as an error:
$ unset build
$ set -u
$ rm -rf "$build"/*
bash: build: unbound variable
set -u
and set -o nounset
are POSIX shell options.
An empty value would not trigger an error though.
For that, use
$ rm -rf "$build:?Error, variable is empty or unset"/*
bash: build: Error, variable is empty or unset
The expansion of $variable:?word
would expand to the value of variable
unless it's empty or unset. If it's empty or unset, the word
would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the :
out would trigger the error only for an unset value, just like under set -u
.
$variable:?word
is a POSIX parameter expansion.
Neither of these would cause an interactive shell to terminate unless set -e
(or set -o errexit
) was also in effect. $variable:?word
causes scripts to exit if the variable is empty or unset. set -u
would cause a script to exit if used together with set -e
.
As for your second question. There is no way to limit rm
to not work outside of the current directory.
The GNU implementation of rm
has a --one-file-system
option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm
call in a function that actually checks the arguments.
As a side note: $build
is exactly equivalent to $build
unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "$buildx"
.
Very good, thanks! 1) Is it$build:?msg
or$build?msg
? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
â Raphael
Jul 11 at 13:31
1
@Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
â Kusalananda
Jul 11 at 13:34
2
@Raphael Ok, I've added a short sentence about what happens without the:
(it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
â Kusalananda
Jul 11 at 14:17
1
Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
â Raphael
Jul 11 at 14:25
2
@MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
â Kusalananda
Jul 13 at 6:52
 |Â
show 5 more comments
up vote
70
down vote
accepted
up vote
70
down vote
accepted
set -u
or
set -o nounset
This would make the current shell treat expansions of unset variables as an error:
$ unset build
$ set -u
$ rm -rf "$build"/*
bash: build: unbound variable
set -u
and set -o nounset
are POSIX shell options.
An empty value would not trigger an error though.
For that, use
$ rm -rf "$build:?Error, variable is empty or unset"/*
bash: build: Error, variable is empty or unset
The expansion of $variable:?word
would expand to the value of variable
unless it's empty or unset. If it's empty or unset, the word
would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the :
out would trigger the error only for an unset value, just like under set -u
.
$variable:?word
is a POSIX parameter expansion.
Neither of these would cause an interactive shell to terminate unless set -e
(or set -o errexit
) was also in effect. $variable:?word
causes scripts to exit if the variable is empty or unset. set -u
would cause a script to exit if used together with set -e
.
As for your second question. There is no way to limit rm
to not work outside of the current directory.
The GNU implementation of rm
has a --one-file-system
option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm
call in a function that actually checks the arguments.
As a side note: $build
is exactly equivalent to $build
unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "$buildx"
.
set -u
or
set -o nounset
This would make the current shell treat expansions of unset variables as an error:
$ unset build
$ set -u
$ rm -rf "$build"/*
bash: build: unbound variable
set -u
and set -o nounset
are POSIX shell options.
An empty value would not trigger an error though.
For that, use
$ rm -rf "$build:?Error, variable is empty or unset"/*
bash: build: Error, variable is empty or unset
The expansion of $variable:?word
would expand to the value of variable
unless it's empty or unset. If it's empty or unset, the word
would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the :
out would trigger the error only for an unset value, just like under set -u
.
$variable:?word
is a POSIX parameter expansion.
Neither of these would cause an interactive shell to terminate unless set -e
(or set -o errexit
) was also in effect. $variable:?word
causes scripts to exit if the variable is empty or unset. set -u
would cause a script to exit if used together with set -e
.
As for your second question. There is no way to limit rm
to not work outside of the current directory.
The GNU implementation of rm
has a --one-file-system
option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm
call in a function that actually checks the arguments.
As a side note: $build
is exactly equivalent to $build
unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "$buildx"
.
edited Jul 12 at 15:04
answered Jul 11 at 13:00
Kusalananda
101k13199312
101k13199312
Very good, thanks! 1) Is it$build:?msg
or$build?msg
? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
â Raphael
Jul 11 at 13:31
1
@Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
â Kusalananda
Jul 11 at 13:34
2
@Raphael Ok, I've added a short sentence about what happens without the:
(it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
â Kusalananda
Jul 11 at 14:17
1
Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
â Raphael
Jul 11 at 14:25
2
@MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
â Kusalananda
Jul 13 at 6:52
 |Â
show 5 more comments
Very good, thanks! 1) Is it$build:?msg
or$build?msg
? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
â Raphael
Jul 11 at 13:31
1
@Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
â Kusalananda
Jul 11 at 13:34
2
@Raphael Ok, I've added a short sentence about what happens without the:
(it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
â Kusalananda
Jul 11 at 14:17
1
Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
â Raphael
Jul 11 at 14:25
2
@MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
â Kusalananda
Jul 13 at 6:52
Very good, thanks! 1) Is it
$build:?msg
or $build?msg
? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.â Raphael
Jul 11 at 13:31
Very good, thanks! 1) Is it
$build:?msg
or $build?msg
? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.â Raphael
Jul 11 at 13:31
1
1
@Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
â Kusalananda
Jul 11 at 13:34
@Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
â Kusalananda
Jul 11 at 13:34
2
2
@Raphael Ok, I've added a short sentence about what happens without the
:
(it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.â Kusalananda
Jul 11 at 14:17
@Raphael Ok, I've added a short sentence about what happens without the
:
(it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.â Kusalananda
Jul 11 at 14:17
1
1
Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
â Raphael
Jul 11 at 14:25
Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
â Raphael
Jul 11 at 14:25
2
2
@MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
â Kusalananda
Jul 13 at 6:52
@MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
â Kusalananda
Jul 13 at 6:52
 |Â
show 5 more comments
up vote
11
down vote
I'm going to suggest normal validation checks using test
/[ ]
You would had been safe if you'd written your script as such:
build="build"
...
[ -n "$build" ] || exit 1
rm -rf "$build/"*
...
The [ -n "$build" ]
checks that "$build"
is a non-zero length string.
The ||
is the logical OR operator in bash. It causes another command to be run if the first one failed.
In this way, had $build
been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).
This also would have protected you in case you removed all uses $build
because [ -n "" ]
will always be false.
The advantage of using test
/[ ]
is there are many other more meaningful checks that it can also use.
For example:
[ -f FILE ] True if FILE exists and is a regular file.
[ -d FILE ] True if FILE exists and is a directory.
[ -O FILE ] True if FILE exists and is owned by the effective user ID.
Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as$variable:?word
@Kusalananda proposes?
â Raphael
Jul 11 at 18:38
@Raphael it works similarly in the case of "does the string have a value" buttest
(i.e.[
) has many other checks that are relevant, such as-d
(the argument is a directory),-f
(the argument is a file),-O
(the file is owned by the current user) and so-on.
â Centimane
Jul 11 at 18:43
1
@Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed$build:?word
along with it? The$variable:?word
format doesn't protect you if you remove all instances of the variable.
â Centimane
Jul 11 at 18:50
1
1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blownif
isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using$v:?w
would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
â Raphael
Jul 12 at 7:29
That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
â Raphael
Jul 12 at 7:31
add a comment |Â
up vote
11
down vote
I'm going to suggest normal validation checks using test
/[ ]
You would had been safe if you'd written your script as such:
build="build"
...
[ -n "$build" ] || exit 1
rm -rf "$build/"*
...
The [ -n "$build" ]
checks that "$build"
is a non-zero length string.
The ||
is the logical OR operator in bash. It causes another command to be run if the first one failed.
In this way, had $build
been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).
This also would have protected you in case you removed all uses $build
because [ -n "" ]
will always be false.
The advantage of using test
/[ ]
is there are many other more meaningful checks that it can also use.
For example:
[ -f FILE ] True if FILE exists and is a regular file.
[ -d FILE ] True if FILE exists and is a directory.
[ -O FILE ] True if FILE exists and is owned by the effective user ID.
Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as$variable:?word
@Kusalananda proposes?
â Raphael
Jul 11 at 18:38
@Raphael it works similarly in the case of "does the string have a value" buttest
(i.e.[
) has many other checks that are relevant, such as-d
(the argument is a directory),-f
(the argument is a file),-O
(the file is owned by the current user) and so-on.
â Centimane
Jul 11 at 18:43
1
@Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed$build:?word
along with it? The$variable:?word
format doesn't protect you if you remove all instances of the variable.
â Centimane
Jul 11 at 18:50
1
1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blownif
isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using$v:?w
would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
â Raphael
Jul 12 at 7:29
That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
â Raphael
Jul 12 at 7:31
add a comment |Â
up vote
11
down vote
up vote
11
down vote
I'm going to suggest normal validation checks using test
/[ ]
You would had been safe if you'd written your script as such:
build="build"
...
[ -n "$build" ] || exit 1
rm -rf "$build/"*
...
The [ -n "$build" ]
checks that "$build"
is a non-zero length string.
The ||
is the logical OR operator in bash. It causes another command to be run if the first one failed.
In this way, had $build
been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).
This also would have protected you in case you removed all uses $build
because [ -n "" ]
will always be false.
The advantage of using test
/[ ]
is there are many other more meaningful checks that it can also use.
For example:
[ -f FILE ] True if FILE exists and is a regular file.
[ -d FILE ] True if FILE exists and is a directory.
[ -O FILE ] True if FILE exists and is owned by the effective user ID.
I'm going to suggest normal validation checks using test
/[ ]
You would had been safe if you'd written your script as such:
build="build"
...
[ -n "$build" ] || exit 1
rm -rf "$build/"*
...
The [ -n "$build" ]
checks that "$build"
is a non-zero length string.
The ||
is the logical OR operator in bash. It causes another command to be run if the first one failed.
In this way, had $build
been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).
This also would have protected you in case you removed all uses $build
because [ -n "" ]
will always be false.
The advantage of using test
/[ ]
is there are many other more meaningful checks that it can also use.
For example:
[ -f FILE ] True if FILE exists and is a regular file.
[ -d FILE ] True if FILE exists and is a directory.
[ -O FILE ] True if FILE exists and is owned by the effective user ID.
edited Jul 11 at 18:45
answered Jul 11 at 18:36
Centimane
3,0791933
3,0791933
Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as$variable:?word
@Kusalananda proposes?
â Raphael
Jul 11 at 18:38
@Raphael it works similarly in the case of "does the string have a value" buttest
(i.e.[
) has many other checks that are relevant, such as-d
(the argument is a directory),-f
(the argument is a file),-O
(the file is owned by the current user) and so-on.
â Centimane
Jul 11 at 18:43
1
@Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed$build:?word
along with it? The$variable:?word
format doesn't protect you if you remove all instances of the variable.
â Centimane
Jul 11 at 18:50
1
1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blownif
isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using$v:?w
would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
â Raphael
Jul 12 at 7:29
That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
â Raphael
Jul 12 at 7:31
add a comment |Â
Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as$variable:?word
@Kusalananda proposes?
â Raphael
Jul 11 at 18:38
@Raphael it works similarly in the case of "does the string have a value" buttest
(i.e.[
) has many other checks that are relevant, such as-d
(the argument is a directory),-f
(the argument is a file),-O
(the file is owned by the current user) and so-on.
â Centimane
Jul 11 at 18:43
1
@Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed$build:?word
along with it? The$variable:?word
format doesn't protect you if you remove all instances of the variable.
â Centimane
Jul 11 at 18:50
1
1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blownif
isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using$v:?w
would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
â Raphael
Jul 12 at 7:29
That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
â Raphael
Jul 12 at 7:31
Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as
$variable:?word
@Kusalananda proposes?â Raphael
Jul 11 at 18:38
Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as
$variable:?word
@Kusalananda proposes?â Raphael
Jul 11 at 18:38
@Raphael it works similarly in the case of "does the string have a value" but
test
(i.e. [
) has many other checks that are relevant, such as -d
(the argument is a directory), -f
(the argument is a file), -O
(the file is owned by the current user) and so-on.â Centimane
Jul 11 at 18:43
@Raphael it works similarly in the case of "does the string have a value" but
test
(i.e. [
) has many other checks that are relevant, such as -d
(the argument is a directory), -f
(the argument is a file), -O
(the file is owned by the current user) and so-on.â Centimane
Jul 11 at 18:43
1
1
@Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed
$build:?word
along with it? The $variable:?word
format doesn't protect you if you remove all instances of the variable.â Centimane
Jul 11 at 18:50
@Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed
$build:?word
along with it? The $variable:?word
format doesn't protect you if you remove all instances of the variable.â Centimane
Jul 11 at 18:50
1
1
1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blown
if
isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using $v:?w
would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.â Raphael
Jul 12 at 7:29
1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blown
if
isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using $v:?w
would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.â Raphael
Jul 12 at 7:29
That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
â Raphael
Jul 12 at 7:31
That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
â Raphael
Jul 12 at 7:31
add a comment |Â
up vote
3
down vote
In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):
# mktemp -d is also a good, reliable choice
trashdir=/tmp/.trash-$USER/trash-`date`
mkdir -p "$trashdir"
...
mv "$build"/* "$trashdir"
...
Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir
destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).
Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.
Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.
2
"This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if/tmp
is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of/tmp
).
â Raphael
Jul 12 at 10:02
1
You could usemktemp -d
to get that temporary directory without treading on another process's toes (and to correctly honour$TMPDIR
).
â Toby Speight
Jul 12 at 15:03
Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
â user117529
Jul 12 at 18:54
add a comment |Â
up vote
3
down vote
In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):
# mktemp -d is also a good, reliable choice
trashdir=/tmp/.trash-$USER/trash-`date`
mkdir -p "$trashdir"
...
mv "$build"/* "$trashdir"
...
Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir
destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).
Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.
Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.
2
"This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if/tmp
is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of/tmp
).
â Raphael
Jul 12 at 10:02
1
You could usemktemp -d
to get that temporary directory without treading on another process's toes (and to correctly honour$TMPDIR
).
â Toby Speight
Jul 12 at 15:03
Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
â user117529
Jul 12 at 18:54
add a comment |Â
up vote
3
down vote
up vote
3
down vote
In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):
# mktemp -d is also a good, reliable choice
trashdir=/tmp/.trash-$USER/trash-`date`
mkdir -p "$trashdir"
...
mv "$build"/* "$trashdir"
...
Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir
destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).
Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.
Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.
In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):
# mktemp -d is also a good, reliable choice
trashdir=/tmp/.trash-$USER/trash-`date`
mkdir -p "$trashdir"
...
mv "$build"/* "$trashdir"
...
Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir
destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).
Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.
Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.
edited Jul 23 at 22:14
answered Jul 12 at 8:07
user117529
1413
1413
2
"This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if/tmp
is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of/tmp
).
â Raphael
Jul 12 at 10:02
1
You could usemktemp -d
to get that temporary directory without treading on another process's toes (and to correctly honour$TMPDIR
).
â Toby Speight
Jul 12 at 15:03
Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
â user117529
Jul 12 at 18:54
add a comment |Â
2
"This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if/tmp
is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of/tmp
).
â Raphael
Jul 12 at 10:02
1
You could usemktemp -d
to get that temporary directory without treading on another process's toes (and to correctly honour$TMPDIR
).
â Toby Speight
Jul 12 at 15:03
Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
â user117529
Jul 12 at 18:54
2
2
"This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if
/tmp
is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of /tmp
).â Raphael
Jul 12 at 10:02
"This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if
/tmp
is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of /tmp
).â Raphael
Jul 12 at 10:02
1
1
You could use
mktemp -d
to get that temporary directory without treading on another process's toes (and to correctly honour $TMPDIR
).â Toby Speight
Jul 12 at 15:03
You could use
mktemp -d
to get that temporary directory without treading on another process's toes (and to correctly honour $TMPDIR
).â Toby Speight
Jul 12 at 15:03
Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
â user117529
Jul 12 at 18:54
Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
â user117529
Jul 12 at 18:54
add a comment |Â
up vote
2
down vote
Use bash paramater substitution to assign defaults when variable are uninitialized, eg:
rm -rf $variable:-"/nonexistent"
add a comment |Â
up vote
2
down vote
Use bash paramater substitution to assign defaults when variable are uninitialized, eg:
rm -rf $variable:-"/nonexistent"
add a comment |Â
up vote
2
down vote
up vote
2
down vote
Use bash paramater substitution to assign defaults when variable are uninitialized, eg:
rm -rf $variable:-"/nonexistent"
Use bash paramater substitution to assign defaults when variable are uninitialized, eg:
rm -rf $variable:-"/nonexistent"
answered Jul 12 at 5:56
rackandboneman
36915
36915
add a comment |Â
add a comment |Â
up vote
1
down vote
The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.
There is most likely no need to glob the contents of the $build
directory in order to delete them but not the actual $build
directory itself. So if you skipped the extraneous *
then an unset value would turn into rm -rf /
which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root
option).
Skipping the trailing /
as well would result in rm ''
which would result in the error message:
rm: can't remove '': No such file or directory
This works even if your rm command does not implement protection for /
.
add a comment |Â
up vote
1
down vote
The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.
There is most likely no need to glob the contents of the $build
directory in order to delete them but not the actual $build
directory itself. So if you skipped the extraneous *
then an unset value would turn into rm -rf /
which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root
option).
Skipping the trailing /
as well would result in rm ''
which would result in the error message:
rm: can't remove '': No such file or directory
This works even if your rm command does not implement protection for /
.
add a comment |Â
up vote
1
down vote
up vote
1
down vote
The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.
There is most likely no need to glob the contents of the $build
directory in order to delete them but not the actual $build
directory itself. So if you skipped the extraneous *
then an unset value would turn into rm -rf /
which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root
option).
Skipping the trailing /
as well would result in rm ''
which would result in the error message:
rm: can't remove '': No such file or directory
This works even if your rm command does not implement protection for /
.
The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.
There is most likely no need to glob the contents of the $build
directory in order to delete them but not the actual $build
directory itself. So if you skipped the extraneous *
then an unset value would turn into rm -rf /
which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root
option).
Skipping the trailing /
as well would result in rm ''
which would result in the error message:
rm: can't remove '': No such file or directory
This works even if your rm command does not implement protection for /
.
answered Jul 24 at 19:57
eschwartz
1016
1016
add a comment |Â
add a comment |Â
up vote
0
down vote
I always try to start my Bash scripts with a line #!/bin/bash -ue
.
-e
means "fail on first uncathed error";
-u
means "fail on first usage of undeclared variable".
Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'nt'
but for my purposes this is overkill.
add a comment |Â
up vote
0
down vote
I always try to start my Bash scripts with a line #!/bin/bash -ue
.
-e
means "fail on first uncathed error";
-u
means "fail on first usage of undeclared variable".
Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'nt'
but for my purposes this is overkill.
add a comment |Â
up vote
0
down vote
up vote
0
down vote
I always try to start my Bash scripts with a line #!/bin/bash -ue
.
-e
means "fail on first uncathed error";
-u
means "fail on first usage of undeclared variable".
Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'nt'
but for my purposes this is overkill.
I always try to start my Bash scripts with a line #!/bin/bash -ue
.
-e
means "fail on first uncathed error";
-u
means "fail on first usage of undeclared variable".
Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'nt'
but for my purposes this is overkill.
answered Jul 24 at 18:25
niya3
214
214
add a comment |Â
add a comment |Â
up vote
-2
down vote
You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.
In this case:
rmdir $build
Since rmdir
will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:
rm -rf $build/$parameter
rmdir $build
Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir
will throw an error. Handle it properly, then:
rmdir $build || build_dir_not_empty "$build"
This way of thinking has served me well because... yep, been there, done that.
6
Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output.make clean
will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that havingrm -rf $build/$parameter
only moves the issue downward a bit.
â Raphael
Jul 11 at 22:13
Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
â Rich
Jul 17 at 23:29
"Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
â Raphael
Jul 18 at 12:44
Hm. I made no assumptions, and wrote a general answer. There's nothing at all aboutmake
in the question. Writing an answer that's only applicable tomake
would be a very specific and surprising assumption. My answer works for cases other thanmake
, other than software development, other than your specific novice problem that you were having by deleting your stuff.
â Rich
Jul 23 at 15:25
1
Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
â Raphael
Jul 25 at 5:17
 |Â
show 3 more comments
up vote
-2
down vote
You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.
In this case:
rmdir $build
Since rmdir
will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:
rm -rf $build/$parameter
rmdir $build
Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir
will throw an error. Handle it properly, then:
rmdir $build || build_dir_not_empty "$build"
This way of thinking has served me well because... yep, been there, done that.
6
Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output.make clean
will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that havingrm -rf $build/$parameter
only moves the issue downward a bit.
â Raphael
Jul 11 at 22:13
Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
â Rich
Jul 17 at 23:29
"Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
â Raphael
Jul 18 at 12:44
Hm. I made no assumptions, and wrote a general answer. There's nothing at all aboutmake
in the question. Writing an answer that's only applicable tomake
would be a very specific and surprising assumption. My answer works for cases other thanmake
, other than software development, other than your specific novice problem that you were having by deleting your stuff.
â Rich
Jul 23 at 15:25
1
Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
â Raphael
Jul 25 at 5:17
 |Â
show 3 more comments
up vote
-2
down vote
up vote
-2
down vote
You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.
In this case:
rmdir $build
Since rmdir
will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:
rm -rf $build/$parameter
rmdir $build
Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir
will throw an error. Handle it properly, then:
rmdir $build || build_dir_not_empty "$build"
This way of thinking has served me well because... yep, been there, done that.
You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.
In this case:
rmdir $build
Since rmdir
will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:
rm -rf $build/$parameter
rmdir $build
Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir
will throw an error. Handle it properly, then:
rmdir $build || build_dir_not_empty "$build"
This way of thinking has served me well because... yep, been there, done that.
answered Jul 11 at 22:03
Rich
307211
307211
6
Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output.make clean
will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that havingrm -rf $build/$parameter
only moves the issue downward a bit.
â Raphael
Jul 11 at 22:13
Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
â Rich
Jul 17 at 23:29
"Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
â Raphael
Jul 18 at 12:44
Hm. I made no assumptions, and wrote a general answer. There's nothing at all aboutmake
in the question. Writing an answer that's only applicable tomake
would be a very specific and surprising assumption. My answer works for cases other thanmake
, other than software development, other than your specific novice problem that you were having by deleting your stuff.
â Rich
Jul 23 at 15:25
1
Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
â Raphael
Jul 25 at 5:17
 |Â
show 3 more comments
6
Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output.make clean
will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that havingrm -rf $build/$parameter
only moves the issue downward a bit.
â Raphael
Jul 11 at 22:13
Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
â Rich
Jul 17 at 23:29
"Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
â Raphael
Jul 18 at 12:44
Hm. I made no assumptions, and wrote a general answer. There's nothing at all aboutmake
in the question. Writing an answer that's only applicable tomake
would be a very specific and surprising assumption. My answer works for cases other thanmake
, other than software development, other than your specific novice problem that you were having by deleting your stuff.
â Rich
Jul 23 at 15:25
1
Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
â Raphael
Jul 25 at 5:17
6
6
Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output.
make clean
will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that having rm -rf $build/$parameter
only moves the issue downward a bit.â Raphael
Jul 11 at 22:13
Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output.
make clean
will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that having rm -rf $build/$parameter
only moves the issue downward a bit.â Raphael
Jul 11 at 22:13
Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
â Rich
Jul 17 at 23:29
Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
â Rich
Jul 17 at 23:29
"Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
â Raphael
Jul 18 at 12:44
"Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
â Raphael
Jul 18 at 12:44
Hm. I made no assumptions, and wrote a general answer. There's nothing at all about
make
in the question. Writing an answer that's only applicable to make
would be a very specific and surprising assumption. My answer works for cases other than make
, other than software development, other than your specific novice problem that you were having by deleting your stuff.â Rich
Jul 23 at 15:25
Hm. I made no assumptions, and wrote a general answer. There's nothing at all about
make
in the question. Writing an answer that's only applicable to make
would be a very specific and surprising assumption. My answer works for cases other than make
, other than software development, other than your specific novice problem that you were having by deleting your stuff.â Rich
Jul 23 at 15:25
1
1
Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
â Raphael
Jul 25 at 5:17
Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
â Raphael
Jul 25 at 5:17
 |Â
show 3 more comments
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f454694%2fhow-can-i-harden-bash-scripts-against-causing-harm-when-changed-in-the-future%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
3
There is no reason to
rm -rf "$build/*
no matter where the quote marks go.rm -rf "$build
will do the same thing because of thef
.â Monty Harder
Jul 11 at 14:36
1
Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
â Charles Duffy
Jul 11 at 15:15
@CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
â Raphael
Jul 11 at 15:18
2
Gotcha. Be sure to note the caveats in BashFAQ #112.
set -u
isn't nearly as frowned on asset -e
is, but it still does have its gotchas.â Charles Duffy
Jul 11 at 15:20
2
joke answer: Just put
#! /usr/bin/env ruby
at the top of every shell script and forget about bash ;)â Pod
Jul 12 at 15:25