How can I harden bash scripts against causing harm when changed in the future?

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
43
down vote

favorite
4












So, I deleted my home folder (or, more precisely, all files I had write access to). What happened is that I had



build="build"
...
rm -rf "$build/"*
...
<do other things with $build>


in a bash script and, after no longer needing $build, removing the declaration and all its usages -- but the rm. Bash happily expands to rm -rf /*. Yea.



I felt stupid, installed the backup, redid the work I lost. Trying to move past the shame.



Now, I wonder: what are techniques to write bash scripts so that such mistakes can't happen, or are at least less likely? For instance, had I written



FileUtils.rm_rf("#build/*")


in a Ruby script, the interpreter would have complained about build not being declared, so there the language protects me.



What I have considered in bash, besides corraling rm (which, as many answers in related questions mention, is not unproblematic):




  1. rm -rf "./$build/"*

    That would have killed my current work (a Git repo) but nothing else.

  2. A variant/parameterization of rm that requires interaction when acting outside of the current directory. (Could not find any.)
    Similar effect.

Is that it, or are there other ways to write bash scripts that are "robust" in this sense?







share|improve this question

















  • 3




    There is no reason to rm -rf "$build/* no matter where the quote marks go. rm -rf "$build will do the same thing because of the f.
    – Monty Harder
    Jul 11 at 14:36






  • 1




    Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
    – Charles Duffy
    Jul 11 at 15:15











  • @CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
    – Raphael
    Jul 11 at 15:18






  • 2




    Gotcha. Be sure to note the caveats in BashFAQ #112. set -u isn't nearly as frowned on as set -e is, but it still does have its gotchas.
    – Charles Duffy
    Jul 11 at 15:20






  • 2




    joke answer: Just put #! /usr/bin/env ruby at the top of every shell script and forget about bash ;)
    – Pod
    Jul 12 at 15:25














up vote
43
down vote

favorite
4












So, I deleted my home folder (or, more precisely, all files I had write access to). What happened is that I had



build="build"
...
rm -rf "$build/"*
...
<do other things with $build>


in a bash script and, after no longer needing $build, removing the declaration and all its usages -- but the rm. Bash happily expands to rm -rf /*. Yea.



I felt stupid, installed the backup, redid the work I lost. Trying to move past the shame.



Now, I wonder: what are techniques to write bash scripts so that such mistakes can't happen, or are at least less likely? For instance, had I written



FileUtils.rm_rf("#build/*")


in a Ruby script, the interpreter would have complained about build not being declared, so there the language protects me.



What I have considered in bash, besides corraling rm (which, as many answers in related questions mention, is not unproblematic):




  1. rm -rf "./$build/"*

    That would have killed my current work (a Git repo) but nothing else.

  2. A variant/parameterization of rm that requires interaction when acting outside of the current directory. (Could not find any.)
    Similar effect.

Is that it, or are there other ways to write bash scripts that are "robust" in this sense?







share|improve this question

















  • 3




    There is no reason to rm -rf "$build/* no matter where the quote marks go. rm -rf "$build will do the same thing because of the f.
    – Monty Harder
    Jul 11 at 14:36






  • 1




    Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
    – Charles Duffy
    Jul 11 at 15:15











  • @CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
    – Raphael
    Jul 11 at 15:18






  • 2




    Gotcha. Be sure to note the caveats in BashFAQ #112. set -u isn't nearly as frowned on as set -e is, but it still does have its gotchas.
    – Charles Duffy
    Jul 11 at 15:20






  • 2




    joke answer: Just put #! /usr/bin/env ruby at the top of every shell script and forget about bash ;)
    – Pod
    Jul 12 at 15:25












up vote
43
down vote

favorite
4









up vote
43
down vote

favorite
4






4





So, I deleted my home folder (or, more precisely, all files I had write access to). What happened is that I had



build="build"
...
rm -rf "$build/"*
...
<do other things with $build>


in a bash script and, after no longer needing $build, removing the declaration and all its usages -- but the rm. Bash happily expands to rm -rf /*. Yea.



I felt stupid, installed the backup, redid the work I lost. Trying to move past the shame.



Now, I wonder: what are techniques to write bash scripts so that such mistakes can't happen, or are at least less likely? For instance, had I written



FileUtils.rm_rf("#build/*")


in a Ruby script, the interpreter would have complained about build not being declared, so there the language protects me.



What I have considered in bash, besides corraling rm (which, as many answers in related questions mention, is not unproblematic):




  1. rm -rf "./$build/"*

    That would have killed my current work (a Git repo) but nothing else.

  2. A variant/parameterization of rm that requires interaction when acting outside of the current directory. (Could not find any.)
    Similar effect.

Is that it, or are there other ways to write bash scripts that are "robust" in this sense?







share|improve this question













So, I deleted my home folder (or, more precisely, all files I had write access to). What happened is that I had



build="build"
...
rm -rf "$build/"*
...
<do other things with $build>


in a bash script and, after no longer needing $build, removing the declaration and all its usages -- but the rm. Bash happily expands to rm -rf /*. Yea.



I felt stupid, installed the backup, redid the work I lost. Trying to move past the shame.



Now, I wonder: what are techniques to write bash scripts so that such mistakes can't happen, or are at least less likely? For instance, had I written



FileUtils.rm_rf("#build/*")


in a Ruby script, the interpreter would have complained about build not being declared, so there the language protects me.



What I have considered in bash, besides corraling rm (which, as many answers in related questions mention, is not unproblematic):




  1. rm -rf "./$build/"*

    That would have killed my current work (a Git repo) but nothing else.

  2. A variant/parameterization of rm that requires interaction when acting outside of the current directory. (Could not find any.)
    Similar effect.

Is that it, or are there other ways to write bash scripts that are "robust" in this sense?









share|improve this question












share|improve this question




share|improve this question








edited Jul 11 at 14:22
























asked Jul 11 at 12:57









Raphael

776820




776820







  • 3




    There is no reason to rm -rf "$build/* no matter where the quote marks go. rm -rf "$build will do the same thing because of the f.
    – Monty Harder
    Jul 11 at 14:36






  • 1




    Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
    – Charles Duffy
    Jul 11 at 15:15











  • @CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
    – Raphael
    Jul 11 at 15:18






  • 2




    Gotcha. Be sure to note the caveats in BashFAQ #112. set -u isn't nearly as frowned on as set -e is, but it still does have its gotchas.
    – Charles Duffy
    Jul 11 at 15:20






  • 2




    joke answer: Just put #! /usr/bin/env ruby at the top of every shell script and forget about bash ;)
    – Pod
    Jul 12 at 15:25












  • 3




    There is no reason to rm -rf "$build/* no matter where the quote marks go. rm -rf "$build will do the same thing because of the f.
    – Monty Harder
    Jul 11 at 14:36






  • 1




    Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
    – Charles Duffy
    Jul 11 at 15:15











  • @CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
    – Raphael
    Jul 11 at 15:18






  • 2




    Gotcha. Be sure to note the caveats in BashFAQ #112. set -u isn't nearly as frowned on as set -e is, but it still does have its gotchas.
    – Charles Duffy
    Jul 11 at 15:20






  • 2




    joke answer: Just put #! /usr/bin/env ruby at the top of every shell script and forget about bash ;)
    – Pod
    Jul 12 at 15:25







3




3




There is no reason to rm -rf "$build/* no matter where the quote marks go. rm -rf "$build will do the same thing because of the f.
– Monty Harder
Jul 11 at 14:36




There is no reason to rm -rf "$build/* no matter where the quote marks go. rm -rf "$build will do the same thing because of the f.
– Monty Harder
Jul 11 at 14:36




1




1




Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
– Charles Duffy
Jul 11 at 15:15





Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used.
– Charles Duffy
Jul 11 at 15:15













@CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
– Raphael
Jul 11 at 15:18




@CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel.
– Raphael
Jul 11 at 15:18




2




2




Gotcha. Be sure to note the caveats in BashFAQ #112. set -u isn't nearly as frowned on as set -e is, but it still does have its gotchas.
– Charles Duffy
Jul 11 at 15:20




Gotcha. Be sure to note the caveats in BashFAQ #112. set -u isn't nearly as frowned on as set -e is, but it still does have its gotchas.
– Charles Duffy
Jul 11 at 15:20




2




2




joke answer: Just put #! /usr/bin/env ruby at the top of every shell script and forget about bash ;)
– Pod
Jul 12 at 15:25




joke answer: Just put #! /usr/bin/env ruby at the top of every shell script and forget about bash ;)
– Pod
Jul 12 at 15:25










7 Answers
7






active

oldest

votes

















up vote
70
down vote



accepted










set -u


or



set -o nounset


This would make the current shell treat expansions of unset variables as an error:



$ unset build
$ set -u
$ rm -rf "$build"/*
bash: build: unbound variable


set -u and set -o nounset are POSIX shell options.



An empty value would not trigger an error though.



For that, use



$ rm -rf "$build:?Error, variable is empty or unset"/*
bash: build: Error, variable is empty or unset


The expansion of $variable:?word would expand to the value of variable unless it's empty or unset. If it's empty or unset, the word would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the : out would trigger the error only for an unset value, just like under set -u.



$variable:?word is a POSIX parameter expansion.



Neither of these would cause an interactive shell to terminate unless set -e (or set -o errexit) was also in effect. $variable:?word causes scripts to exit if the variable is empty or unset. set -u would cause a script to exit if used together with set -e.




As for your second question. There is no way to limit rm to not work outside of the current directory.



The GNU implementation of rm has a --one-file-system option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm call in a function that actually checks the arguments.




As a side note: $build is exactly equivalent to $build unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "$buildx".






share|improve this answer























  • Very good, thanks! 1) Is it $build:?msg or $build?msg? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
    – Raphael
    Jul 11 at 13:31






  • 1




    @Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
    – Kusalananda
    Jul 11 at 13:34






  • 2




    @Raphael Ok, I've added a short sentence about what happens without the : (it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
    – Kusalananda
    Jul 11 at 14:17






  • 1




    Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
    – Raphael
    Jul 11 at 14:25






  • 2




    @MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
    – Kusalananda
    Jul 13 at 6:52

















up vote
11
down vote













I'm going to suggest normal validation checks using test/[ ]



You would had been safe if you'd written your script as such:



build="build"
...
[ -n "$build" ] || exit 1
rm -rf "$build/"*
...



The [ -n "$build" ] checks that "$build" is a non-zero length string.



The || is the logical OR operator in bash. It causes another command to be run if the first one failed.



In this way, had $build been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).



This also would have protected you in case you removed all uses $build because [ -n "" ] will always be false.




The advantage of using test/[ ] is there are many other more meaningful checks that it can also use.



For example:



[ -f FILE ] True if FILE exists and is a regular file.
[ -d FILE ] True if FILE exists and is a directory.
[ -O FILE ] True if FILE exists and is owned by the effective user ID.





share|improve this answer























  • Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as $variable:?word @Kusalananda proposes?
    – Raphael
    Jul 11 at 18:38










  • @Raphael it works similarly in the case of "does the string have a value" but test (i.e. [) has many other checks that are relevant, such as -d (the argument is a directory), -f (the argument is a file), -O (the file is owned by the current user) and so-on.
    – Centimane
    Jul 11 at 18:43







  • 1




    @Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed $build:?word along with it? The $variable:?word format doesn't protect you if you remove all instances of the variable.
    – Centimane
    Jul 11 at 18:50






  • 1




    1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blown if isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using $v:?w would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
    – Raphael
    Jul 12 at 7:29










  • That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
    – Raphael
    Jul 12 at 7:31

















up vote
3
down vote













In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):



# mktemp -d is also a good, reliable choice
trashdir=/tmp/.trash-$USER/trash-`date`
mkdir -p "$trashdir"
...
mv "$build"/* "$trashdir"
...


Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).



Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.



Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.






share|improve this answer



















  • 2




    "This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if /tmp is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of /tmp).
    – Raphael
    Jul 12 at 10:02






  • 1




    You could use mktemp -d to get that temporary directory without treading on another process's toes (and to correctly honour $TMPDIR).
    – Toby Speight
    Jul 12 at 15:03










  • Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
    – user117529
    Jul 12 at 18:54


















up vote
2
down vote













Use bash paramater substitution to assign defaults when variable are uninitialized, eg:



rm -rf $variable:-"/nonexistent"






share|improve this answer




























    up vote
    1
    down vote













    The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.



    There is most likely no need to glob the contents of the $build directory in order to delete them but not the actual $build directory itself. So if you skipped the extraneous * then an unset value would turn into rm -rf / which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root option).



    Skipping the trailing / as well would result in rm '' which would result in the error message:



    rm: can't remove '': No such file or directory


    This works even if your rm command does not implement protection for /.






    share|improve this answer




























      up vote
      0
      down vote













      I always try to start my Bash scripts with a line #!/bin/bash -ue.



      -e means "fail on first uncathed error";



      -u means "fail on first usage of undeclared variable".



      Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'nt' but for my purposes this is overkill.






      share|improve this answer




























        up vote
        -2
        down vote













        You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.



        In this case:



        rmdir $build


        Since rmdir will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:



        rm -rf $build/$parameter
        rmdir $build


        Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir will throw an error. Handle it properly, then:



        rmdir $build || build_dir_not_empty "$build"


        This way of thinking has served me well because... yep, been there, done that.






        share|improve this answer

















        • 6




          Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output. make clean will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that having rm -rf $build/$parameter only moves the issue downward a bit.
          – Raphael
          Jul 11 at 22:13










        • Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
          – Rich
          Jul 17 at 23:29










        • "Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
          – Raphael
          Jul 18 at 12:44











        • Hm. I made no assumptions, and wrote a general answer. There's nothing at all about make in the question. Writing an answer that's only applicable to make would be a very specific and surprising assumption. My answer works for cases other than make, other than software development, other than your specific novice problem that you were having by deleting your stuff.
          – Rich
          Jul 23 at 15:25






        • 1




          Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
          – Raphael
          Jul 25 at 5:17










        Your Answer







        StackExchange.ready(function()
        var channelOptions =
        tags: "".split(" "),
        id: "106"
        ;
        initTagRenderer("".split(" "), "".split(" "), channelOptions);

        StackExchange.using("externalEditor", function()
        // Have to fire editor after snippets, if snippets enabled
        if (StackExchange.settings.snippets.snippetsEnabled)
        StackExchange.using("snippets", function()
        createEditor();
        );

        else
        createEditor();

        );

        function createEditor()
        StackExchange.prepareEditor(
        heartbeatType: 'answer',
        convertImagesToLinks: false,
        noModals: false,
        showLowRepImageUploadWarning: true,
        reputationToPostImages: null,
        bindNavPrevention: true,
        postfix: "",
        onDemand: true,
        discardSelector: ".discard-answer"
        ,immediatelyShowMarkdownHelp:true
        );



        );








         

        draft saved


        draft discarded


















        StackExchange.ready(
        function ()
        StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f454694%2fhow-can-i-harden-bash-scripts-against-causing-harm-when-changed-in-the-future%23new-answer', 'question_page');

        );

        Post as a guest






























        7 Answers
        7






        active

        oldest

        votes








        7 Answers
        7






        active

        oldest

        votes









        active

        oldest

        votes






        active

        oldest

        votes








        up vote
        70
        down vote



        accepted










        set -u


        or



        set -o nounset


        This would make the current shell treat expansions of unset variables as an error:



        $ unset build
        $ set -u
        $ rm -rf "$build"/*
        bash: build: unbound variable


        set -u and set -o nounset are POSIX shell options.



        An empty value would not trigger an error though.



        For that, use



        $ rm -rf "$build:?Error, variable is empty or unset"/*
        bash: build: Error, variable is empty or unset


        The expansion of $variable:?word would expand to the value of variable unless it's empty or unset. If it's empty or unset, the word would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the : out would trigger the error only for an unset value, just like under set -u.



        $variable:?word is a POSIX parameter expansion.



        Neither of these would cause an interactive shell to terminate unless set -e (or set -o errexit) was also in effect. $variable:?word causes scripts to exit if the variable is empty or unset. set -u would cause a script to exit if used together with set -e.




        As for your second question. There is no way to limit rm to not work outside of the current directory.



        The GNU implementation of rm has a --one-file-system option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm call in a function that actually checks the arguments.




        As a side note: $build is exactly equivalent to $build unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "$buildx".






        share|improve this answer























        • Very good, thanks! 1) Is it $build:?msg or $build?msg? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
          – Raphael
          Jul 11 at 13:31






        • 1




          @Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
          – Kusalananda
          Jul 11 at 13:34






        • 2




          @Raphael Ok, I've added a short sentence about what happens without the : (it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
          – Kusalananda
          Jul 11 at 14:17






        • 1




          Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
          – Raphael
          Jul 11 at 14:25






        • 2




          @MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
          – Kusalananda
          Jul 13 at 6:52














        up vote
        70
        down vote



        accepted










        set -u


        or



        set -o nounset


        This would make the current shell treat expansions of unset variables as an error:



        $ unset build
        $ set -u
        $ rm -rf "$build"/*
        bash: build: unbound variable


        set -u and set -o nounset are POSIX shell options.



        An empty value would not trigger an error though.



        For that, use



        $ rm -rf "$build:?Error, variable is empty or unset"/*
        bash: build: Error, variable is empty or unset


        The expansion of $variable:?word would expand to the value of variable unless it's empty or unset. If it's empty or unset, the word would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the : out would trigger the error only for an unset value, just like under set -u.



        $variable:?word is a POSIX parameter expansion.



        Neither of these would cause an interactive shell to terminate unless set -e (or set -o errexit) was also in effect. $variable:?word causes scripts to exit if the variable is empty or unset. set -u would cause a script to exit if used together with set -e.




        As for your second question. There is no way to limit rm to not work outside of the current directory.



        The GNU implementation of rm has a --one-file-system option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm call in a function that actually checks the arguments.




        As a side note: $build is exactly equivalent to $build unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "$buildx".






        share|improve this answer























        • Very good, thanks! 1) Is it $build:?msg or $build?msg? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
          – Raphael
          Jul 11 at 13:31






        • 1




          @Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
          – Kusalananda
          Jul 11 at 13:34






        • 2




          @Raphael Ok, I've added a short sentence about what happens without the : (it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
          – Kusalananda
          Jul 11 at 14:17






        • 1




          Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
          – Raphael
          Jul 11 at 14:25






        • 2




          @MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
          – Kusalananda
          Jul 13 at 6:52












        up vote
        70
        down vote



        accepted







        up vote
        70
        down vote



        accepted






        set -u


        or



        set -o nounset


        This would make the current shell treat expansions of unset variables as an error:



        $ unset build
        $ set -u
        $ rm -rf "$build"/*
        bash: build: unbound variable


        set -u and set -o nounset are POSIX shell options.



        An empty value would not trigger an error though.



        For that, use



        $ rm -rf "$build:?Error, variable is empty or unset"/*
        bash: build: Error, variable is empty or unset


        The expansion of $variable:?word would expand to the value of variable unless it's empty or unset. If it's empty or unset, the word would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the : out would trigger the error only for an unset value, just like under set -u.



        $variable:?word is a POSIX parameter expansion.



        Neither of these would cause an interactive shell to terminate unless set -e (or set -o errexit) was also in effect. $variable:?word causes scripts to exit if the variable is empty or unset. set -u would cause a script to exit if used together with set -e.




        As for your second question. There is no way to limit rm to not work outside of the current directory.



        The GNU implementation of rm has a --one-file-system option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm call in a function that actually checks the arguments.




        As a side note: $build is exactly equivalent to $build unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "$buildx".






        share|improve this answer















        set -u


        or



        set -o nounset


        This would make the current shell treat expansions of unset variables as an error:



        $ unset build
        $ set -u
        $ rm -rf "$build"/*
        bash: build: unbound variable


        set -u and set -o nounset are POSIX shell options.



        An empty value would not trigger an error though.



        For that, use



        $ rm -rf "$build:?Error, variable is empty or unset"/*
        bash: build: Error, variable is empty or unset


        The expansion of $variable:?word would expand to the value of variable unless it's empty or unset. If it's empty or unset, the word would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the : out would trigger the error only for an unset value, just like under set -u.



        $variable:?word is a POSIX parameter expansion.



        Neither of these would cause an interactive shell to terminate unless set -e (or set -o errexit) was also in effect. $variable:?word causes scripts to exit if the variable is empty or unset. set -u would cause a script to exit if used together with set -e.




        As for your second question. There is no way to limit rm to not work outside of the current directory.



        The GNU implementation of rm has a --one-file-system option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm call in a function that actually checks the arguments.




        As a side note: $build is exactly equivalent to $build unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "$buildx".







        share|improve this answer















        share|improve this answer



        share|improve this answer








        edited Jul 12 at 15:04


























        answered Jul 11 at 13:00









        Kusalananda

        101k13199312




        101k13199312











        • Very good, thanks! 1) Is it $build:?msg or $build?msg? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
          – Raphael
          Jul 11 at 13:31






        • 1




          @Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
          – Kusalananda
          Jul 11 at 13:34






        • 2




          @Raphael Ok, I've added a short sentence about what happens without the : (it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
          – Kusalananda
          Jul 11 at 14:17






        • 1




          Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
          – Raphael
          Jul 11 at 14:25






        • 2




          @MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
          – Kusalananda
          Jul 13 at 6:52
















        • Very good, thanks! 1) Is it $build:?msg or $build?msg? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
          – Raphael
          Jul 11 at 13:31






        • 1




          @Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
          – Kusalananda
          Jul 11 at 13:34






        • 2




          @Raphael Ok, I've added a short sentence about what happens without the : (it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
          – Kusalananda
          Jul 11 at 14:17






        • 1




          Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
          – Raphael
          Jul 11 at 14:25






        • 2




          @MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
          – Kusalananda
          Jul 13 at 6:52















        Very good, thanks! 1) Is it $build:?msg or $build?msg? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
        – Raphael
        Jul 11 at 13:31




        Very good, thanks! 1) Is it $build:?msg or $build?msg? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general.
        – Raphael
        Jul 11 at 13:31




        1




        1




        @Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
        – Kusalananda
        Jul 11 at 13:34




        @Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer.
        – Kusalananda
        Jul 11 at 13:34




        2




        2




        @Raphael Ok, I've added a short sentence about what happens without the : (it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
        – Kusalananda
        Jul 11 at 14:17




        @Raphael Ok, I've added a short sentence about what happens without the : (it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment.
        – Kusalananda
        Jul 11 at 14:17




        1




        1




        Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
        – Raphael
        Jul 11 at 14:25




        Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :)
        – Raphael
        Jul 11 at 14:25




        2




        2




        @MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
        – Kusalananda
        Jul 13 at 6:52




        @MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed.
        – Kusalananda
        Jul 13 at 6:52












        up vote
        11
        down vote













        I'm going to suggest normal validation checks using test/[ ]



        You would had been safe if you'd written your script as such:



        build="build"
        ...
        [ -n "$build" ] || exit 1
        rm -rf "$build/"*
        ...



        The [ -n "$build" ] checks that "$build" is a non-zero length string.



        The || is the logical OR operator in bash. It causes another command to be run if the first one failed.



        In this way, had $build been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).



        This also would have protected you in case you removed all uses $build because [ -n "" ] will always be false.




        The advantage of using test/[ ] is there are many other more meaningful checks that it can also use.



        For example:



        [ -f FILE ] True if FILE exists and is a regular file.
        [ -d FILE ] True if FILE exists and is a directory.
        [ -O FILE ] True if FILE exists and is owned by the effective user ID.





        share|improve this answer























        • Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as $variable:?word @Kusalananda proposes?
          – Raphael
          Jul 11 at 18:38










        • @Raphael it works similarly in the case of "does the string have a value" but test (i.e. [) has many other checks that are relevant, such as -d (the argument is a directory), -f (the argument is a file), -O (the file is owned by the current user) and so-on.
          – Centimane
          Jul 11 at 18:43







        • 1




          @Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed $build:?word along with it? The $variable:?word format doesn't protect you if you remove all instances of the variable.
          – Centimane
          Jul 11 at 18:50






        • 1




          1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blown if isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using $v:?w would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
          – Raphael
          Jul 12 at 7:29










        • That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
          – Raphael
          Jul 12 at 7:31














        up vote
        11
        down vote













        I'm going to suggest normal validation checks using test/[ ]



        You would had been safe if you'd written your script as such:



        build="build"
        ...
        [ -n "$build" ] || exit 1
        rm -rf "$build/"*
        ...



        The [ -n "$build" ] checks that "$build" is a non-zero length string.



        The || is the logical OR operator in bash. It causes another command to be run if the first one failed.



        In this way, had $build been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).



        This also would have protected you in case you removed all uses $build because [ -n "" ] will always be false.




        The advantage of using test/[ ] is there are many other more meaningful checks that it can also use.



        For example:



        [ -f FILE ] True if FILE exists and is a regular file.
        [ -d FILE ] True if FILE exists and is a directory.
        [ -O FILE ] True if FILE exists and is owned by the effective user ID.





        share|improve this answer























        • Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as $variable:?word @Kusalananda proposes?
          – Raphael
          Jul 11 at 18:38










        • @Raphael it works similarly in the case of "does the string have a value" but test (i.e. [) has many other checks that are relevant, such as -d (the argument is a directory), -f (the argument is a file), -O (the file is owned by the current user) and so-on.
          – Centimane
          Jul 11 at 18:43







        • 1




          @Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed $build:?word along with it? The $variable:?word format doesn't protect you if you remove all instances of the variable.
          – Centimane
          Jul 11 at 18:50






        • 1




          1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blown if isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using $v:?w would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
          – Raphael
          Jul 12 at 7:29










        • That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
          – Raphael
          Jul 12 at 7:31












        up vote
        11
        down vote










        up vote
        11
        down vote









        I'm going to suggest normal validation checks using test/[ ]



        You would had been safe if you'd written your script as such:



        build="build"
        ...
        [ -n "$build" ] || exit 1
        rm -rf "$build/"*
        ...



        The [ -n "$build" ] checks that "$build" is a non-zero length string.



        The || is the logical OR operator in bash. It causes another command to be run if the first one failed.



        In this way, had $build been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).



        This also would have protected you in case you removed all uses $build because [ -n "" ] will always be false.




        The advantage of using test/[ ] is there are many other more meaningful checks that it can also use.



        For example:



        [ -f FILE ] True if FILE exists and is a regular file.
        [ -d FILE ] True if FILE exists and is a directory.
        [ -O FILE ] True if FILE exists and is owned by the effective user ID.





        share|improve this answer















        I'm going to suggest normal validation checks using test/[ ]



        You would had been safe if you'd written your script as such:



        build="build"
        ...
        [ -n "$build" ] || exit 1
        rm -rf "$build/"*
        ...



        The [ -n "$build" ] checks that "$build" is a non-zero length string.



        The || is the logical OR operator in bash. It causes another command to be run if the first one failed.



        In this way, had $build been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).



        This also would have protected you in case you removed all uses $build because [ -n "" ] will always be false.




        The advantage of using test/[ ] is there are many other more meaningful checks that it can also use.



        For example:



        [ -f FILE ] True if FILE exists and is a regular file.
        [ -d FILE ] True if FILE exists and is a directory.
        [ -O FILE ] True if FILE exists and is owned by the effective user ID.






        share|improve this answer















        share|improve this answer



        share|improve this answer








        edited Jul 11 at 18:45


























        answered Jul 11 at 18:36









        Centimane

        3,0791933




        3,0791933











        • Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as $variable:?word @Kusalananda proposes?
          – Raphael
          Jul 11 at 18:38










        • @Raphael it works similarly in the case of "does the string have a value" but test (i.e. [) has many other checks that are relevant, such as -d (the argument is a directory), -f (the argument is a file), -O (the file is owned by the current user) and so-on.
          – Centimane
          Jul 11 at 18:43







        • 1




          @Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed $build:?word along with it? The $variable:?word format doesn't protect you if you remove all instances of the variable.
          – Centimane
          Jul 11 at 18:50






        • 1




          1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blown if isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using $v:?w would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
          – Raphael
          Jul 12 at 7:29










        • That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
          – Raphael
          Jul 12 at 7:31
















        • Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as $variable:?word @Kusalananda proposes?
          – Raphael
          Jul 11 at 18:38










        • @Raphael it works similarly in the case of "does the string have a value" but test (i.e. [) has many other checks that are relevant, such as -d (the argument is a directory), -f (the argument is a file), -O (the file is owned by the current user) and so-on.
          – Centimane
          Jul 11 at 18:43







        • 1




          @Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed $build:?word along with it? The $variable:?word format doesn't protect you if you remove all instances of the variable.
          – Centimane
          Jul 11 at 18:50






        • 1




          1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blown if isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using $v:?w would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
          – Raphael
          Jul 12 at 7:29










        • That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
          – Raphael
          Jul 12 at 7:31















        Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as $variable:?word @Kusalananda proposes?
        – Raphael
        Jul 11 at 18:38




        Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as $variable:?word @Kusalananda proposes?
        – Raphael
        Jul 11 at 18:38












        @Raphael it works similarly in the case of "does the string have a value" but test (i.e. [) has many other checks that are relevant, such as -d (the argument is a directory), -f (the argument is a file), -O (the file is owned by the current user) and so-on.
        – Centimane
        Jul 11 at 18:43





        @Raphael it works similarly in the case of "does the string have a value" but test (i.e. [) has many other checks that are relevant, such as -d (the argument is a directory), -f (the argument is a file), -O (the file is owned by the current user) and so-on.
        – Centimane
        Jul 11 at 18:43





        1




        1




        @Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed $build:?word along with it? The $variable:?word format doesn't protect you if you remove all instances of the variable.
        – Centimane
        Jul 11 at 18:50




        @Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed $build:?word along with it? The $variable:?word format doesn't protect you if you remove all instances of the variable.
        – Centimane
        Jul 11 at 18:50




        1




        1




        1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blown if isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using $v:?w would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
        – Raphael
        Jul 12 at 7:29




        1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blown if isn't. 2) "would you not have also removed $build:?word along with it" -- the scenario is that I missed a usage of the variable. Using $v:?w would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously.
        – Raphael
        Jul 12 at 7:29












        That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
        – Raphael
        Jul 12 at 7:31




        That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks!
        – Raphael
        Jul 12 at 7:31










        up vote
        3
        down vote













        In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):



        # mktemp -d is also a good, reliable choice
        trashdir=/tmp/.trash-$USER/trash-`date`
        mkdir -p "$trashdir"
        ...
        mv "$build"/* "$trashdir"
        ...


        Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).



        Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.



        Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.






        share|improve this answer



















        • 2




          "This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if /tmp is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of /tmp).
          – Raphael
          Jul 12 at 10:02






        • 1




          You could use mktemp -d to get that temporary directory without treading on another process's toes (and to correctly honour $TMPDIR).
          – Toby Speight
          Jul 12 at 15:03










        • Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
          – user117529
          Jul 12 at 18:54















        up vote
        3
        down vote













        In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):



        # mktemp -d is also a good, reliable choice
        trashdir=/tmp/.trash-$USER/trash-`date`
        mkdir -p "$trashdir"
        ...
        mv "$build"/* "$trashdir"
        ...


        Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).



        Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.



        Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.






        share|improve this answer



















        • 2




          "This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if /tmp is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of /tmp).
          – Raphael
          Jul 12 at 10:02






        • 1




          You could use mktemp -d to get that temporary directory without treading on another process's toes (and to correctly honour $TMPDIR).
          – Toby Speight
          Jul 12 at 15:03










        • Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
          – user117529
          Jul 12 at 18:54













        up vote
        3
        down vote










        up vote
        3
        down vote









        In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):



        # mktemp -d is also a good, reliable choice
        trashdir=/tmp/.trash-$USER/trash-`date`
        mkdir -p "$trashdir"
        ...
        mv "$build"/* "$trashdir"
        ...


        Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).



        Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.



        Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.






        share|improve this answer















        In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):



        # mktemp -d is also a good, reliable choice
        trashdir=/tmp/.trash-$USER/trash-`date`
        mkdir -p "$trashdir"
        ...
        mv "$build"/* "$trashdir"
        ...


        Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).



        Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.



        Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.







        share|improve this answer















        share|improve this answer



        share|improve this answer








        edited Jul 23 at 22:14


























        answered Jul 12 at 8:07









        user117529

        1413




        1413







        • 2




          "This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if /tmp is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of /tmp).
          – Raphael
          Jul 12 at 10:02






        • 1




          You could use mktemp -d to get that temporary directory without treading on another process's toes (and to correctly honour $TMPDIR).
          – Toby Speight
          Jul 12 at 15:03










        • Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
          – user117529
          Jul 12 at 18:54













        • 2




          "This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if /tmp is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of /tmp).
          – Raphael
          Jul 12 at 10:02






        • 1




          You could use mktemp -d to get that temporary directory without treading on another process's toes (and to correctly honour $TMPDIR).
          – Toby Speight
          Jul 12 at 15:03










        • Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
          – user117529
          Jul 12 at 18:54








        2




        2




        "This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if /tmp is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of /tmp).
        – Raphael
        Jul 12 at 10:02




        "This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if /tmp is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of /tmp).
        – Raphael
        Jul 12 at 10:02




        1




        1




        You could use mktemp -d to get that temporary directory without treading on another process's toes (and to correctly honour $TMPDIR).
        – Toby Speight
        Jul 12 at 15:03




        You could use mktemp -d to get that temporary directory without treading on another process's toes (and to correctly honour $TMPDIR).
        – Toby Speight
        Jul 12 at 15:03












        Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
        – user117529
        Jul 12 at 18:54





        Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up.
        – user117529
        Jul 12 at 18:54











        up vote
        2
        down vote













        Use bash paramater substitution to assign defaults when variable are uninitialized, eg:



        rm -rf $variable:-"/nonexistent"






        share|improve this answer

























          up vote
          2
          down vote













          Use bash paramater substitution to assign defaults when variable are uninitialized, eg:



          rm -rf $variable:-"/nonexistent"






          share|improve this answer























            up vote
            2
            down vote










            up vote
            2
            down vote









            Use bash paramater substitution to assign defaults when variable are uninitialized, eg:



            rm -rf $variable:-"/nonexistent"






            share|improve this answer













            Use bash paramater substitution to assign defaults when variable are uninitialized, eg:



            rm -rf $variable:-"/nonexistent"







            share|improve this answer













            share|improve this answer



            share|improve this answer











            answered Jul 12 at 5:56









            rackandboneman

            36915




            36915




















                up vote
                1
                down vote













                The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.



                There is most likely no need to glob the contents of the $build directory in order to delete them but not the actual $build directory itself. So if you skipped the extraneous * then an unset value would turn into rm -rf / which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root option).



                Skipping the trailing / as well would result in rm '' which would result in the error message:



                rm: can't remove '': No such file or directory


                This works even if your rm command does not implement protection for /.






                share|improve this answer

























                  up vote
                  1
                  down vote













                  The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.



                  There is most likely no need to glob the contents of the $build directory in order to delete them but not the actual $build directory itself. So if you skipped the extraneous * then an unset value would turn into rm -rf / which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root option).



                  Skipping the trailing / as well would result in rm '' which would result in the error message:



                  rm: can't remove '': No such file or directory


                  This works even if your rm command does not implement protection for /.






                  share|improve this answer























                    up vote
                    1
                    down vote










                    up vote
                    1
                    down vote









                    The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.



                    There is most likely no need to glob the contents of the $build directory in order to delete them but not the actual $build directory itself. So if you skipped the extraneous * then an unset value would turn into rm -rf / which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root option).



                    Skipping the trailing / as well would result in rm '' which would result in the error message:



                    rm: can't remove '': No such file or directory


                    This works even if your rm command does not implement protection for /.






                    share|improve this answer













                    The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.



                    There is most likely no need to glob the contents of the $build directory in order to delete them but not the actual $build directory itself. So if you skipped the extraneous * then an unset value would turn into rm -rf / which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root option).



                    Skipping the trailing / as well would result in rm '' which would result in the error message:



                    rm: can't remove '': No such file or directory


                    This works even if your rm command does not implement protection for /.







                    share|improve this answer













                    share|improve this answer



                    share|improve this answer











                    answered Jul 24 at 19:57









                    eschwartz

                    1016




                    1016




















                        up vote
                        0
                        down vote













                        I always try to start my Bash scripts with a line #!/bin/bash -ue.



                        -e means "fail on first uncathed error";



                        -u means "fail on first usage of undeclared variable".



                        Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'nt' but for my purposes this is overkill.






                        share|improve this answer

























                          up vote
                          0
                          down vote













                          I always try to start my Bash scripts with a line #!/bin/bash -ue.



                          -e means "fail on first uncathed error";



                          -u means "fail on first usage of undeclared variable".



                          Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'nt' but for my purposes this is overkill.






                          share|improve this answer























                            up vote
                            0
                            down vote










                            up vote
                            0
                            down vote









                            I always try to start my Bash scripts with a line #!/bin/bash -ue.



                            -e means "fail on first uncathed error";



                            -u means "fail on first usage of undeclared variable".



                            Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'nt' but for my purposes this is overkill.






                            share|improve this answer













                            I always try to start my Bash scripts with a line #!/bin/bash -ue.



                            -e means "fail on first uncathed error";



                            -u means "fail on first usage of undeclared variable".



                            Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'nt' but for my purposes this is overkill.







                            share|improve this answer













                            share|improve this answer



                            share|improve this answer











                            answered Jul 24 at 18:25









                            niya3

                            214




                            214




















                                up vote
                                -2
                                down vote













                                You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.



                                In this case:



                                rmdir $build


                                Since rmdir will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:



                                rm -rf $build/$parameter
                                rmdir $build


                                Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir will throw an error. Handle it properly, then:



                                rmdir $build || build_dir_not_empty "$build"


                                This way of thinking has served me well because... yep, been there, done that.






                                share|improve this answer

















                                • 6




                                  Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output. make clean will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that having rm -rf $build/$parameter only moves the issue downward a bit.
                                  – Raphael
                                  Jul 11 at 22:13










                                • Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
                                  – Rich
                                  Jul 17 at 23:29










                                • "Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
                                  – Raphael
                                  Jul 18 at 12:44











                                • Hm. I made no assumptions, and wrote a general answer. There's nothing at all about make in the question. Writing an answer that's only applicable to make would be a very specific and surprising assumption. My answer works for cases other than make, other than software development, other than your specific novice problem that you were having by deleting your stuff.
                                  – Rich
                                  Jul 23 at 15:25






                                • 1




                                  Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
                                  – Raphael
                                  Jul 25 at 5:17














                                up vote
                                -2
                                down vote













                                You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.



                                In this case:



                                rmdir $build


                                Since rmdir will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:



                                rm -rf $build/$parameter
                                rmdir $build


                                Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir will throw an error. Handle it properly, then:



                                rmdir $build || build_dir_not_empty "$build"


                                This way of thinking has served me well because... yep, been there, done that.






                                share|improve this answer

















                                • 6




                                  Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output. make clean will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that having rm -rf $build/$parameter only moves the issue downward a bit.
                                  – Raphael
                                  Jul 11 at 22:13










                                • Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
                                  – Rich
                                  Jul 17 at 23:29










                                • "Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
                                  – Raphael
                                  Jul 18 at 12:44











                                • Hm. I made no assumptions, and wrote a general answer. There's nothing at all about make in the question. Writing an answer that's only applicable to make would be a very specific and surprising assumption. My answer works for cases other than make, other than software development, other than your specific novice problem that you were having by deleting your stuff.
                                  – Rich
                                  Jul 23 at 15:25






                                • 1




                                  Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
                                  – Raphael
                                  Jul 25 at 5:17












                                up vote
                                -2
                                down vote










                                up vote
                                -2
                                down vote









                                You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.



                                In this case:



                                rmdir $build


                                Since rmdir will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:



                                rm -rf $build/$parameter
                                rmdir $build


                                Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir will throw an error. Handle it properly, then:



                                rmdir $build || build_dir_not_empty "$build"


                                This way of thinking has served me well because... yep, been there, done that.






                                share|improve this answer













                                You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.



                                In this case:



                                rmdir $build


                                Since rmdir will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:



                                rm -rf $build/$parameter
                                rmdir $build


                                Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir will throw an error. Handle it properly, then:



                                rmdir $build || build_dir_not_empty "$build"


                                This way of thinking has served me well because... yep, been there, done that.







                                share|improve this answer













                                share|improve this answer



                                share|improve this answer











                                answered Jul 11 at 22:03









                                Rich

                                307211




                                307211







                                • 6




                                  Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output. make clean will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that having rm -rf $build/$parameter only moves the issue downward a bit.
                                  – Raphael
                                  Jul 11 at 22:13










                                • Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
                                  – Rich
                                  Jul 17 at 23:29










                                • "Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
                                  – Raphael
                                  Jul 18 at 12:44











                                • Hm. I made no assumptions, and wrote a general answer. There's nothing at all about make in the question. Writing an answer that's only applicable to make would be a very specific and surprising assumption. My answer works for cases other than make, other than software development, other than your specific novice problem that you were having by deleting your stuff.
                                  – Rich
                                  Jul 23 at 15:25






                                • 1




                                  Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
                                  – Raphael
                                  Jul 25 at 5:17












                                • 6




                                  Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output. make clean will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that having rm -rf $build/$parameter only moves the issue downward a bit.
                                  – Raphael
                                  Jul 11 at 22:13










                                • Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
                                  – Rich
                                  Jul 17 at 23:29










                                • "Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
                                  – Raphael
                                  Jul 18 at 12:44











                                • Hm. I made no assumptions, and wrote a general answer. There's nothing at all about make in the question. Writing an answer that's only applicable to make would be a very specific and surprising assumption. My answer works for cases other than make, other than software development, other than your specific novice problem that you were having by deleting your stuff.
                                  – Rich
                                  Jul 23 at 15:25






                                • 1




                                  Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
                                  – Raphael
                                  Jul 25 at 5:17







                                6




                                6




                                Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output. make clean will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that having rm -rf $build/$parameter only moves the issue downward a bit.
                                – Raphael
                                Jul 11 at 22:13




                                Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output. make clean will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that having rm -rf $build/$parameter only moves the issue downward a bit.
                                – Raphael
                                Jul 11 at 22:13












                                Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
                                – Rich
                                Jul 17 at 23:29




                                Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow.
                                – Rich
                                Jul 17 at 23:29












                                "Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
                                – Raphael
                                Jul 18 at 12:44





                                "Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).)
                                – Raphael
                                Jul 18 at 12:44













                                Hm. I made no assumptions, and wrote a general answer. There's nothing at all about make in the question. Writing an answer that's only applicable to make would be a very specific and surprising assumption. My answer works for cases other than make, other than software development, other than your specific novice problem that you were having by deleting your stuff.
                                – Rich
                                Jul 23 at 15:25




                                Hm. I made no assumptions, and wrote a general answer. There's nothing at all about make in the question. Writing an answer that's only applicable to make would be a very specific and surprising assumption. My answer works for cases other than make, other than software development, other than your specific novice problem that you were having by deleting your stuff.
                                – Rich
                                Jul 23 at 15:25




                                1




                                1




                                Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
                                – Raphael
                                Jul 25 at 5:17




                                Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?"
                                – Raphael
                                Jul 25 at 5:17












                                 

                                draft saved


                                draft discarded


























                                 


                                draft saved


                                draft discarded














                                StackExchange.ready(
                                function ()
                                StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f454694%2fhow-can-i-harden-bash-scripts-against-causing-harm-when-changed-in-the-future%23new-answer', 'question_page');

                                );

                                Post as a guest













































































                                Popular posts from this blog

                                How to check contact read email or not when send email to Individual?

                                Bahrain

                                Postfix configuration issue with fips on centos 7; mailgun relay