shell script inside docker

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP












0















i try run docker with entrypoint.sh, but it doesn't exequite .sh line:



echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_`


I wrote it in dockerfile:



ENTRYPOINT ["bash", "-c", "source /code/entrypoint.sh | ts '[%Y-%m-%d %H:%M:%S]' &>> /output/stderr.log"]


Why echo doesn't execute inside docker run [my_image]



Here is full entrypoint.sh code:



#! /bin/bash

alias time='/usr/bin/time -f "%C --- CPU:t%E real,t%U user,t%S
syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"'

echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 -
participants_`

while read input_bids_path
do
participants_id=$(basename $input_bids_path)
LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH
time fmriprep /input /output participant --fs-license-file
/opt/freesurfer/license.txt --fs-no-reconall --use-aroma --ignore fieldmaps
--n_cpus 12 --force-bbr --participant_label $(cat $participants_id) -w
/output
# rm -r /input/$participants_id
done < <(find /input -name "*participants_*" -type f)
echo `rm -r /input/$participants_id`
wait `jobs -p` && echo __ok__ || echo __err__









share|improve this question






















  • Why do you use echo on command substitution? That rm will only remove the last /input/$participants_id as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?

    – Kusalananda
    Mar 4 at 20:36











  • @Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.

    – Relyativist
    Mar 4 at 20:46











  • You have't mentioned what command you are referring to. You said echo, but there's more than one of those.

    – Kusalananda
    Mar 4 at 20:49











  • @Kusalananda echo ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_

    – Relyativist
    Mar 4 at 20:50












  • The command substitution does not produce anything for echo to output. There is nothing to echo. This is also the case for the second echo. It is unclear what you want these echo calls to actually do.

    – Kusalananda
    Mar 4 at 20:52
















0















i try run docker with entrypoint.sh, but it doesn't exequite .sh line:



echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_`


I wrote it in dockerfile:



ENTRYPOINT ["bash", "-c", "source /code/entrypoint.sh | ts '[%Y-%m-%d %H:%M:%S]' &>> /output/stderr.log"]


Why echo doesn't execute inside docker run [my_image]



Here is full entrypoint.sh code:



#! /bin/bash

alias time='/usr/bin/time -f "%C --- CPU:t%E real,t%U user,t%S
syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"'

echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 -
participants_`

while read input_bids_path
do
participants_id=$(basename $input_bids_path)
LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH
time fmriprep /input /output participant --fs-license-file
/opt/freesurfer/license.txt --fs-no-reconall --use-aroma --ignore fieldmaps
--n_cpus 12 --force-bbr --participant_label $(cat $participants_id) -w
/output
# rm -r /input/$participants_id
done < <(find /input -name "*participants_*" -type f)
echo `rm -r /input/$participants_id`
wait `jobs -p` && echo __ok__ || echo __err__









share|improve this question






















  • Why do you use echo on command substitution? That rm will only remove the last /input/$participants_id as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?

    – Kusalananda
    Mar 4 at 20:36











  • @Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.

    – Relyativist
    Mar 4 at 20:46











  • You have't mentioned what command you are referring to. You said echo, but there's more than one of those.

    – Kusalananda
    Mar 4 at 20:49











  • @Kusalananda echo ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_

    – Relyativist
    Mar 4 at 20:50












  • The command substitution does not produce anything for echo to output. There is nothing to echo. This is also the case for the second echo. It is unclear what you want these echo calls to actually do.

    – Kusalananda
    Mar 4 at 20:52














0












0








0








i try run docker with entrypoint.sh, but it doesn't exequite .sh line:



echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_`


I wrote it in dockerfile:



ENTRYPOINT ["bash", "-c", "source /code/entrypoint.sh | ts '[%Y-%m-%d %H:%M:%S]' &>> /output/stderr.log"]


Why echo doesn't execute inside docker run [my_image]



Here is full entrypoint.sh code:



#! /bin/bash

alias time='/usr/bin/time -f "%C --- CPU:t%E real,t%U user,t%S
syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"'

echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 -
participants_`

while read input_bids_path
do
participants_id=$(basename $input_bids_path)
LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH
time fmriprep /input /output participant --fs-license-file
/opt/freesurfer/license.txt --fs-no-reconall --use-aroma --ignore fieldmaps
--n_cpus 12 --force-bbr --participant_label $(cat $participants_id) -w
/output
# rm -r /input/$participants_id
done < <(find /input -name "*participants_*" -type f)
echo `rm -r /input/$participants_id`
wait `jobs -p` && echo __ok__ || echo __err__









share|improve this question














i try run docker with entrypoint.sh, but it doesn't exequite .sh line:



echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_`


I wrote it in dockerfile:



ENTRYPOINT ["bash", "-c", "source /code/entrypoint.sh | ts '[%Y-%m-%d %H:%M:%S]' &>> /output/stderr.log"]


Why echo doesn't execute inside docker run [my_image]



Here is full entrypoint.sh code:



#! /bin/bash

alias time='/usr/bin/time -f "%C --- CPU:t%E real,t%U user,t%S
syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"'

echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 -
participants_`

while read input_bids_path
do
participants_id=$(basename $input_bids_path)
LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH
time fmriprep /input /output participant --fs-license-file
/opt/freesurfer/license.txt --fs-no-reconall --use-aroma --ignore fieldmaps
--n_cpus 12 --force-bbr --participant_label $(cat $participants_id) -w
/output
# rm -r /input/$participants_id
done < <(find /input -name "*participants_*" -type f)
echo `rm -r /input/$participants_id`
wait `jobs -p` && echo __ok__ || echo __err__






shell docker






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Mar 4 at 20:29









RelyativistRelyativist

317




317












  • Why do you use echo on command substitution? That rm will only remove the last /input/$participants_id as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?

    – Kusalananda
    Mar 4 at 20:36











  • @Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.

    – Relyativist
    Mar 4 at 20:46











  • You have't mentioned what command you are referring to. You said echo, but there's more than one of those.

    – Kusalananda
    Mar 4 at 20:49











  • @Kusalananda echo ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_

    – Relyativist
    Mar 4 at 20:50












  • The command substitution does not produce anything for echo to output. There is nothing to echo. This is also the case for the second echo. It is unclear what you want these echo calls to actually do.

    – Kusalananda
    Mar 4 at 20:52


















  • Why do you use echo on command substitution? That rm will only remove the last /input/$participants_id as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?

    – Kusalananda
    Mar 4 at 20:36











  • @Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.

    – Relyativist
    Mar 4 at 20:46











  • You have't mentioned what command you are referring to. You said echo, but there's more than one of those.

    – Kusalananda
    Mar 4 at 20:49











  • @Kusalananda echo ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_

    – Relyativist
    Mar 4 at 20:50












  • The command substitution does not produce anything for echo to output. There is nothing to echo. This is also the case for the second echo. It is unclear what you want these echo calls to actually do.

    – Kusalananda
    Mar 4 at 20:52

















Why do you use echo on command substitution? That rm will only remove the last /input/$participants_id as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?

– Kusalananda
Mar 4 at 20:36





Why do you use echo on command substitution? That rm will only remove the last /input/$participants_id as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?

– Kusalananda
Mar 4 at 20:36













@Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.

– Relyativist
Mar 4 at 20:46





@Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.

– Relyativist
Mar 4 at 20:46













You have't mentioned what command you are referring to. You said echo, but there's more than one of those.

– Kusalananda
Mar 4 at 20:49





You have't mentioned what command you are referring to. You said echo, but there's more than one of those.

– Kusalananda
Mar 4 at 20:49













@Kusalananda echo ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_

– Relyativist
Mar 4 at 20:50






@Kusalananda echo ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_

– Relyativist
Mar 4 at 20:50














The command substitution does not produce anything for echo to output. There is nothing to echo. This is also the case for the second echo. It is unclear what you want these echo calls to actually do.

– Kusalananda
Mar 4 at 20:52






The command substitution does not produce anything for echo to output. There is nothing to echo. This is also the case for the second echo. It is unclear what you want these echo calls to actually do.

– Kusalananda
Mar 4 at 20:52











1 Answer
1






active

oldest

votes


















0














There's a number of things in this script that are wrong or that can be improved.



The main issue in the question seems to be why the two calls to echo does not produce any output.



When you use a command substitution, like in



echo `rm file`


or the equivalent



echo $(rm file)


then the echo will get the output of the command within the backticks or inside the $(...). Neither of your command substitutions produces any output. Both commands that you use within backticks modify files, but again, neither produces output to their standard output stream (what would ordinarily be displayed in the terminal). This means that both calls to echo will produce no output either, apart from an empty line each.



In general, echo $(...) is an anti-pattern, meaning you can do the same thing in a much better way.



If you do want to output the result of some pipeline pipeline, then instead of writing



echo $(pipeline)


you would simply say



pipeline


The output of pipeline would be displayed, as the output of commands are usually displayed in the terminal.



In the code below, I have inserted a couple of printf statements that will output the relevant "progress information" in the script(s).




Here's a modified version of the script which is totally untested (as I don't have access to the tools that you use, or the input files), but it should mimic what your script is doing, including creating those intermediate files (these are not needed, and I'll show how to get rid of them later).



#!/bin/bash

export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"

timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"

for dirpath in /input/sub-*/; do
name=$(basename "$dirpath")
id=$name#sub-
printf '%sn' "$id"
printf 'Found ID: %sn' "$id" >&2
done | split -l 8 - participants_

for participants_id in participants_*; do
ids=( $(<"$participants_id") )

printf 'Processing ID: %sn' "$ids[@]" >&2

/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output

rm -f "$participants_id"
done


Fixes:



  1. The time command does not need to be an alias just due to having a long option argument to its -f option. Aliases aren't expanded in scripts anyway. I simply save the argument in a string and use that when invoking time.


  2. Your loop added to LD_LIBRARY_PATH in each iteration. This was not needed.


  3. The getting of the IDs from the directory names is better done in a proper loop. This loop will disappear later when we use an array to store the IDs instead.


  4. Rather than using find to locate the intermediate files, we just use them with a simple filename globbing pattern. We know they're right there and what their names are anyway.


  5. The intermediate file just processed is deleted inside the loop.


  6. The code is made readable through use of line continuations.


  7. The call to wait was removed. There are no background tasks to wait for.


The following variation stores the IDs in an array, all_ids instead of in temporary files:



#!/bin/bash

export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"

timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"

all_ids=( /input/sub-*/ )
all_ids=( "$all_ids[@]#/input/sub-" ) # remove "/input/sub-" from each item
all_ids=( "$all_ids[@]%/" ) # remove the trailing "/" from each item

printf 'Found ID: %sn' "$all_ids[@]" >&2

n=0
ids=( "$all_ids[@]:0:8" ) # pick out the first eight IDs

# Loop until the first ID in the ids array is empty
while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2

/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output

n=$(( n + 1 ))
ids=( "$all_ids[@]:n*8:8" ) # pick out the next eight IDs
done





share|improve this answer

























  • first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?

    – Relyativist
    Mar 5 at 12:04











  • @Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.

    – Kusalananda
    Mar 5 at 12:15












  • as i said it doesn't work like that inside container: root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" ) root@9bca6f96475d:/tmp# echo $all_ids *nothing*

    – Relyativist
    Mar 5 at 15:28












  • @Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!

    – Kusalananda
    Mar 5 at 15:34











  • thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks

    – Relyativist
    Mar 5 at 16:18












Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504349%2fshell-script-inside-docker%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









0














There's a number of things in this script that are wrong or that can be improved.



The main issue in the question seems to be why the two calls to echo does not produce any output.



When you use a command substitution, like in



echo `rm file`


or the equivalent



echo $(rm file)


then the echo will get the output of the command within the backticks or inside the $(...). Neither of your command substitutions produces any output. Both commands that you use within backticks modify files, but again, neither produces output to their standard output stream (what would ordinarily be displayed in the terminal). This means that both calls to echo will produce no output either, apart from an empty line each.



In general, echo $(...) is an anti-pattern, meaning you can do the same thing in a much better way.



If you do want to output the result of some pipeline pipeline, then instead of writing



echo $(pipeline)


you would simply say



pipeline


The output of pipeline would be displayed, as the output of commands are usually displayed in the terminal.



In the code below, I have inserted a couple of printf statements that will output the relevant "progress information" in the script(s).




Here's a modified version of the script which is totally untested (as I don't have access to the tools that you use, or the input files), but it should mimic what your script is doing, including creating those intermediate files (these are not needed, and I'll show how to get rid of them later).



#!/bin/bash

export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"

timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"

for dirpath in /input/sub-*/; do
name=$(basename "$dirpath")
id=$name#sub-
printf '%sn' "$id"
printf 'Found ID: %sn' "$id" >&2
done | split -l 8 - participants_

for participants_id in participants_*; do
ids=( $(<"$participants_id") )

printf 'Processing ID: %sn' "$ids[@]" >&2

/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output

rm -f "$participants_id"
done


Fixes:



  1. The time command does not need to be an alias just due to having a long option argument to its -f option. Aliases aren't expanded in scripts anyway. I simply save the argument in a string and use that when invoking time.


  2. Your loop added to LD_LIBRARY_PATH in each iteration. This was not needed.


  3. The getting of the IDs from the directory names is better done in a proper loop. This loop will disappear later when we use an array to store the IDs instead.


  4. Rather than using find to locate the intermediate files, we just use them with a simple filename globbing pattern. We know they're right there and what their names are anyway.


  5. The intermediate file just processed is deleted inside the loop.


  6. The code is made readable through use of line continuations.


  7. The call to wait was removed. There are no background tasks to wait for.


The following variation stores the IDs in an array, all_ids instead of in temporary files:



#!/bin/bash

export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"

timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"

all_ids=( /input/sub-*/ )
all_ids=( "$all_ids[@]#/input/sub-" ) # remove "/input/sub-" from each item
all_ids=( "$all_ids[@]%/" ) # remove the trailing "/" from each item

printf 'Found ID: %sn' "$all_ids[@]" >&2

n=0
ids=( "$all_ids[@]:0:8" ) # pick out the first eight IDs

# Loop until the first ID in the ids array is empty
while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2

/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output

n=$(( n + 1 ))
ids=( "$all_ids[@]:n*8:8" ) # pick out the next eight IDs
done





share|improve this answer

























  • first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?

    – Relyativist
    Mar 5 at 12:04











  • @Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.

    – Kusalananda
    Mar 5 at 12:15












  • as i said it doesn't work like that inside container: root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" ) root@9bca6f96475d:/tmp# echo $all_ids *nothing*

    – Relyativist
    Mar 5 at 15:28












  • @Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!

    – Kusalananda
    Mar 5 at 15:34











  • thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks

    – Relyativist
    Mar 5 at 16:18
















0














There's a number of things in this script that are wrong or that can be improved.



The main issue in the question seems to be why the two calls to echo does not produce any output.



When you use a command substitution, like in



echo `rm file`


or the equivalent



echo $(rm file)


then the echo will get the output of the command within the backticks or inside the $(...). Neither of your command substitutions produces any output. Both commands that you use within backticks modify files, but again, neither produces output to their standard output stream (what would ordinarily be displayed in the terminal). This means that both calls to echo will produce no output either, apart from an empty line each.



In general, echo $(...) is an anti-pattern, meaning you can do the same thing in a much better way.



If you do want to output the result of some pipeline pipeline, then instead of writing



echo $(pipeline)


you would simply say



pipeline


The output of pipeline would be displayed, as the output of commands are usually displayed in the terminal.



In the code below, I have inserted a couple of printf statements that will output the relevant "progress information" in the script(s).




Here's a modified version of the script which is totally untested (as I don't have access to the tools that you use, or the input files), but it should mimic what your script is doing, including creating those intermediate files (these are not needed, and I'll show how to get rid of them later).



#!/bin/bash

export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"

timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"

for dirpath in /input/sub-*/; do
name=$(basename "$dirpath")
id=$name#sub-
printf '%sn' "$id"
printf 'Found ID: %sn' "$id" >&2
done | split -l 8 - participants_

for participants_id in participants_*; do
ids=( $(<"$participants_id") )

printf 'Processing ID: %sn' "$ids[@]" >&2

/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output

rm -f "$participants_id"
done


Fixes:



  1. The time command does not need to be an alias just due to having a long option argument to its -f option. Aliases aren't expanded in scripts anyway. I simply save the argument in a string and use that when invoking time.


  2. Your loop added to LD_LIBRARY_PATH in each iteration. This was not needed.


  3. The getting of the IDs from the directory names is better done in a proper loop. This loop will disappear later when we use an array to store the IDs instead.


  4. Rather than using find to locate the intermediate files, we just use them with a simple filename globbing pattern. We know they're right there and what their names are anyway.


  5. The intermediate file just processed is deleted inside the loop.


  6. The code is made readable through use of line continuations.


  7. The call to wait was removed. There are no background tasks to wait for.


The following variation stores the IDs in an array, all_ids instead of in temporary files:



#!/bin/bash

export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"

timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"

all_ids=( /input/sub-*/ )
all_ids=( "$all_ids[@]#/input/sub-" ) # remove "/input/sub-" from each item
all_ids=( "$all_ids[@]%/" ) # remove the trailing "/" from each item

printf 'Found ID: %sn' "$all_ids[@]" >&2

n=0
ids=( "$all_ids[@]:0:8" ) # pick out the first eight IDs

# Loop until the first ID in the ids array is empty
while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2

/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output

n=$(( n + 1 ))
ids=( "$all_ids[@]:n*8:8" ) # pick out the next eight IDs
done





share|improve this answer

























  • first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?

    – Relyativist
    Mar 5 at 12:04











  • @Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.

    – Kusalananda
    Mar 5 at 12:15












  • as i said it doesn't work like that inside container: root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" ) root@9bca6f96475d:/tmp# echo $all_ids *nothing*

    – Relyativist
    Mar 5 at 15:28












  • @Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!

    – Kusalananda
    Mar 5 at 15:34











  • thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks

    – Relyativist
    Mar 5 at 16:18














0












0








0







There's a number of things in this script that are wrong or that can be improved.



The main issue in the question seems to be why the two calls to echo does not produce any output.



When you use a command substitution, like in



echo `rm file`


or the equivalent



echo $(rm file)


then the echo will get the output of the command within the backticks or inside the $(...). Neither of your command substitutions produces any output. Both commands that you use within backticks modify files, but again, neither produces output to their standard output stream (what would ordinarily be displayed in the terminal). This means that both calls to echo will produce no output either, apart from an empty line each.



In general, echo $(...) is an anti-pattern, meaning you can do the same thing in a much better way.



If you do want to output the result of some pipeline pipeline, then instead of writing



echo $(pipeline)


you would simply say



pipeline


The output of pipeline would be displayed, as the output of commands are usually displayed in the terminal.



In the code below, I have inserted a couple of printf statements that will output the relevant "progress information" in the script(s).




Here's a modified version of the script which is totally untested (as I don't have access to the tools that you use, or the input files), but it should mimic what your script is doing, including creating those intermediate files (these are not needed, and I'll show how to get rid of them later).



#!/bin/bash

export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"

timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"

for dirpath in /input/sub-*/; do
name=$(basename "$dirpath")
id=$name#sub-
printf '%sn' "$id"
printf 'Found ID: %sn' "$id" >&2
done | split -l 8 - participants_

for participants_id in participants_*; do
ids=( $(<"$participants_id") )

printf 'Processing ID: %sn' "$ids[@]" >&2

/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output

rm -f "$participants_id"
done


Fixes:



  1. The time command does not need to be an alias just due to having a long option argument to its -f option. Aliases aren't expanded in scripts anyway. I simply save the argument in a string and use that when invoking time.


  2. Your loop added to LD_LIBRARY_PATH in each iteration. This was not needed.


  3. The getting of the IDs from the directory names is better done in a proper loop. This loop will disappear later when we use an array to store the IDs instead.


  4. Rather than using find to locate the intermediate files, we just use them with a simple filename globbing pattern. We know they're right there and what their names are anyway.


  5. The intermediate file just processed is deleted inside the loop.


  6. The code is made readable through use of line continuations.


  7. The call to wait was removed. There are no background tasks to wait for.


The following variation stores the IDs in an array, all_ids instead of in temporary files:



#!/bin/bash

export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"

timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"

all_ids=( /input/sub-*/ )
all_ids=( "$all_ids[@]#/input/sub-" ) # remove "/input/sub-" from each item
all_ids=( "$all_ids[@]%/" ) # remove the trailing "/" from each item

printf 'Found ID: %sn' "$all_ids[@]" >&2

n=0
ids=( "$all_ids[@]:0:8" ) # pick out the first eight IDs

# Loop until the first ID in the ids array is empty
while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2

/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output

n=$(( n + 1 ))
ids=( "$all_ids[@]:n*8:8" ) # pick out the next eight IDs
done





share|improve this answer















There's a number of things in this script that are wrong or that can be improved.



The main issue in the question seems to be why the two calls to echo does not produce any output.



When you use a command substitution, like in



echo `rm file`


or the equivalent



echo $(rm file)


then the echo will get the output of the command within the backticks or inside the $(...). Neither of your command substitutions produces any output. Both commands that you use within backticks modify files, but again, neither produces output to their standard output stream (what would ordinarily be displayed in the terminal). This means that both calls to echo will produce no output either, apart from an empty line each.



In general, echo $(...) is an anti-pattern, meaning you can do the same thing in a much better way.



If you do want to output the result of some pipeline pipeline, then instead of writing



echo $(pipeline)


you would simply say



pipeline


The output of pipeline would be displayed, as the output of commands are usually displayed in the terminal.



In the code below, I have inserted a couple of printf statements that will output the relevant "progress information" in the script(s).




Here's a modified version of the script which is totally untested (as I don't have access to the tools that you use, or the input files), but it should mimic what your script is doing, including creating those intermediate files (these are not needed, and I'll show how to get rid of them later).



#!/bin/bash

export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"

timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"

for dirpath in /input/sub-*/; do
name=$(basename "$dirpath")
id=$name#sub-
printf '%sn' "$id"
printf 'Found ID: %sn' "$id" >&2
done | split -l 8 - participants_

for participants_id in participants_*; do
ids=( $(<"$participants_id") )

printf 'Processing ID: %sn' "$ids[@]" >&2

/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output

rm -f "$participants_id"
done


Fixes:



  1. The time command does not need to be an alias just due to having a long option argument to its -f option. Aliases aren't expanded in scripts anyway. I simply save the argument in a string and use that when invoking time.


  2. Your loop added to LD_LIBRARY_PATH in each iteration. This was not needed.


  3. The getting of the IDs from the directory names is better done in a proper loop. This loop will disappear later when we use an array to store the IDs instead.


  4. Rather than using find to locate the intermediate files, we just use them with a simple filename globbing pattern. We know they're right there and what their names are anyway.


  5. The intermediate file just processed is deleted inside the loop.


  6. The code is made readable through use of line continuations.


  7. The call to wait was removed. There are no background tasks to wait for.


The following variation stores the IDs in an array, all_ids instead of in temporary files:



#!/bin/bash

export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"

timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"

all_ids=( /input/sub-*/ )
all_ids=( "$all_ids[@]#/input/sub-" ) # remove "/input/sub-" from each item
all_ids=( "$all_ids[@]%/" ) # remove the trailing "/" from each item

printf 'Found ID: %sn' "$all_ids[@]" >&2

n=0
ids=( "$all_ids[@]:0:8" ) # pick out the first eight IDs

# Loop until the first ID in the ids array is empty
while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2

/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output

n=$(( n + 1 ))
ids=( "$all_ids[@]:n*8:8" ) # pick out the next eight IDs
done






share|improve this answer














share|improve this answer



share|improve this answer








edited Mar 5 at 15:33

























answered Mar 4 at 21:29









KusalanandaKusalananda

139k17259430




139k17259430












  • first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?

    – Relyativist
    Mar 5 at 12:04











  • @Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.

    – Kusalananda
    Mar 5 at 12:15












  • as i said it doesn't work like that inside container: root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" ) root@9bca6f96475d:/tmp# echo $all_ids *nothing*

    – Relyativist
    Mar 5 at 15:28












  • @Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!

    – Kusalananda
    Mar 5 at 15:34











  • thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks

    – Relyativist
    Mar 5 at 16:18


















  • first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?

    – Relyativist
    Mar 5 at 12:04











  • @Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.

    – Kusalananda
    Mar 5 at 12:15












  • as i said it doesn't work like that inside container: root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" ) root@9bca6f96475d:/tmp# echo $all_ids *nothing*

    – Relyativist
    Mar 5 at 15:28












  • @Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!

    – Kusalananda
    Mar 5 at 15:34











  • thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks

    – Relyativist
    Mar 5 at 16:18

















first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?

– Relyativist
Mar 5 at 12:04





first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?

– Relyativist
Mar 5 at 12:04













@Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.

– Kusalananda
Mar 5 at 12:15






@Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.

– Kusalananda
Mar 5 at 12:15














as i said it doesn't work like that inside container: root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" ) root@9bca6f96475d:/tmp# echo $all_ids *nothing*

– Relyativist
Mar 5 at 15:28






as i said it doesn't work like that inside container: root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" ) root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" ) root@9bca6f96475d:/tmp# echo $all_ids *nothing*

– Relyativist
Mar 5 at 15:28














@Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!

– Kusalananda
Mar 5 at 15:34





@Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!

– Kusalananda
Mar 5 at 15:34













thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks

– Relyativist
Mar 5 at 16:18






thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks

– Relyativist
Mar 5 at 16:18


















draft saved

draft discarded
















































Thanks for contributing an answer to Unix & Linux Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504349%2fshell-script-inside-docker%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown






Popular posts from this blog

How to check contact read email or not when send email to Individual?

Bahrain

Postfix configuration issue with fips on centos 7; mailgun relay