shell script inside docker
Clash Royale CLAN TAG#URR8PPP
i try run docker with entrypoint.sh, but it doesn't exequite .sh line:
echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_`
I wrote it in dockerfile:
ENTRYPOINT ["bash", "-c", "source /code/entrypoint.sh | ts '[%Y-%m-%d %H:%M:%S]' &>> /output/stderr.log"]
Why echo doesn't execute inside docker run [my_image]
Here is full entrypoint.sh code:
#! /bin/bash
alias time='/usr/bin/time -f "%C --- CPU:t%E real,t%U user,t%S
syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"'
echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 -
participants_`
while read input_bids_path
do
participants_id=$(basename $input_bids_path)
LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH
time fmriprep /input /output participant --fs-license-file
/opt/freesurfer/license.txt --fs-no-reconall --use-aroma --ignore fieldmaps
--n_cpus 12 --force-bbr --participant_label $(cat $participants_id) -w
/output
# rm -r /input/$participants_id
done < <(find /input -name "*participants_*" -type f)
echo `rm -r /input/$participants_id`
wait `jobs -p` && echo __ok__ || echo __err__
shell docker
|
show 3 more comments
i try run docker with entrypoint.sh, but it doesn't exequite .sh line:
echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_`
I wrote it in dockerfile:
ENTRYPOINT ["bash", "-c", "source /code/entrypoint.sh | ts '[%Y-%m-%d %H:%M:%S]' &>> /output/stderr.log"]
Why echo doesn't execute inside docker run [my_image]
Here is full entrypoint.sh code:
#! /bin/bash
alias time='/usr/bin/time -f "%C --- CPU:t%E real,t%U user,t%S
syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"'
echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 -
participants_`
while read input_bids_path
do
participants_id=$(basename $input_bids_path)
LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH
time fmriprep /input /output participant --fs-license-file
/opt/freesurfer/license.txt --fs-no-reconall --use-aroma --ignore fieldmaps
--n_cpus 12 --force-bbr --participant_label $(cat $participants_id) -w
/output
# rm -r /input/$participants_id
done < <(find /input -name "*participants_*" -type f)
echo `rm -r /input/$participants_id`
wait `jobs -p` && echo __ok__ || echo __err__
shell docker
Why do you useecho
on command substitution? Thatrm
will only remove the last/input/$participants_id
as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?
– Kusalananda♦
Mar 4 at 20:36
@Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.
– Relyativist
Mar 4 at 20:46
You have't mentioned what command you are referring to. You saidecho
, but there's more than one of those.
– Kusalananda♦
Mar 4 at 20:49
@Kusalananda echols -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_
– Relyativist
Mar 4 at 20:50
The command substitution does not produce anything forecho
to output. There is nothing toecho
. This is also the case for the secondecho
. It is unclear what you want theseecho
calls to actually do.
– Kusalananda♦
Mar 4 at 20:52
|
show 3 more comments
i try run docker with entrypoint.sh, but it doesn't exequite .sh line:
echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_`
I wrote it in dockerfile:
ENTRYPOINT ["bash", "-c", "source /code/entrypoint.sh | ts '[%Y-%m-%d %H:%M:%S]' &>> /output/stderr.log"]
Why echo doesn't execute inside docker run [my_image]
Here is full entrypoint.sh code:
#! /bin/bash
alias time='/usr/bin/time -f "%C --- CPU:t%E real,t%U user,t%S
syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"'
echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 -
participants_`
while read input_bids_path
do
participants_id=$(basename $input_bids_path)
LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH
time fmriprep /input /output participant --fs-license-file
/opt/freesurfer/license.txt --fs-no-reconall --use-aroma --ignore fieldmaps
--n_cpus 12 --force-bbr --participant_label $(cat $participants_id) -w
/output
# rm -r /input/$participants_id
done < <(find /input -name "*participants_*" -type f)
echo `rm -r /input/$participants_id`
wait `jobs -p` && echo __ok__ || echo __err__
shell docker
i try run docker with entrypoint.sh, but it doesn't exequite .sh line:
echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_`
I wrote it in dockerfile:
ENTRYPOINT ["bash", "-c", "source /code/entrypoint.sh | ts '[%Y-%m-%d %H:%M:%S]' &>> /output/stderr.log"]
Why echo doesn't execute inside docker run [my_image]
Here is full entrypoint.sh code:
#! /bin/bash
alias time='/usr/bin/time -f "%C --- CPU:t%E real,t%U user,t%S
syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"'
echo `ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 -
participants_`
while read input_bids_path
do
participants_id=$(basename $input_bids_path)
LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH
time fmriprep /input /output participant --fs-license-file
/opt/freesurfer/license.txt --fs-no-reconall --use-aroma --ignore fieldmaps
--n_cpus 12 --force-bbr --participant_label $(cat $participants_id) -w
/output
# rm -r /input/$participants_id
done < <(find /input -name "*participants_*" -type f)
echo `rm -r /input/$participants_id`
wait `jobs -p` && echo __ok__ || echo __err__
shell docker
shell docker
asked Mar 4 at 20:29
RelyativistRelyativist
317
317
Why do you useecho
on command substitution? Thatrm
will only remove the last/input/$participants_id
as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?
– Kusalananda♦
Mar 4 at 20:36
@Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.
– Relyativist
Mar 4 at 20:46
You have't mentioned what command you are referring to. You saidecho
, but there's more than one of those.
– Kusalananda♦
Mar 4 at 20:49
@Kusalananda echols -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_
– Relyativist
Mar 4 at 20:50
The command substitution does not produce anything forecho
to output. There is nothing toecho
. This is also the case for the secondecho
. It is unclear what you want theseecho
calls to actually do.
– Kusalananda♦
Mar 4 at 20:52
|
show 3 more comments
Why do you useecho
on command substitution? Thatrm
will only remove the last/input/$participants_id
as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?
– Kusalananda♦
Mar 4 at 20:36
@Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.
– Relyativist
Mar 4 at 20:46
You have't mentioned what command you are referring to. You saidecho
, but there's more than one of those.
– Kusalananda♦
Mar 4 at 20:49
@Kusalananda echols -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_
– Relyativist
Mar 4 at 20:50
The command substitution does not produce anything forecho
to output. There is nothing toecho
. This is also the case for the secondecho
. It is unclear what you want theseecho
calls to actually do.
– Kusalananda♦
Mar 4 at 20:52
Why do you use
echo
on command substitution? That rm
will only remove the last /input/$participants_id
as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?– Kusalananda♦
Mar 4 at 20:36
Why do you use
echo
on command substitution? That rm
will only remove the last /input/$participants_id
as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?– Kusalananda♦
Mar 4 at 20:36
@Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.
– Relyativist
Mar 4 at 20:46
@Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.
– Relyativist
Mar 4 at 20:46
You have't mentioned what command you are referring to. You said
echo
, but there's more than one of those.– Kusalananda♦
Mar 4 at 20:49
You have't mentioned what command you are referring to. You said
echo
, but there's more than one of those.– Kusalananda♦
Mar 4 at 20:49
@Kusalananda echo
ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_
– Relyativist
Mar 4 at 20:50
@Kusalananda echo
ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_
– Relyativist
Mar 4 at 20:50
The command substitution does not produce anything for
echo
to output. There is nothing to echo
. This is also the case for the second echo
. It is unclear what you want these echo
calls to actually do.– Kusalananda♦
Mar 4 at 20:52
The command substitution does not produce anything for
echo
to output. There is nothing to echo
. This is also the case for the second echo
. It is unclear what you want these echo
calls to actually do.– Kusalananda♦
Mar 4 at 20:52
|
show 3 more comments
1 Answer
1
active
oldest
votes
There's a number of things in this script that are wrong or that can be improved.
The main issue in the question seems to be why the two calls to echo
does not produce any output.
When you use a command substitution, like in
echo `rm file`
or the equivalent
echo $(rm file)
then the echo
will get the output of the command within the backticks or inside the $(...)
. Neither of your command substitutions produces any output. Both commands that you use within backticks modify files, but again, neither produces output to their standard output stream (what would ordinarily be displayed in the terminal). This means that both calls to echo
will produce no output either, apart from an empty line each.
In general, echo $(...)
is an anti-pattern, meaning you can do the same thing in a much better way.
If you do want to output the result of some pipeline pipeline
, then instead of writing
echo $(pipeline)
you would simply say
pipeline
The output of pipeline
would be displayed, as the output of commands are usually displayed in the terminal.
In the code below, I have inserted a couple of printf
statements that will output the relevant "progress information" in the script(s).
Here's a modified version of the script which is totally untested (as I don't have access to the tools that you use, or the input files), but it should mimic what your script is doing, including creating those intermediate files (these are not needed, and I'll show how to get rid of them later).
#!/bin/bash
export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"
timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"
for dirpath in /input/sub-*/; do
name=$(basename "$dirpath")
id=$name#sub-
printf '%sn' "$id"
printf 'Found ID: %sn' "$id" >&2
done | split -l 8 - participants_
for participants_id in participants_*; do
ids=( $(<"$participants_id") )
printf 'Processing ID: %sn' "$ids[@]" >&2
/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output
rm -f "$participants_id"
done
Fixes:
The
time
command does not need to be an alias just due to having a long option argument to its-f
option. Aliases aren't expanded in scripts anyway. I simply save the argument in a string and use that when invokingtime
.Your loop added to
LD_LIBRARY_PATH
in each iteration. This was not needed.The getting of the IDs from the directory names is better done in a proper loop. This loop will disappear later when we use an array to store the IDs instead.
Rather than using
find
to locate the intermediate files, we just use them with a simple filename globbing pattern. We know they're right there and what their names are anyway.The intermediate file just processed is deleted inside the loop.
The code is made readable through use of line continuations.
The call to
wait
was removed. There are no background tasks to wait for.
The following variation stores the IDs in an array, all_ids
instead of in temporary files:
#!/bin/bash
export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"
timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"
all_ids=( /input/sub-*/ )
all_ids=( "$all_ids[@]#/input/sub-" ) # remove "/input/sub-" from each item
all_ids=( "$all_ids[@]%/" ) # remove the trailing "/" from each item
printf 'Found ID: %sn' "$all_ids[@]" >&2
n=0
ids=( "$all_ids[@]:0:8" ) # pick out the first eight IDs
# Loop until the first ID in the ids array is empty
while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2
/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output
n=$(( n + 1 ))
ids=( "$all_ids[@]:n*8:8" ) # pick out the next eight IDs
done
first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?
– Relyativist
Mar 5 at 12:04
@Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.
– Kusalananda♦
Mar 5 at 12:15
as i said it doesn't work like that inside container:root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" )
root@9bca6f96475d:/tmp# echo $all_ids
*nothing*
– Relyativist
Mar 5 at 15:28
@Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!
– Kusalananda♦
Mar 5 at 15:34
thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks
– Relyativist
Mar 5 at 16:18
|
show 2 more comments
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504349%2fshell-script-inside-docker%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
There's a number of things in this script that are wrong or that can be improved.
The main issue in the question seems to be why the two calls to echo
does not produce any output.
When you use a command substitution, like in
echo `rm file`
or the equivalent
echo $(rm file)
then the echo
will get the output of the command within the backticks or inside the $(...)
. Neither of your command substitutions produces any output. Both commands that you use within backticks modify files, but again, neither produces output to their standard output stream (what would ordinarily be displayed in the terminal). This means that both calls to echo
will produce no output either, apart from an empty line each.
In general, echo $(...)
is an anti-pattern, meaning you can do the same thing in a much better way.
If you do want to output the result of some pipeline pipeline
, then instead of writing
echo $(pipeline)
you would simply say
pipeline
The output of pipeline
would be displayed, as the output of commands are usually displayed in the terminal.
In the code below, I have inserted a couple of printf
statements that will output the relevant "progress information" in the script(s).
Here's a modified version of the script which is totally untested (as I don't have access to the tools that you use, or the input files), but it should mimic what your script is doing, including creating those intermediate files (these are not needed, and I'll show how to get rid of them later).
#!/bin/bash
export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"
timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"
for dirpath in /input/sub-*/; do
name=$(basename "$dirpath")
id=$name#sub-
printf '%sn' "$id"
printf 'Found ID: %sn' "$id" >&2
done | split -l 8 - participants_
for participants_id in participants_*; do
ids=( $(<"$participants_id") )
printf 'Processing ID: %sn' "$ids[@]" >&2
/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output
rm -f "$participants_id"
done
Fixes:
The
time
command does not need to be an alias just due to having a long option argument to its-f
option. Aliases aren't expanded in scripts anyway. I simply save the argument in a string and use that when invokingtime
.Your loop added to
LD_LIBRARY_PATH
in each iteration. This was not needed.The getting of the IDs from the directory names is better done in a proper loop. This loop will disappear later when we use an array to store the IDs instead.
Rather than using
find
to locate the intermediate files, we just use them with a simple filename globbing pattern. We know they're right there and what their names are anyway.The intermediate file just processed is deleted inside the loop.
The code is made readable through use of line continuations.
The call to
wait
was removed. There are no background tasks to wait for.
The following variation stores the IDs in an array, all_ids
instead of in temporary files:
#!/bin/bash
export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"
timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"
all_ids=( /input/sub-*/ )
all_ids=( "$all_ids[@]#/input/sub-" ) # remove "/input/sub-" from each item
all_ids=( "$all_ids[@]%/" ) # remove the trailing "/" from each item
printf 'Found ID: %sn' "$all_ids[@]" >&2
n=0
ids=( "$all_ids[@]:0:8" ) # pick out the first eight IDs
# Loop until the first ID in the ids array is empty
while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2
/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output
n=$(( n + 1 ))
ids=( "$all_ids[@]:n*8:8" ) # pick out the next eight IDs
done
first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?
– Relyativist
Mar 5 at 12:04
@Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.
– Kusalananda♦
Mar 5 at 12:15
as i said it doesn't work like that inside container:root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" )
root@9bca6f96475d:/tmp# echo $all_ids
*nothing*
– Relyativist
Mar 5 at 15:28
@Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!
– Kusalananda♦
Mar 5 at 15:34
thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks
– Relyativist
Mar 5 at 16:18
|
show 2 more comments
There's a number of things in this script that are wrong or that can be improved.
The main issue in the question seems to be why the two calls to echo
does not produce any output.
When you use a command substitution, like in
echo `rm file`
or the equivalent
echo $(rm file)
then the echo
will get the output of the command within the backticks or inside the $(...)
. Neither of your command substitutions produces any output. Both commands that you use within backticks modify files, but again, neither produces output to their standard output stream (what would ordinarily be displayed in the terminal). This means that both calls to echo
will produce no output either, apart from an empty line each.
In general, echo $(...)
is an anti-pattern, meaning you can do the same thing in a much better way.
If you do want to output the result of some pipeline pipeline
, then instead of writing
echo $(pipeline)
you would simply say
pipeline
The output of pipeline
would be displayed, as the output of commands are usually displayed in the terminal.
In the code below, I have inserted a couple of printf
statements that will output the relevant "progress information" in the script(s).
Here's a modified version of the script which is totally untested (as I don't have access to the tools that you use, or the input files), but it should mimic what your script is doing, including creating those intermediate files (these are not needed, and I'll show how to get rid of them later).
#!/bin/bash
export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"
timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"
for dirpath in /input/sub-*/; do
name=$(basename "$dirpath")
id=$name#sub-
printf '%sn' "$id"
printf 'Found ID: %sn' "$id" >&2
done | split -l 8 - participants_
for participants_id in participants_*; do
ids=( $(<"$participants_id") )
printf 'Processing ID: %sn' "$ids[@]" >&2
/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output
rm -f "$participants_id"
done
Fixes:
The
time
command does not need to be an alias just due to having a long option argument to its-f
option. Aliases aren't expanded in scripts anyway. I simply save the argument in a string and use that when invokingtime
.Your loop added to
LD_LIBRARY_PATH
in each iteration. This was not needed.The getting of the IDs from the directory names is better done in a proper loop. This loop will disappear later when we use an array to store the IDs instead.
Rather than using
find
to locate the intermediate files, we just use them with a simple filename globbing pattern. We know they're right there and what their names are anyway.The intermediate file just processed is deleted inside the loop.
The code is made readable through use of line continuations.
The call to
wait
was removed. There are no background tasks to wait for.
The following variation stores the IDs in an array, all_ids
instead of in temporary files:
#!/bin/bash
export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"
timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"
all_ids=( /input/sub-*/ )
all_ids=( "$all_ids[@]#/input/sub-" ) # remove "/input/sub-" from each item
all_ids=( "$all_ids[@]%/" ) # remove the trailing "/" from each item
printf 'Found ID: %sn' "$all_ids[@]" >&2
n=0
ids=( "$all_ids[@]:0:8" ) # pick out the first eight IDs
# Loop until the first ID in the ids array is empty
while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2
/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output
n=$(( n + 1 ))
ids=( "$all_ids[@]:n*8:8" ) # pick out the next eight IDs
done
first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?
– Relyativist
Mar 5 at 12:04
@Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.
– Kusalananda♦
Mar 5 at 12:15
as i said it doesn't work like that inside container:root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" )
root@9bca6f96475d:/tmp# echo $all_ids
*nothing*
– Relyativist
Mar 5 at 15:28
@Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!
– Kusalananda♦
Mar 5 at 15:34
thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks
– Relyativist
Mar 5 at 16:18
|
show 2 more comments
There's a number of things in this script that are wrong or that can be improved.
The main issue in the question seems to be why the two calls to echo
does not produce any output.
When you use a command substitution, like in
echo `rm file`
or the equivalent
echo $(rm file)
then the echo
will get the output of the command within the backticks or inside the $(...)
. Neither of your command substitutions produces any output. Both commands that you use within backticks modify files, but again, neither produces output to their standard output stream (what would ordinarily be displayed in the terminal). This means that both calls to echo
will produce no output either, apart from an empty line each.
In general, echo $(...)
is an anti-pattern, meaning you can do the same thing in a much better way.
If you do want to output the result of some pipeline pipeline
, then instead of writing
echo $(pipeline)
you would simply say
pipeline
The output of pipeline
would be displayed, as the output of commands are usually displayed in the terminal.
In the code below, I have inserted a couple of printf
statements that will output the relevant "progress information" in the script(s).
Here's a modified version of the script which is totally untested (as I don't have access to the tools that you use, or the input files), but it should mimic what your script is doing, including creating those intermediate files (these are not needed, and I'll show how to get rid of them later).
#!/bin/bash
export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"
timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"
for dirpath in /input/sub-*/; do
name=$(basename "$dirpath")
id=$name#sub-
printf '%sn' "$id"
printf 'Found ID: %sn' "$id" >&2
done | split -l 8 - participants_
for participants_id in participants_*; do
ids=( $(<"$participants_id") )
printf 'Processing ID: %sn' "$ids[@]" >&2
/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output
rm -f "$participants_id"
done
Fixes:
The
time
command does not need to be an alias just due to having a long option argument to its-f
option. Aliases aren't expanded in scripts anyway. I simply save the argument in a string and use that when invokingtime
.Your loop added to
LD_LIBRARY_PATH
in each iteration. This was not needed.The getting of the IDs from the directory names is better done in a proper loop. This loop will disappear later when we use an array to store the IDs instead.
Rather than using
find
to locate the intermediate files, we just use them with a simple filename globbing pattern. We know they're right there and what their names are anyway.The intermediate file just processed is deleted inside the loop.
The code is made readable through use of line continuations.
The call to
wait
was removed. There are no background tasks to wait for.
The following variation stores the IDs in an array, all_ids
instead of in temporary files:
#!/bin/bash
export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"
timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"
all_ids=( /input/sub-*/ )
all_ids=( "$all_ids[@]#/input/sub-" ) # remove "/input/sub-" from each item
all_ids=( "$all_ids[@]%/" ) # remove the trailing "/" from each item
printf 'Found ID: %sn' "$all_ids[@]" >&2
n=0
ids=( "$all_ids[@]:0:8" ) # pick out the first eight IDs
# Loop until the first ID in the ids array is empty
while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2
/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output
n=$(( n + 1 ))
ids=( "$all_ids[@]:n*8:8" ) # pick out the next eight IDs
done
There's a number of things in this script that are wrong or that can be improved.
The main issue in the question seems to be why the two calls to echo
does not produce any output.
When you use a command substitution, like in
echo `rm file`
or the equivalent
echo $(rm file)
then the echo
will get the output of the command within the backticks or inside the $(...)
. Neither of your command substitutions produces any output. Both commands that you use within backticks modify files, but again, neither produces output to their standard output stream (what would ordinarily be displayed in the terminal). This means that both calls to echo
will produce no output either, apart from an empty line each.
In general, echo $(...)
is an anti-pattern, meaning you can do the same thing in a much better way.
If you do want to output the result of some pipeline pipeline
, then instead of writing
echo $(pipeline)
you would simply say
pipeline
The output of pipeline
would be displayed, as the output of commands are usually displayed in the terminal.
In the code below, I have inserted a couple of printf
statements that will output the relevant "progress information" in the script(s).
Here's a modified version of the script which is totally untested (as I don't have access to the tools that you use, or the input files), but it should mimic what your script is doing, including creating those intermediate files (these are not needed, and I'll show how to get rid of them later).
#!/bin/bash
export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"
timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"
for dirpath in /input/sub-*/; do
name=$(basename "$dirpath")
id=$name#sub-
printf '%sn' "$id"
printf 'Found ID: %sn' "$id" >&2
done | split -l 8 - participants_
for participants_id in participants_*; do
ids=( $(<"$participants_id") )
printf 'Processing ID: %sn' "$ids[@]" >&2
/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output
rm -f "$participants_id"
done
Fixes:
The
time
command does not need to be an alias just due to having a long option argument to its-f
option. Aliases aren't expanded in scripts anyway. I simply save the argument in a string and use that when invokingtime
.Your loop added to
LD_LIBRARY_PATH
in each iteration. This was not needed.The getting of the IDs from the directory names is better done in a proper loop. This loop will disappear later when we use an array to store the IDs instead.
Rather than using
find
to locate the intermediate files, we just use them with a simple filename globbing pattern. We know they're right there and what their names are anyway.The intermediate file just processed is deleted inside the loop.
The code is made readable through use of line continuations.
The call to
wait
was removed. There are no background tasks to wait for.
The following variation stores the IDs in an array, all_ids
instead of in temporary files:
#!/bin/bash
export LD_LIBRARY_PATH="/usr/lib/fsl/5.0:$LD_LIBRARY_PATH"
timefmt="%C --- CPU:t%E real,t%U user,t%S syst%PtMem:t%KkiB avg.,t%MkiB max.tExit:t%x"
all_ids=( /input/sub-*/ )
all_ids=( "$all_ids[@]#/input/sub-" ) # remove "/input/sub-" from each item
all_ids=( "$all_ids[@]%/" ) # remove the trailing "/" from each item
printf 'Found ID: %sn' "$all_ids[@]" >&2
n=0
ids=( "$all_ids[@]:0:8" ) # pick out the first eight IDs
# Loop until the first ID in the ids array is empty
while [ -n "$ids[0]" ] ; do
printf 'Processing ID: %sn' "$ids[@]" >&2
/usr/bin/time -f "$timefmt"
fmriprep /input /output participant
--fs-license-file /opt/freesurfer/license.txt
--fs-no-reconall --use-aroma
--ignore fieldmaps --n_cpus 12 --force-bbr
--participant_label "$ids[@]"
-w /output
n=$(( n + 1 ))
ids=( "$all_ids[@]:n*8:8" ) # pick out the next eight IDs
done
edited Mar 5 at 15:33
answered Mar 4 at 21:29
Kusalananda♦Kusalananda
139k17259430
139k17259430
first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?
– Relyativist
Mar 5 at 12:04
@Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.
– Kusalananda♦
Mar 5 at 12:15
as i said it doesn't work like that inside container:root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" )
root@9bca6f96475d:/tmp# echo $all_ids
*nothing*
– Relyativist
Mar 5 at 15:28
@Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!
– Kusalananda♦
Mar 5 at 15:34
thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks
– Relyativist
Mar 5 at 16:18
|
show 2 more comments
first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?
– Relyativist
Mar 5 at 12:04
@Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.
– Kusalananda♦
Mar 5 at 12:15
as i said it doesn't work like that inside container:root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" )
root@9bca6f96475d:/tmp# echo $all_ids
*nothing*
– Relyativist
Mar 5 at 15:28
@Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!
– Kusalananda♦
Mar 5 at 15:34
thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks
– Relyativist
Mar 5 at 16:18
first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?
– Relyativist
Mar 5 at 12:04
first of all - wow, thanks of extended answer. Seconldy, does your script,does you code interpretation allow to iterate throught unknown number of n elements generated with all_ids?
– Relyativist
Mar 5 at 12:04
@Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.
– Kusalananda♦
Mar 5 at 12:15
@Relyativist I believe it should cope with a variable number of IDs, no matter if these are a multiple of 8 or not. As I said, I haven't tested this, so I'm not 100% certain that it works as it should, but you should at least be able to use it to get something working at your end. In particular, I am no Docker user.
– Kusalananda♦
Mar 5 at 12:15
as i said it doesn't work like that inside container:
root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" )
root@9bca6f96475d:/tmp# echo $all_ids
*nothing*
– Relyativist
Mar 5 at 15:28
as i said it doesn't work like that inside container:
root@9bca6f96475d:/tmp# all_ids=( /input/sub-*/ )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]#/input/sub-" )
root@9bca6f96475d:/tmp# all_ids=( "$ids[@]%/" )
root@9bca6f96475d:/tmp# echo $all_ids
*nothing*
– Relyativist
Mar 5 at 15:28
@Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!
– Kusalananda♦
Mar 5 at 15:34
@Relyativist I had typos in my code. Now corrected. Thanks for pointing it out!
– Kusalananda♦
Mar 5 at 15:34
thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks
– Relyativist
Mar 5 at 16:18
thanks, could you please give an idea how to check if subject in output already exists, and not to include into all_ids array? thanks
– Relyativist
Mar 5 at 16:18
|
show 2 more comments
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504349%2fshell-script-inside-docker%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Why do you use
echo
on command substitution? Thatrm
will only remove the last/input/$participants_id
as it's after the loop. Aliases are not expanded in shell scripts by default (and I don't see why you would want to use an alias here anyway). Did you try to run this outside of Docker to debug it? Did you run the individual commands to see that they executed in the way that you expected them to?– Kusalananda♦
Mar 4 at 20:36
@Kusalananda, yes I ran it without docker in shell, everything is working fine. It’s strange that he’ll command doesn’t execute inside of it.
– Relyativist
Mar 4 at 20:46
You have't mentioned what command you are referring to. You said
echo
, but there's more than one of those.– Kusalananda♦
Mar 4 at 20:49
@Kusalananda echo
ls -d /input/sub-*/ | sed -e 's/.*sub-(.*)//1/' | split -l 8 - participants_
– Relyativist
Mar 4 at 20:50
The command substitution does not produce anything for
echo
to output. There is nothing toecho
. This is also the case for the secondecho
. It is unclear what you want theseecho
calls to actually do.– Kusalananda♦
Mar 4 at 20:52