perform the same function to thousands of files with GNU Parallel
Clash Royale CLAN TAG#URR8PPP
up vote
1
down vote
favorite
I need to perform the same function to thousands of files and I would like to use GNU Parallel, but I'm not quite sure how to get it working.
My script is
grep -h ">begin" file* | cut -c1-25|awk 'length ($1) ==25'|sort|uniq -c|sed 's#^ *##'|tr ' ' 't'>OutCombined
Where I have thousands of input files and I want all results sent to OutCombined
shell-script gnu-parallel
add a comment |Â
up vote
1
down vote
favorite
I need to perform the same function to thousands of files and I would like to use GNU Parallel, but I'm not quite sure how to get it working.
My script is
grep -h ">begin" file* | cut -c1-25|awk 'length ($1) ==25'|sort|uniq -c|sed 's#^ *##'|tr ' ' 't'>OutCombined
Where I have thousands of input files and I want all results sent to OutCombined
shell-script gnu-parallel
2
first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â francois P
Jul 3 at 20:49
1
Agreed, chances are very high thatgrep | cut | awk | sort | uniq | sed
can be remade into oneawk
or evenawk | sort -u | sed
at worst.
â DopeGhoti
Jul 3 at 22:06
1
... maybe something likeawk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t'
(I omitted thesort
on the assumption that it's only required for theuniq
- it could be added back quite easily, at least in the case ofgawk
)
â steeldriver
Jul 4 at 0:09
add a comment |Â
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I need to perform the same function to thousands of files and I would like to use GNU Parallel, but I'm not quite sure how to get it working.
My script is
grep -h ">begin" file* | cut -c1-25|awk 'length ($1) ==25'|sort|uniq -c|sed 's#^ *##'|tr ' ' 't'>OutCombined
Where I have thousands of input files and I want all results sent to OutCombined
shell-script gnu-parallel
I need to perform the same function to thousands of files and I would like to use GNU Parallel, but I'm not quite sure how to get it working.
My script is
grep -h ">begin" file* | cut -c1-25|awk 'length ($1) ==25'|sort|uniq -c|sed 's#^ *##'|tr ' ' 't'>OutCombined
Where I have thousands of input files and I want all results sent to OutCombined
shell-script gnu-parallel
edited Jul 3 at 21:16
Jeff Schaller
30.8k846104
30.8k846104
asked Jul 3 at 20:25
mah
554
554
2
first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â francois P
Jul 3 at 20:49
1
Agreed, chances are very high thatgrep | cut | awk | sort | uniq | sed
can be remade into oneawk
or evenawk | sort -u | sed
at worst.
â DopeGhoti
Jul 3 at 22:06
1
... maybe something likeawk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t'
(I omitted thesort
on the assumption that it's only required for theuniq
- it could be added back quite easily, at least in the case ofgawk
)
â steeldriver
Jul 4 at 0:09
add a comment |Â
2
first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â francois P
Jul 3 at 20:49
1
Agreed, chances are very high thatgrep | cut | awk | sort | uniq | sed
can be remade into oneawk
or evenawk | sort -u | sed
at worst.
â DopeGhoti
Jul 3 at 22:06
1
... maybe something likeawk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t'
(I omitted thesort
on the assumption that it's only required for theuniq
- it could be added back quite easily, at least in the case ofgawk
)
â steeldriver
Jul 4 at 0:09
2
2
first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â francois P
Jul 3 at 20:49
first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â francois P
Jul 3 at 20:49
1
1
Agreed, chances are very high that
grep | cut | awk | sort | uniq | sed
can be remade into one awk
or even awk | sort -u | sed
at worst.â DopeGhoti
Jul 3 at 22:06
Agreed, chances are very high that
grep | cut | awk | sort | uniq | sed
can be remade into one awk
or even awk | sort -u | sed
at worst.â DopeGhoti
Jul 3 at 22:06
1
1
... maybe something like
awk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t'
(I omitted the sort
on the assumption that it's only required for the uniq
- it could be added back quite easily, at least in the case of gawk
)â steeldriver
Jul 4 at 0:09
... maybe something like
awk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t'
(I omitted the sort
on the assumption that it's only required for the uniq
- it could be added back quite easily, at least in the case of gawk
)â steeldriver
Jul 4 at 0:09
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
0
down vote
Something like this:
doit()
grep -h ">begin" "$1"
export -f doit
parallel -k doit >OutCombined ::: file*
If you do not need output in the same order as input, you can drop -k
.
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
Something like this:
doit()
grep -h ">begin" "$1"
export -f doit
parallel -k doit >OutCombined ::: file*
If you do not need output in the same order as input, you can drop -k
.
add a comment |Â
up vote
0
down vote
Something like this:
doit()
grep -h ">begin" "$1"
export -f doit
parallel -k doit >OutCombined ::: file*
If you do not need output in the same order as input, you can drop -k
.
add a comment |Â
up vote
0
down vote
up vote
0
down vote
Something like this:
doit()
grep -h ">begin" "$1"
export -f doit
parallel -k doit >OutCombined ::: file*
If you do not need output in the same order as input, you can drop -k
.
Something like this:
doit()
grep -h ">begin" "$1"
export -f doit
parallel -k doit >OutCombined ::: file*
If you do not need output in the same order as input, you can drop -k
.
answered Jul 4 at 5:31
Ole Tange
11.2k1342101
11.2k1342101
add a comment |Â
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f453298%2fperform-the-same-function-to-thousands-of-files-with-gnu-parallel%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
2
first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â francois P
Jul 3 at 20:49
1
Agreed, chances are very high that
grep | cut | awk | sort | uniq | sed
can be remade into oneawk
or evenawk | sort -u | sed
at worst.â DopeGhoti
Jul 3 at 22:06
1
... maybe something like
awk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t'
(I omitted thesort
on the assumption that it's only required for theuniq
- it could be added back quite easily, at least in the case ofgawk
)â steeldriver
Jul 4 at 0:09