perform the same function to thousands of files with GNU Parallel

up vote
1
down vote

favorite

I need to perform the same function to thousands of files and I would like to use GNU Parallel, but I'm not quite sure how to get it working.

My script is

grep -h ">begin" file* | cut -c1-25|awk 'length ($1) ==25'|sort|uniq -c|sed 's#^ *##'|tr ' ' 't'>OutCombined

Where I have thousands of input files and I want all results sent to OutCombined

edited Jul 3 at 21:16

Jeff Schaller

30.8k846104

asked Jul 3 at 20:25

mah

554

2

first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â€“Â francois P
Jul 3 at 20:49

1

Agreed, chances are very high that grep | cut | awk | sort | uniq | sed can be remade into one awk or even awk | sort -u | sed at worst.
â€“Â DopeGhoti
Jul 3 at 22:06

1

... maybe something like awk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t' (I omitted the sort on the assumption that it's only required for the uniq - it could be added back quite easily, at least in the case of gawk)
â€“Â steeldriver
Jul 4 at 0:09

add a commentÂ |Â

up vote
1
down vote

favorite

I need to perform the same function to thousands of files and I would like to use GNU Parallel, but I'm not quite sure how to get it working.

My script is

grep -h ">begin" file* | cut -c1-25|awk 'length ($1) ==25'|sort|uniq -c|sed 's#^ *##'|tr ' ' 't'>OutCombined

Where I have thousands of input files and I want all results sent to OutCombined

edited Jul 3 at 21:16

Jeff Schaller

30.8k846104

asked Jul 3 at 20:25

mah

554

2

first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â€“Â francois P
Jul 3 at 20:49

1

Agreed, chances are very high that grep | cut | awk | sort | uniq | sed can be remade into one awk or even awk | sort -u | sed at worst.
â€“Â DopeGhoti
Jul 3 at 22:06

1

... maybe something like awk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t' (I omitted the sort on the assumption that it's only required for the uniq - it could be added back quite easily, at least in the case of gawk)
â€“Â steeldriver
Jul 4 at 0:09

add a commentÂ |Â

up vote
1
down vote

favorite

I need to perform the same function to thousands of files and I would like to use GNU Parallel, but I'm not quite sure how to get it working.

My script is

grep -h ">begin" file* | cut -c1-25|awk 'length ($1) ==25'|sort|uniq -c|sed 's#^ *##'|tr ' ' 't'>OutCombined

Where I have thousands of input files and I want all results sent to OutCombined

edited Jul 3 at 21:16

Jeff Schaller

30.8k846104

asked Jul 3 at 20:25

mah

554

I need to perform the same function to thousands of files and I would like to use GNU Parallel, but I'm not quite sure how to get it working.

My script is

grep -h ">begin" file* | cut -c1-25|awk 'length ($1) ==25'|sort|uniq -c|sed 's#^ *##'|tr ' ' 't'>OutCombined

Where I have thousands of input files and I want all results sent to OutCombined

edited Jul 3 at 21:16

Jeff Schaller

30.8k846104

asked Jul 3 at 20:25

mah

554

edited Jul 3 at 21:16

Jeff Schaller

30.8k846104

edited Jul 3 at 21:16

Jeff Schaller

30.8k846104

edited Jul 3 at 21:16

Jeff Schaller

30.8k846104

asked Jul 3 at 20:25

mah

554

asked Jul 3 at 20:25

mah

554

asked Jul 3 at 20:25

mah

554

2

first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â€“Â francois P
Jul 3 at 20:49

1

Agreed, chances are very high that grep | cut | awk | sort | uniq | sed can be remade into one awk or even awk | sort -u | sed at worst.
â€“Â DopeGhoti
Jul 3 at 22:06

1

... maybe something like awk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t' (I omitted the sort on the assumption that it's only required for the uniq - it could be added back quite easily, at least in the case of gawk)
â€“Â steeldriver
Jul 4 at 0:09

add a commentÂ |Â

2

first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â€“Â francois P
Jul 3 at 20:49

1

Agreed, chances are very high that grep | cut | awk | sort | uniq | sed can be remade into one awk or even awk | sort -u | sed at worst.
â€“Â DopeGhoti
Jul 3 at 22:06

1

... maybe something like awk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t' (I omitted the sort on the assumption that it's only required for the uniq - it could be added back quite easily, at least in the case of gawk)
â€“Â steeldriver
Jul 4 at 0:09

first try to simplify your own work choose awk or sed but never use on same command a mix of grep/sort/awk/sed :D you will have less difficulties then... try again to generate your OutCombined file. For more help please give an example of line input before & in outCombined file.
â€“Â francois P
Jul 3 at 20:49

... maybe something like awk '/>begin/ && length > 24 a[substr($0,1,25)]++ ENDfor (x in a) print a[x], x' OFS='t' (I omitted the sort on the assumption that it's only required for the uniq - it could be added back quite easily, at least in the case of gawk)
â€“Â steeldriver
Jul 4 at 0:09

add a commentÂ |Â

1 Answer
1

active

oldest

votes

up vote
0
down vote

Something like this:

doit() 
 grep -h ">begin" "$1" 
export -f doit
parallel -k doit >OutCombined ::: file*

If you do not need output in the same order as input, you can drop -k.

answered Jul 4 at 5:31

Ole Tange

11.2k1342101

add a commentÂ |Â

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f453298%2fperform-the-same-function-to-thousands-of-files-with-gnu-parallel%23new-answer', 'question_page');

);

Post as a guest

Name

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

up vote
0
down vote

Something like this:

doit() 
 grep -h ">begin" "$1" 
export -f doit
parallel -k doit >OutCombined ::: file*

If you do not need output in the same order as input, you can drop -k.

answered Jul 4 at 5:31

Ole Tange

11.2k1342101

add a commentÂ |Â

up vote
0
down vote

Something like this:

doit() 
 grep -h ">begin" "$1" 
export -f doit
parallel -k doit >OutCombined ::: file*

If you do not need output in the same order as input, you can drop -k.

answered Jul 4 at 5:31

Ole Tange

11.2k1342101

add a commentÂ |Â

up vote
0
down vote

Something like this:

doit() 
 grep -h ">begin" "$1" 
export -f doit
parallel -k doit >OutCombined ::: file*

If you do not need output in the same order as input, you can drop -k.

answered Jul 4 at 5:31

Ole Tange

11.2k1342101

Something like this:

doit() 
 grep -h ">begin" "$1" 
export -f doit
parallel -k doit >OutCombined ::: file*

If you do not need output in the same order as input, you can drop -k.

answered Jul 4 at 5:31

Ole Tange

11.2k1342101

answered Jul 4 at 5:31

Ole Tange

11.2k1342101

answered Jul 4 at 5:31

Ole Tange

11.2k1342101

answered Jul 4 at 5:31

Ole Tange

11.2k1342101

add a commentÂ |Â

draft saved

draft discarded

draft saved

draft discarded

Post as a guest

Name

9Zvm,tlS2ycFI,kmMHA6v NdqQ95,LsxRTLGCfcyHpvpNtvjkYcVvADnvQAZK8Y433wWvSaJoTq,W,vKftXQ0ffXT5japk0CujZnh

搜尋此網誌

mjhjmtu