Trying to perform two operations on a single huge source file to gain performance
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
I am performing two operations in a single Linux command. The operations are:
Sending data from source file into a new target file.
Counting the number of records in the source file.
For example: source file: a.txt
, target file: b.txt
cat a.txt > b.txt; cat a.txt | wc -l
In the above example, I extract data from a.txt
twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.
How can I accomplish this?
shell-script pipe
New contributor
add a comment |
up vote
0
down vote
favorite
I am performing two operations in a single Linux command. The operations are:
Sending data from source file into a new target file.
Counting the number of records in the source file.
For example: source file: a.txt
, target file: b.txt
cat a.txt > b.txt; cat a.txt | wc -l
In the above example, I extract data from a.txt
twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.
How can I accomplish this?
shell-script pipe
New contributor
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I am performing two operations in a single Linux command. The operations are:
Sending data from source file into a new target file.
Counting the number of records in the source file.
For example: source file: a.txt
, target file: b.txt
cat a.txt > b.txt; cat a.txt | wc -l
In the above example, I extract data from a.txt
twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.
How can I accomplish this?
shell-script pipe
New contributor
I am performing two operations in a single Linux command. The operations are:
Sending data from source file into a new target file.
Counting the number of records in the source file.
For example: source file: a.txt
, target file: b.txt
cat a.txt > b.txt; cat a.txt | wc -l
In the above example, I extract data from a.txt
twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.
How can I accomplish this?
shell-script pipe
shell-script pipe
New contributor
New contributor
edited 12 hours ago
Rui F Ribeiro
38.1k1475123
38.1k1475123
New contributor
asked 12 hours ago
Puneeth
1
1
New contributor
New contributor
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
up vote
4
down vote
tee
is the command you're looking for:
cat a.txt | tee b.txt | wc -l
Also, as suggested in the comments, if you want something shorter you can avoid using cat
entirely and redirect a.txt
straight into tee
's stdin:
tee b.txt < a.txt | wc -l
From the man page:
tee
- read from standard input and write to standard output and files
1
tee b.txt <a.txt | wc -l
. No need for thecat
.
– Kusalananda
12 hours ago
@Kusalananda thank you, added to the answer
– Marco Bonelli
12 hours ago
add a comment |
up vote
0
down vote
To get both actions in one file read you can do:
awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt
Understand that the count will come from stderr
.
add a comment |
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
4
down vote
tee
is the command you're looking for:
cat a.txt | tee b.txt | wc -l
Also, as suggested in the comments, if you want something shorter you can avoid using cat
entirely and redirect a.txt
straight into tee
's stdin:
tee b.txt < a.txt | wc -l
From the man page:
tee
- read from standard input and write to standard output and files
1
tee b.txt <a.txt | wc -l
. No need for thecat
.
– Kusalananda
12 hours ago
@Kusalananda thank you, added to the answer
– Marco Bonelli
12 hours ago
add a comment |
up vote
4
down vote
tee
is the command you're looking for:
cat a.txt | tee b.txt | wc -l
Also, as suggested in the comments, if you want something shorter you can avoid using cat
entirely and redirect a.txt
straight into tee
's stdin:
tee b.txt < a.txt | wc -l
From the man page:
tee
- read from standard input and write to standard output and files
1
tee b.txt <a.txt | wc -l
. No need for thecat
.
– Kusalananda
12 hours ago
@Kusalananda thank you, added to the answer
– Marco Bonelli
12 hours ago
add a comment |
up vote
4
down vote
up vote
4
down vote
tee
is the command you're looking for:
cat a.txt | tee b.txt | wc -l
Also, as suggested in the comments, if you want something shorter you can avoid using cat
entirely and redirect a.txt
straight into tee
's stdin:
tee b.txt < a.txt | wc -l
From the man page:
tee
- read from standard input and write to standard output and files
tee
is the command you're looking for:
cat a.txt | tee b.txt | wc -l
Also, as suggested in the comments, if you want something shorter you can avoid using cat
entirely and redirect a.txt
straight into tee
's stdin:
tee b.txt < a.txt | wc -l
From the man page:
tee
- read from standard input and write to standard output and files
edited 11 hours ago
answered 12 hours ago
Marco Bonelli
22112
22112
1
tee b.txt <a.txt | wc -l
. No need for thecat
.
– Kusalananda
12 hours ago
@Kusalananda thank you, added to the answer
– Marco Bonelli
12 hours ago
add a comment |
1
tee b.txt <a.txt | wc -l
. No need for thecat
.
– Kusalananda
12 hours ago
@Kusalananda thank you, added to the answer
– Marco Bonelli
12 hours ago
1
1
tee b.txt <a.txt | wc -l
. No need for the cat
.– Kusalananda
12 hours ago
tee b.txt <a.txt | wc -l
. No need for the cat
.– Kusalananda
12 hours ago
@Kusalananda thank you, added to the answer
– Marco Bonelli
12 hours ago
@Kusalananda thank you, added to the answer
– Marco Bonelli
12 hours ago
add a comment |
up vote
0
down vote
To get both actions in one file read you can do:
awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt
Understand that the count will come from stderr
.
add a comment |
up vote
0
down vote
To get both actions in one file read you can do:
awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt
Understand that the count will come from stderr
.
add a comment |
up vote
0
down vote
up vote
0
down vote
To get both actions in one file read you can do:
awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt
Understand that the count will come from stderr
.
To get both actions in one file read you can do:
awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt
Understand that the count will come from stderr
.
answered 12 hours ago
Isaac
9,34911442
9,34911442
add a comment |
add a comment |
Puneeth is a new contributor. Be nice, and check out our Code of Conduct.
Puneeth is a new contributor. Be nice, and check out our Code of Conduct.
Puneeth is a new contributor. Be nice, and check out our Code of Conduct.
Puneeth is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f481265%2ftrying-to-perform-two-operations-on-a-single-huge-source-file-to-gain-performanc%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password