How to pipe an output of a list as the input of grep in Linux?
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
I have a table_1
A D G
B E H
C F I
I print the column 1 of the above table using awk 'print $1' table_1
to get list_1
A
B
C
I want to use list_1
above to grep -f
from table_2
A n m
B m n
C n m
D m n
E n m
to get table_3
A n m
B m n
C n m
but I want to do it using a one-liner, not needing to save an intermediate file of list_1
.
How can I structure the command?
text-processing awk grep
add a comment |Â
up vote
0
down vote
favorite
I have a table_1
A D G
B E H
C F I
I print the column 1 of the above table using awk 'print $1' table_1
to get list_1
A
B
C
I want to use list_1
above to grep -f
from table_2
A n m
B m n
C n m
D m n
E n m
to get table_3
A n m
B m n
C n m
but I want to do it using a one-liner, not needing to save an intermediate file of list_1
.
How can I structure the command?
text-processing awk grep
2
grep -f <(awk 'print $1' table_1) table_2
?
â Cyrus
Feb 1 at 6:31
Is it save to assume thatA
or any search term can't be part of any other column oftable_2
? Maybe you should add a^
before and a whitespace after your search terms.
â Philippos
Feb 1 at 6:46
add a comment |Â
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have a table_1
A D G
B E H
C F I
I print the column 1 of the above table using awk 'print $1' table_1
to get list_1
A
B
C
I want to use list_1
above to grep -f
from table_2
A n m
B m n
C n m
D m n
E n m
to get table_3
A n m
B m n
C n m
but I want to do it using a one-liner, not needing to save an intermediate file of list_1
.
How can I structure the command?
text-processing awk grep
I have a table_1
A D G
B E H
C F I
I print the column 1 of the above table using awk 'print $1' table_1
to get list_1
A
B
C
I want to use list_1
above to grep -f
from table_2
A n m
B m n
C n m
D m n
E n m
to get table_3
A n m
B m n
C n m
but I want to do it using a one-liner, not needing to save an intermediate file of list_1
.
How can I structure the command?
text-processing awk grep
edited Feb 1 at 9:13
ñÃÂsýù÷
15k82462
15k82462
asked Feb 1 at 5:45
Johnny Tam
1208
1208
2
grep -f <(awk 'print $1' table_1) table_2
?
â Cyrus
Feb 1 at 6:31
Is it save to assume thatA
or any search term can't be part of any other column oftable_2
? Maybe you should add a^
before and a whitespace after your search terms.
â Philippos
Feb 1 at 6:46
add a comment |Â
2
grep -f <(awk 'print $1' table_1) table_2
?
â Cyrus
Feb 1 at 6:31
Is it save to assume thatA
or any search term can't be part of any other column oftable_2
? Maybe you should add a^
before and a whitespace after your search terms.
â Philippos
Feb 1 at 6:46
2
2
grep -f <(awk 'print $1' table_1) table_2
?â Cyrus
Feb 1 at 6:31
grep -f <(awk 'print $1' table_1) table_2
?â Cyrus
Feb 1 at 6:31
Is it save to assume that
A
or any search term can't be part of any other column of table_2
? Maybe you should add a ^
before and a whitespace after your search terms.â Philippos
Feb 1 at 6:46
Is it save to assume that
A
or any search term can't be part of any other column of table_2
? Maybe you should add a ^
before and a whitespace after your search terms.â Philippos
Feb 1 at 6:46
add a comment |Â
4 Answers
4
active
oldest
votes
up vote
3
down vote
A trivial (but not bash-specific) variation on CyrusâÂÂs comment:
awk 'print $1' table_1 | grep -f - table_2
which uses the widespread convention that a filename of -
means âÂÂread from the standard inputâÂÂ.
but this is not so robust... grep will search any where in the line and would even match part of column...
â Sundeep
Feb 1 at 8:22
1
Where does the question specify which behavior is desired?âÂÂThe question says that the OPâÂÂs current workflow is to create the temporarylist_1
file and use it (implicitly, ingrepâ¯-fâ¯list_1â¯table_2
), and asks how to streamline that operation into a âÂÂone-linerâ with no temporary file.â It doesnâÂÂt say that the OP wants to use the values from columnâ¯1 oftable_1
to match against the values from columnâ¯1 oftable_2
.
â G-Man
Feb 1 at 9:37
add a comment |Â
up vote
2
down vote
Use awk
:
awk 'NR==FNRa[$1];next ($1 in a)' table_1 table_2
add a comment |Â
up vote
1
down vote
HereâÂÂs another way to use join
.ÃÂ
If the files are sorted, and table_2
really has only three columns,
you can use
join -o "2.1 2.2 2.3" table_1 table_2 > table_3
As Kusalananda says,join
is a program that exists specifically to combine two files
by matching values from one column of the first file
against values from one column of the second file.ÃÂ
By default, if uses the first column of each file (you can override this).ÃÂ
By default, it combines the matching lines, like this:
$ join table_1 table_2
A D G n m
B E H m n
C F I n m
The -o "2.1 2.2 2.3"
says âÂÂoutput the first field of the second file,
and then the second field of the second file,
and then the first third of the second file
(but nothing from the first file1)âÂÂ.ÃÂ
Unfortunately, there donâÂÂt seem to be any shortcuts, shorthands,
accelerators, or wildcards here;
thereâÂÂs no way to say âÂÂoutput the entire line from the second fileâÂÂ.ÃÂ
If the second file has many fields,
then the -o
format will have to be very long.
This differs from the grep
-based solutions (yours and mine)
in that it specifically matches the values from columnâ¯1 of table_1
against the values from columnâ¯1 of table_2
,
and will not give you lines from table_2
that have an A
in columnâ¯2 or columnâ¯3, or an AZ
in columnâ¯1.àjoin
, like grep
, recognizes a -i
(--ignore-case
) option,
so, if the first line of table_2
is aÃÂ nÃÂ m
(with a lower-case a
),
you will get that line in your table_3
output
if (and only if) you specify -i
.
________
1 Except for the first field of the first file âÂÂ
which is the same as first field of the second file.
add a comment |Â
up vote
0
down vote
Assuming that both tables are sorted on the first column, and using a shell like bash
or ksh93
that understands process substitution with <(...)
:
$ join <( awk ' print $1 ' table_1 ) table_2
A n m
B m n
C n m
If the tables are not sorted, then we need to sort them:
$ join <( awk ' print $1 ' table_1 | sort ) <( sort table_2 )
A n m
B m n
C n m
The join
utility performs a relational INNER JOIN operation on the first column (by default) of the two files that you give it.
Redirect the output to a new file to create table_3
:
$ join <( awk ' print $1 ' table_1 | sort ) <( sort table_2 ) >table_3
add a comment |Â
4 Answers
4
active
oldest
votes
4 Answers
4
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
A trivial (but not bash-specific) variation on CyrusâÂÂs comment:
awk 'print $1' table_1 | grep -f - table_2
which uses the widespread convention that a filename of -
means âÂÂread from the standard inputâÂÂ.
but this is not so robust... grep will search any where in the line and would even match part of column...
â Sundeep
Feb 1 at 8:22
1
Where does the question specify which behavior is desired?âÂÂThe question says that the OPâÂÂs current workflow is to create the temporarylist_1
file and use it (implicitly, ingrepâ¯-fâ¯list_1â¯table_2
), and asks how to streamline that operation into a âÂÂone-linerâ with no temporary file.â It doesnâÂÂt say that the OP wants to use the values from columnâ¯1 oftable_1
to match against the values from columnâ¯1 oftable_2
.
â G-Man
Feb 1 at 9:37
add a comment |Â
up vote
3
down vote
A trivial (but not bash-specific) variation on CyrusâÂÂs comment:
awk 'print $1' table_1 | grep -f - table_2
which uses the widespread convention that a filename of -
means âÂÂread from the standard inputâÂÂ.
but this is not so robust... grep will search any where in the line and would even match part of column...
â Sundeep
Feb 1 at 8:22
1
Where does the question specify which behavior is desired?âÂÂThe question says that the OPâÂÂs current workflow is to create the temporarylist_1
file and use it (implicitly, ingrepâ¯-fâ¯list_1â¯table_2
), and asks how to streamline that operation into a âÂÂone-linerâ with no temporary file.â It doesnâÂÂt say that the OP wants to use the values from columnâ¯1 oftable_1
to match against the values from columnâ¯1 oftable_2
.
â G-Man
Feb 1 at 9:37
add a comment |Â
up vote
3
down vote
up vote
3
down vote
A trivial (but not bash-specific) variation on CyrusâÂÂs comment:
awk 'print $1' table_1 | grep -f - table_2
which uses the widespread convention that a filename of -
means âÂÂread from the standard inputâÂÂ.
A trivial (but not bash-specific) variation on CyrusâÂÂs comment:
awk 'print $1' table_1 | grep -f - table_2
which uses the widespread convention that a filename of -
means âÂÂread from the standard inputâÂÂ.
answered Feb 1 at 7:34
G-Man
11.5k82657
11.5k82657
but this is not so robust... grep will search any where in the line and would even match part of column...
â Sundeep
Feb 1 at 8:22
1
Where does the question specify which behavior is desired?âÂÂThe question says that the OPâÂÂs current workflow is to create the temporarylist_1
file and use it (implicitly, ingrepâ¯-fâ¯list_1â¯table_2
), and asks how to streamline that operation into a âÂÂone-linerâ with no temporary file.â It doesnâÂÂt say that the OP wants to use the values from columnâ¯1 oftable_1
to match against the values from columnâ¯1 oftable_2
.
â G-Man
Feb 1 at 9:37
add a comment |Â
but this is not so robust... grep will search any where in the line and would even match part of column...
â Sundeep
Feb 1 at 8:22
1
Where does the question specify which behavior is desired?âÂÂThe question says that the OPâÂÂs current workflow is to create the temporarylist_1
file and use it (implicitly, ingrepâ¯-fâ¯list_1â¯table_2
), and asks how to streamline that operation into a âÂÂone-linerâ with no temporary file.â It doesnâÂÂt say that the OP wants to use the values from columnâ¯1 oftable_1
to match against the values from columnâ¯1 oftable_2
.
â G-Man
Feb 1 at 9:37
but this is not so robust... grep will search any where in the line and would even match part of column...
â Sundeep
Feb 1 at 8:22
but this is not so robust... grep will search any where in the line and would even match part of column...
â Sundeep
Feb 1 at 8:22
1
1
Where does the question specify which behavior is desired?âÂÂThe question says that the OPâÂÂs current workflow is to create the temporary
list_1
file and use it (implicitly, in grepâ¯-fâ¯list_1â¯table_2
), and asks how to streamline that operation into a âÂÂone-linerâ with no temporary file.â It doesnâÂÂt say that the OP wants to use the values from columnâ¯1 of table_1
to match against the values from columnâ¯1 of table_2
.â G-Man
Feb 1 at 9:37
Where does the question specify which behavior is desired?âÂÂThe question says that the OPâÂÂs current workflow is to create the temporary
list_1
file and use it (implicitly, in grepâ¯-fâ¯list_1â¯table_2
), and asks how to streamline that operation into a âÂÂone-linerâ with no temporary file.â It doesnâÂÂt say that the OP wants to use the values from columnâ¯1 of table_1
to match against the values from columnâ¯1 of table_2
.â G-Man
Feb 1 at 9:37
add a comment |Â
up vote
2
down vote
Use awk
:
awk 'NR==FNRa[$1];next ($1 in a)' table_1 table_2
add a comment |Â
up vote
2
down vote
Use awk
:
awk 'NR==FNRa[$1];next ($1 in a)' table_1 table_2
add a comment |Â
up vote
2
down vote
up vote
2
down vote
Use awk
:
awk 'NR==FNRa[$1];next ($1 in a)' table_1 table_2
Use awk
:
awk 'NR==FNRa[$1];next ($1 in a)' table_1 table_2
edited Feb 1 at 8:50
answered Feb 1 at 7:00
ñÃÂsýù÷
15k82462
15k82462
add a comment |Â
add a comment |Â
up vote
1
down vote
HereâÂÂs another way to use join
.ÃÂ
If the files are sorted, and table_2
really has only three columns,
you can use
join -o "2.1 2.2 2.3" table_1 table_2 > table_3
As Kusalananda says,join
is a program that exists specifically to combine two files
by matching values from one column of the first file
against values from one column of the second file.ÃÂ
By default, if uses the first column of each file (you can override this).ÃÂ
By default, it combines the matching lines, like this:
$ join table_1 table_2
A D G n m
B E H m n
C F I n m
The -o "2.1 2.2 2.3"
says âÂÂoutput the first field of the second file,
and then the second field of the second file,
and then the first third of the second file
(but nothing from the first file1)âÂÂ.ÃÂ
Unfortunately, there donâÂÂt seem to be any shortcuts, shorthands,
accelerators, or wildcards here;
thereâÂÂs no way to say âÂÂoutput the entire line from the second fileâÂÂ.ÃÂ
If the second file has many fields,
then the -o
format will have to be very long.
This differs from the grep
-based solutions (yours and mine)
in that it specifically matches the values from columnâ¯1 of table_1
against the values from columnâ¯1 of table_2
,
and will not give you lines from table_2
that have an A
in columnâ¯2 or columnâ¯3, or an AZ
in columnâ¯1.àjoin
, like grep
, recognizes a -i
(--ignore-case
) option,
so, if the first line of table_2
is aÃÂ nÃÂ m
(with a lower-case a
),
you will get that line in your table_3
output
if (and only if) you specify -i
.
________
1 Except for the first field of the first file âÂÂ
which is the same as first field of the second file.
add a comment |Â
up vote
1
down vote
HereâÂÂs another way to use join
.ÃÂ
If the files are sorted, and table_2
really has only three columns,
you can use
join -o "2.1 2.2 2.3" table_1 table_2 > table_3
As Kusalananda says,join
is a program that exists specifically to combine two files
by matching values from one column of the first file
against values from one column of the second file.ÃÂ
By default, if uses the first column of each file (you can override this).ÃÂ
By default, it combines the matching lines, like this:
$ join table_1 table_2
A D G n m
B E H m n
C F I n m
The -o "2.1 2.2 2.3"
says âÂÂoutput the first field of the second file,
and then the second field of the second file,
and then the first third of the second file
(but nothing from the first file1)âÂÂ.ÃÂ
Unfortunately, there donâÂÂt seem to be any shortcuts, shorthands,
accelerators, or wildcards here;
thereâÂÂs no way to say âÂÂoutput the entire line from the second fileâÂÂ.ÃÂ
If the second file has many fields,
then the -o
format will have to be very long.
This differs from the grep
-based solutions (yours and mine)
in that it specifically matches the values from columnâ¯1 of table_1
against the values from columnâ¯1 of table_2
,
and will not give you lines from table_2
that have an A
in columnâ¯2 or columnâ¯3, or an AZ
in columnâ¯1.àjoin
, like grep
, recognizes a -i
(--ignore-case
) option,
so, if the first line of table_2
is aÃÂ nÃÂ m
(with a lower-case a
),
you will get that line in your table_3
output
if (and only if) you specify -i
.
________
1 Except for the first field of the first file âÂÂ
which is the same as first field of the second file.
add a comment |Â
up vote
1
down vote
up vote
1
down vote
HereâÂÂs another way to use join
.ÃÂ
If the files are sorted, and table_2
really has only three columns,
you can use
join -o "2.1 2.2 2.3" table_1 table_2 > table_3
As Kusalananda says,join
is a program that exists specifically to combine two files
by matching values from one column of the first file
against values from one column of the second file.ÃÂ
By default, if uses the first column of each file (you can override this).ÃÂ
By default, it combines the matching lines, like this:
$ join table_1 table_2
A D G n m
B E H m n
C F I n m
The -o "2.1 2.2 2.3"
says âÂÂoutput the first field of the second file,
and then the second field of the second file,
and then the first third of the second file
(but nothing from the first file1)âÂÂ.ÃÂ
Unfortunately, there donâÂÂt seem to be any shortcuts, shorthands,
accelerators, or wildcards here;
thereâÂÂs no way to say âÂÂoutput the entire line from the second fileâÂÂ.ÃÂ
If the second file has many fields,
then the -o
format will have to be very long.
This differs from the grep
-based solutions (yours and mine)
in that it specifically matches the values from columnâ¯1 of table_1
against the values from columnâ¯1 of table_2
,
and will not give you lines from table_2
that have an A
in columnâ¯2 or columnâ¯3, or an AZ
in columnâ¯1.àjoin
, like grep
, recognizes a -i
(--ignore-case
) option,
so, if the first line of table_2
is aÃÂ nÃÂ m
(with a lower-case a
),
you will get that line in your table_3
output
if (and only if) you specify -i
.
________
1 Except for the first field of the first file âÂÂ
which is the same as first field of the second file.
HereâÂÂs another way to use join
.ÃÂ
If the files are sorted, and table_2
really has only three columns,
you can use
join -o "2.1 2.2 2.3" table_1 table_2 > table_3
As Kusalananda says,join
is a program that exists specifically to combine two files
by matching values from one column of the first file
against values from one column of the second file.ÃÂ
By default, if uses the first column of each file (you can override this).ÃÂ
By default, it combines the matching lines, like this:
$ join table_1 table_2
A D G n m
B E H m n
C F I n m
The -o "2.1 2.2 2.3"
says âÂÂoutput the first field of the second file,
and then the second field of the second file,
and then the first third of the second file
(but nothing from the first file1)âÂÂ.ÃÂ
Unfortunately, there donâÂÂt seem to be any shortcuts, shorthands,
accelerators, or wildcards here;
thereâÂÂs no way to say âÂÂoutput the entire line from the second fileâÂÂ.ÃÂ
If the second file has many fields,
then the -o
format will have to be very long.
This differs from the grep
-based solutions (yours and mine)
in that it specifically matches the values from columnâ¯1 of table_1
against the values from columnâ¯1 of table_2
,
and will not give you lines from table_2
that have an A
in columnâ¯2 or columnâ¯3, or an AZ
in columnâ¯1.àjoin
, like grep
, recognizes a -i
(--ignore-case
) option,
so, if the first line of table_2
is aÃÂ nÃÂ m
(with a lower-case a
),
you will get that line in your table_3
output
if (and only if) you specify -i
.
________
1 Except for the first field of the first file âÂÂ
which is the same as first field of the second file.
edited Feb 1 at 17:28
answered Feb 1 at 9:37
G-Man
11.5k82657
11.5k82657
add a comment |Â
add a comment |Â
up vote
0
down vote
Assuming that both tables are sorted on the first column, and using a shell like bash
or ksh93
that understands process substitution with <(...)
:
$ join <( awk ' print $1 ' table_1 ) table_2
A n m
B m n
C n m
If the tables are not sorted, then we need to sort them:
$ join <( awk ' print $1 ' table_1 | sort ) <( sort table_2 )
A n m
B m n
C n m
The join
utility performs a relational INNER JOIN operation on the first column (by default) of the two files that you give it.
Redirect the output to a new file to create table_3
:
$ join <( awk ' print $1 ' table_1 | sort ) <( sort table_2 ) >table_3
add a comment |Â
up vote
0
down vote
Assuming that both tables are sorted on the first column, and using a shell like bash
or ksh93
that understands process substitution with <(...)
:
$ join <( awk ' print $1 ' table_1 ) table_2
A n m
B m n
C n m
If the tables are not sorted, then we need to sort them:
$ join <( awk ' print $1 ' table_1 | sort ) <( sort table_2 )
A n m
B m n
C n m
The join
utility performs a relational INNER JOIN operation on the first column (by default) of the two files that you give it.
Redirect the output to a new file to create table_3
:
$ join <( awk ' print $1 ' table_1 | sort ) <( sort table_2 ) >table_3
add a comment |Â
up vote
0
down vote
up vote
0
down vote
Assuming that both tables are sorted on the first column, and using a shell like bash
or ksh93
that understands process substitution with <(...)
:
$ join <( awk ' print $1 ' table_1 ) table_2
A n m
B m n
C n m
If the tables are not sorted, then we need to sort them:
$ join <( awk ' print $1 ' table_1 | sort ) <( sort table_2 )
A n m
B m n
C n m
The join
utility performs a relational INNER JOIN operation on the first column (by default) of the two files that you give it.
Redirect the output to a new file to create table_3
:
$ join <( awk ' print $1 ' table_1 | sort ) <( sort table_2 ) >table_3
Assuming that both tables are sorted on the first column, and using a shell like bash
or ksh93
that understands process substitution with <(...)
:
$ join <( awk ' print $1 ' table_1 ) table_2
A n m
B m n
C n m
If the tables are not sorted, then we need to sort them:
$ join <( awk ' print $1 ' table_1 | sort ) <( sort table_2 )
A n m
B m n
C n m
The join
utility performs a relational INNER JOIN operation on the first column (by default) of the two files that you give it.
Redirect the output to a new file to create table_3
:
$ join <( awk ' print $1 ' table_1 | sort ) <( sort table_2 ) >table_3
answered Feb 1 at 9:02
Kusalananda
103k13202318
103k13202318
add a comment |Â
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f421143%2fhow-to-pipe-an-output-of-a-list-as-the-input-of-grep-in-linux%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
2
grep -f <(awk 'print $1' table_1) table_2
?â Cyrus
Feb 1 at 6:31
Is it save to assume that
A
or any search term can't be part of any other column oftable_2
? Maybe you should add a^
before and a whitespace after your search terms.â Philippos
Feb 1 at 6:46