Remove lines that duplicate each other
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
I have a bash command that lists the directories in HDFS beginning with '/random'.
It goes as follows:
hdfs dfs -ls -R /random | grep '^d' | awk 'print $NF'
It generates the following output:
/random/custom
/random/custom/hive
/random/custom/hive/jars
/random/app
/random/app/nifi
/random/app/nifi/egispub
/random/app/nifi/empgis
/random/app/nifi/ods
/random/app/nifi/ptcsm
/random/app/nifi/tmds
/random/app/nifi/ucmdb
/random/app/oozie
/random/app/oozie/efdm
/random/app/oozie/efdm/ensco
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min/lib
/random/app/oozie/efdm/mermec
/random/app/oozie/efdm/mermec/cleansed_efdm_mermec_5min
/random/app/oozie/efdm/mermec/standardized_efdm_mermec_5min
/random/app/oozie/gis
/random/app/oozie/gis/opk
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/lib
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/schemas
I would like to modify my command so that only the full directory path is listed, as follows:
/random/custom/hive/jars
/random/app/nifi/egispub
/random/app/nifi/empgis
/random/app/nifi/ods
/random/app/nifi/ptcsm
/random/app/nifi/tmds
/random/app/nifi/ucmdb
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min/lib
/random/app/oozie/efdm/mermec/cleansed_efdm_mermec_5min
/random/app/oozie/efdm/mermec/standardized_efdm_mermec_5min
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/lib
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/schemas
How would you do it?
text-processing
asked 9 mins ago
user316697
11
New contributor
user316697 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |Â
up vote
0
down vote
favorite
I have a bash command that lists the directories in HDFS beginning with '/random'.
It goes as follows:
hdfs dfs -ls -R /random | grep '^d' | awk 'print $NF'
It generates the following output:
/random/custom
/random/custom/hive
/random/custom/hive/jars
/random/app
/random/app/nifi
/random/app/nifi/egispub
/random/app/nifi/empgis
/random/app/nifi/ods
/random/app/nifi/ptcsm
/random/app/nifi/tmds
/random/app/nifi/ucmdb
/random/app/oozie
/random/app/oozie/efdm
/random/app/oozie/efdm/ensco
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min/lib
/random/app/oozie/efdm/mermec
/random/app/oozie/efdm/mermec/cleansed_efdm_mermec_5min
/random/app/oozie/efdm/mermec/standardized_efdm_mermec_5min
/random/app/oozie/gis
/random/app/oozie/gis/opk
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/lib
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/schemas
I would like to modify my command so that only the full directory path is listed, as follows:
/random/custom/hive/jars
/random/app/nifi/egispub
/random/app/nifi/empgis
/random/app/nifi/ods
/random/app/nifi/ptcsm
/random/app/nifi/tmds
/random/app/nifi/ucmdb
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min/lib
/random/app/oozie/efdm/mermec/cleansed_efdm_mermec_5min
/random/app/oozie/efdm/mermec/standardized_efdm_mermec_5min
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/lib
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/schemas
How would you do it?
text-processing
asked 9 mins ago
user316697
11
New contributor
user316697 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
"I would like to modify my command so that only the full directory path is listed." What do you mean by this? Do you mean that only/random/custom/hive/jarsshould be output (assumingjarsis a directory)? Do you mean that only the path to the/random/customdirectory should be listed? The full directory path to which location?
– code_dredd
3 mins ago
Also, did you try anything? If so, what was it?
– code_dredd
2 mins ago
add a comment |Â
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have a bash command that lists the directories in HDFS beginning with '/random'.
It goes as follows:
hdfs dfs -ls -R /random | grep '^d' | awk 'print $NF'
It generates the following output:
/random/custom
/random/custom/hive
/random/custom/hive/jars
/random/app
/random/app/nifi
/random/app/nifi/egispub
/random/app/nifi/empgis
/random/app/nifi/ods
/random/app/nifi/ptcsm
/random/app/nifi/tmds
/random/app/nifi/ucmdb
/random/app/oozie
/random/app/oozie/efdm
/random/app/oozie/efdm/ensco
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min/lib
/random/app/oozie/efdm/mermec
/random/app/oozie/efdm/mermec/cleansed_efdm_mermec_5min
/random/app/oozie/efdm/mermec/standardized_efdm_mermec_5min
/random/app/oozie/gis
/random/app/oozie/gis/opk
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/lib
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/schemas
I would like to modify my command so that only the full directory path is listed, as follows:
/random/custom/hive/jars
/random/app/nifi/egispub
/random/app/nifi/empgis
/random/app/nifi/ods
/random/app/nifi/ptcsm
/random/app/nifi/tmds
/random/app/nifi/ucmdb
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min/lib
/random/app/oozie/efdm/mermec/cleansed_efdm_mermec_5min
/random/app/oozie/efdm/mermec/standardized_efdm_mermec_5min
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/lib
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/schemas
How would you do it?
text-processing
asked 9 mins ago
user316697
11
New contributor
user316697 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I have a bash command that lists the directories in HDFS beginning with '/random'.
It goes as follows:
hdfs dfs -ls -R /random | grep '^d' | awk 'print $NF'
It generates the following output:
/random/custom
/random/custom/hive
/random/custom/hive/jars
/random/app
/random/app/nifi
/random/app/nifi/egispub
/random/app/nifi/empgis
/random/app/nifi/ods
/random/app/nifi/ptcsm
/random/app/nifi/tmds
/random/app/nifi/ucmdb
/random/app/oozie
/random/app/oozie/efdm
/random/app/oozie/efdm/ensco
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min/lib
/random/app/oozie/efdm/mermec
/random/app/oozie/efdm/mermec/cleansed_efdm_mermec_5min
/random/app/oozie/efdm/mermec/standardized_efdm_mermec_5min
/random/app/oozie/gis
/random/app/oozie/gis/opk
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/lib
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/schemas
I would like to modify my command so that only the full directory path is listed, as follows:
/random/custom/hive/jars
/random/app/nifi/egispub
/random/app/nifi/empgis
/random/app/nifi/ods
/random/app/nifi/ptcsm
/random/app/nifi/tmds
/random/app/nifi/ucmdb
/random/app/oozie/efdm/ensco/standardized_efdm_ensco_parser_5min/lib
/random/app/oozie/efdm/mermec/cleansed_efdm_mermec_5min
/random/app/oozie/efdm/mermec/standardized_efdm_mermec_5min
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/lib
/random/app/oozie/gis/opk/standardized_gis_opk_parser_10min/schemas
How would you do it?
text-processing
text-processing
asked 9 mins ago
user316697
11
New contributor
user316697 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 9 mins ago
user316697
11
New contributor
user316697 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited 3 mins ago
asked 9 mins ago
user316697
11
New contributor
user316697 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 9 mins ago
user316697
11
asked 9 mins ago
user316697
11
11
New contributor
user316697 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
user316697 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
user316697 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
"I would like to modify my command so that only the full directory path is listed." What do you mean by this? Do you mean that only/random/custom/hive/jarsshould be output (assumingjarsis a directory)? Do you mean that only the path to the/random/customdirectory should be listed? The full directory path to which location?
– code_dredd
3 mins ago
Also, did you try anything? If so, what was it?
– code_dredd
2 mins ago
add a comment |Â
"I would like to modify my command so that only the full directory path is listed." What do you mean by this? Do you mean that only/random/custom/hive/jarsshould be output (assumingjarsis a directory)? Do you mean that only the path to the/random/customdirectory should be listed? The full directory path to which location?
– code_dredd
3 mins ago
Also, did you try anything? If so, what was it?
– code_dredd
2 mins ago
"I would like to modify my command so that only the full directory path is listed." What do you mean by this? Do you mean that only
/random/custom/hive/jars should be output (assuming jars is a directory)? Do you mean that only the path to the /random/custom directory should be listed? The full directory path to which location?– code_dredd
3 mins ago
"I would like to modify my command so that only the full directory path is listed." What do you mean by this? Do you mean that only
/random/custom/hive/jars should be output (assuming jars is a directory)? Do you mean that only the path to the /random/custom directory should be listed? The full directory path to which location?– code_dredd
3 mins ago
Also, did you try anything? If so, what was it?
– code_dredd
2 mins ago
Also, did you try anything? If so, what was it?
– code_dredd
2 mins ago
add a comment |Â
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
user316697 is a new contributor. Be nice, and check out our Code of Conduct.
Â
draft saved
draft discarded
user316697 is a new contributor. Be nice, and check out our Code of Conduct.
user316697 is a new contributor. Be nice, and check out our Code of Conduct.
user316697 is a new contributor. Be nice, and check out our Code of Conduct.
Â
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f476577%2fremove-lines-that-duplicate-each-other%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
"I would like to modify my command so that only the full directory path is listed." What do you mean by this? Do you mean that only
/random/custom/hive/jarsshould be output (assumingjarsis a directory)? Do you mean that only the path to the/random/customdirectory should be listed? The full directory path to which location?– code_dredd
3 mins ago
Also, did you try anything? If so, what was it?
– code_dredd
2 mins ago