text removal, grouped entries based on a tag [on hold]

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
-4
down vote

favorite












Edited again for clarity: sorry for the confusion.



I have the following example log. The events I want to keep do not easily fit a pattern, it is easier to identify what to remove.



Independent processes with names like "fred", "bob", "bill" run and create log entries. The "name" can be considered as a tag that identifies the start of the logging group for that process at that time.



Each process has an automatic process ID associated with it e.g. CRON[3383] and in this example all 3833s are associated with the first "fred" event.
This "name" fred, (or bob, bill), are the only thing I know for sure in advance, I do not know the process IDs. I also do not know the number of rows generated in the associated grouping. So,



Start at the top of the file. Find:
if a line in the log has "fred", determine the Process ID e.g. 3833 (which is between the only square brackets before the reference to "fred" on the identified "fred" line); then group all the same ID lines e.g. all the 3833 including the "fred" line; then delete that group.



Repeat:
Find the next "fred", determine ID e.g. 4099, group all these, delete.
Repeat



If a line has "fred" but no process ID i.e. CRON[num], do not delete.



input:



Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session closed for user root
Jan 2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root


Desired output:



Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root


++Edited with possible solution++:



grep -e "[.*fred" input.txt |sed 's/.*[([^]]*)].*/1/g' |grep -vf /dev/stdin input.txt


Is there a cleaner way?










share|improve this question









New contributor




BillDoor is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











put on hold as unclear what you're asking by Goro, jimmij, Kiwy, Jeff Schaller, Thomas 2 days ago


Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.










  • 1




    Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
    – Goro
    Oct 4 at 8:43










  • sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
    – BillDoor
    Oct 4 at 10:15







  • 1




    do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
    – Kiwy
    Oct 4 at 11:19











  • Two of your three desired output lines do NOT contain uid=0. So - how is that output derived? Except, e.g. < file grep -v "CRON.*]"
    – RudiC
    Oct 4 at 11:38











  • i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
    – BillDoor
    Oct 4 at 16:07














up vote
-4
down vote

favorite












Edited again for clarity: sorry for the confusion.



I have the following example log. The events I want to keep do not easily fit a pattern, it is easier to identify what to remove.



Independent processes with names like "fred", "bob", "bill" run and create log entries. The "name" can be considered as a tag that identifies the start of the logging group for that process at that time.



Each process has an automatic process ID associated with it e.g. CRON[3383] and in this example all 3833s are associated with the first "fred" event.
This "name" fred, (or bob, bill), are the only thing I know for sure in advance, I do not know the process IDs. I also do not know the number of rows generated in the associated grouping. So,



Start at the top of the file. Find:
if a line in the log has "fred", determine the Process ID e.g. 3833 (which is between the only square brackets before the reference to "fred" on the identified "fred" line); then group all the same ID lines e.g. all the 3833 including the "fred" line; then delete that group.



Repeat:
Find the next "fred", determine ID e.g. 4099, group all these, delete.
Repeat



If a line has "fred" but no process ID i.e. CRON[num], do not delete.



input:



Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session closed for user root
Jan 2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root


Desired output:



Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root


++Edited with possible solution++:



grep -e "[.*fred" input.txt |sed 's/.*[([^]]*)].*/1/g' |grep -vf /dev/stdin input.txt


Is there a cleaner way?










share|improve this question









New contributor




BillDoor is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











put on hold as unclear what you're asking by Goro, jimmij, Kiwy, Jeff Schaller, Thomas 2 days ago


Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.










  • 1




    Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
    – Goro
    Oct 4 at 8:43










  • sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
    – BillDoor
    Oct 4 at 10:15







  • 1




    do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
    – Kiwy
    Oct 4 at 11:19











  • Two of your three desired output lines do NOT contain uid=0. So - how is that output derived? Except, e.g. < file grep -v "CRON.*]"
    – RudiC
    Oct 4 at 11:38











  • i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
    – BillDoor
    Oct 4 at 16:07












up vote
-4
down vote

favorite









up vote
-4
down vote

favorite











Edited again for clarity: sorry for the confusion.



I have the following example log. The events I want to keep do not easily fit a pattern, it is easier to identify what to remove.



Independent processes with names like "fred", "bob", "bill" run and create log entries. The "name" can be considered as a tag that identifies the start of the logging group for that process at that time.



Each process has an automatic process ID associated with it e.g. CRON[3383] and in this example all 3833s are associated with the first "fred" event.
This "name" fred, (or bob, bill), are the only thing I know for sure in advance, I do not know the process IDs. I also do not know the number of rows generated in the associated grouping. So,



Start at the top of the file. Find:
if a line in the log has "fred", determine the Process ID e.g. 3833 (which is between the only square brackets before the reference to "fred" on the identified "fred" line); then group all the same ID lines e.g. all the 3833 including the "fred" line; then delete that group.



Repeat:
Find the next "fred", determine ID e.g. 4099, group all these, delete.
Repeat



If a line has "fred" but no process ID i.e. CRON[num], do not delete.



input:



Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session closed for user root
Jan 2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root


Desired output:



Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root


++Edited with possible solution++:



grep -e "[.*fred" input.txt |sed 's/.*[([^]]*)].*/1/g' |grep -vf /dev/stdin input.txt


Is there a cleaner way?










share|improve this question









New contributor




BillDoor is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











Edited again for clarity: sorry for the confusion.



I have the following example log. The events I want to keep do not easily fit a pattern, it is easier to identify what to remove.



Independent processes with names like "fred", "bob", "bill" run and create log entries. The "name" can be considered as a tag that identifies the start of the logging group for that process at that time.



Each process has an automatic process ID associated with it e.g. CRON[3383] and in this example all 3833s are associated with the first "fred" event.
This "name" fred, (or bob, bill), are the only thing I know for sure in advance, I do not know the process IDs. I also do not know the number of rows generated in the associated grouping. So,



Start at the top of the file. Find:
if a line in the log has "fred", determine the Process ID e.g. 3833 (which is between the only square brackets before the reference to "fred" on the identified "fred" line); then group all the same ID lines e.g. all the 3833 including the "fred" line; then delete that group.



Repeat:
Find the next "fred", determine ID e.g. 4099, group all these, delete.
Repeat



If a line has "fred" but no process ID i.e. CRON[num], do not delete.



input:



Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session closed for user root
Jan 2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root


Desired output:



Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root


++Edited with possible solution++:



grep -e "[.*fred" input.txt |sed 's/.*[([^]]*)].*/1/g' |grep -vf /dev/stdin input.txt


Is there a cleaner way?







linux awk sed grep regular-expression






share|improve this question









New contributor




BillDoor is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question









New contributor




BillDoor is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited 2 days ago





















New contributor




BillDoor is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked Oct 4 at 8:34









BillDoor

11




11




New contributor




BillDoor is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





BillDoor is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






BillDoor is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




put on hold as unclear what you're asking by Goro, jimmij, Kiwy, Jeff Schaller, Thomas 2 days ago


Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.






put on hold as unclear what you're asking by Goro, jimmij, Kiwy, Jeff Schaller, Thomas 2 days ago


Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.









  • 1




    Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
    – Goro
    Oct 4 at 8:43










  • sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
    – BillDoor
    Oct 4 at 10:15







  • 1




    do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
    – Kiwy
    Oct 4 at 11:19











  • Two of your three desired output lines do NOT contain uid=0. So - how is that output derived? Except, e.g. < file grep -v "CRON.*]"
    – RudiC
    Oct 4 at 11:38











  • i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
    – BillDoor
    Oct 4 at 16:07












  • 1




    Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
    – Goro
    Oct 4 at 8:43










  • sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
    – BillDoor
    Oct 4 at 10:15







  • 1




    do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
    – Kiwy
    Oct 4 at 11:19











  • Two of your three desired output lines do NOT contain uid=0. So - how is that output derived? Except, e.g. < file grep -v "CRON.*]"
    – RudiC
    Oct 4 at 11:38











  • i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
    – BillDoor
    Oct 4 at 16:07







1




1




Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
– Goro
Oct 4 at 8:43




Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
– Goro
Oct 4 at 8:43












sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
– BillDoor
Oct 4 at 10:15





sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
– BillDoor
Oct 4 at 10:15





1




1




do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
– Kiwy
Oct 4 at 11:19





do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
– Kiwy
Oct 4 at 11:19













Two of your three desired output lines do NOT contain uid=0. So - how is that output derived? Except, e.g. < file grep -v "CRON.*]"
– RudiC
Oct 4 at 11:38





Two of your three desired output lines do NOT contain uid=0. So - how is that output derived? Except, e.g. < file grep -v "CRON.*]"
– RudiC
Oct 4 at 11:38













i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
– BillDoor
Oct 4 at 16:07




i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
– BillDoor
Oct 4 at 16:07










1 Answer
1






active

oldest

votes

















up vote
0
down vote













Try the below code,



grep "uid=0" /var/log/secure | grep -v "CRON.*]"





share|improve this answer




















  • This does not quite deliver the desired output.
    – RudiC
    Oct 4 at 11:36

















1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
0
down vote













Try the below code,



grep "uid=0" /var/log/secure | grep -v "CRON.*]"





share|improve this answer




















  • This does not quite deliver the desired output.
    – RudiC
    Oct 4 at 11:36














up vote
0
down vote













Try the below code,



grep "uid=0" /var/log/secure | grep -v "CRON.*]"





share|improve this answer




















  • This does not quite deliver the desired output.
    – RudiC
    Oct 4 at 11:36












up vote
0
down vote










up vote
0
down vote









Try the below code,



grep "uid=0" /var/log/secure | grep -v "CRON.*]"





share|improve this answer












Try the below code,



grep "uid=0" /var/log/secure | grep -v "CRON.*]"






share|improve this answer












share|improve this answer



share|improve this answer










answered Oct 4 at 10:03









EBIN GLADSON

3316




3316











  • This does not quite deliver the desired output.
    – RudiC
    Oct 4 at 11:36
















  • This does not quite deliver the desired output.
    – RudiC
    Oct 4 at 11:36















This does not quite deliver the desired output.
– RudiC
Oct 4 at 11:36




This does not quite deliver the desired output.
– RudiC
Oct 4 at 11:36


Popular posts from this blog

How to check contact read email or not when send email to Individual?

Bahrain

Postfix configuration issue with fips on centos 7; mailgun relay