text removal, grouped entries based on a tag [on hold]
Clash Royale CLAN TAG#URR8PPP
up vote
-4
down vote
favorite
Edited again for clarity: sorry for the confusion.
I have the following example log. The events I want to keep do not easily fit a pattern, it is easier to identify what to remove.
Independent processes with names like "fred", "bob", "bill" run and create log entries. The "name" can be considered as a tag that identifies the start of the logging group for that process at that time.
Each process has an automatic process ID associated with it e.g. CRON[3383] and in this example all 3833s are associated with the first "fred" event.
This "name" fred, (or bob, bill), are the only thing I know for sure in advance, I do not know the process IDs. I also do not know the number of rows generated in the associated grouping. So,
Start at the top of the file. Find:
if a line in the log has "fred", determine the Process ID e.g. 3833 (which is between the only square brackets before the reference to "fred" on the identified "fred" line); then group all the same ID lines e.g. all the 3833 including the "fred" line; then delete that group.
Repeat:
Find the next "fred", determine ID e.g. 4099, group all these, delete.
Repeat
If a line has "fred" but no process ID i.e. CRON[num], do not delete.
input:
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session closed for user root
Jan 2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root
Desired output:
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root
++Edited with possible solution++:
grep -e "[.*fred" input.txt |sed 's/.*[([^]]*)].*/1/g' |grep -vf /dev/stdin input.txt
Is there a cleaner way?
linux awk sed grep regular-expression
New contributor
put on hold as unclear what you're asking by Goro, jimmij, Kiwy, Jeff Schaller, Thomas 2 days ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, itâÂÂs hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
add a comment |Â
up vote
-4
down vote
favorite
Edited again for clarity: sorry for the confusion.
I have the following example log. The events I want to keep do not easily fit a pattern, it is easier to identify what to remove.
Independent processes with names like "fred", "bob", "bill" run and create log entries. The "name" can be considered as a tag that identifies the start of the logging group for that process at that time.
Each process has an automatic process ID associated with it e.g. CRON[3383] and in this example all 3833s are associated with the first "fred" event.
This "name" fred, (or bob, bill), are the only thing I know for sure in advance, I do not know the process IDs. I also do not know the number of rows generated in the associated grouping. So,
Start at the top of the file. Find:
if a line in the log has "fred", determine the Process ID e.g. 3833 (which is between the only square brackets before the reference to "fred" on the identified "fred" line); then group all the same ID lines e.g. all the 3833 including the "fred" line; then delete that group.
Repeat:
Find the next "fred", determine ID e.g. 4099, group all these, delete.
Repeat
If a line has "fred" but no process ID i.e. CRON[num], do not delete.
input:
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session closed for user root
Jan 2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root
Desired output:
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root
++Edited with possible solution++:
grep -e "[.*fred" input.txt |sed 's/.*[([^]]*)].*/1/g' |grep -vf /dev/stdin input.txt
Is there a cleaner way?
linux awk sed grep regular-expression
New contributor
put on hold as unclear what you're asking by Goro, jimmij, Kiwy, Jeff Schaller, Thomas 2 days ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, itâÂÂs hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
1
Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
â Goro
Oct 4 at 8:43
sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
â BillDoor
Oct 4 at 10:15
1
do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
â Kiwy
Oct 4 at 11:19
Two of your three desired output lines do NOT containuid=0
. So - how is that output derived? Except, e.g.< file grep -v "CRON.*]"
â RudiC
Oct 4 at 11:38
i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
â BillDoor
Oct 4 at 16:07
add a comment |Â
up vote
-4
down vote
favorite
up vote
-4
down vote
favorite
Edited again for clarity: sorry for the confusion.
I have the following example log. The events I want to keep do not easily fit a pattern, it is easier to identify what to remove.
Independent processes with names like "fred", "bob", "bill" run and create log entries. The "name" can be considered as a tag that identifies the start of the logging group for that process at that time.
Each process has an automatic process ID associated with it e.g. CRON[3383] and in this example all 3833s are associated with the first "fred" event.
This "name" fred, (or bob, bill), are the only thing I know for sure in advance, I do not know the process IDs. I also do not know the number of rows generated in the associated grouping. So,
Start at the top of the file. Find:
if a line in the log has "fred", determine the Process ID e.g. 3833 (which is between the only square brackets before the reference to "fred" on the identified "fred" line); then group all the same ID lines e.g. all the 3833 including the "fred" line; then delete that group.
Repeat:
Find the next "fred", determine ID e.g. 4099, group all these, delete.
Repeat
If a line has "fred" but no process ID i.e. CRON[num], do not delete.
input:
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session closed for user root
Jan 2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root
Desired output:
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root
++Edited with possible solution++:
grep -e "[.*fred" input.txt |sed 's/.*[([^]]*)].*/1/g' |grep -vf /dev/stdin input.txt
Is there a cleaner way?
linux awk sed grep regular-expression
New contributor
Edited again for clarity: sorry for the confusion.
I have the following example log. The events I want to keep do not easily fit a pattern, it is easier to identify what to remove.
Independent processes with names like "fred", "bob", "bill" run and create log entries. The "name" can be considered as a tag that identifies the start of the logging group for that process at that time.
Each process has an automatic process ID associated with it e.g. CRON[3383] and in this example all 3833s are associated with the first "fred" event.
This "name" fred, (or bob, bill), are the only thing I know for sure in advance, I do not know the process IDs. I also do not know the number of rows generated in the associated grouping. So,
Start at the top of the file. Find:
if a line in the log has "fred", determine the Process ID e.g. 3833 (which is between the only square brackets before the reference to "fred" on the identified "fred" line); then group all the same ID lines e.g. all the 3833 including the "fred" line; then delete that group.
Repeat:
Find the next "fred", determine ID e.g. 4099, group all these, delete.
Repeat
If a line has "fred" but no process ID i.e. CRON[num], do not delete.
input:
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session closed for user root
Jan 2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (fred)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root
Desired output:
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan 2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root
++Edited with possible solution++:
grep -e "[.*fred" input.txt |sed 's/.*[([^]]*)].*/1/g' |grep -vf /dev/stdin input.txt
Is there a cleaner way?
linux awk sed grep regular-expression
linux awk sed grep regular-expression
New contributor
New contributor
edited 2 days ago
New contributor
asked Oct 4 at 8:34
BillDoor
11
11
New contributor
New contributor
put on hold as unclear what you're asking by Goro, jimmij, Kiwy, Jeff Schaller, Thomas 2 days ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, itâÂÂs hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
put on hold as unclear what you're asking by Goro, jimmij, Kiwy, Jeff Schaller, Thomas 2 days ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, itâÂÂs hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
1
Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
â Goro
Oct 4 at 8:43
sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
â BillDoor
Oct 4 at 10:15
1
do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
â Kiwy
Oct 4 at 11:19
Two of your three desired output lines do NOT containuid=0
. So - how is that output derived? Except, e.g.< file grep -v "CRON.*]"
â RudiC
Oct 4 at 11:38
i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
â BillDoor
Oct 4 at 16:07
add a comment |Â
1
Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
â Goro
Oct 4 at 8:43
sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
â BillDoor
Oct 4 at 10:15
1
do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
â Kiwy
Oct 4 at 11:19
Two of your three desired output lines do NOT containuid=0
. So - how is that output derived? Except, e.g.< file grep -v "CRON.*]"
â RudiC
Oct 4 at 11:38
i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
â BillDoor
Oct 4 at 16:07
1
1
Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
â Goro
Oct 4 at 8:43
Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
â Goro
Oct 4 at 8:43
sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
â BillDoor
Oct 4 at 10:15
sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
â BillDoor
Oct 4 at 10:15
1
1
do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
â Kiwy
Oct 4 at 11:19
do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
â Kiwy
Oct 4 at 11:19
Two of your three desired output lines do NOT contain
uid=0
. So - how is that output derived? Except, e.g. < file grep -v "CRON.*]"
â RudiC
Oct 4 at 11:38
Two of your three desired output lines do NOT contain
uid=0
. So - how is that output derived? Except, e.g. < file grep -v "CRON.*]"
â RudiC
Oct 4 at 11:38
i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
â BillDoor
Oct 4 at 16:07
i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
â BillDoor
Oct 4 at 16:07
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
0
down vote
Try the below code,
grep "uid=0" /var/log/secure | grep -v "CRON.*]"
This does not quite deliver the desired output.
â RudiC
Oct 4 at 11:36
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
Try the below code,
grep "uid=0" /var/log/secure | grep -v "CRON.*]"
This does not quite deliver the desired output.
â RudiC
Oct 4 at 11:36
add a comment |Â
up vote
0
down vote
Try the below code,
grep "uid=0" /var/log/secure | grep -v "CRON.*]"
This does not quite deliver the desired output.
â RudiC
Oct 4 at 11:36
add a comment |Â
up vote
0
down vote
up vote
0
down vote
Try the below code,
grep "uid=0" /var/log/secure | grep -v "CRON.*]"
Try the below code,
grep "uid=0" /var/log/secure | grep -v "CRON.*]"
answered Oct 4 at 10:03
EBIN GLADSON
3316
3316
This does not quite deliver the desired output.
â RudiC
Oct 4 at 11:36
add a comment |Â
This does not quite deliver the desired output.
â RudiC
Oct 4 at 11:36
This does not quite deliver the desired output.
â RudiC
Oct 4 at 11:36
This does not quite deliver the desired output.
â RudiC
Oct 4 at 11:36
add a comment |Â
1
Hello, your question completely unclear. Would you please add clarification for how you got the output. There are many sections in the output are not in the input?
â Goro
Oct 4 at 8:43
sorry about that. the log entries are grouped by CRON[number] or with something else. I am interested in the keyword "uid=0". line 1 has "uid=0" and [3383], line 2 only has [3383] but I want both lines removed. So in the log entries, find "uid=0" and if present the CORN[], then remove the "uid=0 line" and all other lines with the same CRON[]. Lines 6,7,8,9,10 should be removed, only lines 3,4,5 should remain.
â BillDoor
Oct 4 at 10:15
1
do not answer in comments please, edit you question to explain clearly and simply what is your source file was is your input and what output you want. Try to be simple and precise. In its current form it's absolutely not understandable what you want to achieve. Try to add useful information such as I want to filter a log file or anything relevant. Try to use short sentence.
â Kiwy
Oct 4 at 11:19
Two of your three desired output lines do NOT contain
uid=0
. So - how is that output derived? Except, e.g.< file grep -v "CRON.*]"
â RudiC
Oct 4 at 11:38
i'm a bit new to the forum/website. sorry for the confusion and my bad etiquette.
â BillDoor
Oct 4 at 16:07