Extracting 2 dates from log file with specific patterns and compare the 2 dates
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
It seems I have problems with my connection pool tool. There is a big delay when it obtains the DB connection.
What I try to achieve is to get all the cases from log file when this incident occurs.
The related log entries look like
...
2018-03-12 16:18:44,070 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtaining JDBC connection
...
2018-03-12 16:20:23,172 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtained JDBC connection
...
So if the pattern 'DEBUG Obtaining JDBC connection' occurs then extract the date '2018-03-12 16:18:44,070' and when the pattern 'DEBUG Obtained JDBC connection' is found then extract its date and compare the 2 dates. If the difference is more than 2 sec then log then.
I know it is pretty complicated to solve it with one line of code but is it possible without writing a program to do that?
shell-script logs date
add a comment |Â
up vote
0
down vote
favorite
It seems I have problems with my connection pool tool. There is a big delay when it obtains the DB connection.
What I try to achieve is to get all the cases from log file when this incident occurs.
The related log entries look like
...
2018-03-12 16:18:44,070 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtaining JDBC connection
...
2018-03-12 16:20:23,172 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtained JDBC connection
...
So if the pattern 'DEBUG Obtaining JDBC connection' occurs then extract the date '2018-03-12 16:18:44,070' and when the pattern 'DEBUG Obtained JDBC connection' is found then extract its date and compare the 2 dates. If the difference is more than 2 sec then log then.
I know it is pretty complicated to solve it with one line of code but is it possible without writing a program to do that?
shell-script logs date
what does mean "then log then" exactly?
â RomanPerekhrest
May 16 at 8:44
Do the,070
and,172
represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?
â Jeff Schaller
May 16 at 12:52
Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
â Viktor
May 16 at 15:16
Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
â Viktor
May 16 at 15:20
add a comment |Â
up vote
0
down vote
favorite
up vote
0
down vote
favorite
It seems I have problems with my connection pool tool. There is a big delay when it obtains the DB connection.
What I try to achieve is to get all the cases from log file when this incident occurs.
The related log entries look like
...
2018-03-12 16:18:44,070 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtaining JDBC connection
...
2018-03-12 16:20:23,172 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtained JDBC connection
...
So if the pattern 'DEBUG Obtaining JDBC connection' occurs then extract the date '2018-03-12 16:18:44,070' and when the pattern 'DEBUG Obtained JDBC connection' is found then extract its date and compare the 2 dates. If the difference is more than 2 sec then log then.
I know it is pretty complicated to solve it with one line of code but is it possible without writing a program to do that?
shell-script logs date
It seems I have problems with my connection pool tool. There is a big delay when it obtains the DB connection.
What I try to achieve is to get all the cases from log file when this incident occurs.
The related log entries look like
...
2018-03-12 16:18:44,070 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtaining JDBC connection
...
2018-03-12 16:20:23,172 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtained JDBC connection
...
So if the pattern 'DEBUG Obtaining JDBC connection' occurs then extract the date '2018-03-12 16:18:44,070' and when the pattern 'DEBUG Obtained JDBC connection' is found then extract its date and compare the 2 dates. If the difference is more than 2 sec then log then.
I know it is pretty complicated to solve it with one line of code but is it possible without writing a program to do that?
shell-script logs date
edited May 16 at 9:51
Jeff Schaller
31.1k846105
31.1k846105
asked May 16 at 8:19
Viktor
1032
1032
what does mean "then log then" exactly?
â RomanPerekhrest
May 16 at 8:44
Do the,070
and,172
represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?
â Jeff Schaller
May 16 at 12:52
Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
â Viktor
May 16 at 15:16
Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
â Viktor
May 16 at 15:20
add a comment |Â
what does mean "then log then" exactly?
â RomanPerekhrest
May 16 at 8:44
Do the,070
and,172
represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?
â Jeff Schaller
May 16 at 12:52
Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
â Viktor
May 16 at 15:16
Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
â Viktor
May 16 at 15:20
what does mean "then log then" exactly?
â RomanPerekhrest
May 16 at 8:44
what does mean "then log then" exactly?
â RomanPerekhrest
May 16 at 8:44
Do the
,070
and ,172
represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?â Jeff Schaller
May 16 at 12:52
Do the
,070
and ,172
represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?â Jeff Schaller
May 16 at 12:52
Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
â Viktor
May 16 at 15:16
Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
â Viktor
May 16 at 15:16
Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
â Viktor
May 16 at 15:20
Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
â Viktor
May 16 at 15:20
add a comment |Â
2 Answers
2
active
oldest
votes
up vote
2
down vote
accepted
I would approach it with an awk script, below. The script searches for the "obtaining" and "obtained" strings. The typical case is that it finds the "obtaining" string first, followed by the "obtained" string; if an "obtained" line is found first, d1
will be unset or zero, so it won't be reported on. Otherwise, we extract the main part of the date (before the comma), convert that to seconds-since-the-epoch, then add the milliseconds back on. That value is saved into d1
.
Once the "obtained" string is seen, its date is computed in the same way, then we check the difference; if it's more than 2 seconds, we report it. Either way, we then reset d1
to start the process over again.
Save the below into a script, named anything you want (perhaps alert-jdbc.awk
), then run it with awk -f alert-jdbc.awk < log-file-here
.
The script requires the GNU date utility in order to convert arbitrary dates.
/DEBUG Obtaining JDBC connection/
split($1 " " $2, dates, /,/)
"date -d "" dates[1]"" +%s"
/DEBUG Obtained JDBC connection/
if (d1) getline seconds
d2=seconds + dates[2]/1000
if (d2 - d1 > 2)
print "More than 2 seconds for: " $0
d1=0
Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
â Viktor
May 16 at 16:08
add a comment |Â
up vote
2
down vote
Try this:
Delay=$((
($(date -d "`awk '/DEBUG Obtained JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) -
($(date -d "`awk '/DEBUG Obtaining JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) ))
echo "$Delay Sec"
will explain the command if it works for your requirement.
Thanks! I can only see the following output:0 Sec
. I'd expect more lines.
â Viktor
May 16 at 15:13
add a comment |Â
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
accepted
I would approach it with an awk script, below. The script searches for the "obtaining" and "obtained" strings. The typical case is that it finds the "obtaining" string first, followed by the "obtained" string; if an "obtained" line is found first, d1
will be unset or zero, so it won't be reported on. Otherwise, we extract the main part of the date (before the comma), convert that to seconds-since-the-epoch, then add the milliseconds back on. That value is saved into d1
.
Once the "obtained" string is seen, its date is computed in the same way, then we check the difference; if it's more than 2 seconds, we report it. Either way, we then reset d1
to start the process over again.
Save the below into a script, named anything you want (perhaps alert-jdbc.awk
), then run it with awk -f alert-jdbc.awk < log-file-here
.
The script requires the GNU date utility in order to convert arbitrary dates.
/DEBUG Obtaining JDBC connection/
split($1 " " $2, dates, /,/)
"date -d "" dates[1]"" +%s"
/DEBUG Obtained JDBC connection/
if (d1) getline seconds
d2=seconds + dates[2]/1000
if (d2 - d1 > 2)
print "More than 2 seconds for: " $0
d1=0
Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
â Viktor
May 16 at 16:08
add a comment |Â
up vote
2
down vote
accepted
I would approach it with an awk script, below. The script searches for the "obtaining" and "obtained" strings. The typical case is that it finds the "obtaining" string first, followed by the "obtained" string; if an "obtained" line is found first, d1
will be unset or zero, so it won't be reported on. Otherwise, we extract the main part of the date (before the comma), convert that to seconds-since-the-epoch, then add the milliseconds back on. That value is saved into d1
.
Once the "obtained" string is seen, its date is computed in the same way, then we check the difference; if it's more than 2 seconds, we report it. Either way, we then reset d1
to start the process over again.
Save the below into a script, named anything you want (perhaps alert-jdbc.awk
), then run it with awk -f alert-jdbc.awk < log-file-here
.
The script requires the GNU date utility in order to convert arbitrary dates.
/DEBUG Obtaining JDBC connection/
split($1 " " $2, dates, /,/)
"date -d "" dates[1]"" +%s"
/DEBUG Obtained JDBC connection/
if (d1) getline seconds
d2=seconds + dates[2]/1000
if (d2 - d1 > 2)
print "More than 2 seconds for: " $0
d1=0
Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
â Viktor
May 16 at 16:08
add a comment |Â
up vote
2
down vote
accepted
up vote
2
down vote
accepted
I would approach it with an awk script, below. The script searches for the "obtaining" and "obtained" strings. The typical case is that it finds the "obtaining" string first, followed by the "obtained" string; if an "obtained" line is found first, d1
will be unset or zero, so it won't be reported on. Otherwise, we extract the main part of the date (before the comma), convert that to seconds-since-the-epoch, then add the milliseconds back on. That value is saved into d1
.
Once the "obtained" string is seen, its date is computed in the same way, then we check the difference; if it's more than 2 seconds, we report it. Either way, we then reset d1
to start the process over again.
Save the below into a script, named anything you want (perhaps alert-jdbc.awk
), then run it with awk -f alert-jdbc.awk < log-file-here
.
The script requires the GNU date utility in order to convert arbitrary dates.
/DEBUG Obtaining JDBC connection/
split($1 " " $2, dates, /,/)
"date -d "" dates[1]"" +%s"
/DEBUG Obtained JDBC connection/
if (d1) getline seconds
d2=seconds + dates[2]/1000
if (d2 - d1 > 2)
print "More than 2 seconds for: " $0
d1=0
I would approach it with an awk script, below. The script searches for the "obtaining" and "obtained" strings. The typical case is that it finds the "obtaining" string first, followed by the "obtained" string; if an "obtained" line is found first, d1
will be unset or zero, so it won't be reported on. Otherwise, we extract the main part of the date (before the comma), convert that to seconds-since-the-epoch, then add the milliseconds back on. That value is saved into d1
.
Once the "obtained" string is seen, its date is computed in the same way, then we check the difference; if it's more than 2 seconds, we report it. Either way, we then reset d1
to start the process over again.
Save the below into a script, named anything you want (perhaps alert-jdbc.awk
), then run it with awk -f alert-jdbc.awk < log-file-here
.
The script requires the GNU date utility in order to convert arbitrary dates.
/DEBUG Obtaining JDBC connection/
split($1 " " $2, dates, /,/)
"date -d "" dates[1]"" +%s"
/DEBUG Obtained JDBC connection/
if (d1) getline seconds
d2=seconds + dates[2]/1000
if (d2 - d1 > 2)
print "More than 2 seconds for: " $0
d1=0
answered May 16 at 13:12
Jeff Schaller
31.1k846105
31.1k846105
Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
â Viktor
May 16 at 16:08
add a comment |Â
Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
â Viktor
May 16 at 16:08
Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
â Viktor
May 16 at 16:08
Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
â Viktor
May 16 at 16:08
add a comment |Â
up vote
2
down vote
Try this:
Delay=$((
($(date -d "`awk '/DEBUG Obtained JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) -
($(date -d "`awk '/DEBUG Obtaining JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) ))
echo "$Delay Sec"
will explain the command if it works for your requirement.
Thanks! I can only see the following output:0 Sec
. I'd expect more lines.
â Viktor
May 16 at 15:13
add a comment |Â
up vote
2
down vote
Try this:
Delay=$((
($(date -d "`awk '/DEBUG Obtained JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) -
($(date -d "`awk '/DEBUG Obtaining JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) ))
echo "$Delay Sec"
will explain the command if it works for your requirement.
Thanks! I can only see the following output:0 Sec
. I'd expect more lines.
â Viktor
May 16 at 15:13
add a comment |Â
up vote
2
down vote
up vote
2
down vote
Try this:
Delay=$((
($(date -d "`awk '/DEBUG Obtained JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) -
($(date -d "`awk '/DEBUG Obtaining JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) ))
echo "$Delay Sec"
will explain the command if it works for your requirement.
Try this:
Delay=$((
($(date -d "`awk '/DEBUG Obtained JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) -
($(date -d "`awk '/DEBUG Obtaining JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) ))
echo "$Delay Sec"
will explain the command if it works for your requirement.
edited May 16 at 12:33
answered May 16 at 9:01
SivaPrasath
4,74212445
4,74212445
Thanks! I can only see the following output:0 Sec
. I'd expect more lines.
â Viktor
May 16 at 15:13
add a comment |Â
Thanks! I can only see the following output:0 Sec
. I'd expect more lines.
â Viktor
May 16 at 15:13
Thanks! I can only see the following output:
0 Sec
. I'd expect more lines.â Viktor
May 16 at 15:13
Thanks! I can only see the following output:
0 Sec
. I'd expect more lines.â Viktor
May 16 at 15:13
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f444110%2fextracting-2-dates-from-log-file-with-specific-patterns-and-compare-the-2-dates%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
what does mean "then log then" exactly?
â RomanPerekhrest
May 16 at 8:44
Do the
,070
and,172
represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?â Jeff Schaller
May 16 at 12:52
Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
â Viktor
May 16 at 15:16
Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
â Viktor
May 16 at 15:20