Extracting 2 dates from log file with specific patterns and compare the 2 dates

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












It seems I have problems with my connection pool tool. There is a big delay when it obtains the DB connection.

What I try to achieve is to get all the cases from log file when this incident occurs.



The related log entries look like



...
2018-03-12 16:18:44,070 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtaining JDBC connection
...
2018-03-12 16:20:23,172 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtained JDBC connection
...


So if the pattern 'DEBUG Obtaining JDBC connection' occurs then extract the date '2018-03-12 16:18:44,070' and when the pattern 'DEBUG Obtained JDBC connection' is found then extract its date and compare the 2 dates. If the difference is more than 2 sec then log then.



I know it is pretty complicated to solve it with one line of code but is it possible without writing a program to do that?







share|improve this question





















  • what does mean "then log then" exactly?
    – RomanPerekhrest
    May 16 at 8:44











  • Do the ,070 and ,172 represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?
    – Jeff Schaller
    May 16 at 12:52










  • Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
    – Viktor
    May 16 at 15:16










  • Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
    – Viktor
    May 16 at 15:20














up vote
0
down vote

favorite












It seems I have problems with my connection pool tool. There is a big delay when it obtains the DB connection.

What I try to achieve is to get all the cases from log file when this incident occurs.



The related log entries look like



...
2018-03-12 16:18:44,070 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtaining JDBC connection
...
2018-03-12 16:20:23,172 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtained JDBC connection
...


So if the pattern 'DEBUG Obtaining JDBC connection' occurs then extract the date '2018-03-12 16:18:44,070' and when the pattern 'DEBUG Obtained JDBC connection' is found then extract its date and compare the 2 dates. If the difference is more than 2 sec then log then.



I know it is pretty complicated to solve it with one line of code but is it possible without writing a program to do that?







share|improve this question





















  • what does mean "then log then" exactly?
    – RomanPerekhrest
    May 16 at 8:44











  • Do the ,070 and ,172 represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?
    – Jeff Schaller
    May 16 at 12:52










  • Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
    – Viktor
    May 16 at 15:16










  • Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
    – Viktor
    May 16 at 15:20












up vote
0
down vote

favorite









up vote
0
down vote

favorite











It seems I have problems with my connection pool tool. There is a big delay when it obtains the DB connection.

What I try to achieve is to get all the cases from log file when this incident occurs.



The related log entries look like



...
2018-03-12 16:18:44,070 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtaining JDBC connection
...
2018-03-12 16:20:23,172 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtained JDBC connection
...


So if the pattern 'DEBUG Obtaining JDBC connection' occurs then extract the date '2018-03-12 16:18:44,070' and when the pattern 'DEBUG Obtained JDBC connection' is found then extract its date and compare the 2 dates. If the difference is more than 2 sec then log then.



I know it is pretty complicated to solve it with one line of code but is it possible without writing a program to do that?







share|improve this question













It seems I have problems with my connection pool tool. There is a big delay when it obtains the DB connection.

What I try to achieve is to get all the cases from log file when this incident occurs.



The related log entries look like



...
2018-03-12 16:18:44,070 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtaining JDBC connection
...
2018-03-12 16:20:23,172 efault task-166 gine.jdbc.internal.LogicalConnectionImpl DEBUG Obtained JDBC connection
...


So if the pattern 'DEBUG Obtaining JDBC connection' occurs then extract the date '2018-03-12 16:18:44,070' and when the pattern 'DEBUG Obtained JDBC connection' is found then extract its date and compare the 2 dates. If the difference is more than 2 sec then log then.



I know it is pretty complicated to solve it with one line of code but is it possible without writing a program to do that?









share|improve this question












share|improve this question




share|improve this question








edited May 16 at 9:51









Jeff Schaller

31.1k846105




31.1k846105









asked May 16 at 8:19









Viktor

1032




1032











  • what does mean "then log then" exactly?
    – RomanPerekhrest
    May 16 at 8:44











  • Do the ,070 and ,172 represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?
    – Jeff Schaller
    May 16 at 12:52










  • Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
    – Viktor
    May 16 at 15:16










  • Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
    – Viktor
    May 16 at 15:20
















  • what does mean "then log then" exactly?
    – RomanPerekhrest
    May 16 at 8:44











  • Do the ,070 and ,172 represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?
    – Jeff Schaller
    May 16 at 12:52










  • Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
    – Viktor
    May 16 at 15:16










  • Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
    – Viktor
    May 16 at 15:20















what does mean "then log then" exactly?
– RomanPerekhrest
May 16 at 8:44





what does mean "then log then" exactly?
– RomanPerekhrest
May 16 at 8:44













Do the ,070 and ,172 represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?
– Jeff Schaller
May 16 at 12:52




Do the ,070 and ,172 represent milliseconds, and do you care about them? If "obtaining" happens at 16:18:44,999 and "obtained" happens at 16:18:46,001 (a difference of 1.002 seconds, yet apparently 2 seconds if you drop the milliseconds), do you want that reported?
– Jeff Schaller
May 16 at 12:52












Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
– Viktor
May 16 at 15:16




Hi @RomanPerekhrest, any log can be useful to me which helps to find the lines where the delay is too high. For example: the timestamp of the log entry, the full log entry, the difference between the 2 dates, etc.
– Viktor
May 16 at 15:16












Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
– Viktor
May 16 at 15:20




Hi @JeffSchaller, just an example above what has to be examined. There are cases when the difference between starting to get the connection and obtaining it takes many seconds. It seems like a bottleneck and want to investigate why and not just increasing the DB pool.
– Viktor
May 16 at 15:20










2 Answers
2






active

oldest

votes

















up vote
2
down vote



accepted










I would approach it with an awk script, below. The script searches for the "obtaining" and "obtained" strings. The typical case is that it finds the "obtaining" string first, followed by the "obtained" string; if an "obtained" line is found first, d1 will be unset or zero, so it won't be reported on. Otherwise, we extract the main part of the date (before the comma), convert that to seconds-since-the-epoch, then add the milliseconds back on. That value is saved into d1.



Once the "obtained" string is seen, its date is computed in the same way, then we check the difference; if it's more than 2 seconds, we report it. Either way, we then reset d1 to start the process over again.



Save the below into a script, named anything you want (perhaps alert-jdbc.awk), then run it with awk -f alert-jdbc.awk < log-file-here.



The script requires the GNU date utility in order to convert arbitrary dates.



/DEBUG Obtaining JDBC connection/ 
split($1 " " $2, dates, /,/)
"date -d "" dates[1]"" +%s"

/DEBUG Obtained JDBC connection/
if (d1) getline seconds
d2=seconds + dates[2]/1000
if (d2 - d1 > 2)
print "More than 2 seconds for: " $0

d1=0







share|improve this answer





















  • Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
    – Viktor
    May 16 at 16:08

















up vote
2
down vote













Try this:



Delay=$(( 
($(date -d "`awk '/DEBUG Obtained JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) -
($(date -d "`awk '/DEBUG Obtaining JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) ))
echo "$Delay Sec"


will explain the command if it works for your requirement.






share|improve this answer























  • Thanks! I can only see the following output: 0 Sec. I'd expect more lines.
    – Viktor
    May 16 at 15:13











Your Answer







StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);








 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f444110%2fextracting-2-dates-from-log-file-with-specific-patterns-and-compare-the-2-dates%23new-answer', 'question_page');

);

Post as a guest






























2 Answers
2






active

oldest

votes








2 Answers
2






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
2
down vote



accepted










I would approach it with an awk script, below. The script searches for the "obtaining" and "obtained" strings. The typical case is that it finds the "obtaining" string first, followed by the "obtained" string; if an "obtained" line is found first, d1 will be unset or zero, so it won't be reported on. Otherwise, we extract the main part of the date (before the comma), convert that to seconds-since-the-epoch, then add the milliseconds back on. That value is saved into d1.



Once the "obtained" string is seen, its date is computed in the same way, then we check the difference; if it's more than 2 seconds, we report it. Either way, we then reset d1 to start the process over again.



Save the below into a script, named anything you want (perhaps alert-jdbc.awk), then run it with awk -f alert-jdbc.awk < log-file-here.



The script requires the GNU date utility in order to convert arbitrary dates.



/DEBUG Obtaining JDBC connection/ 
split($1 " " $2, dates, /,/)
"date -d "" dates[1]"" +%s"

/DEBUG Obtained JDBC connection/
if (d1) getline seconds
d2=seconds + dates[2]/1000
if (d2 - d1 > 2)
print "More than 2 seconds for: " $0

d1=0







share|improve this answer





















  • Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
    – Viktor
    May 16 at 16:08














up vote
2
down vote



accepted










I would approach it with an awk script, below. The script searches for the "obtaining" and "obtained" strings. The typical case is that it finds the "obtaining" string first, followed by the "obtained" string; if an "obtained" line is found first, d1 will be unset or zero, so it won't be reported on. Otherwise, we extract the main part of the date (before the comma), convert that to seconds-since-the-epoch, then add the milliseconds back on. That value is saved into d1.



Once the "obtained" string is seen, its date is computed in the same way, then we check the difference; if it's more than 2 seconds, we report it. Either way, we then reset d1 to start the process over again.



Save the below into a script, named anything you want (perhaps alert-jdbc.awk), then run it with awk -f alert-jdbc.awk < log-file-here.



The script requires the GNU date utility in order to convert arbitrary dates.



/DEBUG Obtaining JDBC connection/ 
split($1 " " $2, dates, /,/)
"date -d "" dates[1]"" +%s"

/DEBUG Obtained JDBC connection/
if (d1) getline seconds
d2=seconds + dates[2]/1000
if (d2 - d1 > 2)
print "More than 2 seconds for: " $0

d1=0







share|improve this answer





















  • Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
    – Viktor
    May 16 at 16:08












up vote
2
down vote



accepted







up vote
2
down vote



accepted






I would approach it with an awk script, below. The script searches for the "obtaining" and "obtained" strings. The typical case is that it finds the "obtaining" string first, followed by the "obtained" string; if an "obtained" line is found first, d1 will be unset or zero, so it won't be reported on. Otherwise, we extract the main part of the date (before the comma), convert that to seconds-since-the-epoch, then add the milliseconds back on. That value is saved into d1.



Once the "obtained" string is seen, its date is computed in the same way, then we check the difference; if it's more than 2 seconds, we report it. Either way, we then reset d1 to start the process over again.



Save the below into a script, named anything you want (perhaps alert-jdbc.awk), then run it with awk -f alert-jdbc.awk < log-file-here.



The script requires the GNU date utility in order to convert arbitrary dates.



/DEBUG Obtaining JDBC connection/ 
split($1 " " $2, dates, /,/)
"date -d "" dates[1]"" +%s"

/DEBUG Obtained JDBC connection/
if (d1) getline seconds
d2=seconds + dates[2]/1000
if (d2 - d1 > 2)
print "More than 2 seconds for: " $0

d1=0







share|improve this answer













I would approach it with an awk script, below. The script searches for the "obtaining" and "obtained" strings. The typical case is that it finds the "obtaining" string first, followed by the "obtained" string; if an "obtained" line is found first, d1 will be unset or zero, so it won't be reported on. Otherwise, we extract the main part of the date (before the comma), convert that to seconds-since-the-epoch, then add the milliseconds back on. That value is saved into d1.



Once the "obtained" string is seen, its date is computed in the same way, then we check the difference; if it's more than 2 seconds, we report it. Either way, we then reset d1 to start the process over again.



Save the below into a script, named anything you want (perhaps alert-jdbc.awk), then run it with awk -f alert-jdbc.awk < log-file-here.



The script requires the GNU date utility in order to convert arbitrary dates.



/DEBUG Obtaining JDBC connection/ 
split($1 " " $2, dates, /,/)
"date -d "" dates[1]"" +%s"

/DEBUG Obtained JDBC connection/
if (d1) getline seconds
d2=seconds + dates[2]/1000
if (d2 - d1 > 2)
print "More than 2 seconds for: " $0

d1=0








share|improve this answer













share|improve this answer



share|improve this answer











answered May 16 at 13:12









Jeff Schaller

31.1k846105




31.1k846105











  • Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
    – Viktor
    May 16 at 16:08
















  • Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
    – Viktor
    May 16 at 16:08















Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
– Viktor
May 16 at 16:08




Whaoo... absolutely brilliant! I think Sauron really lives, this is really a black magic. :) Thank you very much, it works perfectly!
– Viktor
May 16 at 16:08












up vote
2
down vote













Try this:



Delay=$(( 
($(date -d "`awk '/DEBUG Obtained JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) -
($(date -d "`awk '/DEBUG Obtaining JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) ))
echo "$Delay Sec"


will explain the command if it works for your requirement.






share|improve this answer























  • Thanks! I can only see the following output: 0 Sec. I'd expect more lines.
    – Viktor
    May 16 at 15:13















up vote
2
down vote













Try this:



Delay=$(( 
($(date -d "`awk '/DEBUG Obtained JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) -
($(date -d "`awk '/DEBUG Obtaining JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) ))
echo "$Delay Sec"


will explain the command if it works for your requirement.






share|improve this answer























  • Thanks! I can only see the following output: 0 Sec. I'd expect more lines.
    – Viktor
    May 16 at 15:13













up vote
2
down vote










up vote
2
down vote









Try this:



Delay=$(( 
($(date -d "`awk '/DEBUG Obtained JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) -
($(date -d "`awk '/DEBUG Obtaining JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) ))
echo "$Delay Sec"


will explain the command if it works for your requirement.






share|improve this answer















Try this:



Delay=$(( 
($(date -d "`awk '/DEBUG Obtained JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) -
($(date -d "`awk '/DEBUG Obtaining JDBC connection/print $1" "$2' file.log |
cut -d ',' -f1 | tail -n1`" +%s)) ))
echo "$Delay Sec"


will explain the command if it works for your requirement.







share|improve this answer















share|improve this answer



share|improve this answer








edited May 16 at 12:33


























answered May 16 at 9:01









SivaPrasath

4,74212445




4,74212445











  • Thanks! I can only see the following output: 0 Sec. I'd expect more lines.
    – Viktor
    May 16 at 15:13

















  • Thanks! I can only see the following output: 0 Sec. I'd expect more lines.
    – Viktor
    May 16 at 15:13
















Thanks! I can only see the following output: 0 Sec. I'd expect more lines.
– Viktor
May 16 at 15:13





Thanks! I can only see the following output: 0 Sec. I'd expect more lines.
– Viktor
May 16 at 15:13













 

draft saved


draft discarded


























 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f444110%2fextracting-2-dates-from-log-file-with-specific-patterns-and-compare-the-2-dates%23new-answer', 'question_page');

);

Post as a guest













































































Popular posts from this blog

How to check contact read email or not when send email to Individual?

Bahrain

Postfix configuration issue with fips on centos 7; mailgun relay