mjhjmtu

I am trying to filter a log file based on the following:

date range of an error (yyyy-mm-dd)

timestamp range of an error (01:00:00 - 00:00:00)

keywords (previousFireTime, nextFireTime)

I've been trying to grep the log file, however, I am unable to get the data that I need with using grep. Would I have to use a combination of awk and grep or awk and sed to get the information I need? Or would there be a better more efficient route to go by to filter a log file?

Edit: Example log output

2018-06-06 10:46:43,708 INFO [stdout] (AsyncAppender-Worker-STDOUT) INFO 
[erFactoryBean_Worker-9] [c.c.c.s.i.d.ResendJob] Executing Quartz scheduled 
job: JobExecutionContext: trigger: 'ResendJob.trigger_ResendJob job: 
DEFAULT.ResendJob fireTime: 'Wed Jun 06 10:46:43 UTC 2018 scheduledFireTime: 
Wed Jun 06 10:46:43 UTC 2018 previousFireTime: 'Wed Jun 06 10:45:43 UTC 2018 
nextFireTime: Wed Jun 06 10:47:43 UTC 2018 isRecovering: false refireCount: 0

awk is probably all you need here as it can do regexp matching, split lines into fields and do string comparisons (which works for date comparison as long as you use YYYY-MM-DD HH:MM:SS timestamps and there's no DST change).

If the date is in the first field and time in second:

awk -v date=1 -v time=2 '
 $date > "2018-05-24" && $time < "12:00:00" && /some text/'

The GNU awk implementation of awk has date parsing and formatting extensions which lets you do more advanced stuff like:

gawk -v date=1 -v time=2 '
 function parse_time(t) 
 gsub(/[:-]/, " ", t)
 return mktime(t)
 
 BEGIN 
 start = parse_time("2018-01-01 08:00")
 end = systime() - 86400 # yesterday, same time
 
 t = parse_time($date" "$time)
 t >= start && t <= end && /some test/'

grep filters regular expressions. It is very good at filtering lines containing one particular keyword, but a date range is hard to specify using regular expressions. For example, to get errors between Jan 1st, 20:00 and Jan 3rd, 2:00, you have to accept all times for Jan 2nd, but only evening for Jan 1st and only early morning for Jan 3rd. You can't separate time of day and date, for example.

It is much simpler to use a more expressive tool that can natively compare dates. Perl is a popular language to do this kind of things, and Python is a good alternative.

Here's an example in Python:

import re
import time

f = open('/var/log/syslog')
line = f.readline()
while line:
 # Get the date at the beginning of line with a regex
 m = re.match(r'^([^s]+s+[^s]+s+[^s]+)s+', line)
 # Parse the date
 date = time.strptime(m.group(1), '%b %d %H:%M:%S')
 # Compare with a given date
 if date > time.strptime('Jun 6 14:00:00', '%b %d %H:%M:%S'):
 print(line, end='')

 # Read next line
 line = f.readline()