Weblog entry #3 for PJ_at_Belzabar_Software
See all weblog entries made by PJ_at_Belzabar_Software.
#3
looking at yesterday's logs
Posted by PJ_at_Belzabar_Software on Thu 6 Sep 2007 at 09:36
I just figured out a lazy (short) and robust way to define "yesterday". I think it is pretty cool. Of course I'm probably reinventing the wheel, but I thought I'd gloat about it here anyway.
Imagine you have a log file you want to grep through daily to look at what went on yesterday. The log file is timestamped with the date, and is in a human-readable type of format (eg starting with: Sep 5 2007 18:53). How do you grep through yesterday's logs in an automated way without putting in yesterday's date manually? And keeping it simple?
The problem is that yesterday isn't defined with simple code (think of the rollover for a month or year etc). Which is why we have a zillion perl date handling modules.
But we can avoid worrying about it. The principle is to define yesterday using something like this in bash:
yesterday=`perl -e '$string = localtime($ARGV[0]-86400); print "$string"' \`date +%s\``
(the way this works is that bash's date gives today's date. This is output in the seconds-since-1970 format. This output is the argument used by perl, and has 86400 seconds (1 day) chopped off it. The string that perl then prints out is a human readable timestamp format (eg Wed Sep 5 13:59:57 2007) and this is what becomes the $yesterday variable.)
(Oh, yeah, and you may want to add a
| cut -b5-10
or something like that before the last backtick if you want to handle only some fields).
Then you grep using $yesterday as your pattern through the log file.
The reason I think this is cool is because of the way you dump the problem onto bash date and perl, which handle dates robustly, so the problem is no longer yours.
Well, I thought it was frightfully neat. Now I'm off to pat my back.
PJ
Imagine you have a log file you want to grep through daily to look at what went on yesterday. The log file is timestamped with the date, and is in a human-readable type of format (eg starting with: Sep 5 2007 18:53). How do you grep through yesterday's logs in an automated way without putting in yesterday's date manually? And keeping it simple?
The problem is that yesterday isn't defined with simple code (think of the rollover for a month or year etc). Which is why we have a zillion perl date handling modules.
But we can avoid worrying about it. The principle is to define yesterday using something like this in bash:
yesterday=`perl -e '$string = localtime($ARGV[0]-86400); print "$string"' \`date +%s\``
(the way this works is that bash's date gives today's date. This is output in the seconds-since-1970 format. This output is the argument used by perl, and has 86400 seconds (1 day) chopped off it. The string that perl then prints out is a human readable timestamp format (eg Wed Sep 5 13:59:57 2007) and this is what becomes the $yesterday variable.)
(Oh, yeah, and you may want to add a
| cut -b5-10
or something like that before the last backtick if you want to handle only some fields).
Then you grep using $yesterday as your pattern through the log file.
The reason I think this is cool is because of the way you dump the problem onto bash date and perl, which handle dates robustly, so the problem is no longer yours.
Well, I thought it was frightfully neat. Now I'm off to pat my back.
PJ
Comments on this Entry
Posted by PJ_at_Belzabar_Software (122.162.xx.xx) on Thu 6 Sep 2007 at 10:29
[ Send Message | View Weblogs ]
[ Send Message | View Weblogs ]
Hmmm. More recent versions of GNU date have the ability to go back a date, like this:
date --date=yesterday
This doesn't seem to be documented on my u****u dapper system though.
date --date=yesterday
This doesn't seem to be documented on my u****u dapper system though.
[ Parent | Reply to this comment ]
it's not in the man page, but it is in the output of 'info date' on my debian etch machine. Did you check 'info date' on your ubuntu system?
[ Parent | Reply to this comment ]
Posted by PJ_at_Belzabar_Software (122.162.xx.xx) on Thu 6 Sep 2007 at 12:48
[ Send Message | View Weblogs ]
[ Send Message | View Weblogs ]
Ah. I should have realized that being a GNU utility it would be more up-to-date there in the info pages (like man pages for GNU utilities often remind us).
Yes, indeed, info date had it well-documented. An eye-opener on how clear and readable documentation can be actually - it makes me want to curl up and read it this evening. OK, so that's the geek in me ;-)
Other OSs may have non-GNU date utilities, so this little perl/bash snippet is probably still useful elsewhere too, seeing how ubiquitous perl/bash are.
Yes, indeed, info date had it well-documented. An eye-opener on how clear and readable documentation can be actually - it makes me want to curl up and read it this evening. OK, so that's the geek in me ;-)
Other OSs may have non-GNU date utilities, so this little perl/bash snippet is probably still useful elsewhere too, seeing how ubiquitous perl/bash are.
[ Parent | Reply to this comment ]
Posted by dkg (216.254.xx.xx) on Fri 7 Sep 2007 at 15:43
[ Send Message | View dkg's Scratchpad | View Weblogs ]
[ Send Message | View dkg's Scratchpad | View Weblogs ]
Even better, you can supply arbitrarily tough english to GNU's /bin/date:
[0 dkg@squeak ~]$ date --date='3 weeks ago next Wednesday' Wed Aug 22 00:00:00 EDT 2007 [0 dkg@squeak ~]$
[ Parent | Reply to this comment ]
If you have the good sense to rotate your log files with a log file rotation tool, like logrotate, you can just add the grep to the config files. Alternatively one could just use "logcheck" to grep the items of interest, which might find them more often, but that may not matter.
Only those daft enough [;-)] to invent their own log files, and own rotation method, have this specific problem (especially if they forgot to grep before they ran compress!). Although {Bourne|Korn|Posix|Bash} shell should have had date handling routines built in.
Only those daft enough [;-)] to invent their own log files, and own rotation method, have this specific problem (especially if they forgot to grep before they ran compress!). Although {Bourne|Korn|Posix|Bash} shell should have had date handling routines built in.
[ Parent | Reply to this comment ]
Posted by Anonymous (59.178.xx.xx) on Thu 6 Sep 2007 at 20:54
Heh. Actually it was a syslog file I was applying it to in the real case, before logrotate kicks in daily.
Thanks for reminding us about logcheck, which is the properly sophisticated way to do it (it detects anomalies).
For my purpose I wasn't interested in anomolies, but frequency of an entry yesterday.
PJ
Thanks for reminding us about logcheck, which is the properly sophisticated way to do it (it detects anomalies).
For my purpose I wasn't interested in anomolies, but frequency of an entry yesterday.
PJ
[ Parent | Reply to this comment ]