Re: analysis of java application logs
On 23.05.2011 15:17, Patricia Shanahan wrote:
On 5/23/2011 12:50 AM, Ulrich Scholz wrote:
I'm looking for an approach to the problem of analyzing application
log files.
I need to analyse Java log files from applications (i.e., not logs of
web servers). These logs contain Java exceptions, thread dumps, and
free-form log4j messages issued by log statements inserted by
programmers during development. Right now, these man-made log entries
do not have any specific format.
What I'm looking for is a tool and/or strategy that supports in lexing/
parsing, tagging, and analysing the log entries. Because there is only
little defined syntax and grammar - and because you might not know
what you are looking for - the task requires the quick issuing of
queries against the log data base. Some sort of visualization would be
nice, too.
Pointers to existing tools and approaches as well as appropriate tools/
algorithms to develop the required system would be welcome.
I would use Perl, and begin by recognizing some of the more important
formats, such as thread dumps. I agree with the desirability of
introducing some organized formatting into the log messages, but an
ad-hoc Perl program can often get useful data out of a disorganized log.
Only that Perl is so awful - YMMV of course. But for these kinds of
tasks (more correctly: for *any* task) I very much prefer to use Ruby
because of its cleaner OO and cleaner syntax. In these cases where the
basic format is fixed I place general parsing code in a library (a
single file really) and then I can write ad hoc scripts which do
arbitrary processing of the data. That's very productive.
Kind regards
robert
--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/