On 23/05/2011 15:11, Lew allegedly wrote:
Ulrich Scholz wrote:
I'm looking for an approach to the problem of analyzing application
log files.
I need to analyse Java log files from applications (i.e., not logs of
web servers). These logs contain Java exceptions, thread dumps, and
free-form log4j messages issued by log statements inserted by
programmers during development. Right now, these man-made log entries
do not have any specific format.
What I'm looking for is a tool and/or strategy that supports in lexing/
parsing, tagging, and analysing the log entries. Because there is only
little defined syntax and grammar - and because you might not know
what you are looking for - the task requires the quick issuing of
queries against the log data base. Some sort of visualization would be
nice, too.
Pointers to existing tools and approaches as well as appropriate tools/
algorithms to develop the required system would be welcome.
It helps if you have a logging strategy that mandates a consistent
logging format, specific information in particular positions or marked
by particular markup, logging levels and other such so that your
analysis tool isn't faced with a completely open-ended input. What you
describe requires a general text-analysis approach, as you indicate that
you can make no guarantees about the format. Based on that, your best
tool is "less" or equivalent text-file reader.
What is a tool supposed to do, read your mind?
It's really hard to extract information from a garbage can where people
just randomly dumped whatever they individually felt like dumping
without regard for operational needs. You can't build a skyscraper on a
bad foundation, and you can't build a good log analysis off a crappy log.
Fix the logging system, then the analysis problem will be tractable.
I would argue around the same lines.
I've been faced a while ago with a situation where some orthogonal
organisational unit wanted to exploit my logs. I told them to GTFO.
My logs are my logs. I put in it what I consider necessary. I often
improve them as I step through the code. I might change the message, fix
the level, &c. I don't want to have them set in stone. Neither do I
generally have enough confidence in them to allow them to be used for
analysis.
"The solution, then, is simple", I told them, "spec out the exact
messages and arguments you want, and the exact situations you want them
logged in, and I'll add them for you. But leave me my precious debugging
logs."
Let me emphasize: IMHO debugging logs and logs for analysis are two
different things and should be kept strictly separated -- possibly
logged to a different target respectively.
That last is rather a brilliant idea, to use different targets. Heretofore
support the former. The problem I've always seen is that logging statements
Honi soit qui mal y pense.