Ian Bicking: the old part of his blog


I tried my hand at log analysis, but finally figured that's not where it's at for the more advanced stuff.

I'd suggest looking at how phpOpenTracker uses images/javascript to get the necessary information to generate (incredibly slowly, at least on one of my previous company's about-a-million-of-users-a-month web sites) a lot of the information you'd likely need. I'm not sure if phpOpenTracker is the one that does it (I don't think it is), but there's a tracker which uses the same trick, and you can dynamically replay "click-trails" for individual users and so forth, and generates "average stay per page", and "common paths" and so forth. Now if I could only recall the name...

A nice-ish bonus is that the images/javascript has a very good chance of not picking up bots. And with an additional white-list, you might still be able to pick up Lynx/Links/w3m users. I spend way too much time removing bots - was considering going white-list only at one stage, but lots of the bots fake User Agents to mimic browsers. And doing additional analysis to pick up bots at log-file-analysis time is quite hard.
Comment on Where are the log analyzers?
by Neil Blakey-Milner