So, now that I'm using Selenium (well, planning to), I'm thinking about our all-around testing system. There's unit testing -- but that's not web-specific so I won't talk about that here.

Selenium seems good for acceptance and cross-browser testing. Hopefully with the right toolset even non-programmers can start building and running testing scripts on applications.

There's also a "Driven" mode for Selenium -- basically a process drives the Selenium Javascript code. I'm not 100% clear on how the communication is done, but anyway. If your Selenium test looks like:

Then your drive code might look like:

selenium.open('http://myhost.com')
selenium.click("//a[text()='Next']")
selenium.verifyTextPresent('Page 2')

And you could put that in unittest-based tests or whatever. I'm not that excited about that -- driving browsers on an unattended basis seems like a bit of a pain. Which perhaps is where Twill fits in -- in fact, Twill could even take something looking very much like Seleniums commands, and provide extra value like HTML validation, link checking, performance testing, etc., all in a more controlled environment. More than just a place for both, I see them working in concert.

But two testing strategies don't provide a tripod of stability, and metaphorical stability is top on my list.

So the third testing strategy is to make sure the applications are happily humming away when they are live and no one is looking at them. I don't think it's reasonable to run acceptance tests on live applications -- too much valuable state. So instead I want to start adding self-diagnostic code to those applications. This code will check database connections, make sure files are all in place, etc. Even just a "yes I'm alive" response is a big improvement over nothing. Each of these will be a page, in an out-of-the-way location.

Then we point a monitoring daemon at it. I like the look of Mon, but our sysadmin has experience with Big Brother. So I'm not sure exactly what we'll use (opinions welcome). Either way, I want all of our applications to be pinged on a regular basis, with notification of problems. This will turn into a total system check -- if the network is down, if Apache is down, or if our application server is down, we'll get a report.

But wait, that is not all! No tripod is complete without a fourth leg! The last part is handling all exceptions, no matter who triggers them. Webware can email all exceptions to you -- nice, and easy to configure. We have something ad hoc that does the same in Zope -- a bit ugly and not so easy (what's up with that?!), but it's there. I'm working on a refinement here as well; it won't apply to every system we have -- it can't be in place on a PHP application, for instance -- but it's an exception reporter that can be used outside of a web context, while providing all the contextual information that both Webware and Zope's error reports have. It can also work on background tasks, like cron jobs and scheduled tasks. Extra information (beyond the normal traceback) is important for post-mortem debugging, and so that no error -- no matter how small! -- will go forgotten.