Ian Bicking: the old part of his blog

Good catch all exceptions

In Exception-based code antipatterns Max Ischenko argues that this kind of error checking is bad:

try:
    # do something
except Exception, e:
    # handle it

It's essentially equivalent to using except: without specifying any exception class at all, meaning that it will catch all exceptions. Generally it's better to only catch the exceptions you are expecting, e.g., KeyError or (IOError, OSError). I don't disagree with him, but there are a number of cases where this kind of exception handling is good, though often left out.

The most obvious case: it's not bad at all to catch all exceptions if you re-raise them. In Python you can re-raise the last exception by using raise without any arguments. Basically you are saying if (and only if) something goes wrong, do this, but still signal that something went wrong. This case shouldn't be confused with finally:, which indicates code that should always be called, error or no error. An example of except::

try:
    cursor.execute("INSERT INTO table VALUES (%s, %s)", (a, b))
    obj.setValues(a, b)
except:
    cursor.rollback()
    raise

And one for finally::

conn = pscyopg.connect('dbname=testdb')
try:
    run_transaction(conn)
finally:
    conn.close()

But there's other cases. Anytime you won't be there to babysit a process, you need to handle unexpected exceptions. For a command-line utility, you can read the exception when it occurs, no big deal. But for a long-running or batch process you need to intelligently deal with exceptions. Also for processes that are run by non-developers: you want to capture the error information so a developer can look at it, and then try to keep going if you can.

In these cases you usually want to be careful about putting in except: blocks like the cursor.rollback() example above -- these attempt to clean things up when something unexpected goes wrong, so that the process isn't left in an inconsistent state. Then at some fairly high level you actually handle (and don't re-raise) the exception. Most "frameworks" do this for you -- e.g., Tkinter, Webware, Zope, cgitb, etc. If you don't have a framework, you have to do it yourself. Here's a really simple handler that uses the traceback module:

def catch_exceptions(func, *args, **kw):
    try:
        # We could return the result of the function, but it's
        # a bad idea to make use of that return value since it's
        # undefined when there's an exception; it's better for the
        # function to set a flag or write to some persistent 
        # structure when it finishes successfully.
        func(*args, **kw)
    except Exception, e:
        f = open(log_file, 'a')
        f.write('-'*60 + '\n')
        traceback.print_exc(file=f)
        f.close()

The logging module also has an exception function just for this case.

I'm always very careful about situations where a traceback or error message goes to a black hole. Also, it's annoying for everyone when these error messages have to be handed back to developers from users; expecting a user to copy and email an error message from a web page is unreliable all around. Lastly, it's annoying to lose the result of a long process because you encounter an error. That came up recently with a coworker when he tried to use HTMLParser to spider web pages -- unfortunately HTMLParser doesn't work well on badly-formed HTML (I'd advise using BeautifulSoup, regular expressions, or running the code through tidy first). So the process would run for a while, then fail and die when it found a novel kind of malformed HTML. This

is the kind of unexpected exception that should be expected, and you should write your program knowing you haven't foreseen all the problems you'll encounter.

(BTW, I've been wanting a really good generic library to handle unexpected exceptions, but still haven't found it... anyone know of one?)

Created 25 Jan '05

Comments:

You wrote:

try:
    cursor.execute("INSERT INTO table VALUES (%s, %s)", (a, b))
    obj.setValues(a, b)
except:
    cursor.rollback()
    raise

Beware - there is a subtle bug here which cost me a few hours of debugging time. If the rollback raises an exception then the first exception is silently ignored. For example if the database connection has been lost (triggering the first exception) the cursor.rollback() will also fail, but all you will see is an exception arising from the rollback. Better to log something about the first exception before trying the rollback.

# Mark Russell

some usefull exception-handling routine here: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52215

# Denis

I often end up doing something like the following, especially while debugging/printlining:

try:
    munge(value)
except Exception, x:
    x.args += (value,)
    raise x

When used in a framework that logs exceptions, it's handy. Unfortunately, it loses any traceback from the munge() call on down. :(

# Robert Brewer

If you use just raise (instead of raise x) it should still work, but you won't lose the traceback (at least that's what my brief testing indicates).
# Ian Bicking

source
footnote: "except Exception" doesn't really catch all exceptions:

>>> class Foo:
...     pass
...
>>> try:
...     raise Foo
... except Exception, e:
...     print "got it"
...
Traceback (most recent call last):
  File "<stdin>", line 2, in ?
__main__.Foo: <__main__.Foo instance at 0x00B35FA8>
# Fredrik

Were people talking about fixing that, i.e., disallowing exceptions that weren't subclasses of Exception? (Except string exceptions, which are a separate special case.)
# Ian Bicking

source
In Python 3000, sure.  I don't see how you could possibly
change this before that...
# Fredrik

Really? What kind of crazy/annoying people make alternate exception hierarchies that don't subclass from Exception? There's no good reason for it, and it's really easy to fix if you have done it.
# Ian Bicking

source
> BTW, I've been wanting a really good generic library to handle unexpected exceptions

I've had really good luck with Mongoose (http://themongoose.sourceforge.net/).  You register exception handlers for uncaught 
exceptions.  It comes with one that does logging, one that emails, etc.  One other interesting feature is that it hashes the 
exception type, file name, function name, and other stuff to generate a "Mongoose Incident Identifier(TM)" so you 
(theoretically) have a unique identifier for each independent bug.
# Ian Bicking

Cool -- I hadn't seen Mongoose before, I'll definitely check it out.
# Ian Bicking

source
Catching all exceptions is bad, because it shows that you do not know how your program works. 
Any exception IS predictable, we are talking computer programs, not random static.

Sometimes you need a program to crash or properly raise exceptions. Reraising only works if 
the clean-up action doesn't raise any exceptions, as discussed earlier.  Here's the deal, 
only catch exceptions that you except. If an exception you don't expect is raised that 
means you have a bug, you fix the bug and add the proper handling of that exception.

I've seen many Java programmers do

try {
  yada
} catch (Exception e) {
  System.err.println("Error occurred: " + e.getMessage());
}

and this sucks TREMENDOUSLY if you are trying to debug a program. Sometimes you want your 
program to crash and burn, just to let you know that something is really wrong.
Even user input is predictable enough to write exact error handling.
Catching all exception may cure the symptoms, but not the cause; fix the cause instead.

Finally, it might be OK to catch all exceptions if it is imperitive that your 
process/thread do not stop. In that case, try catching the exception at the highest 
possible level and log with full stacktrace and give as much debug input as possible.
However, on many systems bugs might just as well end the process and send the stacktrace 
to stderr. I've seen too many bad things happen with processes that should have been 
terminated due to bugs, but poor programming let the processes continue doing bad things 
to the rest of the system.
# Ian Bicking

I was surprised to be ego-Googling for my Mongoose project a short time after there was some discussion about it! :-)

I'm intrigued to know who wrote "I've had really good luck with Mongoose" and what they've been using it for, as I wasn't aware of anyone who was actively using outside its original deployment. (I was confused because that comment appeared to be Ian replying to himself.)

I liked the Incident Identifier idea a lot and would be most interested in it being adopted by other similar projects.

# Phil

Sorry, I edited the comment (for formatting) and that messed up the attribution... maybe I have it back now. It was Denis Dube: http://msdl.cs.mcgill.ca/people/denis/

# Ian Bicking