Tricky bugs and release libs

I’m wrapping up a bug right now that has to do with translating python exceptions into C++ exceptions. The translation occurrs as follows:

  • Catch boost::python::error_already_set
  • Import python modules which contain the exception types that I want to compare against
  • Compare the thrown exception with the imported types using PyErr_ExceptionMatches
  • Take appropriate actions (e.g. throw a C++ exception) based on the match

This technique looks something like this:

using namespace boost::python;

void main()
{
  Py_Initialize();

  try {
    do_something();
  } catch (const error_already_set&) {
    object exc = import("my_module").attr("MyException");
    if (PyErr_ExceptionMatches(exc.ptr()))
      throw MyException();
    throw;
  }

  ...
}

In the real code the import calls were only made the first time any translation was requested, but that that’s a minor detail.

Do you see the bug? If you do, I imagine that either this has happened to you before or you know a lot about the CPython’s implementation of module imports.

The import() call eventually calls PyImport_Import(), the recommended method for importing module programatically. This calls PyEval_GetGlobals() to get the global dict for the current execution frame. In my case, there apparently is no current execution frame, and this fact sends PyImport_Import down a branch where it calls…*drumroll*…PyErr_Clear().

What this means is that by the time I call PyErr_ExceptionMatches(), there is no active python exception. It has been silently erased. The simple (and, I think, correct) fix for this is to stash the exceptions before importing, import, and then restore the exceptions:

using namespace boost::python;

void main()
{
  Py_Initialize();

  try {
    do_something();
  } catch (const error_already_set&) {
    PyObject *e, *v, *t;
    PyErr_Fetch(&e, &v, &t);

    object exc = import("my_module").attr("MyException");

    PyErr_Restore(e,v,t);

    if (PyErr_ExceptionMatches(exc.ptr()))
      throw MyException();
    throw;
  }

  ...
}

In the absence of any commentary in the python source, this behavior is a bit puzzling. There may well be a good reason for it, but there is no indication anywhere that I’ve read that PyImport_Import() might clear your error variables. This bears some looking into.

It took me quite a while to track this bug down for two reasons. First, I made the assumption that the bug must exist in my code. I spent all sorts of time looking at things like destructors triggered by unwinding, implicit conversions, and any other crazy thing I could think of that might somehow be triggering clearing of the errors. The one basic fact I had was that a) an error_already_set was reaching my catch and b) when it arrived, there was no active python exception. Based on what I knew of boost::python and CPython, plus my faith in the two, it seemed wise to look for bugs in my code first.

Second, and more egregiously, I was debugging with release versions of boost and python. In hindsight, this fact cost me more time than anything else. I say this because, as soon as I got debugging versions (unoptimized and with symbols), the bug had no place to hide and was easily found. A few well-placed breakpoints plus some stepping through the translation code almost immediately shined a light on boost::python::import(). After that, it was only a few more steps to PyImport_Import() and, ultimately, PyErr_Clear().

So, lesson learned.

Advertisements

,

  1. Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: