(I was composing an e-mail to IMVU's engineering team when I realized this information was generally applicable to anyone embedding XULRunner into their application. Hope it's useful.)

XULRunner is written in a subset of C++ that we'll call XPCOM. An embedded XULRunner window communicates back to the host application through XPCOM interfaces that the host implements. In IMVU, we generally use C++ exceptions to signal failure. On the other hand, XPCOM uses nsresult error codes. Specifically, XULRunner is not written to support C++ exceptions, nor is it compiled with them enabled. (Note that compiling with exceptions enabled is not sufficient to guarantee defined behavior when they're thrown. You must use techniques like RAII to properly unwind and restore state if an exception is thrown.)

If XULRunner is calling into our code, and our code uses exceptions to signal failure, and throwing an exception through a XULRunner call stack results in undefined behavior, what do we do? This is the strategy I took:

In every method of every IMVU implementation of an XPCOM interface, I bracketed the function body with IMVU_BEGIN_DISALLOW_EXCEPTIONS_XPCOM and IMVU_END_DISALLOW_EXCEPTIONS_XPCOM. For example:

nsresult xpcom_method_runtime_error_with_error_info() {
    IMVU_BEGIN_DISALLOW_EXCEPTIONS_XPCOM {
        boost::throw_exception(std::runtime_error("error"));
    } IMVU_END_DISALLOW_EXCEPTIONS_XPCOM;
}

These two macros generate a try ... catch clause that handles every C++ exception thrown from the body, returning NS_ERROR_UNEXPECTED to the caller.

If the exception thrown is a Python error (boost::python::error_already_set), then the Python exception is pulled (PyErr_Fetch) and scheduled to be reraised (PyErr_Restore) in the next iteration through the IMVU client's message loop.

If the exception thrown is a C++ exception, we'd like to take the same approach. However, C++0x has not shipped, so there's no built-in mechanism for transferring exceptions across contexts. Thus, we take advantage of the boost::exception framework to copy and rethrow the exception from the main message loop. Unfortunately, you can't just "throw X()". You have to use boost::throw_exception, which enables the machinery for current_exception() and rethrow_exception(). To enforce this requirement, I have modified our C++ exception hierarchy so that you must use X::throw_(arguments) instead of throw X(arguments).

If the exception thrown is a C++ exception but not a subclass of std::exception, then we catch it with catch (...) or std::uncaught_exception() in a sentry object's destructor, and raise a structured exception to at least indicate that this is occurring in the field.

For reference, here is the implementation:

void handlePythonError();
void handleStandardException(const std::exception& e);

#define IMVU_BEGIN_DISALLOW_EXCEPTIONS         \
    DisallowExceptionsSentry PP_UNIQUE_NAME(); \
    try

#define IMVU_END_DISALLOW_EXCEPTIONS(block)                             \
    catch (const boost::python::error_already_set&) {               \
        handlePythonError();                                            \
        block ;                                                         \
    }                                                                   \
    catch (const std::exception& e) {                               \
        handleStandardException(e);                                     \
        block ;                                                         \
    }

#define IMVU_BEGIN_DISALLOW_EXCEPTIONS_XPCOM IMVU_BEGIN_DISALLOW_EXCEPTIONS
#define IMVU_END_DISALLOW_EXCEPTIONS_XPCOM IMVU_END_DISALLOW_EXCEPTIONS({ return NS_ERROR_UNEXPECTED; })

#define IMVU_DISALLOW_EXCEPTIONS_XPCOM(block)                           \
    IMVU_BEGIN_DISALLOW_EXCEPTIONS_XPCOM {                              \
        block ;                                                         \
    } IMVU_END_DISALLOW_EXCEPTIONS_XPCOM

And the source file:

DisallowExceptionsSentry::~DisallowExceptionsSentry() {
    if (std::uncaught_exception()) {
        RaiseException(EXC_EXCEPTIONS_NOT_ALLOWED, 0, 0, 0);
    }
}

/*
 * On error handling and the GIL.  Gecko's event handlers run during
 * the message loop, which means the GIL is not held.  Calls back into
 * Python require that the GIL be reacquired.  If the Python call
 * fails, the GIL is released (while error_already_set is unwinding
 * the stack).  The GIL must be reacquired to grab the exception
 * information and marshal it to the main thread.
 *
 * However, PumpWaitingMessages releases the GIL too!  Thus, reraising
 * the error on the main thread requires GIL reacquisition.
 *
 * If another thread acquires the GIL and blocks on the main thread's
 * message pump, a deadlock will occur.  Thus, secondary threads
 * should never block on the main thread's message pump.
 */

void reraisePythonError(PyObject* type, PyObject* value, PyObject* traceback) {
    HoldPythonGIL PP_UNIQUE_NAME();
    PyErr_Restore(type, value, traceback);
    boost::python::throw_error_already_set();
}

void error_but_no_error() {
    throw std::runtime_error("error_already_set but no Python error?");
}

void handlePythonError() {
    PyObject* type;
    PyObject* value;
    PyObject* traceback;
    {
        HoldPythonGIL PP_UNIQUE_NAME();
        PyErr_Fetch(&type, &value, &traceback);
    }
    if (type) {
        boost::function<void()> fn(boost::bind(reraisePythonError, type, value, traceback));
        queueMainThreadJob(fn);
    } else {
        queueMainThreadJob(error_but_no_error);
    }
}

void rethrowStandardException(const std::string& s) {
    std::string prefix("Unknown std::exception: ");
    throw std::runtime_error(prefix + s);
}

void handleStandardException(const std::exception& e) {
    if (boost::exception_detail::get_boost_exception(&e)) {
        queueMainThreadJob(boost::bind(
            boost::rethrow_exception,
            boost::current_exception()));
    } else {
        queueMainThreadJob(boost::bind(rethrowStandardException, std::string(e.what())));
    }
}