Custom tracebacks in Python continued

This is just a quick update to my previous blogpost about tracebacks in Python since there hasn't been much news on that topic.

Limitations on frame location/function name

The main problem with the solution in the previous post was that location, the functionname for each frame, had to be a valid Python identifier in order to be able to construct an artificial frame/traceback. Jinja creates frames in a way that fixes this problem: Instead of creating function definitions and eval-ing them, it just creates codeobjects. The following code is Python 3 only:

>>> from types import CodeType
>>> filename = 'foo.rs'
>>> location = 'My::Rust::Type::<with_crazy_punctuation>'
>>> code = compile('1/0', filename, 'exec')
>>> code = CodeType(0, code.co_kwonlyargcount,
...     code.co_nlocals, code.co_stacksize,
...     code.co_flags, code.co_code, code.co_consts,
...     code.co_names, code.co_varnames, filename,
...     location, code.co_firstlineno,
...     code.co_lnotab, (), ())
>>> namespace = {}
>>> exec(code, namespace)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "foo.rs", line 1, in My::Rust::Type::<with_extreme_punctuation>
ZeroDivisionError: division by zero
>>>

(The code works almost the same in Python 2, it's just that the arguments to CodeType have changed since then)

So My::Rust::Type::<with_extreme_punctuation> is obviously not a valid Python identifier (def My::Rust::Type::<with_extreme_punctuation>(): pass is invalid syntax), but we still somehow got this traceback with that identifier it.

I implemented this approach in Shippai, the other larger, but probably less significant change I did was to switch from code objects calling from each other to code objects re-raising the same exception with the same traceback over and over again, but with an additional frame each time. If I want to emulate foo() calling bar(), instead of creating two such code objects that call each other, I do the following:

  1. I create a code object called bar which raises an exception.
  2. I exec it and catch the exception plus traceback: exc_info = sys.exc_info()
  3. I pass exc_info into a new code object called foo, which contains raise exc_info[1].with_traceback(exc_info[2]). The old traceback is reused, but since the exception happens inside the code object, the frame for the code object is added on top as well.

There is no advantage to doing this, but you do save a few lines of code because you only have to generate one type of code object: One that takes an exception and a traceback and reraises it. The traceback can be None at the beginning.

As opposed to one code object that raises an exception and one that calls other code objects.

PDB

If you step through those kind of frames with pdb, the experience is not very great. How can it be? You can only ever access the variables in the namespace dict you passed to exec using pdb, yet see a lot of sourcecode that looks very different from what is actually available.

Jinja is doing fine here because it actually does have local Python variables it can just pass into the artificial frame to make them available to pdb.

Python 3.7 and custom tracebacks

I just recently found out that Python 3.7 lets you create traceback objects... to some degree:

Changed in version 3.7: Traceback objects can now be explicitly instantiated from Python code, and the tb_next attribute of existing instances can be updated.

(Source)

This means we can remove any frame from any traceback:

>>> def baz(): 1/0
...
>>> def bar(): baz()
...
>>> def foo(): bar()
...
>>> import sys
>>> try:
...     foo()
... except:
...     type, exc, tb = sys.exc_info()
...
>>> raise exc.with_traceback(tb)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 1, in foo
  File "<stdin>", line 1, in bar
  File "<stdin>", line 1, in baz
ZeroDivisionError: division by zero
>>> tb.tb_next.tb_next = tb.tb_next.tb_next.tb_next
>>> raise exc.with_traceback(tb)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 1, in foo
  File "<stdin>", line 1, in baz
ZeroDivisionError: division by zero

We just cut out a frame in the middle of a traceback in Python. We could have done that before Python 3.7... by creating the entire traceback with compile and exec.

That's the only obvious usecase to me. The docs say you can create traceback objects too, but you can't create a frame object, which is required by Traceback.__init__. Meaning you will probably have to fall back to a eval/exec-based solution to create at leas the frame.

Removing frames from the inside

This one is a mystery to me which is why I am just going to assume it's unsolvable. How do I raise an exception with a permanently empty traceback?

Assume I have this code:

>>> def foo(): 1/0
...
>>> foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in foo
ZeroDivisionError: division by zero

Now I know how to strip frames if I caught that traceback from outside of foo. But I want to remove the frames at a completely different point: I want to do some magic inside of foo such that the example looks like this:

>>> def foo(): <magic exception raising>
...
>>> foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in foo
ZeroDivisionError: division by zero

Or even present a completely empty traceback to the user. Effectively I want to prevent the Python interpreter from adding frames while unwinding the stack.

My usecase for this would be to remove some annoying ugly middle frames from the exception my own library is raising while being called by somebody else's code.

My "workaround" for this is that I add __tracebackhide__ = True to those annoying frames. Some tools such as py.test or Sentry check for this local variable to determine whether they should skip this frame while formatting the traceback. But that doesn't really produce a nice traceback object but rather just hooks into the formatting logic to hide from the user how ugly the traceback really is. It's similar to overriding sys.excepthook in that regard.