How Python's Internals Became Part of Its Interface

Python makes a + b look simple, but what must an interpreter do to make it work for numbers, strings and user-defined objects? In this 2016 talk, Flask creator Armin Ronacher argues that CPython's internal decisions have become visible enough for programs to depend on them. That visibility gives developers unusual power, but it also raises the cost of changing the interpreter. His examples describe the Python implementations of his time; they are not a specification of current Python internals.

A common explanation says that a + b is equivalent to a.__add__(b). Ronacher traces the operator through the CPython implementation he was discussing: bytecode reaches an addition operation with fast paths for exact integers and strings, then general numeric handling and, if needed, sequence concatenation. Despite its name, PyNumber_Add also participates in adding sequences. The operand types and their relationship affect dispatch. A direct special-method lookup takes a different path and, he says, was sometimes faster in particular cases—not invariably so. Watch the addition walkthrough.

06:40View full image

A slot here is a location in a C type's structure holding an operation pointer, such as numeric addition; it is not Python's separate __slots__ feature for instance attributes. An object points to its type, and the interpreter can use that type's slots to carry out an operation. This route makes the familiar shorthand useful as an introduction, but incomplete as an account of the implementation.

Ronacher recounts built-in C types and Python classes as initially separate mechanisms, while acknowledging uncertainty about the exact chronology. C types supplied operation slots; Python classes supplied special methods such as __add__. The interpreter bridges them: wrappers expose C slots as Python-visible methods, and Python-defined special methods are connected back to type slots. Separate numeric and sequence slots make the mapping more complicated than one name suggests.

12:30View full image

This matters beyond a diagram of implementation: extension authors use the C API and Python programs can observe slot wrappers. A Python-defined function and a C slot wrapper are visibly different objects. Ronacher also points to descriptor binding: a Python method may bind an instance as self, while a class attribute such as conversion = int relies on int not binding in that way. Code can rely on such differences, so an alternative implementation must reproduce behavior that once looked incidental. Watch the type-slot explanation.

One historical example was a compiled re object whose C implementation could support operations while ordinary Python attribute access did not expose expected attributes such as __class__ and __repr__. Ronacher says the example had been fixed by Python 2.7. It illustrates his point about the older C API, not a recipe for reproducing the behavior today.

22:30View full image

He also describes a Python 2 old-style proxy that forwarded attribute access to an underlying value, including special operations; newer-style classes use different special-method lookup. These examples show why replacing one mechanism with a tidier model can change observable behavior. They also show why a second implementation, such as PyPy, faces more than the task of implementing the tutorial explanation. Watch the examples.

24:10View full image

Early byte strings encouraged programmers to expect direct access to positions in a string. Ronacher describes how Unicode support inherited that expectation while characters could require larger values. In his Python 2 example, a narrow build could expose an emoji as two surrogate units; a wide build used four-byte units. UTF-8 instead uses variable-length encodings, which can save space for common text but make finding an arbitrary character position require more work than simply stepping through fixed-width units.

He says the Python 3 implementation he is discussing chooses a storage width according to the largest character in a string. His account of Unicode history and efficiency is a simplified teaching argument, and indexing a code point is not necessarily the same as selecting a user-perceived character. The broader point is the lasting design pressure created when an implementation's behavior becomes an expectation. Watch the Unicode discussion.

The same exposure can be valuable. A frame represents an active call and provides access to information about its execution. Ronacher describes sys._getframe as a way to inspect call-site state, including local variables, filenames and exception information, with some ability in certain circumstances to change locals. He argues that widespread use encouraged the implementation to support it despite its costs.

35:50View full image

His examples include interface machinery that examines a caller, warnings attributed several frames up, and logging that identifies the caller rather than the logging helper. He also describes attaching a debugger to a running interpreter and using traceback and local state in Sentry, Django and Flask. Those are examples of what access can enable, not a promise that a live debugger is available in every deployment. Watch the frame examples.

sys.modules exposes a cache of imported modules by name. That makes a loaded module discoverable: code can look up a module and find a class in it. Ronacher points to pickle, copying and custom import code as consumers of this shared state. Pickle serializes Python objects, and persisted serialized data can carry dependencies across a long period.

In the system he describes, loading two versions of a library under the same name is difficult because consumers expect that name to identify a particular cached module. A version or scope in the lookup would change assumptions already embedded in code and saved data. This is his historical analysis of a compatibility problem, not a claim that all forms of isolation or multiple-version loading are impossible today. Watch the imports discussion.

Ronacher describes a then-current CPython design with built-in static type objects at global addresses. An exact-type check could compare an object's type pointer with the address of the built-in integer type. Those shared objects also had mutable reference counts. In his attempt to make independent interpreters run in separate threads, he encountered dependencies on those objects and on exposed C-level reference counting.

41:40View full image

He connects those constraints to the difficulty of changing the global interpreter lock (GIL) and isolating interpreters, and says that everyday Python work still remains possible despite performance and concurrency limitations. Treat these as the speaker's 2016 assessment and experience, not as a current guarantee about the GIL, subinterpreters or available Python builds. Watch the isolation discussion.

In the audience discussion, an attendee suggests a narrow public API. Ronacher agrees that it could make evolution easier, while acknowledging that developers would lose some of the introspection he values. Asked whether special cases are needed for speed, he proposes that simpler, smaller hot paths could improve cache locality and branch prediction, potentially helping uncommon cases too. That is his design hypothesis in the discussion, not a benchmark established by the talk.

The lesson is to examine both sides of an interface boundary: what useful tools does exposure enable, and which observed details might users come to rely on? The answer is a tradeoff rather than a reason to dismiss either introspection or compatibility. Watch the questions.