Extending opaque types

In the Limited API, most structs are opaque: their size and memory layout are not exposed, so they can be changed in new versions of CPython (or alternate implementations of the C API).

This means that the usual subclassing pattern – making the struct used for instances of the base type be the first element of the struct used for instances of the derived type – does not work. To illustrate with code, the example from the tutorial extends PyListObject (list) using the following struct:

typedef struct {
    PyListObject list;
    int state;
} SubListObject;

This won’t compile in the Limited API, since PyListObject is opaque (to allow changes as features and optimizations are implemented).

Instead, this PEP proposes using a struct with only the state needed in the subclass, that is:

typedef struct {
    int state;
} SubListState;

// (or just `typedef int SubListState;` in this case)

The subclass can now be completely decoupled from the memory layout (and size) of the superclass.

This is possible today. To use such a struct:

• when creating the class, use PyListObject->tp_basicsize + sizeof(SubListState) as PyType_Spec.basicsize;
• when accessing the data, use PyListObject->tp_basicsize as the offset into the instance (PyObject*).

However, this has disadvantages:

• The base’s basicsize may not be properly aligned, causing issues on some architectures if not mitigated. (These issues can be particularly nasty if alignment changes in a new release.)
• PyTypeObject.tp_basicsize is not exposed in the Limited API, so extensions that support Limited API need to use PyObject_GetAttrString(obj, "__basicsize__"). This is cumbersome, and unsafe in edge cases (the Python attribute can be overridden).
• Variable-size objects are not handled (see Extending variable-size objects below).

To make this easy (and even best practice for projects that choose loose coupling over maximum performance), this PEP proposes an API to:

1. During class creation, specify that SubListState should be “appended” to PyListObject, without passing any additional details about list. (The interpreter itself gets all necessary info, like tp_basicsize, from the base).

   This will be specified by a negative PyType_Spec.basicsize: -sizeof(SubListState).

2. Given an instance, and the subclass PyTypeObject*, get a pointer to the SubListState. A new function, PyObject_GetTypeData, will be added for this.

The base class is not limited to PyListObject, of course: it can be used to extend any base class whose instance struct is opaque, unstable across releases, or not exposed at all – including type (PyHeapTypeObject) or third-party extensions (for example, NumPy arrays [1]).

For cases where no additional state is needed, a zero basicsize will be allowed: in that case, the base’s tp_basicsize will be inherited. (This currently works, but lacks explicit documentation and tests.)

The tp_basicsize of the new class will be set to the computed total size, so code that inspects classes will continue working as before.

Extending variable-size objects

Additional considerations are needed to subclass variable-sized objects while maintaining loose coupling: the variable-sized data can collide with subclass data (SubListState in the example above).

Currently, CPython doesn’t provide a way to prevent such collisions. So, the proposed mechanism of extending opaque classes (negative base->tp_itemsize) will fail by default.

We could stop there, but since the motivating type — PyHeapTypeObject — is variable sized, we need a safe way to allow subclassing it. A bit of background first:

PyTupleObject:
┌───────────────────┬───┬───┬╌╌╌╌┐
│ PyObject_VAR_HEAD │var. data   │
└───────────────────┴───┴───┴╌╌╌╌┘

tuple subclass:
┌───────────────────┬───┬───┬╌╌╌╌┬─────────────┐
│ PyObject_VAR_HEAD │var. data   │subclass data│
└───────────────────┴───┴───┴╌╌╌╌┴─────────────┘

heap type:
┌───────────────────┬──────────────┬───┬───┬╌╌╌╌┐
│ PyObject_VAR_HEAD │Heap type data│var. data   │
└───────────────────┴──────────────┴───┴───┴╌╌╌╌┘

type subclass:
┌───────────────────┬──────────────┬─────────────┬───┬───┬╌╌╌╌┐
│ PyObject_VAR_HEAD │Heap type data│subclass data│var. data   │
└───────────────────┴──────────────┴─────────────┴───┴───┴╌╌╌╌┘

Relative member offsets

One more piece of the puzzle is PyMemberDef.offset. Extensions that use a subclass-specific struct (SubListState above) will get a way to specify “relative” offsets (offsets based from this struct) rather than “absolute” ones (based off the PyObject struct).

One way to do it would be to automatically assume “relative” offsets when creating a class using the new API. However, this implicit assumption would be too surprising.

To be more explicit, this PEP proposes a new flag for “relative” offsets. At least initially, this flag will serve only as a check against misuse (and a hint for reviewers). It must be present if used with the new API, and must not be used otherwise.

Relative basicsize

The basicsize member of PyType_Spec will be allowed to be zero or negative. In that case, its absolute value will specify how much extra storage space instances of the new class require, in addition to the basicsize of the base class. That is, the basicsize of the resulting class will be:

type->tp_basicsize = _align(base->tp_basicsize) + _align(-spec->basicsize);

where _align rounds up to a multiple of alignof(max_align_t).

When spec->basicsize is zero, basicsize will be inherited directly instead, i.e. set to base->tp_basicsize without aligning. (This already works; explicit tests and documentation will be added.)

On an instance, the memory area specific to a subclass – that is, the “extra space” that subclass reserves in addition its base – will be available through a new function, PyObject_GetTypeData. In CPython, this function will be defined as:

void *
PyObject_GetTypeData(PyObject *obj, PyTypeObject *cls) {
    return (char *)obj + _align(cls->tp_base->tp_basicsize);
}

Another function will be added to retrieve the size of this memory area:

Py_ssize_t
PyType_GetTypeDataSize(PyTypeObject *cls) {
    return cls->tp_basicsize - _align(cls->tp_base->tp_basicsize);
}

The result may be higher than requested by -basicsize. It is safe to use all of it (e.g. with memset).

The new *Get* functions come with an important caveat, which will be pointed out in documentation: They may only be used for classes created using negative PyType_Spec.basicsize. For other classes, their behavior is undefined. (Note that this allows the above code to assume cls->tp_base is not NULL.)

Inheriting itemsize

When spec->itemsize is zero, tp_itemsize will be inherited from the base. (This already works; explicit tests and documentation will be added.)

A new type flag, Py_TPFLAGS_ITEMS_AT_END, will be added. This flag can only be set on types with non-zero tp_itemsize. It indicates that the variable-sized portion of an instance is stored at the end of the instance’s memory.

The default metatype (PyType_Type) will set this flag.

A new function, PyObject_GetItemData, will be added to access the memory reserved for variable-sized content of types with the new flag. In CPython it will be defined as:

void *
PyObject_GetItemData(PyObject *obj) {
    if (!PyType_HasFeature(Py_TYPE(obj), Py_TPFLAGS_ITEMS_AT_END) {
        <fail with TypeError>
    }
    return (char *)obj + Py_TYPE(obj)->tp_basicsize;
}

This function will initially not be added to the Limited API.

Extending a class with positive base->itemsize using negative spec->basicsize will fail unless Py_TPFLAGS_ITEMS_AT_END is set, either on the base or in spec->flags. (See Extending variable-size objects for a full explanation.)

Extending a class with positive spec->itemsize using negative spec->basicsize will fail.

Relative member offsets

In types defined using negative PyType_Spec.basicsize, the offsets of members defined via Py_tp_members must be relative to the extra subclass data, rather than the full PyObject struct. This will be indicated by a new flag in PyMemberDef.flags: Py_RELATIVE_OFFSET.

In the initial implementation, the new flag will be redundant. It only serves to make the offset’s changed meaning clear, and to help avoid mistakes. It will be an error to not use Py_RELATIVE_OFFSET with negative basicsize, and it will be an error to use it in any other context (i.e. direct or indirect calls to PyDescr_NewMember, PyMember_GetOne, PyMember_SetOne).

CPython will adjust the offset and clear the Py_RELATIVE_OFFSET flag when initializing a type. This means that:

• the created type’s tp_members will not match the input definition’s Py_tp_members slot, and
• any code that reads tp_members will not need to handle the flag.

Alignment & Performance

The proposed implementation may waste some space if instance structs need smaller alignment than alignof(max_align_t). Also, dealing with alignment makes the calculation slower than it could be if we could rely on base->tp_basicsize being properly aligned for the subtype.

In other words, the proposed implementation focuses on safety and ease of use, and trades space and time for it. If it turns out that this is a problem, the implementation can be adjusted without breaking the API:

• The offset to the type-specific buffer can be stored, so PyObject_GetTypeData effectively becomes (char *)obj + cls->ht_typedataoffset, possibly speeding things up at the cost of an extra pointer in the class.
• Then, a new PyType_Slot can specify the desired alignment, to reduce space requirements for instances.

Other layouts for variable-size types

A flag like Py_TPFLAGS_ITEMS_AT_END could be added to signal the “tuple-like” layout described in Extending variable-size objects, and all mechanisms this PEP proposes could be adapted to support it. Other layouts could be added as well. However, it seems there’d be very little practical benefit, so it’s just a theoretical possibility.