PEP 803: Position abi3t as a new variant; use Py_TARGET_ABI3T to select it by encukou · Pull Request #4747 · python/peps

encukou · 2025-12-17T14:44:37Z

Change is either:
- To a Draft PEP
- To an Accepted or Final PEP, with Steering Council approval
- To fix an editorial issue (markup, typo, link, header, etc)
PR title prefixed with PEP number (e.g. PEP 123: Summary of changes)

📚 Documentation preview 📚: https://pep-previews--4747.org.readthedocs.build/

…ct it

rgommers

Thanks for this update @encukou. I will comment on DPO with a few thoughts, but while reading thought it'd be useful to notice the textual little things that could be improved - hence this review.

peps/pep-0803.rst

Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>

encukou · 2026-01-22T14:45:20Z

Thank you!

encukou · 2026-02-12T14:20:26Z

From more discussion, a new knob called Py_TARGET_ABI3T looks best.

ngoldbaum · 2026-02-12T23:43:26Z

peps/pep-0803.rst

+compiled extension to support both at once, and thus be compatible with
+CPython 3.15+ (both free-threaded and GIL-eanbled builds).
+Initially, any extension compiled for ``abi3t`` will be compatible with
+``abi3`` as well.


I'm a little concerned that abi3t as you're proposing it doesn't include PyCriticalSection and maybe PyMutex.

See e.g. my comment here - in practice the restriction that abi3t must be a subset of abi3 means that getting extensions that already support the free-threaded build working on abi3t will be very challenging. CFFI uses PyMutex and critical sections. NumPy uses critical sections and PyMutex in its implementation and many libraries that depend on NumPy use critical sections via Cython. PyO3 exposes wrappers for PyMutex but uses native locks for the most part in its implementation. It does use critical sections in several places in its implementation.

All that to say - I really think an ABI that targets the free-threaded build needs at least PyCriticalSection. I think PyMutex can be replaced with native locks for the most part, but having PyMutex is often convenient.

What I think might make sense is to propose two new ABIs. What you're currently proposing could be called abi3o where o is for opaque. You can build abi3o extensions to support both the free-threaded build and the GIL-enabled build on 3.15 and newer, but you can't rely on the GIL or critical sections for thread safety. There are extensions like this, and if there was a target for that maybe people could update bindings generators to target it by adopting different locking strategies.

But, to get to a place where e.g. cryptography can ship O(1) wheels for Python 3.17, I think we do need an abi3t that looks more like the current free-threaded ABI. This could include an opauque PyObject too.

Another way we can fix this is to add PyMutex, PyCriticalSection, and PyCriticalSection2 to to the stable ABI. I guess you'd have to make operations that use PyCriticalSection no-ops on the GIL-enabled build since there's no per-object locks? But then it becomes much more straightforward to write extensions targeting both builds.

I just double-checked and at least for CFFI and NumPy it’s not a big deal for project that depend on them. The critical sections are all in internals.

That said, PyO3 doesn’t have as clear a separation between internals and generated extensions, so it is a problem there for extensions built with PyO3 on the free-threaded build.

Yeah. This PEP is at a lower level currently.
I fear that adding the PyCriticalSection discussions to this would send it spinning for another month or two. I'd rather keep them out of scope here.

Another way we can fix this is to add PyMutex, PyCriticalSection, and PyCriticalSection2 to to the stable ABI. I guess you'd have to make operations that use PyCriticalSection no-ops on the GIL-enabled build since there's no per-object locks? But then it becomes much more straightforward to write extensions targeting both builds.

That's the plan. No need to complicate things for build tools and such.

OK, fair enough. I hope the case for PyO3 will help convincing everyone it's worth doing this.
I can probably find more examples of load-bearing critical sections if it would help that discussion.

ngoldbaum

I left a few more comments after my main one.

For what it's worth, I have a branch of PyO3 that compiles with _Py_OPAQUE_PYOBJECT set on the GIL-enabled build. If I also define Py_GIL_DISABLED, then I hit issues in PyO3 internals because PyCriticalSection isn't exposed. I could stub out the critical section wrappers like they're currently stubbed on the GIL-enabled build, but then the resulting extensions won't be safe on the free-threaded build.

The tests run, but I get a segfault early on that I'm still debugging - it may be a CPython bug but it may also be a bug in how I set up _Py_OPAQUE_PYOBJECT support in PyO3. I'm meeting with David Hewitt tomorrow morning to figure that out.

ngoldbaum · 2026-02-12T23:44:57Z

peps/pep-0803.rst

+
+* ``Py_LIMITED_API=<version>`` (existing):
+  Compile for ``abi3`` of the given version.
+* ``Py_TARGET_ABI3T=<version>`` (proposed here):


This spelling works for me.

ngoldbaum · 2026-02-12T23:47:16Z

peps/pep-0803.rst

+These two macros are functionally very similar.
+In hindsight, ``Py_TARGET_ABI3`` (without the ``T``) would be a more fitting
+name for :c:macro:`Py_LIMITED_API`.
+We keep the existing name for backwards compatibility.


You could also introduce Py_TARGET_ABI3 as an alias and make the headers error out if you define both. IMO that would be nice for the symmetry. But I guess "there should preferably only be one way to do it"...

This isn't a big deal though and this spelling is fine.

We could error out if Py_TARGET_ABI3 is defined, and suggest the proper spelling.
I don't think the PEP needs to specify that though.

ngoldbaum · 2026-02-12T23:49:56Z

peps/pep-0803.rst

-  - :c:func:`Py_SIZE`
-  - :c:func:`Py_SET_SIZE`
+- :c:func:`Py_SIZE`
+- :c:func:`Py_SET_SIZE`


Isn't this missing Py_IS_TYPE?

Yeah, for completeness -- although the static inline definition Py_TYPE(o) == t is fine, since Py_TYPE is a function.

Suggested change

- :c:func:`Py_SET_SIZE`

- :c:func:`Py_SET_SIZE`

- :c:func:`Py_IS_TYPE`

ngoldbaum · 2026-02-12T23:57:45Z

peps/pep-0803.rst

-similar -- will be added via the C API working group, or in a follow-up PEP.
+Limited API to allow thread-safety without a GIL -- presumably ``PyMutex``,
+``PyCriticalSection``, and similar -- will be added via the C API working group,
+or in a follow-up PEP.


See my comment above about how I think this really needs to be resolved now, otherwise I think it will be difficult to get extensions working in-practice.

Now, yes. But I'd much prefer a separate discussion.

encukou · 2026-02-13T17:13:36Z

I plan to merge this early next week and update the discussion.
If you have a review in progress, let me know.

PEP 803: Position abi3t as a new variant; use Py_GIL_DISABLED to sele…

a10d589

…ct it

rgommers reviewed Jan 15, 2026

View reviewed changes

encukou commented Jan 22, 2026

View reviewed changes

peps/pep-0803.rst Show resolved Hide resolved

Apply suggestions from code review

50b48c6

Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>

encukou marked this pull request as ready for review January 22, 2026 14:56

encukou marked this pull request as draft January 23, 2026 10:14

encukou added 2 commits February 12, 2026 15:16

Add packaging discussion thread

b931652

PEP 803: Use an extra knob, Py_TARGET_ABI3T

c91aeac

encukou changed the title ~~PEP 803: Position abi3t as a new variant; use Py_GIL_DISABLED to select it~~ PEP 803: Position abi3t as a new variant; use Py_TARGET_ABI3T to select it Feb 12, 2026

encukou marked this pull request as ready for review February 12, 2026 14:20

Merge branch 'main' into 803-separate-abi3

c8c8d57

ngoldbaum reviewed Feb 12, 2026

View reviewed changes

ngoldbaum reviewed Feb 13, 2026

View reviewed changes

	- :c:func:`Py_SET_SIZE`
	- :c:func:`Py_SET_SIZE`
	- :c:func:`Py_IS_TYPE`

Uh oh!

Conversation

encukou commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

encukou commented Jan 22, 2026

Uh oh!

encukou commented Feb 12, 2026

Uh oh!

ngoldbaum Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

encukou commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

encukou commented Dec 17, 2025 •

edited

Loading

ngoldbaum Feb 12, 2026 •

edited

Loading