Release CFFI 2.0.0

[ngoldbaum]

I'm opening this now since 3.14.0rc2 came out last week. We should have one more release candidate in September and then the final release in October. It would be nice to have a CFFI release available sometime in September, since that will unblock work in reverse dependencies of cryptography and pynacl.

I'll defer to @nitzmahone on what the broader blockers on a final release are, but here's what I think needs to happen:

• Ensure manylinux2014 wheels are built: Explicitly specify manylinux2014 in wheel building config #184
• Add a top-level thread safety page to the docs Add CFFI thread safety docs #188
• Clean up the Windows testing situation. Fix this commented out line and make this step unconditionally run the same tests on all operating systems. Enable more Windows pytest-run-parallel CI #189 Simplify tests, shorten buildir name on windows #137.

Anything else?

#126 alludes to a need for more multithreaded testing and better multithreaded test coverage. I'd appreciate more detail on what @nitzmahone had in mind there if he considers it a blocker.

I'd also like to make the Py_mod_gil setting configurable. Right now we unconditionally set it to Py_MOD_GIL_NOT_USED, which is true of the wrapper, but not necessarily the wrapped C code. This requires spitballing an API to control that at build time. I don't think this is a blocker since it can be worked around but I'm also happy to work on this now if you'd like.

I'm motivated to work on this but I'm going to need a little guidance on some of these and on whether there are any other blockers. For now I'm going to work on the thread safety docs and the windows testing situation.

[nitzmahone]

I'd be thrilled if all of that was in for a September 2.0.0 - I probably won't have a ton of time to pitch in on implementation before then, but will definitely commit to reviews and release work as needed.

I had similar thoughts about coming up with a nicer way to expose a module init opt in/out for GIL behavior, but didn't want to hold up the initial pre-release trying to design it. IIRC I successfully hacked together a throwaway attempt with a completely manual call to it from a user CFFI module back in a 3.13 pre-release, but it feels like something that should be a little more declarative and less "everyone needs to reinvent this wheel".

I've got mixed feelings on iOS wheels- I haven't spent any time poking at the PR that was opened for it, but if cibuildwheel supports it with near-native performance on a pretty vanilla GHA Mac aarch64 runner (including actually running at least part of the test suite, not just a YOLO cross-compile as many of the early MacOS aarch64 things were doing), I'm okay with it. If it's going to require a bunch of non-standard tooling/config/setup/emulation though, or will result in completely untested artifacts, I'd rather defer that until it doesn't.

I don't think significant new multi-threaded/free-threaded test coverage needs to be a blocker for a 2.0.0. I've wanted to build a stress test around thread-agile generation, use, and collection of ephemeral types to make sure they're all properly cleaned up, and thread-agile handoffs and collection of CFFI wrapper objects to ensure the refcounting is sound. We're getting a little of that for free by running the existing test suite multi-threaded, but I wouldn't call what I remember seeing a stress test by any means.

[ngoldbaum]

Thanks for the quick feedback! I removed the iOS wheels checkbox since it's not a blocker.

[ngoldbaum]

I opened #188 to add some new thread safety documentation.

[ngoldbaum]

I opened #189, but I think re-enabling full Windows CI is going to need something along the lines of #137. @mattip do you think we can resurrect #137 or does that approach have issues?

[mattip]

I think we can resurrect #137. The approach is sound, the devil is in the small details. Some documentation around how tests are structured might help, i.e. there is this comment

cffi0 checks the verify() call that was in cffi 0.x.y, while cffi1 checks the new compile() call.

and there is that thing about cleaning out the test directories that seems fragile

[nitzmahone]

I'm moving house for the next few days, but will try to eyeball these and provide feedback as soon as I can.

[ngoldbaum]

In the meantime I'm going to try to add a way to declare that a module is thread-unsafe when constructing bindings. We could also make it so you have to declare the module to be explicitly thread-safe (following CPython). That'll break people who are using the beta and opted to release free-threaded wheels, which is a little unfortunate, but may be the best if we want to force people to force people to think a little bit about thread safety.

[arigo]

Sorry if I missed that, but can you explain why thread-safety is an issue for the wrapped C code? More precisely, what is the condition for a wrapped C code to behave differently, thread-safety-wise, in the GIL vs non-GIL case? In other words: what is the use case for a way to tell cffi to not include Py_MOD_GIL_NOT_USED ?

[ngoldbaum]

I was thinking about wrapping a thread-safe C library versus wrapping a thread-unsafe library. But you're right, the GIL only provides a very weak form of safety when wrapping a thread-unsafe C library by limiting parallelism. The GIL is only locking CPython internals, not the internals of some other library.

So, maybe you're right and it is correct to always say all CFFI-generated extensions do not rely on the GIL for thread safety. It's just not a very strong statement, since the library may or may not necessarily be thread-safe out of the box. Whether or not the library is thread safe depends on the Python wrapping code and the thread safety guarantees of the wrapped C code.

[mattip]

The python wrapping code generated by cffi should be thread safe, no? Are there ways cffi can generate thread-unsafe wrapping code? Or maybe you meant some other python code?

[arigo]

cffi always calls C code with the GIL released. That has been true since forever, and actually I've resisted a few requests to optionally not release the GIL --- precisely because it would add problems in some cases including the one we're in right now :)

[ngoldbaum]

The python wrapping code generated by cffi should be thread safe, no? Are there ways cffi can generate thread-unsafe wrapping code? Or maybe you meant some other python code?

I was thinking about the C code getting wrapped by CFFI. But @arigo is right, the GIL never protected that.