Repository navigation
gh-150865: Add PyUnstable_Immortalize - #160138
ZeroIntensity wants to merge 9 commits into
Conversation
Documentation build overview
11 files changed ·
|
It's really slow under TSan so it times out.
|
Hi @ZeroIntensity thank you for the PR. Immortal types are fixing the free-threaded slowdown for both PRs numpy/numpy/pull/32853 and numpy/numpy/pull/32552 about as well as the Benchmarks: numpy/numpy/pull/32853 types (
numpy/numpy/pull/32552 types (neighborhood iterator, MapIter), free-threaded, million calls/s**
GIL build, one thread, ns per call (lower is better)
|
|
@ZeroIntensity wasn't there another approach you were thinking about for this? |
|
Per our discussion during the sprint, I think we reached a consensus that NumPy can use static types for things it would otherwise immortalize, right? As far as I can tell, it should be possible to share third-party static types across interpreters; the only requirement for subinterpreters is to use multi-phase initialization, which doesn't disallow static types. (If you find bugs with that, we should fix those!) |
|
@prathamhole14 @mattip does that work for you two? |
|
As far as I know, static types are incompatible with the limited API. If CPython wishes to allow c-extension authors to support subinterpreters, free-threading, and the limited API in a performant way, I think we need to be able to declare heap types immutable and immortal. |
|
The limited API already has performance problems (all the |
|
The |
Is this the general consensus? |
|
I think it'd be useful for NumPy to be able to optionally distribute limited API wheels on new platforms where we don't necessarily want to incur the cost of building several version-specific wheels. Long-term, I think it would be great for NumPy to be able to migrate to using the limited API. But also NumPy is just about the hardest package to do this for because of the NumPy C API. I'm not at all surprised to find there are significant issues that might require several iterations to get right, allowing us to migrate without incurring large trade-offs. |
I'm not sure it's written down, but I think that's been the understanding whenever I've talked to people about this. But now that I think about it, this wouldn't help for limited API builds anyway, because it won't be in the limited API. (We don't want to tie ourselves to immortality forever, so unfortunately there's a very slim chance that anything like this will make it into the limited API.) |
Although CPython only requires multi-phase init and nothing checks for static types but for the kind of subinterpreter I tried this with a small extension (multi-phase init, declaring I asked AI to investigate: CPython keeps one For NumPy this matters because |
Ok, I think we should fix that. I was under the impression that was what we already did for user-defined static types, but I guess not. |
Largely based on my old implementation for 3.14.
PyUnstable_SetImmortalto objects allocated in that thread? #150865