Repository navigation
[mypyc] Fix race in the __weakref__ getter on free-threaded builds - #22486
Open
Jahongir-Qurbonov wants to merge 1 commit into
Open
Jahongir-Qurbonov wants to merge 1 commit into
Jahongir-Qurbonov wants to merge 1 commit into
Conversation
The __weakref__ getter of native classes read the head of the weak reference list and increfed it without synchronization. On a free-threaded build, another thread could free the head in between. CPython fixed the same race in subtype_getweakref by holding its internal weak reference list lock, which extensions can't take. On free-threaded builds, call CPython's own getter instead, which we get from the __weakref__ descriptor of a class created with type().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
__weakref__getter of native classes (CPy_GetWeakrefs, added in #22387) reads the head of the object's weak reference list and increfs it without any synchronization. On a free-threaded build another thread can drop the last reference to that weak reference object between the two steps.weakref_deallocthen unlinks it from the list and frees it, and the getter increfs freed memory.CPython had the same race in
subtype_getweakref(the getter that classes defined in Python use). It was fixed in python/cpython#150247 (for python/cpython#149816; backported to 3.14 and 3.15, not to 3.13): the getter now holds the interpreter's weak reference list lock and uses_Py_TryIncref. Both are internal APIs, and the lock is a striped mutex in the interpreter state, so an extension can't take the same lock.PyUnstable_TryIncRefalone isn't enough, because it relies on the caller holding a lock that the deallocator also takes.Reproducer
Running two
churnthreads and tworeadthreads on one instance withn = 100_000, compiled for 3.14t: on master 8 of 10 runs failed (5 segfaults, 3 hangs). With this PR all 10 passed, and the same script with a non-native class passes on both.Fix
On free-threaded builds,
CPy_GetWeakrefsnow calls CPython's own getter. It gets the__weakref__descriptor of a small class created withtype()on first use and calls itsPyGetSetDef.getfunction directly.subtype_getweakrefonly looks atPy_TYPE(obj)->tp_weaklistoffset, so it works for native classes too. That way the getter takes the right lock, and native classes behave the same as classes defined in Python on every version: they are safe wherever CPython is, and on versions where CPython's own getter is still racy (3.13t, 3.14t before 3.14.9) they are no worse than interpreted classes.The descriptor is cached with a compare-and-swap rather than a mutex, since creating the class can run arbitrary code through the GC, which could reenter the getter. GIL builds are unchanged.
Tests
I added
testConcurrentWeakrefGettertorun-threading.test, with the same pattern as the other concurrency tests there. On 3.14t it segfaulted in 3 of 3 runs without the fix and passed in 3 of 3 runs with it. It also passes on regular builds, where it's only a smoke test. The weakref and threading run tests pass on 3.14t, and the fulltest_run.pypasses on 3.13.