Repository navigation
Closed
Labels
topic-free-threadingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Bug report
Bug description:
(originally noticed by @davidhewitt in #116738 (comment))
Very infrequently, PyO3 tests for timezone conversions will fail with errors like:
---- conversions::chrono_tz::tests::test_into_pyobject stdout ----
thread 'conversions::chrono_tz::tests::test_into_pyobject' panicked at src/conversions/chrono_tz.rs:120:17:
zoneinfo.ZoneInfo(key='UTC') != zoneinfo.ZoneInfo(key='UTC')
Unfortunately, I don't have an easier way to trigger this besides "run the PyO3 tests over and over until you see a failure", since I think triggering it requires many threads simultaneously using the C API which happens automatically with cargo for the PyO3 tests.
The test code is here:
I believe that rust is more or less doing the equivalent of this C code:
#include <Python.h>
int main()
{
if (Py_IsInitialized() == 0) {
Py_InitializeEx(0);
PyObject *mod = PyImport_ImportModule("zoneinfo");
if (mod == NULL) {
PyErr_PrintEx(0);
return -1;
}
PyObject *zoneinfo = PyObject_GetAttrString(mod, "ZoneInfo");
if (zoneinfo == NULL) {
PyErr_PrintEx(0);
return -1;
}
PyObject *arg = PyUnicode_FromString("Europe/Paris");
PyObject *obj1 = PyObject_CallOneArg(zoneinfo, arg);
PyObject *obj2 = PyObject_CallOneArg(zoneinfo, arg);
PyObject *result = PyObject_RichCompare(obj1, obj2, Py_EQ);
PyObject_Print(result, stderr, Py_PRINT_RAW);
}
return 0;
}
This doesn't fail, which makes me suspect triggering it requires touching the C API in other threads.
CPython versions tested on:
3.13
Operating systems tested on:
macOS
Linked PRs
Reacted by Sam Gross
Metadata
Metadata
Assignees
Labels
topic-free-threadingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Projects
- StatusShow more project fieldsDone
Activity
ngoldbaum commentedon Oct 10, 2024
To trigger it, have a rust toolchain and Python installed, clone the PyO3 repo and execute the following command:
You may also hit some other test failures related to importing errors in the
socketmodule, I'm opening a separate issue about that.colesbury commentedon Oct 10, 2024
I'm having a bit of trouble reproducing this. I first removed the
#[cfg(not(Py_GIL_DISABLED))].When I run:
I get:
Is that expected?
If I run all the tests, like in your command, I get lots of failures, including:
Is that expected?
I also tried with the environment variable
PYTHON_GIL=1set. Does that get passed throughcargo test? I still see lots of failures and a panic:colesbury commentedon Oct 10, 2024
@ngoldbaum here's a draft PR that fixes the non-threadsafe cache in
_zoneinfo.c. And here's the same change, but on the 3.13 branch if that's helpful.I'm still having trouble testing this and it looks like the accesses to
PyDateTimeAPI_implin PyO3 are not thread-safe without the GIL:https://github.com/PyO3/pyo3/blob/50a254dd8431c2eb680184898db3997c0c2cd77e/pyo3-ffi/src/datetime.rs#L611-L627
(I posted this on the wrong issue originally)
pythongh-125243: Fix ZoneInfo data race in free threading build
colesbury commentedon Oct 10, 2024
OK, it turns out that it's very important to use a Python installation built with
--enable-sharedor PyO3's imports of Python modules that use native extensions fail.davidhewitt commentedon Oct 11, 2024
Thanks for investigating and the quick turnaround!
Yes, there is some crude support for "static embedding" but it's horribly broken. Sorry that sunk time for you, we should probably make it harder to allow that.
👍 that looks very likely, we can look into that downstream.
gh-125243: Fix ZoneInfo data race in free threading build (#125281)
pythongh-125243: Fix ZoneInfo data race in free threading build (pyth…
ngoldbaum commentedon Oct 15, 2024
Good point, completely missed that!
I'm also having trouble reproducing this today. I am hitting the issue with
_PyMem_ProcessedDelayedquite a bit though, so I'll check again once #125540 is merged.colesbury commentedon Oct 15, 2024
Ok, just merged the 3.13 backport.
ngoldbaum commentedon Oct 15, 2024
Thanks!
Unfortunately I'm completely unable to reproduce the ZoneInfo comparison issue I reported last week. That said, I know it did happen in the PyO3 CI and I definitely triggered it at least once last week on my laptop using a free-threaded build of 3.13.0.
I'll try to keep my eyes peeled for this to see if I can come up with a way to more reliably trigger the problem.
[3.13] gh-125243: Fix ZoneInfo data race in free threading build (GH-…
colesbury commentedon Oct 15, 2024
Sorry if my series of comments was confusing. I was able to reproduce the issue reliably once I recompiled CPython with
--enable-shared. I just now merged theZoneInfofix to 3.13 (I had forgotten about the backport). I'm reasonably confident that it fixes the comparison issue.ngoldbaum commentedon Oct 24, 2024
This should be fixed and I haven't seen it in a while, so closing.