Repository navigation
Closed
Labels
plugin: subtestsrelated to the subtests builtin pluginrelated to the subtests builtin pluginplugin: unittestrelated to the unittest integration builtin pluginrelated to the unittest integration builtin plugintype: performanceperformance or memory problem/improvementperformance or memory problem/improvementtype: regressionindicates a problem that was introduced in a release which was working previouslyindicates a problem that was introduced in a release which was working previously
Description
Running the following test using pytest 9.0.0 in python 3.10 leads to a quadratic blowup:
from unittest import TestCase
N = 1 # 1, 2, 3, 4, ...
class TestQuadraticBlowup(TestCase):
def test_quadratic_blowup(self) -> None:
for i in range(1000 * N): # multiplying by 1000 to make bigger steps
with self.subTest():
passFor a given N, it takes O(N^2) time to complete. Here are a few samples:
N=10-> 10sN=20-> 35sN=30-> 73sN=40-> 130s
When running this test using python -m unittest test.py, it takes less than 0.5s for N=40.
I traced this test a little:
TestCaseFunction.runtest()is called (unittest.py#360)- it calls
testcase(result=self)(unittest.py#389) which isTestCase.run()(case.py#557) - it constructs
outcome = _Outcome(result)(case.py#582) and assigns toselfa line later - the actual test is executed with
self._callTestMethod(testMethod)(case.py#591) - the test repeatedly calls
with self.subTest()which isTestCase.subTest()(case.py#480) TestCase.subTest()constructs_SubTest(case.py#495)- it enters that subtest with
with self._outcome.testPartExecutor(self._subtest, isTest=True): yield(case.py#497) which is_Outcome.testPartExecutor()(case.py#55) - when the subtest finishes successfully, that
_Subtestinstance is appended to the_Outcome.errorslist withNoneas error (case.py#79) - after all subtests finish,
TestCase.run()callsself._feedErrorsToResult(result, outcome.errors)(case.py#599)- note that
outcome.errorshas1000 * Nelements
- note that
TestCase._feedErrorsToResult()iterates over all those errors (case.py#511)- this gives the first, outer
O(N)
- this gives the first, outer
- for each error, it calls
result.addSubTest(test.test_case, test, exc_info)(case.py#513) which isTestCaseFunction.addSubTest()(unittest.py#404) TestCaseFunction.addSubTest()eventually iterates overself.instance._outcome.errors(unittest.py#455) which is the same list asoutcome.errorsabove (1000 * Nelements)- this gives us the second, nested
O(N), which in total gives usO(N^2)for the entire test
- this gives us the second, nested
There might be similar issues with skipped tests (unittest.py#450). Another potential hot spot is unittest.py#318.
pip list output:
Package Version
----------------- -------
exceptiongroup 1.3.0
iniconfig 2.3.0
packaging 25.0
pip 23.0.1
pluggy 1.6.0
Pygments 2.19.2
pytest 9.0.1
setuptools 65.5.0
tomli 2.3.0
typing_extensions 4.15.0
Python: 3.10.17
Metadata
Metadata
Assignees
Labels
plugin: subtestsrelated to the subtests builtin pluginrelated to the subtests builtin pluginplugin: unittestrelated to the unittest integration builtin pluginrelated to the unittest integration builtin plugintype: performanceperformance or memory problem/improvementperformance or memory problem/improvementtype: regressionindicates a problem that was introduced in a release which was working previouslyindicates a problem that was introduced in a release which was working previously
Activity
bluetech commentedon Nov 14, 2025
Interesting. From your analysis of the code, do you think there is some workaround that pytest can do on Python 3.10 only to fix it?
(I was going to suggest a workaround of disabling the subtests plugin using
-p no:subtests, but that doesn't work currently because the unittest plugin does subtest stuff even if the subtests plugin is disabled, we might want to fix that)AkaZecik commentedon Nov 15, 2025
I'm not familiar with the codebase to be able to suggest a well-thought-out solution, and I didn't see any quick and obvious workarounds.
My ideas all revolve around some kind of caching of the values
non_subtest_skip(unittest.py#448) andsubtest_errors(unittest.py#453). I crafted a somewhat hacky solution that replaces the original_Outcomeclass with a class that caches those values, and invalidates the cache when_Outcome.skippedor_Outcome.errorsis modified (either replaced completely, or some elements are removed/added/replaced). This patch makes the test withN=100in python 3.10 finish in roughly the same amount of time (6~8s) as in python 3.11. Precisely, it runs ~1.25 times longer, but this might be because the code performs additional checks in the first place in python 3.10.Let me know if this kind of approach seems reasonable.
RonnyPfannschmidt commentedon Nov 15, 2025
Please open a draft pr for easy review
Fix quadratic-time behavior when handling `unittest` subtests in Pyth…
Fix quadratic-time behavior when handling `unittest` subtests in Pyth…
Fix quadratic-time behavior when handling `unittest` subtests in Pyth…
Fix quadratic-time behavior when handling `unittest` subtests in Pyth…
chore(deps): update pytest (major)
Merge branch 'renovate/pytest-9.x' into 'main'
🔒️ Update dependency pytest to v9 [SECURITY] (#16)
[SECURITY] Update dependency pytest to v9 [SECURITY] (#2)
Update dependency pytest to v9 (#35)