Repository navigation
Description
Following an upgrade to Pytest 8, we are seeing a change in the way duplicate items are handled, which does not seem logical/expected to me.
Current behavior
Let's assume we have the following directory structure:
> tree tests
tests
├── subdirectory
│ └── test_two.py
└── test_one.py
With Pytest 7, if we call pytest test_one.py tests --collect-only, it returns 3
tests:
============================= test session starts ==============================
platform linux -- Python 3.12.2, pytest-7.0.0, pluggy-1.4.0
rootdir: /home
collected 3 items
<Module tests/test_one.py>
<Function test_one>
<Module tests/test_one.py>
<Function test_one>
<Module tests/subdirectory/test_two.py>
<Function test_two>
With Pytest 8, we get only one:
============================= test session starts ==============================
platform linux -- Python 3.12.2, pytest-8.0.2, pluggy-1.4.0
rootdir: /home
collected 1 item
<Dir home>
<Dir tests>
<Module test_one.py>
<Function test_one>
After looking a bit at Pytest's internals, the change looks related to the refactoring done in #11646, where the duplicates handling logic was moved away from the former _collectfile method (which, I guess, operate on a file-per-file basis), to be consolidated in genitems() (in which a single duplicate in a node will result in the complete node being ignored).
Expected behavior
I would expect test_one.py not to prevent the further collection of tests in subdirectory, even though it's a duplicate.
To be honest, I'm also a bit puzzled by the previous behavior in Pytest 7, as I wouldn't have expected to see tests/test_one.py twice (as I didn't pass the --keep-duplicates option).
Are my expectations correct in the first place, or did I misunderstand the changes to the test collection and the behavior is expected?
Additional information
- With a bit of guidance, I would be willing to provide a patch (if it's a real bug of course 🙂)
- Reproduced on Pytest 8.0.2 / Linux & macOS
- I made a repository that contains a Dockerfile to reproduce the issue easily, please let me know if you need any additional details
Activity
bluetech commentedon Mar 13, 2024
Thanks for the bug report @nonatomiclabs, I agree the new behavior is buggy. I will take a look.
The duplicate handling is a bit of a headache and as you said it was quite broken also before #11646.
bluetech commentedon Mar 30, 2024
I started looking at this, but it's surprisingly tricky to come up with a self-consistent, intuitive and reasonably backward-compatible logic for the duplicate handling, if you think about it. Though I'll keep trying :)
bluetech commentedon Apr 28, 2024
First, some explanations of the details that are relevant to duplicate handling, then some thoughts on how it should work.
Collection arguments
The collection arguments are the inputs that pytest starts collecting from. These are usually the positional command line arguments but can also be
testpathsand others - doesn't matter.A collection arg has two parts - the path and (optionally) parts within the file.
All of the collection args are given to
Session.collect()which collects them and yields the initial set of nodes.Session.collect()Suppose the collection arguments are
a/aa/aaa.py,/ab/. ThenSession.collect()will produce two nodes (the order is important):But remember that nodes form a tree, so to get the full picture we need to look at the parents of the nodes:
Note that the
<Dir a>parent is the same for both collection args.genitemsAfter
Session.collect()takes the collection arguments and returns the initial nodes, the functiongenitemstakes each node and recursively expands by callingcollect()on each collector node and yielding item nodes (the leaves, i.e. the tests).So it can look something like this:
keep-duplicatesflagPytest has a
--keep-duplicatesflag (off by default) documented here but is mostly unspecified.I think we can ignore whatever it does currently and make it mean what we want.
How should duplicates work?
I think first we need to decide on the semantics. I quickly wrote a test case with some scenarios, it is incomplete but can be a basis for discussion. It includes roughly the behavior that I think it should have but it will definitely change with more consideration (and there are some more interesting cases I should add).
Test cases
Here are some guidelines I based my expected outcomes on (these are debatable):
--keep-duplicatesAlexander-Shukaev commentedon Sep 30, 2024
Hi, any plan to work on this, someone?
WIP: main: add explicit handling of overlapping collection arguments
WIP: main: add explicit handling of overlapping collection arguments
WIP: main: add explicit handling of overlapping collection arguments
WIP: main: add explicit handling of overlapping collection arguments
main: add explicit handling of overlapping collection arguments
testpaths = A A/Bdoesn't test folder A (used to work in v7.4.4) #12605 as a duplicate of this issuetestpaths = A A/Bdoesn't test folder A (used to work in v7.4.4) #12605 as a duplicate of this issuechore(deps): update pytest (major)
Merge branch 'renovate/pytest-9.x' into 'main'
chore(test-fix): add Python 3.14 to tox test matrix (#701)
test.py: deduplicate CLI test arguments before passing to pytest
test.py: deduplicate CLI test arguments before passing to pytest
test.py: deduplicate CLI test arguments before passing to pytest
test.py: deduplicate CLI test arguments before passing to pytest
Merge 'test: fix disabled test handling and deduplicate CLI test argu…
Merge 'test: fix disabled test handling and deduplicate CLI test argu…
test.py: deduplicate CLI test arguments before passing to pytest
Merge 'test: fix disabled test handling and deduplicate CLI test argu…
🔒️ Update dependency pytest to v9 [SECURITY] (#16)
[SECURITY] Update dependency pytest to v9 [SECURITY] (#2)
Update dependency pytest to v9 (#35)