Skip to content

Dont list directorymarkers in iterdir - #536

Merged
ap-- merged 4 commits into
fsspec:mainfrom
ap--:dont-list-directorymarkers-in-iterdir
Nov 10, 2026
Merged

ap-- merged 4 commits into
fsspec:mainfrom
ap--:dont-list-directorymarkers-in-iterdir

Conversation

@ap--

@ap-- ap-- commented Jan 13, 2026

Copy link
Copy Markdown
Collaborator

Should close #226

@danielgafni

Copy link
Copy Markdown

Perhaps we should also check for "ContentType": "application/x-directory; charset=UTF-8", as a precaution? This would prevent the special key from filtering out in case the user has actually set it.

@ap--

ap-- commented Jan 13, 2026

Copy link
Copy Markdown
Collaborator Author

It seems, the way info is implemented in s3fs, ContentType is currently not propagated for these directory markers.

@ap--

ap-- commented Jan 13, 2026

Copy link
Copy Markdown
Collaborator Author

If you could explicitly confirm that the PR as-is fixes your issue I'd be happy to merge ☺️

@danielgafni

danielgafni commented Jan 13, 2026 •

Copy link
Copy Markdown
  1. I don't have access to AWS S3 right now (I can set it up later), so I am testing it with Cloudflare R2. I also artificially created an empty object at s3://anam-data-dev/rofl/ via s3fs I've also created another empty object at s3://anam-data-dev/rofl/object .
  2. I've encountered a different issue now:
/tmp/rofl via 🐍 v3.12.7 (rofl) via   on 🅰 (auto)
› uv pip install git+https://github.com/ap--/universal_pathlib.git@dont-list-directorymarkers-in-iterdir
Resolved 3 packages in 540ms
Uninstalled 1 package in 2ms
░░░░░░░░░░░░░░░░░░░░ [0/1] Installing wheels...                                                                              
Installed 1 package in 6ms
 - universal-pathlib==0.3.8
 + universal-pathlib==0.0.12.post330+gb5d91d25e (from git+https://github.com/ap--/universal_pathlib.git@b5d91d25e2f7acb0e533617a1e38184d375944d9)

/tmp/rofl via 🐍 v3.12.7 (rofl) via   on 🅰 (auto)
› ipython
Python 3.12.7 (main, Oct 16 2024, 04:37:19) [Clang 18.1.8 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.9.0 -- An enhanced Interactive Python. Type '?' for help.
Tip: You can find how to type a LaTeX symbol by back-completing it, eg `\θ<tab>` will expand to `\theta`.

In [1]: from upath import UPath

In [2]: path = UPath("s3://anam-data-dev/")

In [3]: list(path.iterdir())
Out[3]:
[S3Path('anam-data-dev/rofl', protocol='s3'),
 S3Path('anam-data-dev/rofl', protocol='s3'),
 S3Path('anam-data-dev/test', protocol='s3'),
 S3Path('anam-data-dev/test.delta', protocol='s3'),
 S3Path('anam-data-dev/video-factory', protocol='s3'),
 S3Path('anam-data-dev/video-generation', protocol='s3')]

In [4]: path = UPath("s3://anam-data-dev/rofl")

In [5]: list(path.iterdir())
---------------------------------------------------------------------------
NotADirectoryError                        Traceback (most recent call last)
Cell In[5], line 1
----> 1 list(path.iterdir())

File /tmp/rofl/.venv/lib/python3.12/site-packages/upath/core.py:1192, in UPath.iterdir(self)
   1190 base_path = base.path
   1191 if not fs.isdir(base_path):
-> 1192     raise NotADirectoryError(str(self))
   1193 name: str
   1194 for info in fs.listdir(base_path):
   1195     # fsspec returns dictionaries

NotADirectoryError: s3://anam-data-dev/rofl

In [6]: path = UPath("s3://anam-data-dev/rofl/")

In [7]: list(path.iterdir())
---------------------------------------------------------------------------
NotADirectoryError                        Traceback (most recent call last)
Cell In[7], line 1
----> 1 list(path.iterdir())

File /tmp/rofl/.venv/lib/python3.12/site-packages/upath/core.py:1192, in UPath.iterdir(self)
   1190 base_path = base.path
   1191 if not fs.isdir(base_path):
-> 1192     raise NotADirectoryError(str(self))
   1193 name: str
   1194 for info in fs.listdir(base_path):
   1195     # fsspec returns dictionaries

NotADirectoryError: s3://anam-data-dev/rofl

So the artificial marker object breaks universal-pathlib right now (for me) when trying to list the marker path. It looks like the requirement here is to have both s3://anam-data-dev/rofl/ and s3://anam-data-dev/rofl/object empty.

@danielgafni

Copy link
Copy Markdown

Also this seems a little weird to me

image

@ap--

ap-- commented Jan 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

I don't have access to AWS S3 right now (I can set it up later), so I am testing it with Cloudflare R2. I also artificially created an empty object at s3://anam-data-dev/rofl/ via s3fs I've also created another empty object at s3://anam-data-dev/rofl/object

s3fs doesn't let you do this like the aws console would because it strips the trailing slash

@ap--

ap-- commented Jan 13, 2026

Copy link
Copy Markdown
Collaborator Author

Hmmm, seems like yet another edge case. Okay I'll see if this can be handled in a better way.

@ap-- ap-- added this to the v0.4.0 milestone Jan 31, 2026
ap-- added 4 commits November 10, 2026 23:00
On object stores an object key can also be the prefix of other objects.
s3fs then lists the name twice (as file and as directory), and once the
parent listing is cached, isdir() returns False for that name, so
iterdir() yielded duplicates and raised NotADirectoryError when listing
the colliding directory (see fsspec/s3fs#999).

iterdir() now skips duplicate names and re-checks isdir() once after
invalidating the filesystem cache before raising NotADirectoryError.
The directory marker check only matched s3 listings of non-root prefixes
(base paths without a trailing separator, with s3fs specific info keys).
Listing entries whose name equals the listed directory are never children,
so skip them for all filesystems. This also fixes iterdir() yielding a
bogus 's3://bucket/bucket' child for buckets with keys starting with '/'
(fsspec#144).
@ap--
ap-- force-pushed the dont-list-directorymarkers-in-iterdir branch from b5d91d2 to 910c969 Compare November 10, 2026 22:20
@ap--
ap-- merged commit d3d6c83 into fsspec:main Nov 10, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

path.iterdir() yields path itself as the first item (with S3)

2 participants