Skip to content

Split INI data only on \n, \r\n and \r - #89

Open
abhishekcpp007 wants to merge 1 commit into
pytest-dev:mainfrom
abhishekcpp007:fix/split-lines-on-newlines-only
Open

abhishekcpp007 wants to merge 1 commit into
pytest-dev:mainfrom
abhishekcpp007:fix/split-lines-on-newlines-only

Conversation

@abhishekcpp007

Copy link
Copy Markdown

What's wrong

parse_ini_data() splits the file with str.splitlines(). That method doesn't just break on \n, \r\n and \r. It also breaks on \x0b, \x0c (form feed), \x1c-\x1e, \x85, U+2028 and U+2029. So:

  • a value that contains one of these gets cut in half, and the second half becomes its own line (usually a ParseError)
  • a form feed page break line (the ^L some editors use) counts as two lines, so every lineof() result and every ParseError line number after it is off by one

configparser doesn't do this since it iterates the file by real newlines.

Reproduction

from iniconfig import IniConfig

IniConfig("x.ini", data="[a]\nk = one\x0btwo\n")
# ParseError: x.ini:3: unexpected line: 'two'

c = IniConfig("x.ini", data="[a]\n\x0c\nk = v\n")
c.lineof("a", "k")
# 4, but k is on line 3

Root cause

src/iniconfig/_parse.py:45 (on main) passes data.splitlines(True) to parse_lines().

Fix

Read the lines through io.StringIO(data, newline=None).readlines(). That's universal newlines mode, so \n, \r\n and \r still end a line (same as before) and nothing else does. Data read from a file already goes through text-mode open(), so this only changes how the other separators are treated.

I added tests for the six separators staying inside a value, the form feed line number shift in lineof() and ParseError, and a check that \n, \r\n and \r endings still parse the same. Plus a CHANGELOG entry under "Unreleased".

Tests

I ran these in Docker (python:3.12, package installed as a wheel, src/ removed so the tests hit the installed copy):

  • main: 54 passed
  • this branch: 65 passed
  • the new tests against main's src/: 8 failed, 57 passed (the six separator cases plus both form feed tests; the line ending tests pass on both, as they should)

pre-commit run on the changed files: ruff check, ruff format and mypy all pass.

str.splitlines() also breaks on form feeds, vertical tabs, U+2028 and
other Unicode separators. That split values containing them into bogus
lines and shifted the line numbers reported by lineof() and ParseError.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant