Repository navigation
Split INI data only on \n, \r\n and \r - #89
Open
abhishekcpp007 wants to merge 1 commit into
Open
abhishekcpp007 wants to merge 1 commit into
abhishekcpp007 wants to merge 1 commit into
Conversation
str.splitlines() also breaks on form feeds, vertical tabs, U+2028 and other Unicode separators. That split values containing them into bogus lines and shifted the line numbers reported by lineof() and ParseError.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's wrong
parse_ini_data()splits the file withstr.splitlines(). That method doesn't just break on\n,\r\nand\r. It also breaks on\x0b,\x0c(form feed),\x1c-\x1e,\x85, U+2028 and U+2029. So:ParseError)^Lsome editors use) counts as two lines, so everylineof()result and everyParseErrorline number after it is off by oneconfigparserdoesn't do this since it iterates the file by real newlines.Reproduction
Root cause
src/iniconfig/_parse.py:45(on main) passesdata.splitlines(True)toparse_lines().Fix
Read the lines through
io.StringIO(data, newline=None).readlines(). That's universal newlines mode, so\n,\r\nand\rstill end a line (same as before) and nothing else does. Data read from a file already goes through text-modeopen(), so this only changes how the other separators are treated.I added tests for the six separators staying inside a value, the form feed line number shift in
lineof()andParseError, and a check that\n,\r\nand\rendings still parse the same. Plus a CHANGELOG entry under "Unreleased".Tests
I ran these in Docker (
python:3.12, package installed as a wheel,src/removed so the tests hit the installed copy):54 passed65 passedsrc/:8 failed, 57 passed(the six separator cases plus both form feed tests; the line ending tests pass on both, as they should)pre-commit runon the changed files: ruff check, ruff format and mypy all pass.