GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — bpo-40059: tomllib by hukkin · Pull Request #31498 · python/cpython

Conversation

hukkin commented Feb 22, 2022 •

edited by encukou

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

This adds a new standard library module, tomllib, for parsing TOML. The recently accepted PEP 680 -- tomllib is relevant here.

This PR has already seen some review in a PR under my personal fork: hukkin#2 (thanks to @encukou, @merwok, @hauntsaninja, @JelleZijlstra (I hope I'm not forgetting anyone)).

The implementation is based on Tomli which I plan to keep maintaining as a backport for Python versions 3.7, 3.8, 3.9 and 3.10, until finally Python 3.10 goes EOL.

Steps taken (converting `tomli` to `tomllib`)

Move everything in tomli:src/tomli to Lib/tomllib. Exclude py.typed.
Remove __version__ = ... line from Lib/tomllib/__init__.py
Move everything in tomli:tests to Lib/test/test_tomllib. Exclude the following test data dirs recursively:
- tomli:tests/data/invalid/_external/
- tomli:tests/data/valid/_external/

Create Lib/test/test_tomllib/__main__.py:

import unittest

from . import load_tests


unittest.main()

Add the following to Lib/test/test_tomllib/__init__.py:

import os
from test.support import load_package_tests

def load_tests(*args):
    return load_package_tests(os.path.dirname(__file__), *args)

Also change import tomli as tomllib to import tomllib.

In cpython/Lib/tomllib/_parser.py replace __fp with fp and __s with s. Add the / to load and loads function signatures.
Run make regen-stdlib-module-names
Create Doc/library/tomllib.rst and reference it in Doc/library/fileformats.rst

edit: For reference, there's one more step – Add tomllib to Makefile.pre.in

https://bugs.python.org/issue40059

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk, petsuter, astrojuanlu, erlend-aasland, hauntsaninja, hmc-cs-mdrissi, barneygale, elprans, AA-Turner, ajoino, and 5 more reacted with hooray emoji

jacopofar, petsuter, astrojuanlu, barneygale, and patrick91 reacted with heart emoji

pradyunsg, barneygale, and patrick91 reacted with rocket emoji

All reactions

🎉 15 reactions
❤️ 5 reactions
🚀 3 reactions

hukkin added 7 commits

February 2, 2022 01:02


          Add tomllib and tests

2f5edf2


          Run make regen-stdlib-module-names

026a48b


          Add tomllib docs

1cdda7c


          Document parse_float limitations

eafc4e0


          Add conversion table to docs

1c9b341


          Sync with Tomli

876a43c


          Remove commented out test code

ab5e488

hukkin requested review from abalkin and pganssle as code owners

February 22, 2022 13:24

bedevere-bot added the awaiting review label

Feb 22, 2022

the-knights-who-say-ni added the CLA signed label

Feb 22, 2022

hukkin mentioned this pull request

Feb 22, 2022

Please consider pushing tomli into stdlib hukkin/tomli#141

Closed

hugovk added the type-feature A feature request or enhancement label

Feb 22, 2022

hukkin commented Feb 22, 2022

Copy link

Copy Markdown

Contributor Author

A question: upstream (Tomli) is formatted with Black, using Black's defaults. This means a line length of 88. Should I reformat with line length at 79?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk commented Feb 22, 2022

Copy link

Copy Markdown

Member

This will need NEWS and "What's new" entries:

https://devguide.python.org/committing/#updating-news-and-what-s-new-in-python

You can add this to https://github.com/python/cpython/blob/main/.github/CODEOWNERS

hukkin reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mgorny commented Feb 22, 2022

Copy link

Copy Markdown

Contributor

Perhaps it'd make sense to include the tomli version used (or even the git commit hash), to make future syncing easier.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hukkin commented Feb 22, 2022

Copy link

Copy Markdown

Contributor Author

Perhaps it'd make sense to include the tomli version used (or even the git commit hash), to make future syncing easier.

It was previously suggested that I remove Tomli version from the standard library port: hukkin#2 (comment)

I plan to include a migration guide in Tomli repository, also including commit hash that tracks what is currently in the stdlib.

merwok reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

TeamSpen210 commented Feb 22, 2022

Copy link

Copy Markdown

Would it be good to mention in the docs why load() takes only binary files? The encoding requirement probably isn't obvious for first-time users.

ciupicri reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hukkin added 3 commits

February 23, 2022 01:12


          Add NEWS

4316ae6


          Add whatsnew

13edc69


          Add CODEOWNERS

6cd95a5

hukkin mentioned this pull request

Feb 22, 2022

bpo-40059: tomllib hukkin/cpython#2

Closed

encukou commented Feb 23, 2022

Copy link

Copy Markdown

Member

I should get to the review this week or the next.

A question: upstream (Tomli) is formatted with Black, using Black's defaults. This means a line length of 88. Should I reformat with line length at 79?

No, I don't think that's worth it. And anyway, line length is not the only point where Black disagrees with PEP8 (starting with, like, the whole philosophy).
But future edits probably won't use Black.

Would it be good to mention in the docs why load() takes only binary files? The encoding requirement probably isn't obvious for first-time users.

Let's leave docs improvements to future PRs, so they get a proper discussion and don't delay this PR?

hukkin reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

astrojuanlu mentioned this pull request

Feb 23, 2022

ENH: BLD: enable building SciPy with Meson scipy/scipy#14847

Merged

Contextualist mentioned this pull request

Feb 27, 2022

Migrate Python dependency uiri/toml to tomllib / hukkin/tomli PyO3/maturin#821

Merged

encukou reviewed

Mar 2, 2022

View reviewed changes

Comment thread

.github/CODEOWNERS Resolved

Uh oh!

There was an error while loading. Please reload this page.


          Update .github/CODEOWNERS

d5e8053

Co-authored-by: Petr Viktorin <[email protected]>

encukou approved these changes

Mar 2, 2022

View reviewed changes

encukou left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few test nitpicks, but I'm happy to merge this!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Lib/test/test_tomllib/test_data.py

Comment on lines +50 to +53

		if isinstance(expected, MissingFile):
		# Would be nice to xfail here, but unittest doesn't seem
		# to allow that in a nice way.
		continue

encukou Mar 2, 2022

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MissingFile looks unnecessary. Does tomli need it?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

encukou Mar 2, 2022

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see it does.
For a poor man's xfail, you could you assert that p.stem is one of the expected failing cases.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

hukkin Mar 2, 2022

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MissingFile looks unnecessary. Does tomli need it?

Yeah one of the two external test suites has a couple test cases where the expected data is missing.

For a poor man's xfail, you could you assert that p.stem is one of the expected failing cases.

I'm not sure I understand what you mean. Maybe you can show with the "Add a suggestion" feature?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

hauntsaninja Mar 2, 2022

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think something like:

if isinstance(expected, MissingFile):
    assert valid.stem in ("xfail_test_case1, "xfail_test_case2", ...)
    continue

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hukkin and encukou reacted with thumbs up emoji

All reactions

👍 2 reactions

hukkin Mar 2, 2022

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, of course, thanks. Committed that.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Lib/test/test_tomllib/test_data.py

		INVALID_FILES = tuple((DATA_DIR / "invalid").glob("*/.toml"))


		class TestData(unittest.TestCase):

encukou Mar 2, 2022

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For peace of mind, could you assert len(VALID_FILES) > 0, and same for INVALID_FILES?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

hukkin Mar 2, 2022

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Makes sense. Added the asserts.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bedevere-bot added awaiting merge and removed awaiting review labels

Mar 2, 2022

hukkin added 2 commits

March 3, 2022 00:00


          Add asserts

ea27f36


          Add a poor man's xfail

2898cc3

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label

Mar 3, 2022

bedevere-bot commented Mar 3, 2022

Copy link

Copy Markdown

🤖 New build scheduled with the buildbot fleet by @encukou for commit 2898cc3 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label

Mar 3, 2022

brandtbucher mentioned this pull request

Mar 3, 2022

bpo-46841: Use inline caching for attribute accesses #31640

Merged

hukkin mentioned this pull request

Mar 3, 2022

Add type hints for tomllib python/typeshed#7432

Merged

hukkin commented Mar 4, 2022

Copy link

Copy Markdown

Contributor Author

It seems the failing CI job (AMD64 Arch Linux Usan PR) errors in other PRs too so should be unrelated to this PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou commented Mar 4, 2022

Copy link

Copy Markdown

Member

The alpha 6 release is a bit bumpy and I don't want to destabilize it, so I'm holding off the merge until it's out.
I tested with all buildbots to see if there's an unforeseen platform-specific issue. It's common to see a few buildbots fail.

hukkin reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou merged commit 591f675 into python:main

Mar 8, 2022

bedevere-bot removed the awaiting merge label

Mar 8, 2022

encukou commented Mar 8, 2022

Copy link

Copy Markdown

Member

Let's get it in!

hauntsaninja, AlexWaygood, hukkin, hugovk, erlend-aasland, Contextualist, and hroncok reacted with hooray emoji

erlend-aasland reacted with rocket emoji

All reactions

🎉 7 reactions
🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hukkin deleted the tomllib branch

March 8, 2022 08:29

mkniewallner mentioned this pull request

Mar 8, 2022

Replace toml with tomli PyCQA/bandit#829

Merged

messense mentioned this pull request

Mar 15, 2022

No TOML module installed for Python 3.11 PyO3/maturin#850

Closed

2 tasks

DanielNoord mentioned this pull request

Mar 15, 2022

Can't install on the CPython 3.11 (broken indirect wrapt deps) pylint-dev/pylint#5919

Closed

Pierre-Sassoulas mentioned this pull request

Mar 15, 2022

Add Python 3.11-dev to Github actions pylint-dev/pylint#5920

Closed

mgorny mannequin mentioned this pull request

Apr 10, 2022

Provide a toml module in the standard library #84240

Closed

coderabbitai Bot mentioned this pull request

Jul 26, 2026

Add reproducible Linux release packaging and compatibility CI JoshCLWren/cdisplayagain#49

Merged

coderabbitai Bot mentioned this pull request

Aug 9, 2026

feat(labels): one ordered axis, as data, with a sync script tinyhumansai/tinysweeper#70

Merged

Reviewers

hauntsaninja

hauntsaninja left review comments

encukou

encukou approved these changes

pganssle

Awaiting requested review from pganssle

abalkin

Awaiting requested review from abalkin

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

type-feature

A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

8 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

Conversation

hukkin commented Feb 22, 2022 • edited by encukou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps taken (converting tomli to tomllib)

Uh oh!

hukkin commented Feb 22, 2022

Uh oh!

hugovk commented Feb 22, 2022

Uh oh!

mgorny commented Feb 22, 2022

Uh oh!

hukkin commented Feb 22, 2022

Uh oh!

TeamSpen210 commented Feb 22, 2022

Uh oh!

encukou commented Feb 23, 2022

Uh oh!

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Mar 3, 2022

Uh oh!

hukkin commented Mar 4, 2022

Uh oh!

encukou commented Mar 4, 2022

Uh oh!

encukou commented Mar 8, 2022

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

hukkin commented Feb 22, 2022 •

edited by encukou

Loading

Uh oh!

There was an error while loading. Please reload this page.

Steps taken (converting `tomli` to `tomllib`)