GeezerVison is showing github.com — bpo-43926: Cleaner metadata with PEP 566 JSON support. by jaraco · Pull Request #25565 · python/cpython
bpo-43926: Cleaner metadata with PEP 566 JSON support.
#25565
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement . We’ll occasionally send you account related emails.
Already on GitHub?
Sign in
to your account
Uh oh!
There was an error while loading. Please reload this page.
Show file tree
Hide file tree
Changes from all commits
Commits
File filter
Filter by extension
.py
(10)
.rst
(2)
All 2 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Diff view
Hide whitespace
Diff view
Hide whitespace
There are no files selected for viewing
Original file line number
Diff line number
Diff line change
Expand Up
@@ -170,6 +170,19 @@ the values are returned unparsed from the distribution metadata::
>>> wheel_metadata['Requires-Python'] # doctest: +SKIP
'>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*'
``PackageMetadata`` also presents a ``json`` attribute that returns
Comment thread
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more .
This is a Python 3.10 only feature and hence a versionadded directive can be added.
Slightly out of topic, are Distribution objects and metadata attribute immutable? Accessing json attribute from metadata objects repeats the computation though it looks like a computed once attribute so just curious if metadata attribute of Distribution, json attribute etc. can be made a cached_property.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
jaraco reacted with thumbs up emoji
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more .
Fixed in 289665029e (also versionchanged for changes to the interpretation of the metadata).
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more .
can be made a cached_property?
Although technically the PackageMetadata object is mutable, the protocol for it doesn't allow for mutation, so it's effectively immutable, so yes, a cached_property could be employed. Such an optimization may be premature, so I'm slightly inclined to defer that concern until a use-case presents itself. I don't feel strongly about it though and would likely accept a PR to importlib_metadata.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
all the metadata in a JSON-compatible form per :PEP:`566`::
>>> wheel_metadata.json['requires_python']
'>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*'
.. versionchanged:: 3.10
The ``Description`` is now included in the metadata when presented
through the payload. Line continuation characters have been removed.
.. versionadded:: 3.10
The ``json`` attribute was added.
.. _version:
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -14,6 +14,7 @@
import posixpath
import collections
from . import _adapters, _meta
from ._collections import FreezableDefaultDict, Pair
from ._functools import method_cache
from ._itertools import unique_everseen
Expand All
@@ -22,7 +23,7 @@
from importlib import import_module
from importlib.abc import MetaPathFinder
from itertools import starmap
from typing import Any, List, Mapping, Optional, Protocol, TypeVar , Union
from typing import List, Mapping, Optional, Union
__all__ = [
Expand Down
Expand Up
@@ -385,25 +386,6 @@ def __repr__(self):
return '<FileHash mode: {} value: {}>'.format(self.mode, self.value)
_T = TypeVar("_T")
class PackageMetadata(Protocol):
def __len__(self) -> int:
... # pragma: no cover
def __contains__(self, item: str) -> bool:
... # pragma: no cover
def __getitem__(self, key: str) -> str:
... # pragma: no cover
def get_all(self, name: str, failobj: _T = ...) -> Union[List[Any], _T]:
"""
Return all values associated with a possibly multi-valued key.
"""
class Distribution:
"""A Python distribution package."""
Expand Down
Expand Up
@@ -488,7 +470,7 @@ def _local(cls, root='.'):
return PathDistribution(zipfile.Path(meta.build_as_zip(builder)))
@property
def metadata(self) -> PackageMetadata:
def metadata(self) -> _meta. PackageMetadata:
"""Return the parsed metadata for this Distribution.
The returned object will have keys that name the various bits of
Expand All
@@ -502,7 +484,7 @@ def metadata(self) -> PackageMetadata:
# (which points to the egg-info file) attribute unchanged.
or self.read_text('')
)
return email.message_from_string(text)
return _adapters.Message( email.message_from_string(text) )
@property
def name(self):
Expand Down
Expand Up
@@ -829,7 +811,7 @@ def distributions(**kwargs):
return Distribution.discover(**kwargs)
def metadata(distribution_name) -> PackageMetadata:
def metadata(distribution_name) -> _meta. PackageMetadata:
"""Get the metadata for the named package.
:param distribution_name: The name of the distribution package to query.
Expand Down
Original file line number
Diff line number
Diff line change
@@ -0,0 +1,67 @@
import re
import textwrap
import email.message
from ._text import FoldedCase
class Message(email.message.Message):
multiple_use_keys = set(
map(
FoldedCase,
[
'Classifier',
'Obsoletes-Dist',
'Platform',
'Project-URL',
'Provides-Dist',
'Provides-Extra',
'Requires-Dist',
'Requires-External',
'Supported-Platform',
],
)
)
"""
Keys that may be indicated multiple times per PEP 566.
"""
def __new__(cls, orig: email.message.Message):
res = super().__new__(cls)
vars(res).update(vars(orig))
return res
def __init__(self, *args, **kwargs):
self._headers = self._repair_headers()
# suppress spurious error from mypy
def __iter__(self):
return super().__iter__()
def _repair_headers(self):
def redent(value):
"Correct for RFC822 indentation"
if not value or '\n' not in value:
return value
return textwrap.dedent(' ' * 8 + value)
headers = [(key, redent(value)) for key, value in vars(self)['_headers']]
if self._payload:
headers.append(('Description', self.get_payload()))
return headers
@property
def json(self):
"""
Convert PackageMetadata to a JSON-compatible format
per PEP 0566.
"""
def transform(key):
value = self.get_all(key) if key in self.multiple_use_keys else self[key]
if key == 'Keywords':
value = re.split(r'\s+', value)
tk = key.lower().replace('-', '_')
return tk, value
return dict(map(transform, map(FoldedCase, self)))
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number
Diff line number
Diff line change
@@ -0,0 +1,29 @@
from typing import Any, Dict, Iterator, List, Protocol, TypeVar, Union
_T = TypeVar("_T")
class PackageMetadata(Protocol):
def __len__(self) -> int:
... # pragma: no cover
def __contains__(self, item: str) -> bool:
... # pragma: no cover
def __getitem__(self, key: str) -> str:
... # pragma: no cover
def __iter__(self) -> Iterator[str]:
... # pragma: no cover
def get_all(self, name: str, failobj: _T = ...) -> Union[List[Any], _T]:
"""
Return all values associated with a possibly multi-valued key.
"""
@property
def json(self) -> Dict[str, Union[str, List[str]]]:
"""
A JSON-compatible form of the metadata.
"""
Original file line number
Diff line number
Diff line change
@@ -0,0 +1,99 @@
import re
from ._functools import method_cache
# from jaraco.text 3.5
class FoldedCase(str):
"""
A case insensitive string class; behaves just like str
except compares equal when the only variation is case.
>>> s = FoldedCase('hello world')
>>> s == 'Hello World'
True
>>> 'Hello World' == s
True
>>> s != 'Hello World'
False
>>> s.index('O')
4
>>> s.split('O')
['hell', ' w', 'rld']
>>> sorted(map(FoldedCase, ['GAMMA', 'alpha', 'Beta']))
['alpha', 'Beta', 'GAMMA']
Sequence membership is straightforward.
>>> "Hello World" in [s]
True
>>> s in ["Hello World"]
True
You may test for set inclusion, but candidate and elements
must both be folded.
>>> FoldedCase("Hello World") in {s}
True
>>> s in {FoldedCase("Hello World")}
True
String inclusion works as long as the FoldedCase object
is on the right.
>>> "hello" in FoldedCase("Hello World")
True
But not if the FoldedCase object is on the left:
>>> FoldedCase('hello') in 'Hello World'
False
In that case, use in_:
>>> FoldedCase('hello').in_('Hello World')
True
>>> FoldedCase('hello') > FoldedCase('Hello')
False
"""
def __lt__(self, other):
return self.lower() < other.lower()
def __gt__(self, other):
return self.lower() > other.lower()
def __eq__(self, other):
return self.lower() == other.lower()
def __ne__(self, other):
return self.lower() != other.lower()
def __hash__(self):
return hash(self.lower())
def __contains__(self, other):
return super(FoldedCase, self).lower().__contains__(other.lower())
def in_(self, other):
"Does self appear in other?"
return self in FoldedCase(other)
# cache lower since it's likely to be called frequently.
@method_cache
def lower(self):
return super(FoldedCase, self).lower()
def index(self, sub):
return self.lower().index(sub.lower())
def split(self, splitter=' ', maxsplit=0):
pattern = re.compile(re.escape(splitter), re.I)
return pattern.split(self, maxsplit)
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,6 @@
import os
import sys
import copy
import shutil
import pathlib
import tempfile
Expand Down
Expand Up
@@ -108,6 +109,16 @@ def setUp(self):
super(DistInfoPkg, self).setUp()
build_files(DistInfoPkg.files, self.site_dir)
def make_uppercase(self):
"""
Rewrite metadata with everything uppercase.
"""
shutil.rmtree(self.site_dir / "distinfo_pkg-1.0.0.dist-info")
files = copy.deepcopy(DistInfoPkg.files)
info = files["distinfo_pkg-1.0.0.dist-info"]
info["METADATA"] = info["METADATA"].upper()
build_files(files, self.site_dir)
class DistInfoPkgWithDot(OnSysPath, SiteDir):
files: FilesDef = {
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -125,7 +125,7 @@ def pkg_with_non_ascii_description(site_dir):
metadata_dir.mkdir()
metadata = metadata_dir / 'METADATA'
with metadata.open('w', encoding='utf-8') as fp:
fp.write('Description: pôrˈtend\n ')
fp.write('Description: pôrˈtend')
return 'portend'
@staticmethod
Expand All
@@ -145,7 +145,7 @@ def pkg_with_non_ascii_description_egg_info(site_dir):
pôrˈtend
"""
).lstrip ()
).strip ()
)
return 'portend'
Expand All
@@ -157,7 +157,7 @@ def test_metadata_loads(self):
def test_metadata_loads_egg_info(self):
pkg_name = self.pkg_with_non_ascii_description_egg_info(self.site_dir)
meta = metadata(pkg_name)
assert meta.get_payload() == 'pôrˈtend\n '
assert meta['Description'] == 'pôrˈtend'
class DiscoveryTests(fixtures.EggInfoPkg, fixtures.DistInfoPkg, unittest.TestCase):
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -231,6 +231,29 @@ def test_more_complex_deps_requires_text(self):
assert deps == expected
def test_as_json(self):
md = metadata('distinfo-pkg').json
assert 'name' in md
assert md['keywords'] == ['sample', 'package']
desc = md['description']
assert desc.startswith('Once upon a time\nThere was')
assert len(md['requires_dist']) == 2
def test_as_json_egg_info(self):
md = metadata('egginfo-pkg').json
assert 'name' in md
assert md['keywords'] == ['sample', 'package']
desc = md['description']
assert desc.startswith('Once upon a time\nThere was')
assert len(md['classifier']) == 2
def test_as_json_odd_case(self):
self.make_uppercase()
md = metadata('distinfo-pkg').json
assert 'name' in md
assert len(md['requires_dist']) == 2
assert md['keywords'] == ['SAMPLE', 'PACKAGE']
class LegacyDots(fixtures.DistInfoPkgWithDotLegacy, unittest.TestCase):
def test_name_normalization(self):
Expand Down
Original file line number
Diff line number
Diff line change
@@ -0,0 +1,4 @@
In ``importlib.metadata``, provide a uniform interface to ``Description``,
allow for any field to be encoded with multiline values, remove continuation
lines from multiline values, and add a ``.json`` property for easy access to
the PEP 566 JSON-compatible form. Sync with ``importlib_metadata 4.0``.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.