GeezerVison is showing github.com — Consider making `Template` and `Interpolation` generic at runtime · Issue #133970 · python/cpython
Consider making Template and Interpolation generic at runtime #133970DescriptionFeature or enhancement
Template and Interpolation will once be supported by type-checkers. And it might be worth adding generics support for these types.
Example
from string.templatelib import Template, Interpolation
from urllib.parse import quote_plus
domain = 'example.com'
query = 'python string formatting is too complex'
template = t'https://{domain}?q={query}'
def quote_url(interp: Interpolation[str]) -> str:
return quote_plus(interp.value)
def format_url(template: Template) -> str:
parts = []
for part in template:
match part:
case str() as s: # regular string
parts.append(s)
case Interpolation(str(), expression='query') as interp:
parts.append(quote_url(interp))
case Interpolation(value):
parts.append(value)
return ''.join(parts)
print(format_url(template))Here we can see that Interpolation[str] can be useful in users' code.
We can also benefit from generic Template, where we can make it generic based on exact TypeVarTuple. Demo:
from typing import Any, TypeVarTuple, Generic, Unpack
Ts = TypeVarTuple('Ts', default=Unpack[tuple[Any, ...]])
class Template(Generic[*Ts]):
def __new__(cls, *args: *Ts) -> Template[*Ts]: ... # type: ignore[empty-body]
class Interpolation: ...
reveal_type(Template('a', 'b', Interpolation(), 'd'))
# Revealed type is "__main__.Template[Literal['a']?, Literal['b']?, __main__.Interpolation, Literal['d']?]"https://mypy-play.net/?mypy=latest&python=3.13&gist=0dc13b3b926e1efb9783ab9b70d39ceb
This can potentially help type checkers to infer correct Template type.
Current state
Current typeshed definitions: https://github.com/python/typeshed/blob/main/stdlib/string/templatelib.pyi
Here's how Template is defined:
@final
class Template: # TODO: consider making `Template` generic on `TypeVarTuple`
strings: tuple[str, ...]
interpolations: tuple[Interpolation, ...]
def __new__(cls, *args: str | Interpolation) -> Template: ...
def __iter__(self) -> Iterator[str | Interpolation]: ...
def __add__(self, other: Template | str) -> Template: ...
@property
def values(self) -> tuple[Any, ...]: ... # Tuple of interpolation values, which can have any typeIdeally, it should be generic on TypeVarTuple
Here's how Interpolation is defined:
@final
class Interpolation:
value: Any # TODO: consider making `Interpolation` generic in runtime
expression: str
conversion: Literal["a", "r", "s"] | None
format_spec: str
__match_args__ = ("value", "expression", "conversion", "format_spec")
def __new__(
cls, value: Any, expression: str, conversion: Literal["a", "r", "s"] | None = None, format_spec: str = ""
) -> Interpolation: ...Ideally, it should be generic on value. Interpolation[str] means that value is str and Interpolation[tuple[int, int]] means that Interpolation has a value for type tuple[int, int]
Proposal
From runtime's part with only need to add Py_GenericAlias, that's it.
Is it too late for feature freeze? This is a minor detail, so I hope it is not :)
I have a PR ready.
CC @srittau @JelleZijlstra @lysnikolaou
Linked PRs
Reacted by Sebastian Rittau, Alistair Pattison and jaenw Activitysrittau commented on May 13, 2025 One alternative is to make Template generic over the interpolation value types instead. So __new__ would remain typed as it is, but type checkers could special case this (which they already have to do to support template strings) so that:
foo: str = ""
bar: int = 42
tmpl = t"x{foo}{bar}y"
assert_type(tmpl, Template[str, int])
assert_type(tmpl.values, tuple[str, int])sobolevn commented on May 13, 2025 @srittau this would be also possible, if we just add Py_GenericAlias to Template. Exact typing would be determined later by type-checkers / typing council. Reacted by Sebastian Rittau JelleZijlstra commented on May 13, 2025 Yes, I think we should do this, and it should be fine to do it during the beta phase as part of stabilizing the new feature.
We'll have to decide in typeshed what the actual generic parameters are. I think it makes the most sense to use a single type parameter for both Template and Interpolation, so that for example Template[str] holds a series of Interpolation[str] and each has a value of type str. sobolevn commented on May 13, 2025 AA-Turner commented on May 13, 2025 AA-Turner commented on May 13, 2025 I think having Interpolation[T], where T is the type of i.value makes sense. For Template, my first reaction is that t'a {1} b {dict()} c {0.0}' could be type-hinted as Template[LiteralString, int, LiteralString, dict, ...]. This is very verbose though, and it would probably be annoying to type! Jelle's suggestion of a single parameter may be simpler in practice, though more restrictive for heterogeneous types.
# Revealed type is "__main__.Template[Literal['a']?, Literal['b']?, __main__.Interpolation, Literal['d']?]"
What do the question marks mean here? This isn't legal syntax.
A sobolevn commented on May 13, 2025 @AA-Turner
What do the question marks mean here? This isn't legal syntax.
This is how mypy pretty-prints type when reveal_type is called. ? means that this type was inferenced, mostly. davepeck commented on May 14, 2025 Considering the two suggested approaches:
-
Template[T] implies that Template.interpolations is typed as tuple[Interpolation[T], ...]: I worry that real-world templates are nearly always heterogeneous and we'll end up with big unions.
-
Template[A, B] implies Template.interpolations is typed as tuple[Interpolation[A], Interpolation[B]]: The types seem likely to get very large. Consider, say, building a full HTML page from a deeply nested collection of templates.
Walking through a simple example:
def user_image(user: User, classes: Sequence[str] = None):
return t"<img src={user.image_url} alt={user.image_name} class={classes} />"
def user_details(user: User, attribs: Mapping = None, classes: Sequence[str] = None):
return t"""
<div {attribs}>
<span>{user.name} ({user.star_count})</span>
{user_image(user, classes)}
</div>
"""
# What's the type of this `Template`?
template = user_details(
some_user,
{"aria-label": user.name, "aria-hidden": False},
["rounded"]
)In this example:
{attribs} is a dict[str, str | bool]{user.name} is a str{user.star_count} is an int{user_image(...)} is a nested Template{classes} is list[str]
In approach (1) we probably end up with Template[object] in practice?
In approach (2) I suppose we get something like Template[dict[str, str | bool], str, int, Template[str, str, list[str]]]? JelleZijlstra commented on May 14, 2025 I would think of this in terms of the type that you write if you accept a template. In your example, there might be a function render_html(template). If it can interpolate say str and int in a template, it would be typed as def render_html(template: Template[str | int]):. Type checkers can then tell you if you pass some object that isn't allowed as a template value. TeamSpen210 commented on May 15, 2025 The big unions might be alleviated a bit by aliases, especially if checkers can display them unexpanded. Probably they'll be necessary anyway, since you'd want to specify sub-templates recursively. For the default, object might be better - templates are immutable and therefore covariant, so that'll accept any template, but require users to verify interpolation values before use. It's inconsistent with other generics though... sobolevn commented on May 15, 2025 I merged the runtime generics PR, but I won't close the issue, we can continue talking about semantics here :) lysnikolaou commented on May 15, 2025 I'm not sure about this. Yes, Interpolation being generic makes sense, but Template is much harder and both choices that have been presented here are far from ideal and will mostly go unused in practice. I think we need to think bigger about this and, while I'm not the right person to do it, we might also want to explore options for how our type system could evolve around t-strings and the new options they open up.
Tying ourselves to this specific design of making Template generic and then having to adhere to backwards-compatibility rules is not really something I'd like us to do. Maybe we should revert the PR and do it only for Interpolation for now? JelleZijlstra commented on May 15, 2025 The runtime support is very simple; it just allows writing Template[whatever] creating a types.GenericAlias. It's up to type checkers and typeshed to decide what exactly will be allowed within the brackets, and we can decide that later without impacting backwards compatibility at the CPython level.
It's hard to do much with a vague "we need to think bigger", but it seems likely that any type system support for templates will involve some variation of Template + square brackets, whether those square brackets hold the types of the interpolations or something else. Having this support in CPython now means that even if we specify a type system feature later, people will be able to use it even in CPython 3.14.
I'd encourage you to open a discussion in the typing section on Discuss to brainstorm what this bigger thing would look like. What would be helpful would be examples of how templates are expected to be used and what kinds of things you'd like type checkers to be able to help you catch. Reacted by Sebastian Rittau sobolevn commented on May 15, 2025 Probably any reference implementation for any of the type-checkers will also help a lot to decide what is best for the static typing. lysnikolaou commented on May 15, 2025
The runtime support is very simple; it just allows writing Template[whatever] creating a types.GenericAlias. It's up to type checkers and typeshed to decide what exactly will be allowed within the brackets, and we can decide that later without impacting backwards compatibility at the CPython level.
Okay, I misunderstood the PR then and this looks good.
It's hard to do much with a vague "we need to think bigger", but it seems likely that any type system support for templates will involve some variation of Template + square brackets, whether those square brackets hold the types of the interpolations or something else. Having this support in CPython now means that even if we specify a type system feature later, people will be able to use it even in CPython 3.14.
I'd encourage you to open a discussion in the typing section on Discuss to brainstorm what this bigger thing would look like. What would be helpful would be examples of how templates are expected to be used and what kinds of things you'd like type checkers to be able to help you catch.
I know that this is vague and I apologize for that. It's just that I can't really do better than that because I don't know what that "bigger thinking" might look like. There's ideas about combining type checking with static analysis for DSLs, but I don't think that's in scope for this thread.
The recommendation to open a thread on Discuss is good. I also did a lightning talk in the language summit to get people to think about this stuff. I'll try to do it ASAP, but I'm not able to lead that discussion. erictraut commented on Jun 11, 2025
Probably any reference implementation for any of the type-checkers will also help a lot to decide what is best for the static typing.
Pyright 1.1.402 includes support for PEP 750. Here's an example.
One small complication with making Template and Interpolation generic is that there is no way to explicitly specialize an Interpolation[T] by calling its constructor in the context of a t-string since its construction is implicitly called by the runtime. That means bidirectional type inference is the only option for influencing the type of the interpolations. I think this could work, but it's complicated by the concatenation rules for t-strings and regular strings (e.g. x: Template[int] = "foo" t"{bar}"). The inference context needs to span the concatenation. JelleZijlstra commented on Jun 12, 2025 Note there's still an open discussion about changing the implicit concatenation behavior that the SC needs to decide on: python/steering-council#289 picnixz commented on Jun 13, 2025 (adding triaged so that I don't remove the 3.14 label by mistake)
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|