GeezerVison is showing github.com — Compile time textwrap.dedent() equivalent for str or bytes literals · Issue #81087 · python/cpython
Compile time textwrap.dedent() equivalent for str or bytes literals #81087Description
| BPO |
36906 |
| Nosy |
@rhettinger, @terryjreedy, @gpshead, @stevendaprano, @methane, @serhiy-storchaka, @MojoVampire, @Carreau, @pablogsal, @remilapeyre, @Marco-Sulla, @iforapsy, @jtojnar |
| PRs |
gh-81087: Add str.dedent and cache it at compile time #13445 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/gpshead'
closed_at = None
created_at = <Date 2019-05-13.18:40:31.178>
labels = ['interpreter-core', 'type-feature', '3.10']
title = 'Compile time textwrap.dedent() equivalent for str or bytes literals'
updated_at = <Date 2021-12-20.23:57:51.965>
user = 'https://github.com/gpshead' bugs.python.org fields:
activity = <Date 2021-12-20.23:57:51.965>
actor = 'iforapsy'
assignee = 'gregory.p.smith'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2019-05-13.18:40:31.178>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 36906
keywords = ['patch']
message_count = 46.0
messages = ['342373', '342407', '342420', '342429', '342477', '342488', '342569', '342600', '342909', '342914', '342915', '342916', '342917', '342918', '342927', '342928', '342931', '342938', '342962', '342965', '342968', '342972', '343961', '343991', '350162', '356153', '356154', '356155', '356160', '356162', '356182', '356193', '356198', '356203', '356204', '365688', '381540', '381544', '381545', '381546', '381584', '381599', '381602', '381605', '382456', '398270']
nosy_count = 14.0
nosy_names = ['rhettinger', 'terry.reedy', 'gregory.p.smith', 'steven.daprano', 'methane', 'serhiy.storchaka', 'josh.r', 'mbussonn', 'pablogsal', 'remi.lapeyre', 'Marco Sulla', 'iforapsy', 'Miguel Amaral', 'jtojnar']
pr_nums = ['13445']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue36906'
versions = ['Python 3.10'] Reacted by 叶芝秋, Segev Finer and Oliver Sanders Activitygpshead commented on May 13, 2019 A Python pattern in code is to keep everything indented to look pretty while, yet when the triple quoted multiline string in question needs to not have leading whitespace, calling textwrap.dedent("""long multiline constant""") is a common pattern.
rather than doing this computation at runtime, this is something that'd make sense to do at compilation time. A natural suggestion for this would be a new letter prefix for multiline string literals that triggers this.
Probably not worth "wasting" a letter on this, so I'll understand if we reject the idea, but it'd be nice to have rather than importing textwrap and calling it all over the place just for this purpose.
There are many workarounds but an actual syntax would enable writing code that looked like this:
class Castle:
def __init__(self, name, lyrics=None):
if not lyrics:
lyrics = df"""\
We're knights of the round table
We dance whene'er we're able
We do routines and scenes
With footwork impeccable.
We dine well here in {name}
We eat ham and jam and spam a lot.
"""
self._name = name
self._lyrics = lyricsWithout generating a larger temporary always in memory string literal in the code object that gets converted at runtime to the desired dedented form via a textwrap.dedent() call. I chose "d" as the the letter to mean dedent. I don't have a strong preference if we ever do make this a feature. rhettinger commented on May 14, 2019 I agree that this is a recurring need and would be nice to have. stevendaprano commented on May 14, 2019 gpshead commented on May 14, 2019 Oh good, I thought this had come up before. Your method idea that could be optimized on literals makes a lot of sense, and is notably more readable than yet another letter prefix. remilapeyre commented on May 14, 2019 Hi, I have been looking to get more acquainted with the peephole optimizer. Is it okay if I work on this? gpshead commented on May 14, 2019 I'd say go for it. We can't guarantee we'll accept the feature yet, but I think the .dedent() method with an optimization pass approach is worthwhile making a proof of concept of regardless. stevendaprano commented on May 15, 2019 For the record, I just came across this proposed feature for Java:
https://openjdk.java.net/jeps/8222530
Add text blocks to the Java language. A text block is a multi-line
string literal that avoids the need for most escape sequences,
automatically formats the string in predictable ways, and gives the
developer control over format when desired.
It seems to be similar to Python triple-quoted strings except that the
compiler automatically dedents the string using a "re-indentation
algorithm". (Which sounds to me similar to, if not identical, to that
used by textwrap.)
The JEP proposal says:
A study of Google's large internal repository of Java source code
showed that 98% of string literals, once converted to text blocks
and formatted appropriately, would require removal of incidental
white space. If Java introduced multi-line string solution without
support for automatically removing incidental white space, then many
developers would write a method to remove it themselves and/or lobby
for the String class to include a removal method.
which matches my own experience: *most* but not all of my indented
triple-quotes strings start with incidental whitespace that I don't care
about. But not quite all, so I think backwards compatibility requires
that *by default* triple-quoted strings are not dedented.
Note that there are a couple of major difference between the JEP
proposal and this:
-
The JEP proposes to automatically dedent triple-quoted strings;
this proposal requires an explicit call to .dedent().
-
The JEP proposal allows the user to control the dedent by
indenting, or not, the trailing end-quote;
-
however that means that in Java you won't be able to control
the dedent if the string doesn't end with a final blank line;
-
Should the dedent method accept an optional int argument
specifying the number of spaces to dedent by? (Defaulting to
None, meaning "dedent by the common indent".) If so, that won't
affect the compile-time optimization so long as the argument is
a literal.
-
the JEP performs the dedent before backslash escapes are
interpreted; in this proposal backslash escapes will
occur before the dedent.
The JEP also mentions considering multi-line string literals as Swift
and Rust do them:
https://github.com/apple/swift-evolution/blob/master/proposals/0168-multi-line-string-literals.md
https://stackoverflow.com/questions/29483365/what-is-the-syntax-for-a-multiline-string-literal
I mention these for completeness, not to suggest them as alternatives. gpshead commented on May 15, 2019 Thanks, it's actually good to see this being a feature accepted in other languages. remilapeyre commented on May 20, 2019 Hi @steven.daprano, @gregory.p.smith. I added the first version of my PR for review.
One issue with it is that in:
def f():
return " foo".dedent()f will have both " foo" and "foo" in its constants even if the first is not used anymore. Removing it requires looping over the code once more while marking the constants seen in a set and I was not sure if this was ok. serhiy-storchaka commented on May 20, 2019 Perform the optimization at the AST level, not in the peepholer. stevendaprano commented on May 20, 2019
One issue with it is that in:
def f():
return " foo".dedent()
f will have both " foo" and "foo" in its constants even if the first is not used anymore.
That seems to be what happens with other folded constants:
>>> def f():
... return 99.0 + 0.9
...
>>> f.__code__.co_consts
(None, 99.0, 0.9, 99.9) so I guess that this is okay for a first draft. One difference is that
strings tend to be much larger than floats, so this will waste more
memory. We ought to consider removing unused constants at some point.
(But not me, sorry, I don't have enough C.)
Removing it requires looping over the code once more while marking
the constants seen in a set and I was not sure if this was ok.
That should probably be a new issue. stevendaprano commented on May 20, 2019 Serhiy's message crossed with mine -- you should probably listen to
him over me :-) jtojnar commented on Dec 3, 2020 One benefit of using a compile time feature over a runtime method is that the former allows for more predictable dedenting by first dedenting and only then interpolating variables.
For example, the following code does not dedent the test string at all:
import textwrap
foo = "\n".join(["aaa", "bbb"])
test = textwrap.dedent(
f"""
block xxx:
{foo}
"""
)It would be much nicer if we had syntactical dedents based on the leftmost non-whitespace column in the string, as supported by Nix language, for example.
test = (
df"""
block xxx:
{textwrap.indent(foo, ' '*4)}
"""
)It would be even nicer if the interpolated strings could be indented based on the column the interpolation occurs in but that can at least be done at runtime with only minor inconvenience. stevendaprano commented on Jul 26, 2021 julianfortune commented on Jan 9, 2023 What do folks think about taking an approach similar to Scala's stripMargin:
>>> my_str = """
... |Foo
... | Bar
... | Baz
... """.stripmargin()
Foo
Bar
BazSeems it could be achieved with only the addition of a new method to str, no? stevendaprano commented on Jan 9, 2023 How is stripmargin different from dedent? julianfortune commented on Jan 10, 2023 @stevendaprano my apologies, I did not realize dedent took into account shared whitespace; it seems dedent completely fills the need although it is a little more clunky that it is a separate function versus being implemented as a member of str. stevendaprano commented on Jan 10, 2023 The proposal is to make dedent a string method, and then allow
interpreters to optimise it into a compile-time operation (perhaps in
the keyhole optimiser), *not* to use a function like this:
```
def func(x):
textwrap.dedent("""blah blah blah
multiple lines
"""
)
```
That won't work because it won't be seen as a docstring.
This issue hasn't been touched for 18 months. It looks like this issue
is languishing, possibly over the question whether the method
should implement textwrap.dedent or inspect.cleandoc. gpshead commented on Jan 10, 2023 That question is valid but seems a mere implementation detail. A PR implementing the optimization would help motivate making that decision.
I'd still love it to be automatic for docstrings as it'd save memory for everyone in the world, but that can be considered a separate follow-up feature. stevendaprano commented on Jan 11, 2023 cleandoc and dedent are functionally different, so it's not just an
implementation detail, its a difference of semantics.
Looking at the output of the two, I think cleandoc is the more correct
behaviour for docstrings:
```
s = """First
Second
Third
Indented fourth
Fifth
"""
inspect.cleandoc(s) # --> 'First\nSecond\nThird\n Indented fourth\nFifth.'
textwrap.dedent(s) # --> 'First\n Second\n Third\n Indented fourth\n Fifth.\n'
```
I'd still love it to be automatic for docstrings as it'd save memory for everyone in the world,
What are the consequences of just automatically dedenting docstrings?
Technically its a breaking change, but is there code or people who rely
on the current leading whitespace?
help() already reformats the docstring, as does inspect, so maybe we
should seriously consider just changing the rules for docstrings to
automatically dedent them at function build time. It would probably have
to go through a future import first. methane commented on Jan 11, 2023
Looking at the output of the two, I think cleandoc is the more correct behaviour for docstrings:
This issue is premierly about general multine string dedenting.
So please forget about docstring now and consider what is the best dedent behevior for general multiline strings like SQL.
s = """First
Second
Third
Indented fourth
Fifth
"""
inspect.cleandoc(s) # --> 'First\nSecond\nThird\n Indented fourth\nFifth.'
textwrap.dedent(s) # --> 'First\n Second\n Third\n Indented fourth\n Fifth.\n'
I don't think special casing first line is not needed for general dedenting. stevendaprano commented on Jan 11, 2023 On Tue, Jan 10, 2023 at 10:02:47PM -0800, Inada Naoki wrote:
> Looking at the output of the two, I think cleandoc is the more correct behaviour for docstrings:
This issue is premierly about general multine string dedenting.
But we already have a solution for that: the textwrap module.
The textwrap module is inconvenient to use with docstrings, which is
also when we want to put extra indents in the text block to make it
look good in the .py file, but we need to remove that leading
space before displaying the text.
I agree that we should *also* think about other uses of dedent, but we
should not forget docstrings. They are an important use-case, and as far
as I am concerned, they are the driving motivation for moving dedent
into the str type as a method. methane commented on Jan 12, 2023
I agree that we should also think about other uses of dedent, but we
should not forget docstrings. They are an important use-case, and as far
as I am concerned, they are the driving motivation for moving dedent
into the str type as a method.
As I wrote in this comment, new syntax can be different from inspect.cleandoc().
I prefer having the best multiline syntax by learning from Julia, Java, Swift, etc...
Please read this comment too. Compile time docstring cleandoc is tracked in #81283. terryjreedy commented on Feb 11, 2023 @Carreau "Is this a supposed to deprecating textwrap.dedent?"
@stevendaprano "I don't think so, but eventually it might."
I definitely feel that this should NOT be part of the initial PR without being discussed here.
One of the removed uses in the PR is a 16 line, about 500 char, """-quoted string literal in idlelib. At the momemt I don't really like having 'dedent' moved from the starting line to the last. (I could use inspect.cleandoc instead. Or literal concatenation as elsewhere in the file. )
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|