GeezerVison is showing github.com — Ideas for making ast.literal_eval() usable · Issue #83340 · python/cpython
Ideas for making ast.literal_eval() usable #83340Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2019-12-29.22:22:49.960>
labels = ['type-feature', 'library', '3.9']
title = 'Ideas for making ast.literal_eval() usable'
updated_at = <Date 2020-12-22.02:41:55.180>
user = 'https://github.com/rhettinger' bugs.python.org fields:
activity = <Date 2020-12-22.02:41:55.180>
actor = 'rhettinger'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-12-29.22:22:49.960>
creator = 'rhettinger'
dependencies = []
files = []
hgrepos = []
issue_num = 39159
keywords = ['patch']
message_count = 6.0
messages = ['359011', '361995', '361998', '368042', '383563', '383568']
nosy_count = 4.0
nosy_names = ['rhettinger', 'serhiy.storchaka', 'eamanu', 'BTaskaya']
pr_nums = ['19899']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue39159'
versions = ['Python 3.9'] Activityrhettinger commented on Dec 29, 2019 ContributorAuthor More actions A primary goal for ast.literal_eval() is to "Safely evaluate an expression node or a string".
In the context of a real application, we need to do several things to make it possible to fulfill its design goal:
-
We should document possible exceptions that need to be caught. So far, I've found TypeError, MemoryError, SyntaxError, ValueError.
-
Define a size limit guaranteed not to give a MemoryError. The smallest unsafe size I've found so far is 301 characters:
s = '(' * 100 + '0' + ',)' * 100
literal_eval(s) # Raises a MemoryError
-
Consider writing a standalone expression compiler that doesn't have the same small limits as our usual compile() function. This would make literal_eval() usable for evaluating tainted inputs with bigger datasets. (Imagine if the json module could only be safely used with inputs under 301 characters).
-
Perhaps document an example of how we suggest that someone process tainted input:
expr = input('Enter a dataset in Python format: ')
if len(expr) > 300:
error(f'Maximum supported size is 300, not {len(expr)}')
try:
data = literal_eval(expr)
except (TypeError, MemoryError, SyntaxError, ValueError):
error('Input cannot be evaluated')isidentical commented on Feb 14, 2020 SponsorMember More actions
- We should document possible exceptions that need to be caught. So far, I've found TypeError, MemoryError, SyntaxError, ValueError.
Maybe we should wrap all of these into something like LiteralEvalError to easily catch all of them, LiteralEvalError can be subclass of that four but I guess in some cases this change might break code.
- Define a size limit guaranteed not to give a MemoryError. The smallest unsafe size I've found so far is 301 characters:
>>> s = "(" * 101 + ")" * 101
>>> len(s)
202
>>> ast.literal_eval(s)
s_push: parser stack overflow
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.9/ast.py", line 61, in literal_eval
node_or_string = parse(node_or_string, mode='eval')
File "/usr/local/lib/python3.9/ast.py", line 49, in parse
return compile(source, filename, mode, flags,
MemoryError
- Consider writing a standalone expression compiler that doesn't have the same small limits as our usual compile() function. This would make literal_eval() usable for evaluating tainted inputs with bigger datasets. (Imagine if the json module could only be safely used with inputs under 301 characters).
Can you explain it a bit more detailed, how does this standalone expression compiler should work? isidentical commented on Feb 14, 2020 SponsorMember More actions
- We should document possible exceptions that need to be caught. So far, I've found TypeError, MemoryError, SyntaxError, ValueError.
Also, an addition to these errors is RecursionError
>>> t = ast.Tuple(elts=[], ctx=ast.Load())
>>> t.elts.append(t)
>>> ast.literal_eval(t)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.9/ast.py", line 101, in literal_eval
return _convert(node_or_string)
File "/usr/local/lib/python3.9/ast.py", line 81, in _convert
return tuple(map(_convert, node.elts))
File "/usr/local/lib/python3.9/ast.py", line 81, in _convert
return tuple(map(_convert, node.elts))
File "/usr/local/lib/python3.9/ast.py", line 81, in _convert
return tuple(map(_convert, node.elts))
[Previous line repeated 496 more times]
RecursionError: maximum recursion depth exceededserhiy-storchaka commented on May 4, 2020 It can also crash.
ast.literal_eval('+0'*10**6)The cause is that all AST handling C code (in particularly converting the AST from C to Python) is recursive, and therefore can overflow the C stack. Some recursive code has arbitrary limits which cause raising exceptions like MemoryError in the initial example, but not all code has such checks. Reacted by Gregory P. Smith and donhatch rhettinger commented on Dec 22, 2020 ContributorAuthor More actions rhettinger commented on Dec 22, 2020 ContributorAuthor More actions
Can you explain it a bit more detailed,
how does this standalone expression compiler should work?
Aim for something like JSON parser but for the supported Python constant expressions and with the existing tokenize module to feed a hand-rolled on-recursive parser gpshead commented on Apr 14, 2022 related: the old #55552 issue about producing better errors from the parser. not crashing due to a C stack overflow counts as better. ;)
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|