GeezerVison is showing github.com — argparse.REMAINDER doesn't work as first argument · Issue #61252 · python/cpython
argparse.REMAINDER doesn't work as first argument #61252Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-05-24.21:31:38.776>
created_at = <Date 2013-01-27.08:51:01.514>
labels = ['type-bug', '3.8', '3.9', 'docs']
title = "argparse.REMAINDER doesn't work as first argument"
updated_at = <Date 2020-05-24.21:31:38.776>
user = 'https://github.com/cjerdonek' bugs.python.org fields:
activity = <Date 2020-05-24.21:31:38.776>
actor = 'rhettinger'
assignee = 'docs@python'
closed = True
closed_date = <Date 2020-05-24.21:31:38.776>
closer = 'rhettinger'
components = ['Documentation']
creation = <Date 2013-01-27.08:51:01.514>
creator = 'chris.jerdonek'
dependencies = []
files = ['48894']
hgrepos = []
issue_num = 17050
keywords = ['patch']
message_count = 19.0
messages = ['180752', '187178', '187182', '312173', '312174', '312179', '312180', '312184', '312185', '312187', '312255', '315711', '315716', '361968', '362006', '362041', '362249', '369818', '369830']
nosy_count = 6.0
nosy_names = ['rhettinger', 'docs@python', 'python-dev', 'miss-islington', 'shihai1991', 'dHannasch']
pr_nums = ['18661', '20363']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17050'
versions = ['Python 3.8', 'Python 3.9'] Activitycjerdonek commented on Jan 27, 2013 Works:
>>> p = ArgumentParser(prog='test.py')
>>> p.add_argument('pos')
>>> p.add_argument('remainder', nargs=argparse.REMAINDER)
>>> p.parse_args(['abc', '--def'])
Namespace(pos='abc', remainder=['--def'])Doesn't work:
>>> p = ArgumentParser(prog='test.py')
>>> p.add_argument('remainder', nargs=argparse.REMAINDER)
>>> p.parse_args(['--def'])
usage: test.py [-h] ...
test.py: error: unrecognized arguments: --defThis use case comes up, for example, if you would like to extract all the arguments passed to a subparser in order to pass to a different program. paulj3 commented on Apr 17, 2013 The problem isn't with REMAINDER, but with the distinction between optionals and arguments. If you change '--def' to 'def', the parse should work:
>> p = ArgumentParser(prog='test.py')
>> p.add_argument('remainder', nargs=argparse.REMAINDER)
>> p.parse_args(['def'])
'--def' would give problems with almost all of the nargs options, especially '*' and '?'.
The issue is that '--def' looks like an optional. Since it is not found in the defined arguments, it is classed as an unknown extra and skipped (try p.parse_known_args(['--def'])). All of this takes place before 'REMAINDER' has a chance to look at the argument strings.
In http://bugs.python.org/issue9334 I submitted a patch that defines a 'args_default_to_positional' parser option. If this is True, that unrecognized '--def' would be classed as a 'positional' and would be captured by the REMAINDER. paulj3 commented on Apr 17, 2013 Here's a way of passing an optional-like argument to a subparser:
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='cmd')
sub1 = subparsers.add_parser('cmd')
sub1.add_argument('foo',nargs='*')
args = parser.parse_args('cmd -- --def 1 2 3'.split())producing
Namespace(cmd='cmd', foo=['--def', '1', '2', '3']) The '--' forces the parser to treat '--def' as a positional. If nargs='REMAINDER', foo=['--', '--def', ...].
But the following subparser definition would be even better:
sub1.add_argument('--def', action='store_true')
sub1.add_argument('rest',nargs='...')Here the '--def' is handle explicitly, as opposed to being passed on.
You don't need the whole subparsers mechanism if you are just going to pass those arguments (unparsed) to another program. akvadrako commented on Feb 14, 2018 Can we at least document that argparse.REMAINDER cannot be used as the first argument? akvadrako commented on Feb 14, 2018 I don't understand how this is about positionals vs optionals. REMAINDER is supposed to capture everything from that point forward, not just positionals. paulj3 commented on Feb 14, 2018 This is another expression of the bigger problem of handling arguments that look like flags. In optparse the 'nargs' (or equivalent, it doesn't handle positionals), control how many arguments an Option takes, regardless of their form. In argparse, the distinction between a option (flag) and argument has priority. So it is difficult to treat strings like '--def' as a plain argument. The default behavior is to treat it as an optional flag.
https://bugs.python.org/issue9334 akvadrako commented on Feb 14, 2018 I still don't understand how that corresponds to the described behavior of REMAINDER and what it has to do with this bug.
How can REMAINDER possibly ever work if optionals take priority? However it does when it's not the first argument. paulj3 commented on Feb 14, 2018 Oops, I see I already mentioned 9334. Here the parsing sequence is a bit different, and the fix I suggest there would not apply here. But the underlying issue is still there - the parser has, in its first iteration, determined that the '--def' looks like an optional. This first scan focuses on the form, not on possible 'nargs' sequences.
In _parse_known_args() it alternates between 'consume_positionals' and 'consume_optional'
In the docs example:
'--foo B cmd --arg1 XX ZZ'
It finds the '--foo' and parses that as optional, consuming the 'B'
Next it finds 'cmd', and so starts to parse positionals. Here is pays attention to the REMAINDER, and thus consumes 'cmd' and the following strings. In other words, once it starts to parse positionals, it can parse as many as match their nargs.
The same applies to the 'abc --def' example given at the start of this question.
But in the 2nd example, with just the REMAINDER and a ['--def'], it doesn't parse any positionals. The code has this comment:
# consume any Positionals preceding the next option
There aren't any strings preceding the '--def', so it moves on to parsing this 'optional'. Yes, I know you think it really is a positional, because you know about the REMAINDER nargs, but the code doesn't know this (or at least doesn't check for that possibility.
As stressed in 9334, when a dashed string is used in a argument like slot, there's an inherent ambiguity. Should it treat as a (potential) error, or accept the programmer and/or user is going against convention?
https://bugs.python.org/issue9334#msg111691 paulj3 commented on Feb 14, 2018 REMAINDER is not widely used, and probably was not tested thoroughly during development. It works for the example given in the docs.
A variant, argparse.PARSER ('A...') is widely used. This is, effectively, REMAINDER ('...') that requires an initial non-optional string. Sort of what '+' is to '*'.
I suspect REMAINDER is most reliable when used as nargs for an optional, e.g.
add_argument('--rest', nargs=argparse.REMAINDER)That way it's clear to everyone, developer, user, and the parser that the following strings are to be taken is.
When parsing the command line, clarity should have priority over convenience. akvadrako commented on Feb 15, 2018 This bug is 5 years old and you are arguing what? That it doesn't matter because it's rarely used compared to some undocumented useless alternative?
It's mildly interesting to hear about some implementation detail but I really don't care. I think you're just wasting your time. I want the docs to match the implementation.
But it doesn't matter - argparse is shit anyway - I'll just write my own parser. paulj3 commented on Feb 16, 2018 A REMAINDER that would work with a flag-like string would be too powerful, too greedy.
In [64]: p = argparse.ArgumentParser();
In [65]: p.add_argument('--foo');
In [66]: p.add_argument('rest', nargs='...');
If the flag is first, its Action works:
In [67]: p.parse_args('--foo x a b c'.split())
Out[67]: Namespace(foo='x', rest=['a', 'b', 'c'])
If there's a non-flag string, REMAINDER grabs everything:
In [68]: p.parse_args('d --foo x a b c'.split())
Out[68]: Namespace(foo=None, rest=['d', '--foo', 'x', 'a', 'b', 'c'])
Imagine a REMAINDER could act with '--foo' as the first string. In[67] would then parse as Out[68] but without the 'd'.
In documented use 'cmd' acts as a gatekeeper, allowing the REMAINDER to grab the rest. So does the '--rest' flag in:
p.add_argument('--rest', nargs='...')Double dash is another gatekeeper:
In [69]: p.parse_args('-- --foo x a b c'.split())
Out[69]: Namespace(foo=None, rest=['--', '--foo', 'x', 'a', 'b', 'c'])
If you don't want such a gatekeeper, why used argparse at all? Why not use sys.argv[1:] directly?
So some sort of warning about the limitations of REMAINDER would be good. But the trick is to come up with something that is clear but succinct. The argparse documentation is already daunting to beginners.
A closed request to document the argparse.PARSER option:
https://bugs.python.org/issue16988
A closed request to document '...'
https://bugs.python.org/issue24647
There was also an issue asking to treat unrecognized flags as plain arguments. I don't recall the status of that issue. With that, REMAINDER could grab '--bar a b c', but 'fail' with '--foo a b c'. It would interesting to test such a variation, but that would be even harder to document. aldencolerain commented on Apr 24, 2018 Paul. This is a bug, not a feature in argparse. Devin is 100% correct. According to the docs REMAINDER should be greedy and is used for passing arguments to sub commands. In your example the expected behavior is that if you do put "d --foo x a b c" that --foo is none and args gets everything. We shouldn't need to use a gatekeeper or resort to manually parsing the remainder arguments. It also shouldn't take 5 years to acknowledged that it needs to be fixed. I'm happy to make a patch if its a bandwidth issue. Am I misunderstanding and you feel like its not possible to fix? I guess if there is a backward compatibility issue then we need to write a new option that does literally return the remainder arguments as documented. paulj3 commented on Apr 24, 2018 Since this feature is buggy, and there isn't an easy fix, we should probably remove any mention of it from the docs. We can still leave it as an undocumented legacy feature.
There is precedent for leaving nargs constants undocumented. argparse.PARSER ('+...') is used by the subparser mechanism, but is not documented. https://bugs.python.org/issue16988 dHannasch commented on Feb 13, 2020 I've attached a file that can be run, but it's a simple script that I can include here inline, too:
"""
Context:
I am trying to set up a cookiecutter so that newly-created packages will come with a Jupyter notebook users can play with.
That is, python -m package_name jupyter would open up a Jupyter quickstart notebook demonstrating the package's features.
argparse.REMAINDER as the first argument isn't important for a top-level parser, since we can work around it by not using argparse at all,
but using argparse.REMAINDER in a subparser seems like a pretty straightforward use case.
Any time we want to dispatch a subcommand to a separate tool --- forwarding all following arguments --- we're going to need it.
"""
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('command', default='cmdname')
parser.add_argument('cmdname_args', nargs=argparse.REMAINDER)
args = parser.parse_args('cmdname --arg1 XX ZZ --foobar'.split())
if args != argparse.Namespace(cmdname_args=['--arg1', 'XX', 'ZZ', '--foobar'], command='cmdname'):
raise Exception(args)
print('This is how argparse.REMAINDER works when there is an argument in front.')
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
parser.add_argument('command', default='cmdname')
parser.add_argument('cmdname_args', nargs=argparse.REMAINDER)
args = parser.parse_args('--foo B cmdname --arg1 XX ZZ --foobar'.split())
if args != argparse.Namespace(cmdname_args=['--arg1', 'XX', 'ZZ', '--foobar'], command='cmdname', foo='B'):
raise Exception(args)
print('This is how argparse.REMAINDER works there is an option in front.')
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
subparsers = parser.add_subparsers(dest='command')
commandParser = subparsers.add_parser('cmdname')
commandParser.add_argument('--filler-boundary-marker', dest='cmdname_args', nargs=argparse.REMAINDER)
args = parser.parse_args('--foo B cmdname --filler-boundary-marker --arg1 XX ZZ --foobar'.split())
if args != argparse.Namespace(cmdname_args=['--arg1', 'XX', 'ZZ', '--foobar'], command='cmdname', foo='B'):
raise Exception(args)
print('This is how argparse.REMAINDER works with a visible "filler" name for the list of arguments.')
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
subparsers = parser.add_subparsers(dest='command')
commandParser = subparsers.add_parser('cmdname')
commandParser.add_argument('--filler-boundary-marker', dest='cmdname_args', nargs=argparse.REMAINDER)
args = parser.parse_args('cmdname --filler-boundary-marker --arg1 XX ZZ --foobar --foo B'.split())
if args != argparse.Namespace(cmdname_args=['--arg1', 'XX', 'ZZ', '--foobar', '--foo', 'B'], command='cmdname', foo=None):
raise Exception(args)
print("If an optional argument is provided after cmdname instead of before, it will get interpreted as part of the argparse.REMAINDER instead of normally. And that's great! We don't even need to be paranoid about other functions of our command-line tool sharing arguments with the tool we want to wrap. Everything will be forwarded.")
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
subparsers = parser.add_subparsers(dest='command')
commandParser = subparsers.add_parser('cmdname')
commandParser.add_argument('positional_arg')
commandParser.add_argument('cmdname_args', nargs=argparse.REMAINDER)
args = parser.parse_args('cmdname can_put_anything_here --arg1 XX ZZ --foobar --foo B'.split())
if args != argparse.Namespace(cmdname_args=['--arg1', 'XX', 'ZZ', '--foobar', '--foo', 'B'], command='cmdname', positional_arg='can_put_anything_here', foo=None):
raise Exception(args)
print("If an optional argument is provided after cmdname instead of before, it will get interpreted as part of the argparse.REMAINDER instead of normally. And that's great! We don't even need to be paranoid about other functions of our command-line tool sharing arguments with the tool we want to wrap. Everything will be forwarded.")"""
Note that this means we can fix the bug simply by,
whenever the cmdname subparser is invoked and the cmdname subparser uses argparse.REMAINDER,
automatically adding an imaginary first positional argument to the subparser
and inserting that imaginary first positional argument into the stream before parsing the arguments to cmdname.
https://github.com/python/cpython/blob/master/Lib/argparse.py#L1201
(Obviously it would be better to fix the underlying cause.)
"""
print('What we want to do is have a subparser that, in the case of one particular selection, forwards all following arguments to another tool.')
print('script.py --foo B cmdname --arg1 XX ZZ --foobar should dispatch to cmdname --arg1 XX ZZ --foobar.')
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
subparsers = parser.add_subparsers(dest='command')
commandParser = subparsers.add_parser('cmdname')
commandParser.add_argument('cmdname_args', nargs=argparse.REMAINDER)
parser.parse_args('--foo B cmdname --arg1 XX ZZ --foobar'.split())
# error: unrecognized arguments: --arg1rhettinger commented on Feb 15, 2020 I concur with Paul's suggestion. dHannasch commented on Feb 16, 2020 Okay. Would it be all right if I submit a fix to get it working at least in the subparser case? shihai1991 commented on Feb 19, 2020
Okay. Would it be all right if I submit a fix to get it working at least in the subparser case?
Hi, dHannasch. According raymond and paul's opinion, you could try to create a PR to update argparse's doc. miss-islington commented on May 24, 2020 rhettinger commented on May 24, 2020
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|