GeezerVison

Nice, I didn't know if this was ever going to actually happen or not.

In particular, I'm not 100% confident in the terminology I chose for the documentation ("Flags", "Options", "Positionals").

There's an IEEE standard for this, which might help.

The utility in the example is named utility_name. It is followed by options, option-arguments, and operands. The arguments that consist of characters and single letters or digits, such as 'a', are known as "options" (or, historically, "flags"). Certain options are followed by an "option-argument", as shown with [ -c option_argument]. The arguments following the last options and option-arguments are named "operands".
— https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html

You mention the possibility of renaming the expectsValue option to something else, so you might be interested in the excellent getopt API documentation, which calls this has_arg.

This field says whether the option takes an argument. It is an integer, and there are three legitimate values: no_argument, required_argument and optional_argument.
— https://www.gnu.org/software/libc/manual/html_node/Getopt-Long-Options.html

Hope to have time for a more thorough review later, but I do agree that command-line argument parsing is a fundamental capability that should be readily available. What I've noticed is that there are basically three major styles:

1. No frills (what you propose)
2. Subcommands (i.e., the Git CLI git commit -m …)
3. Docker-style (multiple subcommands & args)

@DerekNonGeneric Thanks, didn't know about the IEEE thing. I can adopt this terminology if we think it'd be more clear.

This seems like something that should live off of util instead.

Would like someone else to weigh in on the _ convention for positionals, e.g., @bcoe

I'm interested in feedback on all "possible alternatives" and questions listed in the description. Consolidating it into a list here:

1. Should we rename the expectsValue option? For example, optionArguments (to follow the IEEE guidelines)
2. For args expecting a value (e.g., --foo=bar) should the parsed result always be an array of values? {foo: 'bar'} vs {foo: ['bar']} in this case
3. Should we remove the "count" behavior for repeated flags? What should happen instead, in the case of --foo --foo --foo?
4. Should we adopt IEEE terminology in the code and documentation?
5. Should I revert the change to the order of files in node.gyp?
6. Should the function be named process.parseArgv()?
7. Should more exceptions be thrown somewhere?

In addition, I would like further input on these:

1. Should this function live somewhere else besides process, e.g., util?
2. Should we abandon the _ convention, and return a nested object instead (with e.g., options and positionals properties)?

When answering these, please consider the expected users of this API and the developer experience. What would be easiest to understand at first glance? Once familiar, will the API be ergonomic or tedious?

@devsnek I will not be changing the API to force expectsValue to be an array (though I'm happy to rename it!), as I strongly believe providing a smoother developer experience is more important than somebody's idea of theoretical purity. It's my hope you will not block the PR for this reason.

I don't think confusing apis provide a smoother developer experience.

@devsnek There's nothing confusing about the API; as I mentioned, this is widely supported across userland.

Sounds like @nodejs/tsc may need to weigh in.

Some comments on the issue summary first, will look at the code later.

Another one is combining (or concatenating?) "short flags"--those using a single hyphen, like -v--where -vD would be equivalent to -v -D. While this is a POSIX convention, it is not universally supported by the popular command-line parsers. Since it is inherently sugar (and makes the implementation more complicated), we chose not to implement it.

How much more complicated is it to implement this behavior, and why is it more complicated vs more complex?

Every argument expecting a value (as declared in expectsValue) will parse to Array of strings, even if there is only one string in the Array (e.g., --require c.js becomes {require: ['c.js']}. That makes the API more consistent at the expense of making the common case (no repetition) slightly more awkward.

String type when there's no repetition seems fine.

Remove the "count" behavior. While this is widely supported by modules, I don't often see it used in the wild in Node.js CLI apps.

I never seen this behavior on any CLI apps (Node.js or otherwise), IMO it can be removed

In particular, I'm not 100% confident in the terminology I chose for the documentation ("Flags", "Options", "Positionals"). This does align with other documentation I've read on the subject of CLI arguments, I am unsure if introducing this terminology to our documentation is a Good Idea. Perhaps it can be expressed without new terminology.

Python uses the terminology "arguments" (https://docs.python.org/3/library/argparse.html), but we don't have to use the same terminology. We need to use a terminology, the one you proposed seems as good as any, as long as we are consistent across our documentation (including documentation about Node.js CLI options) about it we're good.

I sorted some files around my modification in in node.gyp, which looked like it wanted to be in order, but was not. It did not seem to affect the build, but I can revert these changes if need be.

We tend to avoid reordering things unless necessary because it makes git archaeology harder.

IMO if a flag with = is provided by the user but the option is not present on expectsValue, parseArgs should throw. I want to hear other's opinion wrt this though.

I think the proper solution would be a system you have to pass in all the args you expect, whether they're flags or options, if it's an option whether it's required, whether certain positional arguments are required, etc.

@devsnek I kinda agree, this API is kind of a mix of low level and high level API: high level due to the magic bits, but low level because the user has to implement so many boilerplate things that are common on CLI tools (--help, options validation, type conversion, etc.). The way it is implemented today it would be possible to add a "strict" mode in the future though, or a more comprehensive parser builder.

The API clearly has a goal of simplicity and is not intended to fully replace existing parser modules, nor is intended to cater for more complex use cases without offloading the heavy work to the user, or at least that's my understanding.

Regarding the "count" feature: ssh uses it; see man page. But that's the only one I can think of offhand. It's not in wide use, but it's also an information channel that is lost if we do not support it.

Regarding returning diff't data structures (a string for singular option values and an array for multiples), I would rather @bcoe could speak to the relative wisdom of such an API, because that is yargs's default behavior, and it is the most popular userland arg parsing lib. Does it trip people up?

Regarding whether options should be explicitly defined... maybe, but, a better developer experience is to not throw on unknown options, and let the API consumer decide what to do with them. In a CLI app, throwing an exception should be avoided wherever possible--end-users typically don't need to see a stack trace, and whether the exception message would actually provide actionable context is a crapshoot. I think we'd rather let the API consumer decide how to express such an error (or silently accept unknown options).

But otherwise, I'm unclear on what explicit definition of boolean-flag-style options really buys an API consumer. If it doesn't throw (and IMO it should not), should this API silently ignore the unknown options? The API consumer would not know if unknown options were passed, unless we threw them into another bucket property, e.g. unknownFlags or provided another option to instruct parseArgs how to handle unknown options. Adding more ways to control its behavior is a very slippery slope (see yargs-parser; apologies Ben 😉).

Also: @nodejs/tooling in case you missed it

Many popular languages don't have argument parsers in their standard libraries, including Java, R, C, C++, C#, etc.

But a few standard libraries do have built-in option parsers.

• For C/C++, there are getopt, getopt_long, and getopt_long_only in the POSIX/GNU standard library. Even though these are historic remnants and most C/C++ projects implement their own parsing logic or use external libraries, the newer getopt variants do provide a bit of flexibility. For example, they allow defining options before parsing.
• Boost has probably almost reached the point of being a standard library for C++, and provides the powerful Boost.Program_options class.
• Python has an older, deprecated option parser getopt that is very similar to the traditional getopt function in POSIX C/C++. This parser has since been replaced with the new optparse module, which is much more flexible and configurable.
• Similarly, Go has flag, which is also quite flexible.

All popular languages provide option parsers as external libraries. Node.js probably has hundreds via npm, C# has System.CommandLine. Java has many, including Apache Commons CLI. R has optparse.

So it seems to me that any modern framework either brings no option parser with it, or a much more powerful parser than the one being proposed here. Also, I am afraid that adding this to Node.js might reduce awareness for better solutions in the ecosystem.

See also Sam's concerns in nodejs/tooling#19 (comment), nodejs/tooling#19 (comment), nodejs/tooling#19 (comment).

Regarding whether options should be explicitly defined... maybe, but, a better developer experience is to not throw on unknown options, and let the API consumer decide what to do with them. In a CLI app, throwing an exception should be avoided wherever possible--end-users typically don't need to see a stack trace, and whether the exception message would actually provide actionable context is a crapshoot. I think we'd rather let the API consumer decide how to express such an error (or silently accept unknown options).

But otherwise, I'm unclear on what explicit definition of boolean-flag-style options really buys an API consumer. If it doesn't throw (and IMO it should not), should this API silently ignore the unknown options? The API consumer would not know if unknown options were passed, unless we threw them into another bucket property, e.g. unknownFlags or provided another option to instruct parseArgs how to handle unknown options. Adding more ways to control its behavior is a very slippery slope (see yargs-parser; apologies Ben wink).

With the approach on this PR the developer has to validate all options through if statements or similar and then manually write a help message for their end users, whereas with explicit declaration that throws the developer can catch the error and print the help message (because we can generate a help message with explicit declaration). It seems a lot more work for developers or the developer will take the simple road and it will be a worse experience for their final user.

So it seems to me that any modern framework either brings no option parser with it, or a much more powerful parser than the one being proposed here. Also, I am afraid that adding this to Node.js might reduce awareness for better solutions in the ecosystem.

I would like to chime in from the perspective of someone who is both supportive of this functionality, and leads development yargs (one of the most widely adopted argument parsers in the npm ecosystem).

Thinking from the perspective of my day job

For my job, I work at Google as a DPE. We behave as customer zero for Google Cloud Products:

• helping build the client libraries that interact with APIs.
• writing samples and documentation for our various products.
• interacting with language communities (👋 hello).

It's my work around samples that has me advocating for a built-in command line argument parser in Node.js.

It allows you to provide an elegant snippet of code to folks that just works:

const {projectId, datasetName} = process.parseArgs()
const bigqueryClient = new BigQuery({
  projectId: args
});
const [dataset] = await bigqueryClient.createDataset(datasetName);
console.log(`Dataset ${dataset.id} created.`);

It creates confusion for users that they need to bring other dependencies to the table to use an exemplary snippet of code (beyond the code you're writing an example for).

Thinking from the perspective of a Node.js tooling author

Node.js is a powerful platform, and I would love to be able to do more with it without pulling in dependencies:

Node.js has great http built-ins, fs built-ins, encryption built-ins, etc., and yet when I write a new library I tend to pull in dependencies like: mkdirp, rimraf, or serve. Why? so I can write npm scripts like this:

{
  "scripts": {
    "precompile": "rimraf ./build",
    "compile": "... some build step"
  }
}

Python exposes a variety of useful built-ins as modules, and allows you to run them as command line applications:

python -m http.server 8000 --bind 127.0.0.1

What if I could do this in Node.js?

{
  "scripts": {
    "precompile": "node -m rmdir --recursive ./build",
    "compile": "... some build step"
  }
}

Having a built-in command line argument parser I'm convinced could help unlock a variety of interesting ideas like this.

In terms of opinions for repeated flags: I have seen too many bugs caused by repeated query params breaking apps that didn't bother to check for isArray. In practice this kind of "almost always a string but if you pass 2+ it becomes an array" leads to one of two things: super verbose defensive code that needs to type check every step of the way OR tools that throw errors because the file named "[a,b]" couldn't be found (or even worse "x.substring isn't a function").

So... I personally would prefer if support for repetition would be dropped (it's the much less often used kind afaict), made explicit via a 2nd option, or if it's always an array and never a string.

So... I personally would prefer if support for repetition would be dropped (it's the much less often used kind afaict), made explicit via a 2nd option, or if it's always an array and never a string.

I don't think it's uncommon enough for us to drop it (personally I've used it on quite a few CLIs I wrote). Always an array seems like a good option for me.

One feature I don't think this proposal supports, which I know is quite popular, is the concept of arrays

The current proposal supports arrays :D

@gabrielschulhof So essentially you're proposing:

• if you do not use optionsWithValues, you can expect a boolean true, string, or undefined
• if you use optionsWithValues, you can expect a string (empty string in the case of a missing value) or undefined

(Is that correct?)

It's important that @tniessen gives the green light to this strategy.

please describe how it should work instead of waiting for me to make a change as @tniessen is suggesting, then coming up with something else.

I don't think I suggested any changes directly, at least none that you implemented. I did highlight problems, but I never said I had solutions to these problems (apart from leaving option parsing to the ecosystem).

You are looking for an API that should be as simple as possible, should work with basically zero configuration, and should never throw exceptions based on argv. From the very beginning, I was doubtful towards such an API in core, and I expressed that, like multiple others have in nodejs/tooling#19. My specific concerns are correctness and safety, first mentioned here and later here. Based on that concern, I requested changes here. My position did not change, and I never suggested any changes.

You expect a solution from me, but I don't have one. I simply don't know how to combine the properties I am concerned about (safety and correctness) with the properties you are focusing on (simplicity and not throwing).

This is nothing personal @boneskull, and I understand your frustration. Nobody has proposed any complete solutions, and at this point, I don't even know if a good solution exists :)

if you do not use optionsWithValues, you can expect a boolean true, string, or undefined

As you correctly mentioned in #35015 (comment), this is not safe either, but probably at least closer to being correct. However, it introduces return values with unpredictable types, and I don't understand why this behavior would be preferable over throwing an error. I know that you are not considering that an option (no pun intended), but I personally don't get why.

To me, it seems illogical to accept values for flags, which, according to the documentation, don't have values.

To use the proposed API safely, developers must check whether values were passed for flags (which don't accept values). If a value is permitted for a flag, why didn't the developer declare it as an option, which takes an optional value? If a value is passed for a flag, most applications should probably throw or display an error. In other words, it seems to me that not throwing an error in this API makes safe usage much more difficult for developers, compared to just handling an error. I simply don't see the logic behind this.

This is what I assume safe usage would look like:

const { options, positionals } = util.parseArgs({ optionsWithValues: ['foo'] });

// Since Node.js allows passing anything for flags, we need to check every single flag for a value.
// The value might be 'false' or something similar, which would lead to incorrect behavior if ignored.
// Node.js could absolutely do this itself, but it doesn't want to.
for (const name of ['bar', 'baz', 'qux', 'quux', 'quuz', 'corge', 'grault']) {
  if (options[name] !== undefined && options[name] !== true) {
    console.error(`I did not expect a value for the flag ${name}, but Node.js accepted one.`);
    console.error(`I cannot really help you, other than tell you not to specify a value for this flag.`);
    process.exit(1);
  }
}

// Okay, after manually checking whether Node.js accepted any values for
// flags that should not accept values, we can finally actually use the values.
const { foo, bar, baz, qux, quux, quuz, corge, grault } = options;

// Oh wait, we also need to make sure no unexpected positionals were passed...
// (Not included here)

I honestly don't understand how this is better than simply handling one exception. I know you don't want to consider this, but I am interested in the reason.

It's important that @tniessen gives the green light to this strategy.

If the PR is somewhat close to implementing somewhat correct behavior, and if the resulting unsafe behavior is properly documented, I'll dismiss my request for changes. That doesn't mean I'll approve the PR, but I also won't stand in the way.

There must be some way for the API to be used safely. With its current state, I don't see how that would be possible, and adding inherently unsafe APIs to Node.js does not seem logical to me.

As much as I am not a fan of the proposed solution, it seems to allow safe usage of the API, assuming developers are aware of the potential pitfalls caused by it, and implement their own error checking.

Of course, you can also leave it to the TSC to overrule my request for changes. Maybe others are fine with landing it as it is, despite the resulting safety and correctness issues.

fwiw I also raised concerns about correctness and safety, but I think those comments got lost when the PR was updated at some point.

@mmarchini I haven't rebased... what are you referring to?

Also as a point of order: given @tniessen objection and the fact that to reach a correct and safe implementation we might end up with material changes to the API, I don't think it's worth discussing Gus objection right now (as the code they're objecting might not exist on a future solution). If that piece of code remains once @tniessen objection is either resolved or dismissed, the TSC must reach a decision on Gus' objection in a timely manner.

@boneskull I'll try to find my comments, but as I said they disappeared (at least in the code view), so it might take a while to find it.

Won't be pushing any changes until Tuesday. I plan on implementing this and updating the documentation accordingly.

Ok, so here are my comments related to correctness and safety (although I didn't use those specific words, and I apologize if I wasn't as clear as @tniessen on my concerns):

• util: add util.parseArgs() #35015 (comment)
• util: add util.parseArgs() #35015 (comment)
• util: add util.parseArgs() #35015 (comment)
• util: add util.parseArgs() #35015 (comment)

(note I didn't make it an explicit objection because I do believe we can have a follow up API with more strict/structured format, but I definitely see why others might think a loose API is not ideal in the first place)

I don't believe balancing these concerns is impossible.

OKAY, what about this:

We want to know if an end-user gave us weird or unexpected input. A flag having a value, for instance. e.g., --verbose=yes is unexpected; --verbose was the correct usage.

Instead of just returning {options: {verbose: 'yes'}}, we return {options: {}, errors: [SomeError]}, where errors is one or more Problems.

This way, we can:

1. keep the value of "flags" as boolean true (consistent types) and
2. we lose no information and
3. we do not throw.

This means that --verbose=false is an "error." The developer can choose how to interpret what was provided. SomeError could be an actual Error or not, but a code and contextual information would be helpful, so I'm leaning towards yes.

Likewise, if optionsWithValues was specified, a missing value would also be considered an "error", instead of returning an empty string.

@mmarchini I think that is reasonable.

Edited my comment just to clarify that my objection is resolved with the change above (just to avoid confusion). Also, I'm happy with other solutions for the passthrough problem, I only suggested that one because it's the simplest solution I could think of.

Other options I find reasonable:

• preserving the -- being optional via parseArgs parameter
• return passthrough items in a separate array ({options, positionals, passthrough})
• return the index of the first passthrough option (same return structure as previous suggestion, but passthrough is a numeric attribute)

@boneskull Why was this closed?

@kibertoad there are efforts to reopen the discussion around this feature in the new year.

Awesome!
If any development support would be needed, more than happy to help.

Questions
In particular, I'm not 100% confident in the terminology I chose for the documentation ("Flags", "Options", "Positionals"). This does align with other documentation I've read on the subject of CLI arguments, [...]

For possible interest, I researched what terminology to use consistently in Commander docs last year. I settled on different terms, but did mention these as used elsewhere.

Short version, Commander terminology: The command line arguments are made up of options, option-arguments, commands, and command-arguments.

Long version: https://github.com/tj/commander.js/blob/master/docs/terminology.md