GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — tools: non-Ascii linter for /lib only by SirR4T · Pull Request #18043 · nodejs/node

Conversation

@SirR4T

@SirR4T SirR4T commented Jan 8, 2018 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Non-ASCII characters in /lib get compiled into the node binary,
and may bloat the binary size unnecessarily. A linter rule may
help prevent this.

Fixes: #11209

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • commit message follows commit guidelines
Affected core subsystem(s)

eslint

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@nodejs-github-bot nodejs-github-bot added the tools Issues and PRs related to the tools directory. label Jan 8, 2018
addaleax
addaleax approved these changes Jan 8, 2018
View reviewed changes

@addaleax addaleax left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would make it clear somewhere in a comment that this only catches characters in literals, not the source code itself (which is what we would eventually want to have instead)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@SirR4T

SirR4T commented Jan 8, 2018

Copy link
Copy Markdown
Author

oh? then @hkal 's approach of parsing all tokens would be preferred? I had assumed the eslint selector would select the source code too, i see where i might have gone wrong 😅

Also, doubts:

  • the regexp /[^\n\x20-\x7e]/ doesn't capture \r, which I have seen some instances of, while running the linter. I should be adding that in the regexp, right?
  • I have seen instances of \u0000 and \u000c etc. in code. I should be disabling this rule for these, right?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@addaleax

addaleax commented Jan 8, 2018

Copy link
Copy Markdown
Member

@SirR4T I’m not sure how to do it, but yes, you want all tokens and even comments.

I have seen instances of \u0000 and \u000c etc. in code. I should be disabling this rule for these, right?

It’s fine to have the escape sequences spelled out as escape sequences in the source code. What matters is whether the files themselves contain characters that don’t fit into the ASCII range, because that’s what the script that bakes those files into the node binary looks for.

I hope that’s helpful. :)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@SirR4T

SirR4T commented Jan 8, 2018

Copy link
Copy Markdown
Author

now I'm running into a very peculiar error, while doing make lint-js:

I'm unable to disable the below error, via the regular methods (//eslint-disable-line, //eslint-disable-next-line, etc.)

$ make lint-js
Running JS linter...

/Users/sarat/Play/Github/node/lib/internal/test/unicode.js
  1:1  error  Non-ASCII character '✓' detected  non-ascii-character

✖ 1 problem (1 error, 0 warnings)
  1 error, 0 warnings potentially fixable with the `--fix` option.

make: *** [lint-js] Error 1
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@SirR4T

SirR4T commented Jan 9, 2018

Copy link
Copy Markdown
Author

@addaleax : review latest commit? This was the approach followed in the (now closed) pull #11371, but I think that this is catching much lesser violations.

addaleax reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@SirR4T
SirR4T force-pushed the eslintForLib branch from 1233f7b to 0403094 Compare January 9, 2018 03:37
@not-an-aardvark

not-an-aardvark commented Jan 10, 2018

Copy link
Copy Markdown
Contributor

I'm a bit confused about why non-ascii characters would bloat the binary size. Aren't the files encoded as UTF-8?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@SirR4T

SirR4T commented Jan 10, 2018

Copy link
Copy Markdown
Author

@not-an-aardvark that is what I understood, from #11129 (comment) :

I would have to assume it is because the external string API does not directly support UTF8. That is, if you look at ExternalStringResource, it assumes a utf16_t* buffer while ExternalOneByteStringResource assumes const char*. If the strings were stored as UTF8 we would incur an additional cost at startup that is not necessary.

Is that not correct?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@addaleax

addaleax commented Jan 10, 2018

Copy link
Copy Markdown
Member

@SirR4T Yes, that is correct. Files that don’t fit into ASCII are currently saved as UTF-16 instead:

node/tools/js2c.py

Lines 230 to 240 in 1e0f331

def Render(var, data):
# Treat non-ASCII as UTF-8 and convert it to UTF-16.
if any(ord(c) > 127 for c in data):
template = TWO_BYTE_STRING
data = map(ord, data.decode('utf-8').encode('utf-16be'))
data = [data[i] * 256 + data[i+1] for i in xrange(0, len(data), 2)]
data = ToCArray(data)
else:
template = ONE_BYTE_STRING
data = ToCString(data)
return template.format(var=var, data=data)

not-an-aardvark reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

addaleax
addaleax approved these changes Jan 10, 2018
View reviewed changes

@addaleax addaleax left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, this still looks fine to me, but I’m not an expert for eslint anyway.

If it doesn’t catch the diagram in lib/timers.js and the check mark in lib/internal/test/unicode.js (for both of which we’ll want eslint-disable comments), it’s probably not quite strict enough, though.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Trott
Trott reviewed Jan 10, 2018
View reviewed changes
Comment thread tools/eslint-rules/non-ascii-character.js Outdated

@Trott Trott Jan 10, 2018

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason not to use a RegExp literal like this?:

const nonAsciiRegexPattern = /[^\r\n\x20-\x7e]/;

Seems more readable to me and also more in line with our general coding style. (I'm kind of surprised this isn't caught by a lint rule itself, to be honest. Or maybe it is?)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
not-an-aardvark
not-an-aardvark reviewed Jan 10, 2018
View reviewed changes
Comment thread tools/eslint-rules/non-ascii-character.js Outdated

@not-an-aardvark not-an-aardvark Jan 10, 2018

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be better to match on source.text rather than each individual token, to ensure that non-ascii whitespace is detected (which would not be part of any token or comment).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@SirR4T
SirR4T force-pushed the eslintForLib branch from 0403094 to cf8e821 Compare January 11, 2018 08:54
@SirR4T

SirR4T commented Jan 11, 2018

Copy link
Copy Markdown
Author

Yay! 🎉

Could catch the lib/timers.js too, using source.text. But am still unable to bypass / skip the rule, for it ☹️

Any ideas? @Trott @not-an-aardvark

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@SirR4T
SirR4T force-pushed the eslintForLib branch from d4fb97c to 004ffb3 Compare January 11, 2018 16:58
@SirR4T

SirR4T commented Jan 19, 2018

Copy link
Copy Markdown
Author

cc: @Trott @not-an-aardvark @addaleax

can someone help me out here? I'm unable to skip the lint checks, inside lib/ folder. Need to skip the rules for lib/timers.js, and the check mark in lib/internal/test/unicode.js.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@not-an-aardvark

not-an-aardvark commented Jan 19, 2018

Copy link
Copy Markdown
Contributor

I think there are a few issues:

  • You're using a comment like // eslint-disable non-ascii-character, but I think it should be a block comment (/* eslint-disable non-ascii-character */)
  • When the rule reports a problem, the location of the problem is currently always the top of the file, because you're passing the Program node to context.report. Instead, it would be better if the report location were the point in the file that contains the non-ascii character. To do this, you could find the index of the non-ascii character in sourceCode.text, and then pass { ... loc: sourceCode.getLocFromIndex(theIndex) } to context.report.

If you make both of those changes, I think the disable comments will work as expected.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sarat Addepalli added 3 commits January 19, 2018 22:36
Sarat Addepalli
tools: non-Ascii linter for /lib only
3d63b03
Non-ASCII characters in /lib get compiled into the node binary,
and may bloat the binary size unnecessarily. A linter rule may
help prevent this.

Fixes: nodejs#11209
Sarat Addepalli
tools: fix non-ascii linter detection
1399bf5
the linter should detect not just literals, but also source code
and all comments too.
Sarat Addepalli
tools: using source.text instead of tokens
bd452d5
@SirR4T

SirR4T commented Jan 19, 2018

Copy link
Copy Markdown
Author

Thanks @not-an-aardvark ! worked like a charm! pushing the latest changes, once the build is done.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sarat Addepalli
Adding loc to report
1e0ab23
@SirR4T
SirR4T force-pushed the eslintForLib branch from 004ffb3 to 1e0ab23 Compare January 19, 2018 18:10
@maclover7
maclover7 force-pushed the master branch from bb5575a to 993b716 Compare January 26, 2018 22:02
@cjihrig
cjihrig force-pushed the master branch from 993b716 to 082f952 Compare January 26, 2018 22:36
BridgeAR
BridgeAR approved these changes Feb 2, 2018
View reviewed changes
@BridgeAR

BridgeAR commented Feb 2, 2018

Copy link
Copy Markdown
Member

CI https://ci.nodejs.org/job/node-test-pull-request/12908/

@not-an-aardvark PTAL

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

not-an-aardvark
not-an-aardvark approved these changes Feb 2, 2018
View reviewed changes
@BridgeAR BridgeAR added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Feb 2, 2018
addaleax
addaleax approved these changes Feb 2, 2018
View reviewed changes
@addaleax

addaleax commented Feb 4, 2018

Copy link
Copy Markdown
Member

Landed in c45afe8

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@addaleax addaleax closed this Feb 4, 2018
@addaleax addaleax removed the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Feb 4, 2018
addaleax pushed a commit that referenced this pull request Feb 4, 2018
Sarat Addepalli @addaleax
tools: non-Ascii linter for /lib only
c45afe8
Non-ASCII characters in /lib get compiled into the node binary,
and may bloat the binary size unnecessarily. A linter rule may
help prevent this.

PR-URL: #18043
Fixes: #11209
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: Teddy Katz <[email protected]>
MylesBorins pushed a commit that referenced this pull request Feb 21, 2018
Sarat Addepalli @MylesBorins
tools: non-Ascii linter for /lib only
46b8481
Non-ASCII characters in /lib get compiled into the node binary,
and may bloat the binary size unnecessarily. A linter rule may
help prevent this.

PR-URL: #18043
Fixes: #11209
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: Teddy Katz <[email protected]>
MylesBorins pushed a commit that referenced this pull request Feb 21, 2018
Sarat Addepalli @MylesBorins
tools: non-Ascii linter for /lib only
e1fffbc
Non-ASCII characters in /lib get compiled into the node binary,
and may bloat the binary size unnecessarily. A linter rule may
help prevent this.

PR-URL: #18043
Fixes: #11209
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: Teddy Katz <[email protected]>
MylesBorins pushed a commit that referenced this pull request Feb 21, 2018
Sarat Addepalli @MylesBorins
tools: non-Ascii linter for /lib only
e00bb16
Non-ASCII characters in /lib get compiled into the node binary,
and may bloat the binary size unnecessarily. A linter rule may
help prevent this.

PR-URL: #18043
Fixes: #11209
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: Teddy Katz <[email protected]>
@MylesBorins MylesBorins mentioned this pull request Feb 21, 2018
v9.6.0 proposal #18902
Merged
@MylesBorins MylesBorins added backport-requested-v6.x labels Mar 20, 2018
@MylesBorins

MylesBorins commented Mar 20, 2018

Copy link
Copy Markdown
Contributor

Should this be backported to v8.x-staging or v6.x-staging? If yes please follow the guide and raise a backport PR, if not let me know or add the dont-land-on label.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@SirR4T SirR4T mentioned this pull request Mar 20, 2018
[v6.x backport] tools: non-Ascii linter for /lib only #19493
Closed
2 of 4 tasks
@SirR4T SirR4T mentioned this pull request Mar 21, 2018
[v8.x backport] tools: non-Ascii linter for /lib only #19499
Closed
2 of 4 tasks
SirR4T pushed a commit to SirR4T/node that referenced this pull request Mar 29, 2018
Sarat Addepalli
tools: non-Ascii linter for /lib only
6ba7fd0
Non-ASCII characters in /lib get compiled into the node binary,
and may bloat the binary size unnecessarily. A linter rule may
help prevent this.

PR-URL: nodejs#18043
Fixes: nodejs#11209
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: Teddy Katz <[email protected]>
SirR4T pushed a commit to SirR4T/node that referenced this pull request Mar 29, 2018
Sarat Addepalli
tools: non-Ascii linter for /lib only
bc22bf0
Non-ASCII characters in /lib get compiled into the node binary,
and may bloat the binary size unnecessarily. A linter rule may
help prevent this.

PR-URL: nodejs#18043
Fixes: nodejs#11209
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: Teddy Katz <[email protected]>
BethGriggs pushed a commit that referenced this pull request Apr 10, 2018
Sarat Addepalli @BethGriggs
tools: non-Ascii linter for /lib only
016a28a
Non-ASCII characters in /lib get compiled into the node binary,
and may bloat the binary size unnecessarily. A linter rule may
help prevent this.

PR-URL: #18043
Backport-PR-URL: #19499
Fixes: #11209
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: Teddy Katz <[email protected]>
@MylesBorins MylesBorins mentioned this pull request May 2, 2018
v8.11.2 proposal #20478
Merged
MayaLekova pushed a commit to MayaLekova/node that referenced this pull request May 8, 2018
Sarat Addepalli @MayaLekova
tools: non-Ascii linter for /lib only
ee8bdf3
Non-ASCII characters in /lib get compiled into the node binary,
and may bloat the binary size unnecessarily. A linter rule may
help prevent this.

PR-URL: nodejs#18043
Fixes: nodejs#11209
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Ruben Bridgewater <[email protected]>
Reviewed-By: Teddy Katz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Trott Trott Trott left review comments
@addaleax addaleax addaleax approved these changes
@BridgeAR BridgeAR BridgeAR approved these changes
+1 more reviewer
@not-an-aardvark not-an-aardvark not-an-aardvark approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

tools Issues and PRs related to the tools directory.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Linter rule: use ASCII quotes not Unicode ones

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@SirR4T @addaleax @not-an-aardvark @BridgeAR @MylesBorins @Trott @nodejs-github-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.