GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — crypto, string_bytes: treat `buffer` str as `utf8` by indutny · Pull Request #5522 · nodejs/node

Conversation

@indutny

@indutny indutny commented Mar 2, 2016

Copy link
Copy Markdown
Member

Pull Request check-list

Please make sure to review and check all of these items:

  • Does make -j8 test (UNIX) or vcbuild test nosign (Windows) pass with
    this change (including linting)?
  • Is the commit message formatted according to CONTRIBUTING.md?
  • If this change fixes a bug (or a performance problem), is a regression
    test (or a benchmark) included?
  • Is a documentation update included (if this change modifies
    existing APIs, or introduces new ones)?

NOTE: these things are not required to open a PR and can be done
afterwards / while the PR is open.

Affected core subsystem(s)

Please provide affected core subsystem(s) (like buffer, cluster, crypto, etc)

Description of change

Do not treat crypto inputs as binary strings, convert them to Buffers
using new Buffer(..., 'utf8'), or using newly updated StringBytes
APIs.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@indutny
crypto, string_bytes: treat buffer str as utf8
6bd1a8e
Do not treat crypto inputs as `binary` strings, convert them to Buffers
using `new Buffer(..., 'utf8')`, or using newly updated StringBytes
APIs.
@indutny

indutny commented Mar 2, 2016

Copy link
Copy Markdown
Member Author

See #5504

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@indutny

indutny commented Mar 2, 2016

Copy link
Copy Markdown
Member Author

cc @nodejs/collaborators @nodejs/ctc

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@indutny indutny mentioned this pull request Mar 2, 2016
src: remove BINARY encoding #5504
Closed
4 tasks done
@Trott Trott added the crypto Issues and PRs related to the crypto subsystem. label Mar 2, 2016
@indutny

indutny commented Mar 2, 2016

Copy link
Copy Markdown
Member Author

CI: https://ci.nodejs.org/job/node-test-pull-request/1806/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Fishrock123

Fishrock123 commented Mar 2, 2016

Copy link
Copy Markdown
Contributor

Is this semver-major?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@indutny indutny added the semver-major PRs that contain breaking changes and should be released in the next major version. label Mar 2, 2016
@indutny

indutny commented Mar 2, 2016

Copy link
Copy Markdown
Member Author

It is.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bnoordhuis

bnoordhuis commented Mar 2, 2016

Copy link
Copy Markdown
Member

Can you remove BINARY from enum encoding in src/node.h as well?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bnoordhuis

bnoordhuis commented Mar 2, 2016

Copy link
Copy Markdown
Member

Oh wait, the intent is to change the default, not remove it altogether? LGTM in that case.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@indutny

indutny commented Mar 2, 2016

Copy link
Copy Markdown
Member Author

Thank you, @bnoordhuis .

How should we handle this major breakage @nodejs/ctc ? There is hardly a way to deprecate it, because it is a change of default value. Should we just land it and write some blog posts on this change?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jasnell

jasnell commented Mar 2, 2016

Copy link
Copy Markdown
Member

Yeah, I think that's likely best. Not sure there's a better way.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jasnell

jasnell commented Mar 2, 2016

Copy link
Copy Markdown
Member

LGTM

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@indutny

indutny commented Mar 2, 2016

Copy link
Copy Markdown
Member Author

Landed in b010c87, thank you!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@indutny indutny closed this Mar 2, 2016
@indutny
indutny deleted the feature/no-default-binary-in-crypto.js branch March 2, 2016 18:25
indutny added a commit that referenced this pull request Mar 2, 2016
@indutny
crypto, string_bytes: treat buffer str as utf8
b010c87
Do not treat crypto inputs as `binary` strings, convert them to Buffers
using `new Buffer(..., 'utf8')`, or using newly updated StringBytes
APIs.

PR-URL: #5522
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: James M Snell <[email protected]>
@Fishrock123

Fishrock123 commented Mar 2, 2016

Copy link
Copy Markdown
Contributor

@indutny Please don't land major changes so fast?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jasnell

jasnell commented Mar 2, 2016

Copy link
Copy Markdown
Member

Yeah... for stuff like this perhaps give it a day or two to give more people a chance to review. I wouldn't anticipate any issues with this one but still...

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@indutny

indutny commented Mar 2, 2016

Copy link
Copy Markdown
Member Author

Oh, sorry... What's our process on this?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jasnell

jasnell commented Mar 2, 2016

Copy link
Copy Markdown
Member

For a major I think it's something like at least 48 hours.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mscdex
mscdex reviewed Mar 2, 2016
View reviewed changes
Comment thread doc/api/crypto.markdown

The `crypto.DEFAULT_ENCODING` mechanism is provided for backwards compatibility
with legacy programs that expect `'binary'` to be the default encoding.
with legacy programs that expect `'utf8'` to be the default encoding.

@mscdex mscdex Mar 2, 2016

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence doesn't make much sense anymore since legacy programs were not assuming utf8, but binary.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@indutny indutny Mar 2, 2016

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gosh, this is true.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@indutny indutny Mar 2, 2016

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May I ask you to submit follow-up PR to fix this?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@Trott

Trott commented Mar 2, 2016

Copy link
Copy Markdown
Member

For reference (so there can be broad agreement or else we can talk about updating it to be more accurate) from the current version of the Collaborator Guide:

Before landing pull requests, sufficient time should be left for input from other Collaborators. Leave at least 48 hours during the week and 72 hours over weekends to account for international time differences and work schedules. Trivial changes (e.g. those which fix minor bugs or improve performance without affecting API or causing other wide-reaching impact) may be landed after a shorter delay.

It does not appear to be in the Collaborator Guide, but the doc for onboarding, or at least the one I bookmarked at my onboarding, says:

semver-major (breaking) changes must be reviewed in some form by the CTC

If that's still correct, it's probably worth mentioning in the Collaborator Guide. (We may also want to get at least a tiny bit more specific than "in some form".)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@rvagg

rvagg commented Mar 3, 2016

Copy link
Copy Markdown
Member

For anything but trivial or otherwise urgent changes we should be leaving at least 48 hours. This is kind of major and deserves the chance for more discussion to flush out any possible objections.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jasnell jasnell mentioned this pull request Mar 17, 2016
Planning for v6 #5766
Closed
@nolanlawson nolanlawson mentioned this pull request Apr 28, 2016
node6 is permafail apache/pouchdb#5102
Closed
@fanatid fanatid mentioned this pull request Apr 29, 2016
utf8 as default encoding browserify/hash-base#4
Merged
fanatid added a commit to browserify/hash-base that referenced this pull request Apr 29, 2016
@fanatid
utf8 as default encoding
5d95d75
details: nodejs/node#5522
@ArtemGovorov ArtemGovorov mentioned this pull request Apr 29, 2016
Node 6.0 support for browser tests wallabyjs/public#568
Closed
omsmith added a commit to omsmith/pouchdb that referenced this pull request Apr 29, 2016
@omsmith
(apache#5102) - specify md5 data encoding
c3704f3
The default encoding was switched to utf8 in nodejs/node#5522
@omsmith omsmith mentioned this pull request Apr 29, 2016
(#5102) - specify md5 data encoding apache/pouchdb#5107
Merged
@ArtemGovorov ArtemGovorov mentioned this pull request Apr 29, 2016
hixie 76 is broken in node 6 websockets/ws#716
Closed
omsmith added a commit to omsmith/pouchdb that referenced this pull request Apr 29, 2016
@omsmith
(apache#5102) - specify md5 data encoding
48bda89
The default encoding was switched to utf8 in nodejs/node#5522
daleharvey pushed a commit to apache/pouchdb that referenced this pull request Apr 29, 2016
@omsmith @daleharvey
(#5102) - Specify md5 data encoding
3d7d497
The default encoding was switched to utf8 in nodejs/node#5522
@ChALkeR ChALkeR mentioned this pull request May 1, 2016
Doesn't work under Node.js 6.0.0 brianc/node-postgres#1000
Closed
@tejasmanohar

tejasmanohar commented May 11, 2016 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Just saw websockets/ws#716. Did someone write a blog post / example in docs on what Node users need to change (for example, with hash#update calls)?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@fanatid fanatid mentioned this pull request May 19, 2016
Bug in sha implementation (likely due to international characters) browserify/sha.js#37
Closed
@tejasmanohar tejasmanohar mentioned this pull request Jun 13, 2016
specify data encoding type 'binary' at hash#update websockets/ws#746
Merged
@addaleax addaleax mentioned this pull request Jul 13, 2016
Crypto - sign.update, verifier.update default parameter for strings 'utf8' not really true? #7712
Closed
This was referenced Sep 15, 2016
Code and Learn at Node.js Interactive in Amsterdam nodejs/code-and-learn#56
Closed
crypto: fix default encoding of LazyTransform #8611
Closed
@sam-github

sam-github commented Dec 14, 2016

Copy link
Copy Markdown
Contributor

This PR made sweeping backwards-incompatible changes to the crypto API, but did not include documentation. cf. #10265

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@indutny

indutny commented Dec 14, 2016

Copy link
Copy Markdown
Member Author

@sam-github clearly this PR changes all occurrences of binary to utf8 in the docs. What kind of documentation do you mean?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sam-github

sam-github commented Dec 14, 2016

Copy link
Copy Markdown
Contributor

So, that basically boils down to "existing behaviour was undocumented, so there were no docs to change", which is one way to look at it.

I'm not necessarily saying that this PR shouldn't have landed without you being forced to document the crypto APIs, but I am saying that users of APIs like pbkdf2() (see #10265) could never have figured out from the docs that there has been a change, which is a problem that needs flagging.

Which is why I commented here, so that future readers of this PR can know that the changes it made weren't documented.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@indutny

indutny commented Dec 14, 2016

Copy link
Copy Markdown
Member Author

I agree we could do better on this. However, careful reader of docs would notice:

https://github.com/nodejs/node/pull/5522/files#diff-efc9444c79e30e68a167275387a55aa6R813

Which says that default encoding is utf8.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@indutny

indutny commented Dec 14, 2016

Copy link
Copy Markdown
Member Author

So I can't say that it was completely undocumented, just wasn't in pbkdf documentation itself.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@dschenkelman dschenkelman mentioned this pull request Jan 20, 2017
Fix for node 6.x. Be explicit about crypto update input encoding latin1 mcastany/passport-wsfed-saml2#1
Merged
lmoe pushed a commit to lmoe/node that referenced this pull request Feb 16, 2017
Lukas Möller
crypto: fix default encoding of LazyTransform
fe9de5d
PullRequest nodejs#5522 and nodejs#5500 described the change
of the default encoding into UTF8 in crypto functions.

This however was only changed for the non-streaming API.
The streaming API still used binary as the default encoding.

This commit will change the default streaming API encoding to UTF8
to make both APIs behave the same.

It will also add tests to validate the behavior.
addaleax pushed a commit that referenced this pull request Mar 11, 2017
Lukas Möller @addaleax
crypto: fix default encoding of LazyTransform
443691a
PullRequest #5522 and #5500 described the change
of the default encoding into UTF8 in crypto functions.

This however was only changed for the non-streaming API.
The streaming API still used binary as the default encoding.

This commit will change the default streaming API encoding to UTF8
to make both APIs behave the same.

It will also add tests to validate the behavior.

Refs: #5522
Refs: #5500
PR-URL: #8611
Reviewed-By: Fedor Indutny <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
jungx098 pushed a commit to jungx098/node that referenced this pull request Mar 21, 2017
Lukas Möller @jungx098
crypto: fix default encoding of LazyTransform
98659fd
PullRequest nodejs#5522 and nodejs#5500 described the change
of the default encoding into UTF8 in crypto functions.

This however was only changed for the non-streaming API.
The streaming API still used binary as the default encoding.

This commit will change the default streaming API encoding to UTF8
to make both APIs behave the same.

It will also add tests to validate the behavior.

Refs: nodejs#5522
Refs: nodejs#5500
PR-URL: nodejs#8611
Reviewed-By: Fedor Indutny <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
@gits2501 gits2501 mentioned this pull request Sep 28, 2017
More node version for testing gits2501/Hmac_Sha1#4
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

crypto Issues and PRs related to the crypto subsystem. semver-major PRs that contain breaking changes and should be released in the next major version.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@indutny @Fishrock123 @bnoordhuis @jasnell @Trott @rvagg @tejasmanohar @sam-github @mscdex
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.