GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings by joyeecheung · Pull Request #55217 · nodejs/node

Conversation

@joyeecheung

@joyeecheung joyeecheung commented Oct 1, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This backports the following PRs that are related to the unflagging of --experimental-require-module:

  • module: remove bogus assertion in CJS entrypoint handling with --import #54592
  • module: refator ESM loader for adding future synchronous hooks #54769
  • module: report unfinished TLA in ambiguous modules #54980
  • module: implement the "module-sync" exports condition #54648
  • module: unflag --experimental-require-module #55085

Note that I don't think this backport PR is strictly required - the PRs seem to land cleanly on v22.x except the unflagging commit has a small conflict in the YAML changelog description. This PR is mostly opened to look for regressions in v22 with CITGM.

Refs: #52697

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

kirillgroshkov, snewell92, styfle, sedghi, and MomenNano reacted with hooray emoji bricss, acommodari, marco-ippolito, jonkoops, kirillgroshkov, snewell92, styfle, and MomenNano reacted with rocket emoji
All reactions
  • 🎉 5 reactions
  • 🚀 8 reactions
@nodejs-github-bot

nodejs-github-bot commented Oct 1, 2024

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/loaders
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nodejs-github-bot nodejs-github-bot added lib / src Issues and PRs involving general changes in the lib/ or src/ directories. needs-ci PRs that need a full CI run. v22.x Issues that can be reproduced on v22.x or PRs targeting the v22.x-staging branch. labels Oct 1, 2024
@avivkeller avivkeller added module Issues and PRs related to the module subsystem. commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. labels Oct 1, 2024
@joyeecheung

joyeecheung commented Oct 1, 2024

Copy link
Copy Markdown
Member Author

CITGM for v22.x-staging: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3481/
CITGM for this PR: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3482/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@avivkeller avivkeller added the needs-citgm PRs that need a CITGM CI run. label Oct 1, 2024
@joyeecheung

joyeecheung commented Oct 2, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Some regressions found:

  1. yarn is expecting ERR_REQUIRE_ESM in its tests, probably to check that they didn't do anything to alter Node.js module loading https://github.com/yarnpkg/berry/blob/33f249b6cc06db1803f03eb5f17306a23d11115c/packages/acceptance-tests/pkg-tests-specs/sources/pnp-esm.test.ts#L617
  2. Gulp tests are breaking because they don't expect the experimental warning coming out of Node.js https://github.com/gulpjs/gulp/blob/2fa4981a910d7bdedb758bd09868620c9bb21d54/test/index.test.js#L90 (I am not sure how they load the .mjs configs, though it seems they are also doing a try-catch of require(config) somewhere)
  3. 2 also happened to Jest.
  4. In local testing, I also noticed that the debug package (used by npm, etc.) has a try { require(esm) } catch { /* fallback */ } pattern here, which only gets improvements out of require(esm) support after the unflagging, but then the experimental warning would also break the output.

I think for backporting to v22.x we should probably do some changes:

a. Provides process.features.require_module for feature detection
b. Do not emit the experimental warning if it's coming out of node_modules

b would not fix the test regressions 2-4, the packages still need to update their tests which is to be expected. But at least it would not break the CLIs for end users.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@joyeecheung

joyeecheung commented Oct 2, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

As for esm, from what I can tell in standard-things/esm#883 it seems to have been broken in different ways since many major releases of Node.js ago. The regression that predated unflagging of require(esm) I found in #55085 (comment) also seem to come from some TDZ issue in the way the package is written (it relies on access to Node.js internals and seem to expect certain internals to be invoked in certain order). It doesn't seem to be very cost-effective to do anything about it on v22.

kirillgroshkov reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@richardlau

richardlau commented Oct 2, 2024

Copy link
Copy Markdown
Member

I think for backporting to v22.x we should probably do some changes:

a. Provides process.features.require_module for feature detection
b. Do not emit the experimental warning if it's coming out of node_modules

b would not fix the test regressions 2-4, the packages still need to update their tests which is to be expected. But at least it would not break the CLIs for end users.

These SGTM

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@joyeecheung

joyeecheung commented Oct 2, 2024

Copy link
Copy Markdown
Member Author

Opened #55241 for feature detection.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@joyeecheung

joyeecheung commented Oct 4, 2024

Copy link
Copy Markdown
Member Author

Added a commit from #55243 and a commit for v22.x specifically (and maybe v20.x in the future) to disable the warning when require() comes from node_modules.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@targos

targos commented Oct 4, 2024

Copy link
Copy Markdown
Member

I just updated v22.x-staging so this now has conflicts because of duplicated commits I guess.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tats-u tats-u mentioned this pull request Oct 4, 2024
Enable consuming of ES modules in extensions microsoft/vscode#130367
Closed
@targos targos mentioned this pull request Oct 5, 2024
doc: fix typos #55066
Merged
This was referenced Oct 5, 2024
src: implement IsInsideNodeModules() in C++ #55286
Merged
Tracking issue: require(esm) #52697
Closed
@joyeecheung
joyeecheung force-pushed the backport-require-esm-2 branch from dda6fc3 to 9c282c3 Compare October 7, 2024 15:45
@joyeecheung
joyeecheung marked this pull request as ready for review October 7, 2024 15:47
@joyeecheung

joyeecheung commented Oct 7, 2024

Copy link
Copy Markdown
Member Author

Rebased to drop the duplicate commits. Also cherry-picked #55243 and #55241 from the main branch. I think for backport to v22.x it is ready, note that the last commit (silence the experimental warning when it comes from node_modules) is only targeting v22.x, as proposed in #55217 (comment) and agreed in the last TSC meeting.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@joyeecheung
joyeecheung force-pushed the backport-require-esm-2 branch from 9c282c3 to 3d0afe0 Compare October 7, 2024 15:52
@joyeecheung joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Oct 7, 2024
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Oct 7, 2024
@nodejs-github-bot

nodejs-github-bot commented Oct 7, 2024

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/62972/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@aduh95

aduh95 commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Can you rebase please?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@joyeecheung
joyeecheung force-pushed the backport-require-esm-2 branch 2 times, most recently from 5ed6462 to 898396a Compare October 8, 2024 19:00
mcollina
mcollina approved these changes Oct 9, 2024
View reviewed changes

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@joyeecheung
joyeecheung force-pushed the backport-require-esm-2 branch from 898396a to 5ea60dc Compare October 9, 2024 17:08
@nodejs-github-bot

nodejs-github-bot commented Nov 23, 2024

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/63677/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nodejs-github-bot

nodejs-github-bot commented Nov 24, 2024

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/63679/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ruyadorno

ruyadorno commented Nov 25, 2024

Copy link
Copy Markdown
Member

Thank you @joyeecheung!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

aduh95 pushed a commit that referenced this pull request Nov 26, 2024
@joyeecheung @aduh95
module: do not warn when require(esm) comes from node_modules
4be5047
As part of the standard experimental feature graduation
policy, when we unflagged require(esm) we moved the
experimental warning to be emitted when require() is
actually used to load ESM, which previously was an error.
However, some packages in the ecosystem have already
being using try-catch to load require(esm) to e.g.
resolve optional dependency, and emitting warning from
there instead of throwing directly could break the CLI
output.

To reduce the disruption for releases, as a compromise, this
patch skips the warning if require(esm) comes from
node_modules, where users typically don't have much control
over the code. This warning will be eventually removed
when require(esm) becomes stable.

This patch was originally intended for the LTS releases,
though it seems there's appetite for it on v23.x as
well so it's re-targeted to the main branch.

PR-URL: #55960
Refs: #55217
Refs: #52697
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Jacob Smith <[email protected]>
ruyadorno pushed a commit that referenced this pull request Nov 27, 2024
@joyeecheung @ruyadorno
module: unflag --experimental-require-module
58d6871
This unflags --experimental-require-module so require(esm) can be
used without the flag. For now, when require() actually encounters
an ESM, it will still emit an experimental warning. To opt out
of the feature, --no-experimental-require-module can be used.

There are some tests specifically testing ERR_REQUIRE_ESM. Some
of them are repurposed to test --no-experimental-require-module.
Some of them are modified to just expect loading require(esm) to
work, when it's appropriate.

PR-URL: #55085
Backport-PR-URL: #55217
Refs: #52697
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Marco Ippolito <[email protected]>
Reviewed-By: Rafael Gonzaga <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: LiviaMedeiros <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Filip Skokan <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
Reviewed-By: Richard Lau <[email protected]>
ruyadorno pushed a commit that referenced this pull request Nov 27, 2024
@guybedford @ruyadorno
module: support 'module.exports' interop export in require(esm)
545c069
PR-URL: #54563
Backport-PR-URL: #55217
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
Refs: #52697
ruyadorno pushed a commit that referenced this pull request Nov 27, 2024
@aduh95 @ruyadorno
doc: update require(ESM) history and stability status
f1b9791
PR-URL: #55199
Backport-PR-URL: #55217
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Guy Bedford <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
Refs: #52697
ruyadorno pushed a commit that referenced this pull request Nov 27, 2024
@joyeecheung @ruyadorno
module: use kNodeModulesRE to detect node_modules
d8c34ce
This is faster and more consistent with other places using the
regular expression to detect node_modules.

PR-URL: #55243
Backport-PR-URL: #55217
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Jacob Smith <[email protected]>
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Marco Ippolito <[email protected]>
Refs: #52697
ruyadorno pushed a commit that referenced this pull request Nov 27, 2024
@joyeecheung @ruyadorno
module: check --experimental-require-module separately from detection
fcdd616
Previously we assumed if `--experimental-detect-module` is true, then
`--experimental-require-module` is true, which isn't the case, as
the two can be enabled/disabled separately. This patch fixes the
checks so `--no-experimental-require-module` is still effective when
`--experimental-detect-module` is enabled.

Drive-by: make the assertion messages more informative and remove
obsolete TODO about allowing TLA in entrypoints handled by
require(esm).

PR-URL: #55250
Backport-PR-URL: #55217
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Jacob Smith <[email protected]>
Reviewed-By: Rafael Gonzaga <[email protected]>
Refs: #52697
ruyadorno pushed a commit that referenced this pull request Nov 27, 2024
@joyeecheung @ruyadorno
src: implement IsInsideNodeModules() in C++
2af72a7
This previously compiles a script and run it in a new context
to avoid global pollution, which is more complex than necessary
and can be too slow for it to be reused in other cases. The
new implementation just checks the frames in C++ which is safe
from global pollution, faster and simpler.

The previous implementation also had a bug when the call site
is in a ESM, because ESM have URLs as their script names,
which don't start with '/' or '\' and will be skipped. The new
implementation removes the skipping to fix it for ESM.

PR-URL: #55286
Backport-PR-URL: #55217
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Chengzhong Wu <[email protected]>
Refs: #52697
ruyadorno pushed a commit that referenced this pull request Nov 27, 2024
@joyeecheung @ruyadorno
module: include module information in require(esm) warning
6ac3400
When emitting the experimental warning for `require(esm)`, include
information about the parent module and the module being require()-d
to help users locate and update them.

PR-URL: #55397
Backport-PR-URL: #55217
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Stephen Belanger <[email protected]>
Reviewed-By: Chemi Atlow <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Chengzhong Wu <[email protected]>
Refs: #52697
ruyadorno pushed a commit that referenced this pull request Nov 27, 2024
@joyeecheung @ruyadorno
module: allow ESM that failed to be required to be re-imported
28b5b9a
When a ESM module cannot be loaded by require due to the presence
of TLA, its module status would be stopped at kInstantiated. In
this case, when it's imported again, we should allow it to be
evaluated asynchronously, as it's also a common pattern for users
to retry with dynamic import when require fails.

PR-URL: #55502
Backport-PR-URL: #55217
Fixes: #55500
Refs: #52697
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Chemi Atlow <[email protected]>
ruyadorno pushed a commit that referenced this pull request Nov 27, 2024
@joyeecheung @ruyadorno
module: trim off internal stack frames for require(esm) warnings
16119f2
Trim off irrelevant internal stack frames for require(esm) warnings
so it's easier to locate where the call comes from when
--trace-warnings is used.

PR-URL: #55496
Backport-PR-URL: #55217
Reviewed-By: Marco Ippolito <[email protected]>
Reviewed-By: Paolo Insogna <[email protected]>
Refs: #52697
ruyadorno pushed a commit that referenced this pull request Nov 27, 2024
@joyeecheung @ruyadorno
module: do not warn when require(esm) comes from node_modules
cb527a9
As part of the standard experimental feature graduation
policy, when we unflagged require(esm) we moved the
experimental warning to be emitted when require() is
actually used to load ESM, which previously was an error.
However, some packages in the ecosystem have already
being using try-catch to load require(esm) to e.g.
resolve optional dependency, and emitting warning from
there instead of throwing directly could break the CLI
output.

To reduce the disruption for releases, as a compromise, this
patch skips the warning if require(esm) comes from
node_modules, where users typically don't have much control
over the code. This warning will be eventually removed
when require(esm) becomes stable.

This patch was originally intended for the LTS releases,
though it seems there's appetite for it on v23.x as
well so it's re-targeted to the main branch.

PR-URL: #55960
Backport-PR-URL: #55217
Refs: #55217
Refs: #52697
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Jacob Smith <[email protected]>
@ruyadorno

ruyadorno commented Nov 27, 2024

Copy link
Copy Markdown
Member

Landed in 9ffd2dd...cb527a9

bernardobelchior, nicolo-ribaudo, robpalme, snewell92, ggoodman, styfle, jacobroschen, bjohansebas, jonkoops, sxzz, and 2 more reacted with hooray emoji bernardobelchior reacted with heart emoji
All reactions
  • 🎉 12 reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ruyadorno ruyadorno closed this Nov 27, 2024
@BPScott BPScott mentioned this pull request Nov 29, 2024
Improve debug message when json parsing fails in e2e tests prettier/stylelint-prettier#376
Merged
@Murderlon Murderlon mentioned this pull request Dec 2, 2024
feat(webdav): initial implementation for a WebDAV simple auth provider transloadit/uppy#5528
Closed
@zachleat zachleat mentioned this pull request Dec 3, 2024
Node unflagged require(esm) in Node 22 11ty/buildawesome#3568
Closed
@tats-u tats-u mentioned this pull request Dec 10, 2024
yarn test (node --run test) fails in Node.js 22.12 due to requiring top-level await module prettier/prettier#16930
Closed
joyeecheung added a commit to joyeecheung/node that referenced this pull request Feb 6, 2025
@joyeecheung
module: do not warn when require(esm) comes from node_modules
3403846
As part of the standard experimental feature graduation
policy, when we unflagged require(esm) we moved the
experimental warning to be emitted when require() is
actually used to load ESM, which previously was an error.
However, some packages in the ecosystem have already
being using try-catch to load require(esm) to e.g.
resolve optional dependency, and emitting warning from
there instead of throwing directly could break the CLI
output.

To reduce the disruption for releases, as a compromise, this
patch skips the warning if require(esm) comes from
node_modules, where users typically don't have much control
over the code. This warning will be eventually removed
when require(esm) becomes stable.

This patch was originally intended for the LTS releases,
though it seems there's appetite for it on v23.x as
well so it's re-targeted to the main branch.

PR-URL: nodejs#55960
Refs: nodejs#55217
Refs: nodejs#52697
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Jacob Smith <[email protected]>
joyeecheung added a commit to joyeecheung/node that referenced this pull request Feb 10, 2025
@joyeecheung
module: do not warn when require(esm) comes from node_modules
f6fc5c6
As part of the standard experimental feature graduation
policy, when we unflagged require(esm) we moved the
experimental warning to be emitted when require() is
actually used to load ESM, which previously was an error.
However, some packages in the ecosystem have already
being using try-catch to load require(esm) to e.g.
resolve optional dependency, and emitting warning from
there instead of throwing directly could break the CLI
output.

To reduce the disruption for releases, as a compromise, this
patch skips the warning if require(esm) comes from
node_modules, where users typically don't have much control
over the code. This warning will be eventually removed
when require(esm) becomes stable.

This patch was originally intended for the LTS releases,
though it seems there's appetite for it on v23.x as
well so it's re-targeted to the main branch.

PR-URL: nodejs#55960
Refs: nodejs#55217
Refs: nodejs#52697
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Jacob Smith <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Feb 11, 2025
@joyeecheung @marco-ippolito
module: do not warn when require(esm) comes from node_modules
e01dd4b
As part of the standard experimental feature graduation
policy, when we unflagged require(esm) we moved the
experimental warning to be emitted when require() is
actually used to load ESM, which previously was an error.
However, some packages in the ecosystem have already
being using try-catch to load require(esm) to e.g.
resolve optional dependency, and emitting warning from
there instead of throwing directly could break the CLI
output.

To reduce the disruption for releases, as a compromise, this
patch skips the warning if require(esm) comes from
node_modules, where users typically don't have much control
over the code. This warning will be eventually removed
when require(esm) becomes stable.

This patch was originally intended for the LTS releases,
though it seems there's appetite for it on v23.x as
well so it's re-targeted to the main branch.

PR-URL: #55960
Backport-PR-URL: #56927
Refs: #55217
Refs: #52697
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Jacob Smith <[email protected]>
@aladdin-add aladdin-add mentioned this pull request Feb 23, 2025
bug report: eslint-plugin-no-autofix with ESM-only packages aladdin-add/eslint-plugin#104
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aduh95 aduh95 aduh95 approved these changes
@mcollina mcollina mcollina approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. lib / src Issues and PRs involving general changes in the lib/ or src/ directories. module Issues and PRs related to the module subsystem. needs-ci PRs that need a full CI run. needs-citgm PRs that need a CITGM CI run. v22.x Issues that can be reproduced on v22.x or PRs targeting the v22.x-staging branch.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

12 participants

@joyeecheung @nodejs-github-bot @richardlau @targos @aduh95 @nicolo-ribaudo @slorber @ruyadorno @mcollina @panva @avivkeller @guybedford
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.