GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — build,tools: make addons tests work with GN by zcbenz · Pull Request #50737 · nodejs/node

Conversation

@zcbenz

@zcbenz zcbenz commented Nov 15, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This PR refactors the install and build_addons scripts so they can be reused by node-ci and electron projects to build the addons tests.

In detail, this PR does following things:

  1. Minor modifications of the GN configurations to fix loading native modules.
  2. Add some flags to tools/install.py script to allow customize locations of config.gypi, v8, etc. With some refactoring to remove global variables.
  3. Rewrite the tools/build_addons.py script:
    1. Rewrite it from js to python, because implementing cmd args handling with vanilla Node.js would take much longer time than a simple rewrite.
    2. Add a few more flags to make it easier to use for embedders.
  4. Modify the Makefile to generate headers in out dir first, and then calls build_addons.py --headers-dir=out_dir/headers (which calls node-gyp --nodedir=out_dir/headers) to build addons. Previously the build system was just calling node-gyp --nodedir=., which relied on quirky behavior of node-gyp and did not really test whether the generated headers work.

Users that rely on make or vcbuild.bat to work on Node are not affected.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@nodejs-github-bot

nodejs-github-bot commented Nov 15, 2023

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/tsc
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nodejs-github-bot nodejs-github-bot added build Issues and PRs related to Node.js builds or CI infrastructure. dependencies PRs that add, update, or configure Node.js dependencies. libuv Issues and PRs related to the libuv dependency or the uv binding. needs-ci PRs that need a full CI run. openssl Issues and PRs related to the OpenSSL dependency. python PRs and issues that require attention from people who are familiar with Python. tools Issues and PRs related to the tools directory. windows Issues and PRs related to the Windows platform. labels Nov 15, 2023
@zcbenz
zcbenz force-pushed the build-addons-gn branch 2 times, most recently from 507f7ab to 353afc9 Compare November 15, 2023 11:06
@zcbenz zcbenz mentioned this pull request Nov 15, 2023
Things blocking a full-featured GN build of upstream Node.js photoionization/node_with_gn#1
Closed
26 of 27 tasks
@zcbenz
zcbenz force-pushed the build-addons-gn branch 2 times, most recently from 0940864 to 13f87fc Compare November 16, 2023 07:32
@zcbenz

zcbenz commented Nov 16, 2023

Copy link
Copy Markdown
Contributor Author

I have separated some changes out of this PR so review should be easier.

/cc @nodejs/gyp @joyeecheung @richardlau for reviewing changes on tools.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

joyeecheung
joyeecheung reviewed Nov 16, 2023
View reviewed changes
Comment thread tools/build_addons.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@gengjiawen gengjiawen added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Nov 17, 2023
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Nov 17, 2023
@nodejs-github-bot

nodejs-github-bot commented Nov 17, 2023

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/55698/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zcbenz

zcbenz commented Nov 28, 2023

Copy link
Copy Markdown
Contributor Author

Ping @nodejs/gyp @joyeecheung @anonrig for review after the holiday.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

joyeecheung
joyeecheung approved these changes Nov 28, 2023
View reviewed changes

@joyeecheung joyeecheung left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % a correction but I think ideally we should have someone from @nodejs/build to confirm that it works with our releases.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Makefile Resolved

Uh oh!

There was an error while loading. Please reload this page.

@zcbenz
zcbenz force-pushed the build-addons-gn branch from 118f738 to 5348591 Compare November 29, 2023 00:09
benjamingr
benjamingr approved these changes Nov 29, 2023
View reviewed changes
@zcbenz zcbenz mentioned this pull request Dec 1, 2023
Create a branch on latest main v8/node#172
Closed
@gengjiawen gengjiawen added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Dec 4, 2023
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Dec 4, 2023
@nodejs-github-bot

nodejs-github-bot commented Dec 4, 2023

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/56066/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zcbenz
zcbenz force-pushed the build-addons-gn branch 3 times, most recently from 974401b to 3be1111 Compare January 4, 2024 02:28
@zcbenz
zcbenz deleted the build-addons-gn branch February 23, 2024 07:17
marco-ippolito pushed a commit that referenced this pull request Feb 26, 2024
@zcbenz @marco-ippolito
build,tools: make addons tests work with GN
0f1bbea
PR-URL: #50737
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Feb 27, 2024
@zcbenz @marco-ippolito
build,tools: make addons tests work with GN
5911731
PR-URL: #50737
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
@batrla

batrla commented Feb 28, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Sorry to report it, but this PR caused regression, build in --shared mode now fails:

11:02:41: stdout installing /some_dir/build/prototype/i386/some_bindir/node
11:02:41: stderr Traceback (most recent call last):
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 419, in <module>
11:02:41: stderr     run(parse_options(sys.argv[1:]))
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 372, in run
11:02:41: stderr     files(options, install)
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 179, in files
11:02:41: stderr     action(options, [os.path.join(output_prefix, output_lib)],
11:02:41: stderr                                   ^^^^^^^^^^^^^
11:02:42: stderr UnboundLocalError: cannot access local variable 'output_prefix' where it is not associated with a value   

The problem in source code is obvious, the variable 'output_prefix' is only defined in some branches.
Later in code the variable is used in a branch where it was not defined before.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zcbenz

zcbenz commented Feb 28, 2024

Copy link
Copy Markdown
Contributor Author

Sorry for breaking it! I'll create a fix soon.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

zcbenz added a commit to zcbenz/node that referenced this pull request Feb 28, 2024
@zcbenz
tools: fix installing node with shared mode
a0e52cb
Fix a bug caused by nodejs#50737 that,
make install fails for --shared mode.
@zcbenz zcbenz mentioned this pull request Feb 28, 2024
tools: fix installing node with shared mode #51910
Merged
@marco-ippolito marco-ippolito mentioned this pull request Mar 1, 2024
v21.7.0 proposal #51932
Merged
nodejs-github-bot pushed a commit that referenced this pull request Mar 1, 2024
@zcbenz
tools: fix installing node with shared mode
6aa1d98
Fix a bug caused by #50737 that,
make install fails for --shared mode.

PR-URL: #51910
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Mar 1, 2024
@zcbenz @marco-ippolito
tools: fix installing node with shared mode
27f8549
Fix a bug caused by #50737 that,
make install fails for --shared mode.

PR-URL: #51910
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
zcbenz pushed a commit to photoionization/node_ci that referenced this pull request Mar 6, 2024
pthier Node-ci LUCI CQ
Update node
625aecc
- Use node in-tree gn configs when possible.
  Some in-tree configs (deps/openssl) were modified in our mirror until
  they are fixed in the upstream node repo [1].
- Trigger landmine to clean out directory (required due to changed
  targets with new build files)
- Update to node-ci-2024-01-17
- Add fp16 to update_deps.py

[1] nodejs/node#50737.

Bug: v8:14572

Change-Id: I86cc19d23151b2a64d7173cf8f0aa6adc417c091
Reviewed-on: https://chromium-review.googlesource.com/c/v8/node-ci/+/5203209
Commit-Queue: Patrick Thier <[email protected]>
Reviewed-by: Victor Gomes <[email protected]>
richardlau pushed a commit that referenced this pull request Mar 25, 2024
@zcbenz @richardlau
build,tools: make addons tests work with GN
ffe467b
PR-URL: #50737
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
richardlau pushed a commit that referenced this pull request Mar 25, 2024
@zcbenz @richardlau
tools: fix installing node with shared mode
f4ac7ba
Fix a bug caused by #50737 that,
make install fails for --shared mode.

PR-URL: #51910
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
@richardlau richardlau mentioned this pull request Mar 25, 2024
v20.12.0 proposal #52212
Merged
@vmoroz vmoroz mentioned this pull request Apr 22, 2024
build,tools,node-api: fix building node-api tests for Windows Debug #52632
Merged
@codebytere codebytere mentioned this pull request Jun 6, 2024
build: fix generate_gn_filenames_json.py electron/electron#42389
Merged
2 of 3 tasks
codebytere added a commit to electron/electron that referenced this pull request Oct 16, 2024
@codebytere
build,tools: make addons tests work with GN
5ee874b
nodejs/node#50737
codebytere added a commit to electron/electron that referenced this pull request Oct 21, 2024
@codebytere
build,tools: make addons tests work with GN
e4038c5
nodejs/node#50737
codebytere added a commit to electron/electron that referenced this pull request Oct 21, 2024
@codebytere
build,tools: make addons tests work with GN
527b190
nodejs/node#50737
codebytere added a commit to electron/electron that referenced this pull request Oct 22, 2024
@codebytere
build,tools: make addons tests work with GN
2cea186
nodejs/node#50737
codebytere added a commit to electron/electron that referenced this pull request Oct 28, 2024
@codebytere
build,tools: make addons tests work with GN
3785fdc
nodejs/node#50737
codebytere added a commit to electron/electron that referenced this pull request Oct 29, 2024
@codebytere
build,tools: make addons tests work with GN
4d357a8
nodejs/node#50737
codebytere added a commit to electron/electron that referenced this pull request Oct 31, 2024
@codebytere
build,tools: make addons tests work with GN
7ae4c77
nodejs/node#50737
jkleinsc pushed a commit to electron/electron that referenced this pull request Nov 4, 2024
@codebytere
chore: bump Node.js to v22.9.0 (#44281)
c63d0d6
* chore: bump Node.js to v22.9.0

* build: drop base64 dep in GN build

nodejs/node#52856

* build,tools: make addons tests work with GN

nodejs/node#50737

* fs: add fast api for InternalModuleStat

nodejs/node#51344

* src: move package_json_reader cache to c++

nodejs/node#50322

* crypto: disable PKCS#1 padding for privateDecrypt

nodejs-private/node-private#525

* src: move more crypto code to ncrypto

nodejs/node#54320

* crypto: ensure valid point on elliptic curve in SubtleCrypto.importKey

nodejs/node#50234

* src: shift more crypto impl details to ncrypto

nodejs/node#54028

* src: switch crypto APIs to use Maybe<void>

nodejs/node#54775

* crypto: remove DEFAULT_ENCODING

nodejs/node#47182

* deps: update libuv to 1.47.0

nodejs/node#50650

* build: fix conflict gyp configs

nodejs/node#53605

* lib,src: drop --experimental-network-imports

nodejs/node#53822

* esm: align sync and async load implementations

nodejs/node#49152

* esm: remove unnecessary toNamespacedPath calls

nodejs/node#53656

* module: detect ESM syntax by trying to recompile as SourceTextModule

nodejs/node#52413

* test: adapt debugger tests to V8 11.4

nodejs/node#49639

* lib: update usage of always on Atomics API

nodejs/node#49639

* test: adapt test-fs-write to V8 internal changes

nodejs/node#49639

* test: adapt to new V8 trusted memory spaces

nodejs/node#50115

* deps: update libuv to 1.47.0

nodejs/node#50650

* src: use non-deprecated v8::Uint8Array::kMaxLength

nodejs/node#50115

* src: update default V8 platform to override functions with location

nodejs/node#51362

* src: add missing TryCatch

nodejs/node#51362

* lib,test: handle new Iterator global

nodejs/node#51362

* src: use non-deprecated version of CreateSyntheticModule

nodejs/node#50115

* src: remove calls to recently deprecated V8 APIs

nodejs/node#52996

* src: use new V8 API to define stream accessor

nodejs/node#53084

* src: do not use deprecated V8 API

nodejs/node#53084

* src: do not use soon-to-be-deprecated V8 API

nodejs/node#53174

* src: migrate to new V8 interceptors API

nodejs/node#52745

* src: use supported API to get stalled TLA messages

nodejs/node#51362

* module: print location of unsettled top-level await in entry points

nodejs/node#51999

* test: make snapshot comparison more flexible

nodejs/node#54375

* test: do not set concurrency on parallelized runs

nodejs/node#52177

* src: move FromNamespacedPath to path.cc

nodejs/node#53540

* test: adapt to new V8 trusted memory spaces

nodejs/node#50115

* build: add option to enable clang-cl on Windows

nodejs/node#52870

* chore: fixup patch indices

* chore: add/remove changed files

* esm: drop support for import assertions

nodejs/node#54890

* build: compile with C++20 support

nodejs/node#52838

* deps: update nghttp2 to 1.62.1

nodejs/node#52966

* src: parse inspector profiles with simdjson

nodejs/node#51783

* build: add GN build files

nodejs/node#47637

* deps,lib,src: add experimental web storage

nodejs/node#52435

* build: add missing BoringSSL dep

* src: rewrite task runner in c++

nodejs/node#52609

* fixup! build: add GN build files

* src: stop using deprecated fields of v8::FastApiCallbackOptions

nodejs/node#54077

* fix: shadow variable

* build: add back incorrectly removed SetAccessor patch

* fixup! fixup! build: add GN build files

* crypto: fix integer comparison in crypto for BoringSSL

* src,lib: reducing C++ calls of esm legacy main resolve

nodejs/node#48325

* src: move more crypto_dh.cc code to ncrypto

nodejs/node#54459

* chore: fixup GN files for previous commit

* src: move more crypto code to ncrypto

nodejs/node#54320

* Fixup Perfetto ifdef guards

* fix: missing electron_natives dep

* fix: node_use_node_platform = false

* fix: include src/node_snapshot_stub.cc in libnode

* 5507047: [import-attributes] Remove support for import assertions

https://chromium-review.googlesource.com/c/v8/v8/+/5507047

* fix: restore v8-sandbox.h in filenames.json

* fix: re-add original-fs generation logic

* fix: ngtcp2 openssl dep

* test: try removing NAPI_VERSION undef

* chore(deps): bump @types/node

* src: move more crypto_dh.cc code to ncrypto

nodejs/node#54459

* esm: remove unnecessary toNamespacedPath calls

nodejs/node#53656

* buffer: fix out of range for toString

nodejs/node#54553

* lib: rewrite AsyncLocalStorage without async_hooks

nodejs/node#48528

* module: print amount of load time of a cjs module

nodejs/node#52213

* test: skip reproducible snapshot test on 32-bit

nodejs/node#53592

* fixup! src: move more crypto_dh.cc code to ncrypto

* test: adjust emittedUntil return type

* chore: remove redundant wpt streams patch

* fixup! chore(deps): bump @types/node

* fix: gn executable name on Windows

* fix: build on Windows

* fix: rename conflicting win32 symbols in //third_party/sqlite

On Windows otherwise we get:

lld-link: error: duplicate symbol: sqlite3_win32_write_debug
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:47987
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_sleep
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48042
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_is_nt
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48113
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_unicode
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48470
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_unicode_to_utf8
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48486
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_mbcs_to_utf8
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48502
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_mbcs_to_utf8_v2
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48518
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_mbcs
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48534
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_mbcs_v2
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48550
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

* docs: remove unnecessary ts-expect-error after types bump

* src: move package resolver to c++

nodejs/node#50322

* build: set ASAN detect_container_overflow=0

nodejs/node#55584

* chore: fixup rebase

* test: disable failing ASAN test

* win: almost fix race detecting ESRCH in uv_kill

libuv/libuv#4341
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joyeecheung joyeecheung joyeecheung approved these changes
@benjamingr benjamingr benjamingr approved these changes
@anonrig anonrig anonrig approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

build Issues and PRs related to Node.js builds or CI infrastructure. commit-queue-failed PRs whose Commit Queue landing failed and need manual intervention before retrying. commit-queue-squash PRs the Commit Queue should land as one squashed commit. dependencies PRs that add, update, or configure Node.js dependencies. libuv Issues and PRs related to the libuv dependency or the uv binding. needs-ci PRs that need a full CI run. openssl Issues and PRs related to the OpenSSL dependency. python PRs and issues that require attention from people who are familiar with Python. tools Issues and PRs related to the tools directory. windows Issues and PRs related to the Windows platform.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@zcbenz @nodejs-github-bot @joyeecheung @batrla @benjamingr @anonrig @gengjiawen
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.