GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — tools: prefilter commit queue metadata · nodejs/node@aa3b598

File tree

Expand file treeCollapse file tree

Open diff view settings

Filter options

.github/workflows
- commit-queue.yml
doc/contributing
- commit-queue.md

Expand file treeCollapse file tree

Open diff view settings

Collapse file

`‎.github/workflows/commit-queue.yml‎`

Copy file name to clipboardExpand all lines: .github/workflows/commit-queue.yml

+100-28Lines changed: 100 additions & 28 deletions

Original file line number	Diff line number	Diff line change
`@@ -22,51 +22,44 @@ permissions:`
`22`	`22`	`contents: read`
`23`	`23`
`24`	`24`	`jobs:`
`25`		`- get_mergeable_prs:`
	`25`	`+ get_candidate_prs:`
`26`	`26`	`permissions:`
`27`	`27`	`pull-requests: read`
`28`	`28`	`if: github.repository == 'nodejs/node'`
`29`	`29`	`runs-on: ubuntu-slim`
`30`	`30`	`outputs:`
`31`		`- numbers: ${{ steps.get_mergeable_prs.outputs.numbers }}`
	`31`	`+ candidates: ${{ steps.get_candidate_prs.outputs.candidates }}`
`32`	`32`	`steps:`
`33`		`- - name: Get Pull Requests`
`34`		`- id: get_mergeable_prs`
	`33`	`+ - name: Get Pull Request Candidates`
	`34`	`+ id: get_candidate_prs`
`35`	`35`	`run: \|`
`36`		`- prs=$(gh pr list \`
	`36`	`+ list_prs() {`
	`37`	`+ gh pr list \`
`37`	`38`	`--repo "$GITHUB_REPOSITORY" \`
`38`	`39`	`--base "$GITHUB_REF_NAME" \`
`39`	`40`	`--label 'commit-queue' \`
	`41`	`+ "$@" \`
`40`	`42`	`--json 'number' \`
`41`		`- --search "created:<=$(date --date="2 days ago" +"%Y-%m-%dT%H:%M:%S%z") -label:blocked" \`
`42`	`43`	`-t '{{ range . }}{{ .number }} {{ end }}' \`
`43`		`- --limit 100)`
`44`		`- fast_track_prs=$(gh pr list \`
`45`		`- --repo "$GITHUB_REPOSITORY" \`
`46`		`- --base "$GITHUB_REF_NAME" \`
`47`		`- --label 'commit-queue' \`
	`44`	`+ --limit 100`
	`45`	`+ }`
	`46`	`+ aged_prs=$(list_prs \`
	`47`	`+ --search "created:<=$(date --date="2 days ago" +"%Y-%m-%dT%H:%M:%S%z") -label:blocked")`
	`48`	`+ fast_track_prs=$(list_prs \`
`48`	`49`	`--label 'fast-track' \`
`49`		`- --search "-label:blocked" \`
`50`		`- --json 'number' \`
`51`		`- -t '{{ range . }}{{ .number }} {{ end }}' \`
`52`		`- --limit 100)`
`53`		`- numbers=$(echo $prs' '$fast_track_prs \| jq -r -s 'unique \| join(" ")')`
`54`		`- echo "numbers=$numbers" >> "$GITHUB_OUTPUT"`
	`50`	`+ --search "-label:blocked")`
	`51`	`+ queued_prs=$(list_prs \`
	`52`	`+ --search "-label:blocked")`
	`53`	`+ candidates=$(printf '%s %s %s\n' "$aged_prs" "$fast_track_prs" "$queued_prs" \|`
	`54`	`+ jq -r -s 'reduce .[] as $pr ([]; if index($pr) then . else . + [$pr] end) \| join(" ")')`
	`55`	`+ echo "candidates=$candidates" >> "$GITHUB_OUTPUT"`
`55`	`56`	`env:`
`56`	`57`	`GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}`
`57`	`58`	`commitQueue:`
`58`		`- needs: get_mergeable_prs`
`59`		`- if: needs.get_mergeable_prs.outputs.numbers != ''`
	`59`	`+ needs: get_candidate_prs`
	`60`	`+ if: needs.get_candidate_prs.outputs.candidates != ''`
`60`	`61`	`runs-on: ubuntu-slim`
`61`	`62`	`steps:`
`62`		`- - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0`
`63`		`- with:`
`64`		`- # A personal token is required because pushing with GITHUB_TOKEN will`
`65`		`- # prevent commits from running CI after they land. It needs`
`66`		- # to be set here because `checkout` configures GitHub authentication
`67`		`- # for push as well.`
`68`		`- token: ${{ secrets.GH_USER_TOKEN }}`
`69`		`-`
`70`	`63`	`# Install dependencies`
`71`	`64`	`- name: Install Node.js`
`72`	`65`	`uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0`
`@@ -93,7 +86,86 @@ jobs:`
`93`	`86`	`GITHUB_TOKEN: ${{ secrets.GH_USER_TOKEN }}`
`94`	`87`	`JENKINS_TOKEN: ${{ secrets.JENKINS_TOKEN }}`
`95`	`88`
	`89`	`+ - name: Filter Pull Requests`
	`90`	`+ id: get_mergeable_prs`
	`91`	`+ run: \|`
	`92`	`+ readme="${RUNNER_TEMP}/README.md"`
	`93`	`+ curl -fsSLo "$readme" "https://github.com/${GITHUB_REPOSITORY}/raw/${GITHUB_SHA}/README.md"`
	`94`	`+`
	`95`	`+ numbers=`
	`96`	`+ # shellcheck disable=SC2086`
	`97`	`+ for pr in $CANDIDATES; do`
	`98`	`+ metadata="${RUNNER_TEMP}/metadata-${pr}.json"`
	`99`	`+ output="${RUNNER_TEMP}/metadata-${pr}.txt"`
	`100`	`+ if git node metadata "$pr" \`
	`101`	`+ --owner "$GITHUB_REPOSITORY_OWNER" \`
	`102`	`+ --repo "$REPOSITORY" \`
	`103`	`+ --readme "$readme" \`
	`104`	`+ --json > "$metadata" 2> "$output"; then`
	`105`	`+ metadata_status=0`
	`106`	`+ else`
	`107`	`+ metadata_status=$?`
	`108`	`+ fi`
	`109`	`+`
	`110`	`+ if [ -s "$output" ]; then`
	`111`	`+ cat "$output"`
	`112`	`+ fi`
	`113`	`+`
	`114`	`+ case "$metadata_status" in`
	`115`	`+ 0\|2[0-9]\|4[0-9]) ;;`
	`116`	`+ *)`
	`117`	`+ echo "git node metadata failed for pr ${pr} with exit code ${metadata_status}"`
	`118`	`+ exit 1`
	`119`	`+ ;;`
	`120`	`+ esac`
	`121`	`+`
	`122`	`+ metadata_exit_code=$(jq -r '.exitCode' "$metadata") \|\| {`
	`123`	`+ echo "failed to parse metadata JSON for pr ${pr}"`
	`124`	`+ exit 1`
	`125`	`+ }`
	`126`	`+ if [ "$metadata_exit_code" != "$metadata_status" ]; then`
	`127`	`+ echo "metadata JSON exitCode mismatch for pr ${pr}"`
	`128`	`+ exit 1`
	`129`	`+ fi`
	`130`	`+ metadata_reason_codes=$(jq -r '.reasonCodes \| join(", ")' "$metadata") \|\| {`
	`131`	`+ echo "failed to parse metadata reason codes for pr ${pr}"`
	`132`	`+ exit 1`
	`133`	`+ }`
	`134`	`+`
	`135`	`+ if [ "$metadata_status" -eq 0 ]; then`
	`136`	`+ echo "pr ${pr} is ready for the commit queue"`
	`137`	`+ numbers="$numbers $pr"`
	`138`	`+ continue`
	`139`	`+ fi`
	`140`	`+`
	`141`	`+ if [ "$metadata_status" -ge 20 ] && [ "$metadata_status" -le 29 ]; then`
	`142`	`+ echo "pr ${pr} skipped, not ready to land"`
	`143`	`+ echo "reason codes: ${metadata_reason_codes}"`
	`144`	`+ continue`
	`145`	`+ fi`
	`146`	`+`
	`147`	`+ echo "pr ${pr} will be handled by the commit queue"`
	`148`	`+ echo "reason codes: ${metadata_reason_codes}"`
	`149`	`+ numbers="$numbers $pr"`
	`150`	`+ done`
	`151`	`+`
	`152`	`+ numbers=$(echo "$numbers" \| xargs)`
	`153`	`+ echo "numbers=$numbers" >> "$GITHUB_OUTPUT"`
	`154`	`+ env:`
	`155`	`+ CANDIDATES: ${{ needs.get_candidate_prs.outputs.candidates }}`
	`156`	`+ GITHUB_TOKEN: ${{ secrets.GH_USER_TOKEN }}`
	`157`	`+`
	`158`	`+ - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0`
	`159`	`+ if: steps.get_mergeable_prs.outputs.numbers != ''`
	`160`	`+ with:`
	`161`	`+ # A personal token is required because pushing with GITHUB_TOKEN will`
	`162`	`+ # prevent commits from running CI after they land. It needs`
	`163`	+ # to be set here because `checkout` configures GitHub authentication
	`164`	`+ # for push as well.`
	`165`	`+ token: ${{ secrets.GH_USER_TOKEN }}`
	`166`	`+`
`96`	`167`	`- name: Start the Commit Queue`
`97`		`- run: ./tools/actions/commit-queue.sh "${GITHUB_REPOSITORY_OWNER}" "${REPOSITORY}" ${{ needs.get_mergeable_prs.outputs.numbers }}`
	`168`	`+ if: steps.get_mergeable_prs.outputs.numbers != ''`
	`169`	`+ run: ./tools/actions/commit-queue.sh "${GITHUB_REPOSITORY_OWNER}" "${REPOSITORY}" ${{ steps.get_mergeable_prs.outputs.numbers }}`
`98`	`170`	`env:`
`99`	`171`	`GITHUB_TOKEN: ${{ secrets.GH_USER_TOKEN }}`

Collapse file

`‎doc/contributing/commit-queue.md‎`

Copy file name to clipboardExpand all lines: doc/contributing/commit-queue.md

+108-53Lines changed: 108 additions & 53 deletions

Display the source diff
Display the rich diff

Original file line number	Diff line number	Diff line change
`@@ -1,36 +1,48 @@`
`1`	`1`	`# Commit queue`
`2`	`2`
`3`		-_tl;dr: You can land pull requests by adding the `commit-queue` label to it._
	`3`	`+_tl;dr: You can ask the queue to land pull requests by adding the`
	`4`	+`commit-queue` label to them._
`4`	`5`
`5`	`6`	`Commit Queue is a feature for the project which simplifies the`
`6`	`7`	`landing process by automating it via GitHub Actions. With it, collaborators can`
`7`		-land pull requests by adding the `commit-queue` label to a PR. All
`8`		-checks will run via `@node-core/utils`, and if the pull request is ready to
`9`		-land, the Action will rebase it and push to `main`.
	`8`	+queue pull requests for landing by adding the `commit-queue` label to a PR. The
	`9`	+selector checks readiness with `@node-core/utils`. If the pull request is only
	`10`	`+blocked on a deferrable condition, currently wait time, the queue leaves the`
	`11`	`+label in place and retries later. Other failures continue to the existing`
	`12`	`+landing and failure-reporting path.`
`10`	`13`
`11`	`14`	`This document gives an overview of how the Commit Queue works, as well as`
`12`	`15`	`implementation details, reasoning for design choices, and current limitations.`
`13`	`16`
`14`	`17`	`## Overview`
`15`	`18`
`16`		`-From a high-level, the Commit Queue works as follow:`
`17`		`-`
`18`		-1. Collaborators will add `commit-queue` label to pull requests ready to land
`19`		`-2. Every five minutes the queue will do the following for each mergeable pull request`
`20`		`- with the label:`
`21`		- 1. Check if the PR also has a `request-ci` label (if it has, skip this PR
	`19`	`+From a high-level, the Commit Queue works as follows:`
	`20`	`+`
	`21`	+1. Collaborators will add `commit-queue` label to pull requests they want the
	`22`	`+ queue to land. The label can be added before the pull request has completed`
	`23`	`+ its wait time, or before requested CI has finished. Required approvals must`
	`24`	`+ already be in place. The commit queue does not request CI on its own.`
	`25`	`+2. On each scheduled run, the queue builds a candidate list from open pull`
	`26`	+ requests with the `commit-queue` label and without the `blocked` label. The
	`27`	`+ workflow uses a five-minute cron, but GitHub Actions scheduled workflows are`
	`28`	`+ not guaranteed to run exactly every five minutes. For each candidate, the`
	`29`	`+ queue will:`
	`30`	+ 1. In the landing job, install and configure `@node-core/utils`, then run a
	`31`	`+ metadata-only readiness check without checking out the repository`
	`32`	`+ 2. If the metadata check exits with a deferrable readiness code, meaning`
	`33`	+ the PR is only blocked on wait time, keep the `commit-queue` label and
	`34`	`+ skip this PR until a later queue run`
	`35`	+ 3. Check if the PR also has a `request-ci` label (if it has, skip this PR
`22`	`36`	`since it's pending a CI run)`
`23`		`- 2. Check if the last Jenkins CI is finished running (if it is not, skip this`
`24`		`- PR)`
`25`		- 3. Remove the `commit-queue` label
`26`		- 4. Run `git node land <pr> --oneCommitMax`
`27`		`- 5. If it fails:`
`28`		- 1. Abort `git node land` session
`29`		- 2. Add `commit-queue-failed` label to the PR
`30`		- 3. Leave a comment on the PR with the output from `git node land`
`31`		`- 4. Skip next steps, go to next PR in the queue`
`32`		`- 6. If it succeeds:`
`33`		`- 1. Push the changes to nodejs/node`
	`37`	`+ 4. Check whether GitHub checks are still running (if they are, skip this PR)`
	`38`	+ 5. Remove the `commit-queue` label and run `git node land`
	`39`	`+ 6. If it fails:`
	`40`	+ 1. Add the `commit-queue-failed` label to the PR
	`41`	+ 2. Leave a comment on the PR with the output from `git node land`
	`42`	+ 3. Abort the `git node land` session. If the abort succeeds, continue to
	`43`	`+ the next PR; otherwise, stop the queue in an unknown state`
	`44`	`+ 7. If it succeeds:`
	`45`	`+ 1. Push or merge the changes into nodejs/node`
`34`	`46`	2. Leave a comment on the PR with `Landed in ...`
`35`	`47`	`3. Close the PR`
`36`	`48`	`4. Go to next PR in the queue`
`@@ -51,18 +63,23 @@ of the commit queue:`
`51`	`63`	guidelines or be a valid [`fixup!`](https://git-scm.com/docs/git-commit#Documentation/git-commit.txt---fixupamendrewordltcommitgt)
`52`	`64`	commit that will be correctly handled by the [`--autosquash`](https://git-scm.com/docs/git-rebase#Documentation/git-rebase.txt---autosquash)
`53`	`65`	`option`
`54`		`-2. A CI must've ran and succeeded since the last change on the PR`
	`66`	`+2. A CI must have run and succeeded since the last change on the PR`
`55`	`67`	`3. A collaborator must have approved the PR since the last change`
`56`	`68`	`4. Only Jenkins CI and GitHub Actions are checked (V8 CI and CITGM are ignored)`
`57`	`69`	5. The PR must target the `main` branch (PRs opened against other branches, such
`58`	`70`	`as backport PRs, are ignored)`
`59`	`71`
`60`	`72`	`## Implementation`
`61`	`73`
`62`		`-The [action](../../.github/workflows/commit-queue.yml) will run on scheduler`
`63`		`-events every five minutes. Five minutes is the smallest number accepted by`
`64`		`-the scheduler. The scheduler is not guaranteed to run every five minutes, it`
`65`		`-might take longer between runs.`
	`74`	`+The [action](../../.github/workflows/commit-queue.yml) runs on scheduled events.`
	`75`	`+It uses a five-minute cron because that is the smallest interval accepted by`
	`76`	`+GitHub Actions. Scheduled workflows are not guaranteed to run exactly at that`
	`77`	`+cadence and might take longer between runs.`
	`78`	`+`
	`79`	`+The workflow also uses a concurrency group so only one commit queue run can be`
	`80`	`+active at a time. If a scheduled run starts while a previous run is still`
	`81`	`+running, GitHub Actions keeps at most one pending run for the same concurrency`
	`82`	`+group. A newer pending run replaces an older pending run.`
`66`	`83`
`67`	`84`	`Using the scheduler is preferable over using pull\_request\_target for two`
`68`	`85`	`reasons:`
`@@ -76,41 +93,79 @@ reasons:`
`76`	`93`	`commit, meaning we wouldn't be able to use it for already opened PRs`
`77`	`94`	`without rebasing them first.`
`78`	`95`
`79`		-`@node-core/utils` is configured with a personal token and
`80`		`-a Jenkins token from`
`81`		`-[@nodejs-github-bot](https://github.com/nodejs/github-bot).`
`82`		-`octokit/graphql-action` is used to fetch all pull requests with the
`83`		-`commit-queue` label. The output is a JSON payload, so `jq` is used to turn
`84`		`-that into a list of PR ids we can pass as arguments to`
`85`		-[`commit-queue.sh`](../../tools/actions/commit-queue.sh).
`86`		`-`
`87`		`-> The personal token only needs permission for public repositories and to read`
`88`		`-> profiles, we can use the GITHUB\_TOKEN for write operations. Jenkins token is`
	`96`	`+The workflow starts with a small candidate job that uses GitHub CLI to fetch`
	`97`	+pull requests with the `commit-queue` label. It first fetches the same
	`98`	`+age-based and fast-track buckets the queue used before accepting early queue`
	`99`	`+requests, then fetches the broader queue and de-duplicates the result. This`
	`100`	`+keeps not-yet-ready PRs from crowding out PRs that the previous query would`
	`101`	`+have selected if GitHub paginates or caps a query result.`
	`102`	`+`
	`103`	`+If there are candidate PRs, the landing job installs and configures`
	`104`	+`@node-core/utils` once with a personal token and a Jenkins token from
	`105`	`+[@nodejs-github-bot](https://github.com/nodejs/github-bot). It then downloads`
	`106`	`+the workflow commit's README without checking out the repository and runs`
	`107`	+`git node metadata --readme --json` for each candidate. This uses the same
	`108`	+`@node-core/utils` PR readiness checks as `git node land`, but does not clone,
	`109`	`+fetch, or merge the PR. The filter consumes the structured metadata result`
	`110`	`+and its exit code instead of matching human-readable output:`
	`111`	`+`
	`112`	+* exit code `0`: the PR is ready and is passed to
	`113`	+ [`commit-queue.sh`](../../tools/actions/commit-queue.sh)
	`114`	+* exit codes `20`-`29`: the PR is not ready for a deferrable metadata reason,
	`115`	+ currently wait time, so it keeps the `commit-queue` label and is retried
	`116`	`+ later`
	`117`	+* exit codes `40`-`49`: the PR has a hard or mixed metadata readiness failure
	`118`	+ and is passed to [`commit-queue.sh`](../../tools/actions/commit-queue.sh)
	`119`	`+`
	`120`	+The `20`-`29` exit code range is reserved by `@node-core/utils` for deferrable
	`121`	+metadata readiness states, and `40`-`49` is reserved for hard metadata failure
	`122`	`+states. Unknown filter failures fail the workflow before starting the landing`
	`123`	`+script and leave PR labels unchanged so the queue can retry on a later`
	`124`	+scheduled run. PRs passed through with exit code `40`-`49` continue through
	`125`	+`commit-queue.sh`. The workflow checks out the repository only when at least
	`126`	`+one PR remains after filtering. The script still applies its existing`
	`127`	+`request-ci` and pending-check deferrals before removing the queue label and
	`128`	`+reporting a hard failure.`
	`129`	`+`
	`130`	`+> The personal token needs permission for public repositories and to read`
	`131`	+> profiles. It is used by `@node-core/utils` and by the landing job for
	`132`	`+> checkout, label and comment updates, merging, and pushing. Jenkins token is`
`89`	`133`	`> required to check CI status.`
`90`	`134`
`91`	`135`	`commit-queue.sh` receives the following positional arguments:
`92`	`136`
`93`	`137`	`1. The repository owner`
`94`	`138`	`2. The repository name`
`95`		`-3. The Action GITHUB\_TOKEN`
`96`		`-4. Every positional argument starting at this one will be a pull request ID of`
	`139`	`+3. Every positional argument starting at this one will be a pull request ID of`
`97`	`140`	`a pull request with commit-queue set.`
`98`	`141`
`99`		-The script will iterate over the pull requests. `ncu-ci` is used to check if
`100`		`-the last CI is still pending, and calls to the GitHub API are used to check if`
`101`		-the PR is waiting for CI to start (`request-ci` label). The PR is skipped if CI
`102`		-is pending. No other CI validation is done here since `git node land` will fail
`103`		`-if the last CI failed.`
`104`		`-`
`105`		-The script removes the `commit-queue` label. It then runs `git node land`,
`106`		`-forwarding stdout and stderr to a file. If any errors happen,`
`107`		-`git node land --abort` is run, and then a `commit-queue-failed` label is added
`108`		-to the PR, as well as a comment with the output of `git node land`.
`109`		`-`
`110`		-If no errors happen during `git node land`, the script will use the
`111`		-`GITHUB_TOKEN` to push the changes to `main`, and then will leave a
`112`		-`Landed in ...` comment in the PR, and then will close it. Iteration continues
`113`		`-until all PRs have done the steps above.`
	`142`	`+The script will iterate over the pull requests. GitHub CLI is used to check if`
	`143`	+the PR is waiting for CI to start (`request-ci` label) or still has pending
	`144`	`+GitHub checks. The PR is skipped if CI is pending. No other CI validation is`
	`145`	+done here since `git node land` will fail if the last CI failed.
	`146`	`+`
	`147`	+The script removes the `commit-queue` label, then runs `git node land`,
	`148`	`+forwarding stdout and stderr to a file. PRs that are only blocked on wait time`
	`149`	`+should have already been filtered by the metadata check. If a hard readiness`
	`150`	+failure appears between the metadata filter and `git node land`, the landing
	`151`	+job adds a `commit-queue-failed` label to the PR, leaves a comment with the
	`152`	+output of `git node land`, and then aborts the landing session. If the abort
	`153`	`+fails, the queue stops instead of continuing in an unknown state.`
	`154`	`+`
	`155`	`+Fast-tracked PRs use the metadata check before checkout and the landing script.`
	`156`	`+If the fast-track request has not yet received enough collaborator thumbs-up,`
	`157`	+the queue keeps the `commit-queue` label and retries until either the
	`158`	`+fast-track request is approved or the PR becomes landable through the regular`
	`159`	`+wait-time rules. The commit queue does not create the fast-track request`
	`160`	+comment; that is handled when the `fast-track` label is added. If that comment
	`161`	`+is missing, the queue reports the failure instead of keeping the PR queued.`
	`162`	`+`
	`163`	+If no errors happen during `git node land`, the script either pushes the direct
	`164`	+rebase landing to `main` or uses GitHub's squash merge API for single-commit and
	`165`	+fixup landings. It then leaves a `Landed in ...` comment in the PR. GitHub
	`166`	`+closes PRs merged through the merge API automatically; for direct pushes, the`
	`167`	`+script closes the PR. Iteration continues until all PRs have done the steps`
	`168`	`+above.`
`114`	`169`
`115`	`170`	`## Reverting broken commits`
`116`	`171`

0 commit comments

Comments

0 (0)

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.