GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — feat(run-engine,sdk,webapp): cap a queue's total concurrency across keyed and keyless runs by matt-aitken · Pull Request #4823 · triggerdotdev/trigger.dev

Conversation

matt-aitken commented Aug 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

Summary

Adds a totalConcurrencyLimit option to queues. On a queue used with concurrencyKey, concurrencyLimit applies to each key value independently, so ten active keys with a limit of 5 can run 50 at once and nothing bounds the queue as a whole short of the environment limit. totalConcurrencyLimit caps in-flight runs across all keys while each key still gets at most concurrencyLimit:

export const perUserQueue = queue({
  name: "per-user-queue",
  concurrencyLimit: 1,
  totalConcurrencyLimit: 10,
});

Enforcement is gated behind RUN_ENGINE_TOTAL_CONCURRENCY_LIMITS_ENABLED (default off) and applies to runs triggered with a concurrencyKey. With the gate off, admit paths are unchanged.

Design

The engine keeps a per-base-queue groupConcurrency set shared by every concurrency-key variant; its cardinality is the queue's total in-flight count. The concurrency-key dequeue script bounds each batch by min(totalLimit, envLimit) - SCARD(group) and adds admitted runs to the set. The enqueue fast path checks the same gate and falls back to a normal enqueue when the queue is at its total limit.

Every release path (ack, nack, dead-letter, concurrency release, TTL expiry, sweeper clear) removes a run from the group set whenever it removes it from the per-key currentConcurrency set. Those removals run regardless of the gate, so the set stays correct if the gate is later turned off, and the existing reconciliation sweep self-heals the group set because it is always a subset of the per-key sets it acks against.

The stored limit is the raw declared value; readers clamp to the environment concurrency limit. The option flows through the queue manifest into a new nullable TaskQueue.totalConcurrencyLimit column (additive migration, internal name) and syncs to the engine on deploy.

The group set self-heals against release paths that miss the removal (an instance on an older build during a rollout, or a future release script such as the one #4398 adds). Every terminal release deletes the run's message key, so when a queue sits at its total the dequeue gate prunes members whose message key no longer exists, throttled to one pass per interval per queue. Runs in flight before the gate is enabled are the opposite case: they are absent from the set, so a queue can transiently exceed its total by at most that count, converging as each one completes.

Not included here, planned as follow-ups: a runtime override API for the total limit, dashboard and metrics surfacing, and skipping queues at their total during fair-queue selection.

A follow-up in this stack renames the public option to combinedConcurrencyLimit before release; engine internals and storage keep these names.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

changeset-bot Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

🦋 Changeset detected

Latest commit: 42b3d23

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 27 packages

Name	Type
@trigger.dev/sdk	Patch
@trigger.dev/core	Patch
@trigger.dev/python	Patch
@internal/dashboard-agent	Patch
@trigger.dev/build	Patch
trigger.dev	Patch
@trigger.dev/redis-worker	Patch
@trigger.dev/schema-to-json	Patch
@internal/clickhouse	Patch
@internal/llm-model-catalog	Patch
@internal/metrics-pipeline	Patch
@trigger.dev/rbac	Patch
@internal/redis	Patch
@internal/replication	Patch
@internal/run-engine	Patch
@internal/run-store	Patch
@internal/schedule-engine	Patch
@internal/tracing	Patch
@internal/webhook-engine	Patch
@internal/webhook-sources	Patch
@internal/cache	Patch
@trigger.dev/react-hooks	Patch
@trigger.dev/rsc	Patch
@trigger.dev/database	Patch
@trigger.dev/otlp-importer	Patch
@trigger.dev/sso	Patch
@internal/testcontainers	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

coderabbitai Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

Adds an optional totalConcurrencyLimit queue setting for keyed runs. The SDK and queue schemas expose and register the setting. The database stores it, and the webapp synchronizes it with RunQueue. RunQueue tracks concurrency across key variants with Redis sets and enforces the limit during enqueue and dequeue operations. Release, retry, dead-letter, clear, and expiry paths update aggregate state. Integration tests cover enabled, disabled, per-key, fallback, acknowledgment, nack, and leaked-member reconciliation behavior.

Merge Risk: 🟡 Moderate · up to 588ac

The change adds a gated queue-wide concurrency cap, but saturated queues may currently perform large repeated cleanup operations that delay processing, while rollout or missed cleanup can strand capacity and prevent runs from starting; enabling the cap with existing runs can also temporarily exceed it. The feature is disabled by default, but these bounded availability and rollout risks require explicit owner follow-up before broad enablement.

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 30.77% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 13 functions across 14 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	⚠️ Warning	The title describes the total concurrency feature, but it incorrectly states that the limit applies to keyless runs. The implementation applies it only to runs with a concurrencyKey.	Update the title to state that the limit applies across keyed runs, for example: "feat(run-engine,sdk,webapp): cap total concurrency across concurrency-keyed runs".
Description check	⚠️ Warning	The description gives a detailed and relevant design summary, but it omits the required issue reference, checklist, Testing section, Changelog section, and Screenshots section from the repository temp…	Add the required template sections. Include the issue closure, completed checklist items, exact test commands and results, a short changelog entry, and screenshots or an explicit statement that screenshots are not applicable.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description gives a detailed and relevant design summary, but it omits the required issue reference, checklist, Testing section, Changelog section, and Screenshots section from the repository template.

Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/queue-total-concurrency-limit

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

devin-ai-integration Bot reviewed

Aug 28, 2026

View reviewed changes

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

This comment was marked as resolved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

matt-aitken force-pushed the feat/queue-total-concurrency-limit branch from 2d1f985 to b895c82 Compare

August 28, 2026 17:12

pkg-pr-new Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@42b3d23

trigger.dev

npm i https://pkg.pr.new/trigger.dev@42b3d23

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@42b3d23

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@42b3d23

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@42b3d23

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@42b3d23

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@42b3d23

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@42b3d23

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@42b3d23

commit: 42b3d23

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This comment was marked as resolved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

matt-aitken force-pushed the feat/queue-total-concurrency-limit branch from 468323d to 023c282 Compare

August 28, 2026 17:57

This comment was marked as resolved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This comment was marked as resolved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

coderabbitai Bot reviewed

Aug 28, 2026

View reviewed changes

coderabbitai Bot left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

internal-packages/run-engine/src/run-queue/index.ts (1)

3827-3865: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add crumbs for aggregate-concurrency state changes.

Add // @Crumbs markers or `// `#region` `@crumbs blocks around the new total-cap admission and group-set release paths. This includes the tracked CK enqueue, dequeue, acknowledgment, nack, dead-letter, release, and clear commands.

As per coding guidelines, “Add crumbs as you write code — not just when debugging.”

Also applies to: 4639-4661, 5249-5725

Source: Coding guidelines

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e99334b8-fb48-462b-a79f-b9683eac6e60

📥 Commits

Reviewing files that changed from the base of the PR and between 468323d and 023c282.

📒 Files selected for processing (1)

internal-packages/run-engine/src/run-queue/index.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 7 remain after this review.

📜 Review details

⏰ Context from checks skipped due to timeout. (2)

GitHub Check: webapp / 🧪 Unit Tests: Webapp (13, 24)
GitHub Check: internal / 🧪 Unit Tests: Internal

🧰 Additional context used

📓 Path-based instructions (5)

**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:

📄 CodeRabbit inference engine (AGENTS.md)

Files:

internal-packages/run-engine/src/run-queue/index.ts

Add crumbs as you write code — not just when debugging. Mark lines with

📄 CodeRabbit inference engine (AGENTS.md)

Files:

internal-packages/run-engine/src/run-queue/index.ts

Use function declarations instead of default exports

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

internal-packages/run-engine/src/run-queue/index.ts

Use types over interfaces for TypeScript

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

internal-packages/run-engine/src/run-queue/index.ts

When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

Files:

internal-packages/run-engine/src/run-queue/index.ts

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

This comment was marked as resolved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

devin-ai-integration Bot reviewed

Aug 28, 2026

View reviewed changes

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

devin-ai-integration Bot reviewed

Aug 28, 2026

View reviewed changes

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

This was referenced Aug 29, 2026

feat(run-engine): queue gates enforced from the message payload #4826

Closed

feat(sdk,core,webapp,run-engine): runtime overrides for queue total concurrency limits #4829

Closed

matt-aitken changed the title ~~feat(run-engine,sdk,webapp): queue total concurrency limit across keys~~ feat(run-engine,sdk,webapp): queue combined concurrency limit across keys

Aug 29, 2026

This comment was marked as resolved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

matt-aitken force-pushed the feat/queue-total-concurrency-limit branch from 5c57e37 to 6290352 Compare

August 29, 2026 18:21

matt-aitken changed the title ~~feat(run-engine,sdk,webapp): queue combined concurrency limit across keys~~ feat(run-engine,sdk,webapp): queue total concurrency limit across keys

Aug 29, 2026

matt-aitken force-pushed the feat/queue-total-concurrency-limit branch from 6290352 to b87eb36 Compare

August 31, 2026 12:02

This comment was marked as resolved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This comment was marked as resolved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This comment was marked as resolved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

matt-aitken added 6 commits

September 6, 2026 13:12


          feat(run-engine,sdk,webapp): queue total concurrency limit across keys

a7131de

On a queue used with concurrencyKey, concurrencyLimit applies to each key
value independently, so nothing bounds the queue as a whole short of the
environment limit. The new totalConcurrencyLimit queue option caps in-flight
runs across all keys of the queue while each key still gets at most
concurrencyLimit.

Enforcement lives in the concurrency-key dequeue and enqueue fast-path
scripts, gated behind RUN_ENGINE_TOTAL_CONCURRENCY_LIMITS_ENABLED (default
off). A per-base-queue groupConcurrency set tracks total in-flight; every
release path mirrors its per-key removal into that set unconditionally so
the set stays correct across flag toggles.


          fix(run-engine): address review on total concurrency limit

129e5b6

Strengthen the nack test so it proves the group slot is released (a second
key's run must be admitted after the nack), document the enable-time
convergence window on the flag, and correct the totalConcurrencyOfQueue doc
to describe the drain-on-disable behavior.


          feat(run-engine): self-heal leaked total-concurrency members at the gate

A release path that misses the group-set mirror (an instance on an older
build during rollout, or a future release script) would otherwise leave a
member the gate counts forever. Every terminal release deletes the run's
message key, so when a queue sits at its total the dequeue gate prunes
members whose message key no longer exists, throttled to one pass per
interval per queue. Members of re-queued runs keep their message key and
clear through the mirrored ack when the run completes.


          fix(run-engine,core): non-blocking test polls and document the zero t…

68a2560

…otal limit

The waitFor polls dequeued with the default 10s blocking pop, which could
blow past the helper deadline on a slow runner; poll non-blocking instead.
Document that totalConcurrencyLimit: 0 holds every keyed run, matching
concurrencyLimit's zero semantics.


          perf(run-engine): bound group-set reconciliation to one SSCAN batch p…

d83d971

…er pass

Reconciling a saturated queue's group set with SMEMBERS plus one EXISTS per
member runs the whole traversal inside a single Lua call, which blocks Redis
for the duration on a large set. Scan one bounded batch per pass instead,
persisting the SSCAN cursor between passes so successive intervals cover the
whole set. Covered by a test that drains a 1,200-member leaked backlog.


          chore: knip ignore for the run-queue module during the stack

23c5131

The module exports an error class whose consumer is introduced and removed
at different levels of this stack; the ignore keeps every level clean and
lifts naturally when the stack squashes to main.


          feat(webapp): enable combined concurrency limit enforcement by default

42b3d23

The env var stays as a kill switch: set
RUN_ENGINE_TOTAL_CONCURRENCY_LIMITS_ENABLED=0 to disable enforcement
without redeploying code. Off by default previously meant the SDK,
dashboard, and API surfaced combined limits that nothing enforced.

matt-aitken force-pushed the feat/queue-total-concurrency-limit branch from ad05459 to 42b3d23 Compare

September 6, 2026 12:20

devin-ai-integration Bot reviewed

Sep 7, 2026

View reviewed changes

devin-ai-integration Bot left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

matt-aitken force-pushed the feat/queue-total-concurrency-limit branch from e70cfc9 to 42b3d23 Compare

September 7, 2026 00:25

matt-aitken changed the title ~~feat(run-engine,sdk,webapp): queue total concurrency limit across keys~~ feat(run-engine,sdk,webapp): cap a queue's total concurrency across keyed and keyless runs

Sep 7, 2026

ericallam approved these changes

Sep 17, 2026

View reviewed changes

matt-aitken added this pull request to the merge queue

Sep 17, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks

Sep 17, 2026

Uh oh!

There was an error while loading. Please reload this page.

matt-aitken commented Sep 18, 2026

Copy link

Copy Markdown

Member Author

Closing: this stack was flattened into a single change that has now been merged and will land in this repository via the automated sync. All review feedback from these PRs is incorporated, along with further review rounds on the flattened change.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

matt-aitken closed this

Sep 18, 2026

Reviewers

coderabbitai[bot]

coderabbitai[bot] left review comments

devin-ai-integration[bot]

devin-ai-integration[bot] left review comments

ericallam

ericallam approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

matt-aitken commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design

Uh oh!

changeset-bot Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

coderabbitai Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

❌ Failed checks (3 warnings)

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

This comment was marked as resolved.

Uh oh!

pkg-pr-new Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

matt-aitken commented Sep 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

matt-aitken commented Aug 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

changeset-bot Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

coderabbitai Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

pkg-pr-new Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.