GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — improvement(credentials): keep valid client-credential tokens through refresh outages and pace token retries by failure kind by waleedlatif1 · Pull Request #9925 · simstudioai/sim

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Makes client-credential token requests more resilient, following the patterns used by MSAL, Entra's resilience guidance and botocore. The changes apply to the per-instance mint cache, the shared coordinated resolver (Shopify, Linear, Atlassian, Vanta) and Vanta itself.

  • Still-valid tokens keep working. Tokens are refreshed early, inside the 5-minute window before expiry. If that refresh fails with an outage or throttling, the cached or stored token is served until 30 s before it expires. That margin is checked again after the failed refresh, so a slow attempt cannot carry the token past it. A refusal of the credentials still fails at once.

  • Retry timing depends on the kind of failure (shared policy in lib/credentials/token-request-resilience.ts):

    • Refusal (bad or revoked credentials): replayed for 30 s, keyed by the secret's fingerprint.
    • Throttled (429, or 503 with Retry-After): the provider's Retry-After is honoured (seconds or HTTP-date), clamped to 1 s to 5 min. Without one, wait 1 min, doubling up to 5 min.
    • Outage (other 5xx, timeout, network error, malformed or unreadable response): 5 s, doubling to 30 s while it lasts. Before the backoff, one retry inside the mint after about 1 s, only if the failed attempt was quick.
    • Each kind keeps its own streak. A 429 after a run of outages starts its backoff at 1 min and does not inherit the outage count.
  • Retry-After is carried on errors from every client-credential minter. TokenServiceAccountValidationError gets an optional retryAfterMs, read by throttledRetryAfterMs (429/503 only). It is filled in by:

    • the shared helpers: throwForProviderResponse, exchangeClientCredentialsGrant, Shopify and Vanta
    • every bespoke minter: Box, Zoom, DocuSign, NetSuite, Help Scout, Ramp, Jama, Zendesk, Salesforce, Zoho Desk
    • the CrowdStrike, SailPoint and ZoomInfo client errors, through a new optional field

    A replayed failure carries an optional replayForMs, which the per-instance layer honours exactly.

  • Coordinated resolver:

    • The failure is remembered once for every worker in the existing row. Remembered failures and the still-valid fallback are served without taking the application lock.
    • A refusal whose code the row cannot store (the previous build reads it as provider_unavailable) gets an additive refusal: true flag, so a replay never serves the stale token.
    • A throttled failure gets an additive throttled: true flag, so the throttling streak continues across workers and stays separate from an outage streak.
    • Vanta asks for a token at most every 15 s while exchanges fail, because Vanta allows 5 tokens a minute and each new token revokes the previous one. That includes a 200 whose body cannot be read.
  • Single-flight is unchanged: coalesceLocally per instance, the advisory lock across workers.

  • The new fallback warning leaves out the credential id in selector privacy mode, like every other log in credential-service.ts.

Type of Change

  • Bug fix
  • Other: resilience improvement

Testing

  • Unit tests in credential-service-cancellation.test.ts, errors.test.ts and client-credentials-grant.test.ts:
    • fallback while the token is valid, error once it has run out (including after a slow failed refresh), and no fallback after a refusal
    • the single in-mint retry
    • each replay window, the Retry-After cap and the backoff floors
    • a 429 after two outages waits about a minute, not the outage streak's longer wait
  • A Retry-After test for each bespoke minter, in the minter test files (new helpscout.test.ts).
  • Real-Postgres integration tests in coordinated-client-token.integration.ts and vanta-token.integration.ts, run one file at a time:
    • fallback across workers
    • the retry while holding the lock, shared by waiting resolutions
    • serving without taking the lock
    • backoff growth and reset after recovery
    • a throttled answer after an outage backs off from 1 min, and throttling that continues backs off longer for every worker
    • sweep retention
    • a refusal the row cannot store stays a refusal on replay
    • Vanta's 15 s pacing, including an unreadable 200 body
  • Rolling-deploy fixtures pin both earlier failure decoders. A refusal memo is still exactly {code,status}. An outage memo adds only consecutiveFailures, a throttle memo adds consecutiveFailures and throttled, and an unstorable refusal adds only refusal; the old decoders ignore all three. The replay length rides on the existing expiresAt.
  • Each guard was reverted and its tests went red.
  • For reviewers: check the 5-minute Retry-After cap, and that the retry while holding the lock stays inside the 15 s lock_timeout (2 s budget + about 1.2 s pause + one 10 s attempt).
  • Follow-up, not in this PR: the Microsoft app (Entra client-credentials) token cache has its own LRU and still has no stale fallback or failure memo.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
waleedlatif1 added 6 commits November 11, 2026 19:17
@waleedlatif1
feat(credentials): carry a provider's Retry-After on token request fa…
ee743a4
…ilures

A 429 or 503 token answer now records its Retry-After (delay-seconds or
HTTP-date, RFC 9110 section 10.2.3) as an optional retryAfterMs on
TokenServiceAccountValidationError, through throwForProviderResponse and the
shared client-credentials grant, whose provider classifier receives it.
@waleedlatif1
feat(credentials): keep valid client-credential tokens through refres…
3c307df
…h outages

The per-instance client-credential mint cache now:
- serves a cached token that is inside its five-minute early-refresh window
  but not expired when the refresh fails with an outage or throttling, as
  MSAL, Azure.Core and botocore do; a refusal of the credentials still fails
- retries a quick transient mint failure once after about a second
- replays failures by kind instead of a flat 30 s: a refusal 30 s, a
  throttled answer for its Retry-After (1 s to 5 min), an outage 5 s growing
  to 30 s while it continues

The policy lives in lib/credentials/token-request-resilience.ts so the
coordinated resolver can share it.
@waleedlatif1
feat(credentials): share token refresh resilience across workers in t…
5d21da6
…he coordinated resolver

The coordinated client-credentials resolver (Shopify, Linear, Atlassian,
Vanta) now applies the same policy as the per-instance mint cache, through
the shared row every worker reads:
- a quick outage is retried once under the application lock, within the 15 s
  its waiters allow
- a refusal is remembered 30 s, a throttled answer for its Retry-After (1 s to
  5 min, Vanta's now included), an outage 5 s growing to 30 s while it lasts
- while an outage or throttling is remembered, the stored token for the same
  secret is served until 30 s before it expires; a refusal fails at once

Rolling deploys stay safe: a refusal is stored exactly as before, and outages
add only a consecutiveFailures key that previous decoders ignore (both prior
decoders are pinned in the integration tests). The replay length rides on the
row's existing expiresAt. Expired memos are kept five more minutes so the next
failure can extend the backoff; an older build's sweep only resets it.
@waleedlatif1
fix(credentials): pace throttled and Vanta token retries and replay r…
c5f2e1f
…emembered failures without the lock

- A 429 that names no Retry-After is replayed for at least a minute, doubling
  up to the five-minute cap, instead of the 5 s outage curve.
- The outage backoff never undercuts Entra's 5 s first-retry floor through jitter.
- Vanta asks for a token at most once per 15 s while exchanges fail (no
  in-request retry, replay floored), within its documented 5 tokens a minute,
  through a new optional minExchangeIntervalMs on the coordinated resolver.
- A remembered failure and the still-valid fallback are served from committed
  rows before the application lock is taken, so a refresh outage no longer
  serializes every resolution behind the lock.
- retryAfterMs now only carries a provider's Retry-After; a replayed memo
  reports its remainder as the additive replayForMs, which the per-instance
  layer honors exactly.
- One decode/verify path for stored tokens; the stored-failure permanence
  check uses isPermanentTokenFailureCode instead of building an error.
@waleedlatif1
fix(credentials): honor every minter's Retry-After and close token fa…
7b7eb35
…llback gaps

- Carry the token endpoint's Retry-After on 429/503 failures from every
  bespoke client-credential minter (Box, Zoom, DocuSign, NetSuite, Help
  Scout, Ramp, Jama, Zendesk, Salesforce, Zoho Desk) and through the
  CrowdStrike, SailPoint and ZoomInfo client errors, so their throttled
  answers wait as long as the provider asks instead of the fixed backoff.
- Recheck the 30 s expiry margin when the per-instance cache falls back
  to a still-valid token after a slow failed refresh.
- Map an unreadable Vanta token body to provider_unavailable so it is
  remembered for every worker and paced at 15 s like any failed exchange.
- Remember a refusal whose code the shared failure row cannot store with
  an additive `refusal` flag, so a replay never serves the stale token.
- Rename the credentials helper to throttledRetryAfterMs so it no longer
  shares a name with providers/retry.ts.
@waleedlatif1
fix(credentials): keep throttling and outage backoff streaks apart an…
24a4664
…d redact the fallback log in selector mode
@vercel

vercel Bot commented Nov 12, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 12, 2026 12:03pm UTC

Request Review

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vercel
vercel Bot temporarily deployed to Preview November 12, 2026 11:44 Inactive
@greptile-apps

greptile-apps Bot commented Nov 12, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[High impact] The PR changes retry and token-fallback behavior across several credential workflows.

Fix SailPoint's handling of unreadable 429 bodies before merging.

Findings

  1. P1 SailPoint loses throttle delays ▶

Summary

The PR keeps still-valid client-credential tokens available through refresh outages and gives refusals, throttling, and outages different retry delays.

  • Client-credential callers keep using valid tokens through temporary refresh failures.
  • Token errors carry provider retry timing to the credential resolvers.
  • Vanta token requests slow down during failures to stay within its mint limit.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Request[Resolve token] --> Fresh{Token fresh?}
  Fresh -->|Yes| Serve[Return token]
  Fresh -->|No| Refresh[Try refresh]
  Refresh -->|Success| Save[Save and return new token]
  Refresh -->|Refusal| Drop[Drop token and remember refusal]
  Refresh -->|Outage or throttling| Remember[Remember failure and retry delay]
  Remember --> Valid{Old token still valid?}
  Valid -->|Yes| Serve
  Valid -->|No| Fail[Return failure]
Loading

Reviews (2) · Last reviewed commit: "fix(credentials): end a refused token, r..." · Reviewed by Greptile

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps[bot]
greptile-apps Bot reviewed Nov 12, 2026
View reviewed changes
Comment thread apps/sim/lib/oauth/credential-service.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/internal/sailpoint/client.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Nov 12, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 41 files

Turn on auto-fix | Re-trigger cubic

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread apps/sim/lib/credentials/coordinated-client-token.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/internal/crowdstrike/client.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/oauth/credential-service.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/credentials/coordinated-client-token.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/internal/sailpoint/client.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/credentials/client-credential-accounts/vanta-token.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

@waleedlatif1
fix(credentials): end a refused token, read legacy refusals as refusa…
d014680
…ls, and keep throttle delays through SailPoint and CrowdStrike

- A refusal of the credentials drops the per-instance cached token and expires the shared
  token minted with that secret, so a later outage cannot serve it again.
- A previous build's provider_unavailable memo with a non-transient 4xx status replays as a
  refusal instead of serving the stored token.
- The SailPoint saved-credential minter returns the first 429 with its Retry-After instead of
  retrying until the exchange timeout swallows it.
- A CrowdStrike error whose body cannot be read keeps its status and Retry-After.
- Vanta logs a malformed or oversized token response apart from an unreadable stream.
@vercel
vercel Bot temporarily deployed to Preview November 12, 2026 12:03 Inactive
@waleedlatif1

waleedlatif1 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile

greptile-apps[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@waleedlatif1

waleedlatif1 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

cubic-dev-ai[bot] reacted with eyes emoji
All reactions
  • 👀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps[bot]
greptile-apps Bot reviewed Nov 12, 2026
View reviewed changes
Comment thread apps/sim/lib/credentials/client-credential-accounts/minters/sailpoint.ts
clientSecret,
options?.signal ? AbortSignal.any([options.signal, timeout]) : timeout
options?.signal ? AbortSignal.any([options.signal, timeout]) : timeout,
{ maxThrottleRetries: 0 }

@greptile-apps greptile-apps Bot Nov 12, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 SailPoint loses throttle delays

With maxThrottleRetries: 0, the first 429 reaches readBoundedBody. If its body is cut off or exceeds the size limit, the read throws before SailPointAuthError preserves the status and Retry-After. The minter reports a 502 instead, so saved credentials retry after about a second rather than waiting for the provider's delay. This can prolong throttling and keep workflows failing.

Catch body-read failures on non-success responses and preserve the status and header, while still rethrowing caller cancellation.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Nov 12, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 41 files

Confidence score: 3/5

  • In credential-service.ts, a local refresh-failure memo can keep returning stale credentials after another worker has refreshed the shared token, causing avoidable auth failures. Check the coordinated token before honoring the memo.
  • In client.ts, if reading a 429/503 response body fails, the error loses Retry-After, so the resolver may retry without the server's backoff. Preserve the status and header when constructing SailPointAuthError even if the body read fails.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/oauth/credential-service.ts">

<violation number="1" location="apps/sim/lib/oauth/credential-service.ts:798">
P2: This local failure memo can hide a recovered shared application token: another credential or worker may successfully refresh the coordinated token, but this credential returns its stale memo before consulting the coordinated resolver. Restrict this memo lookup to non-`sharedToken` providers, whose tokens are actually cached here, and let the coordinated layer pace shared-token failures.</violation>
</file>

<file name="apps/sim/lib/internal/sailpoint/client.ts">

<violation number="1" location="apps/sim/lib/internal/sailpoint/client.ts:237">
P2: SailPoint still loses `Retry-After` when a 429/503 error body cannot be read. Catch the error-body read for non-success responses and construct `SailPointAuthError` from the status and header anyway, so the resolver can apply the provider's backoff.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread apps/sim/lib/oauth/credential-service.ts
const stillValid = () =>
cached && cached.expiresAtMs - Date.now() > TOKEN_EXPIRY_MARGIN_MS ? cached : undefined

const failedEntry = clientCredentialMintFailureCache.get(cacheIdentity)

@cubic-dev-ai cubic-dev-ai Bot Nov 12, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This local failure memo can hide a recovered shared application token: another credential or worker may successfully refresh the coordinated token, but this credential returns its stale memo before consulting the coordinated resolver. Restrict this memo lookup to non-sharedToken providers, whose tokens are actually cached here, and let the coordinated layer pace shared-token failures.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At apps/sim/lib/oauth/credential-service.ts, line 798:

<comment>This local failure memo can hide a recovered shared application token: another credential or worker may successfully refresh the coordinated token, but this credential returns its stale memo before consulting the coordinated resolver. Restrict this memo lookup to non-`sharedToken` providers, whose tokens are actually cached here, and let the coordinated layer pace shared-token failures.</comment>

<file context>
@@ -761,29 +786,30 @@ async function resolveMintedServiceAccountToken(
+    const stillValid = () =>
+      cached && cached.expiresAtMs - Date.now() > TOKEN_EXPIRY_MARGIN_MS ? cached : undefined
+
+    const failedEntry = clientCredentialMintFailureCache.get(cacheIdentity)
+    const failed = failedEntry?.secretFingerprint === secretFingerprint ? failedEntry : undefined
+    if (failed && Date.now() < failed.retryAtMs) {
</file context>
Suggested change
const failedEntry = clientCredentialMintFailureCache.get(cacheIdentity)
const failedEntry = sharedToken ? undefined : clientCredentialMintFailureCache.get(cacheIdentity)
Fix with cubic

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread apps/sim/lib/internal/sailpoint/client.ts
getSailPointErrorMessage(data, 'Failed to authenticate with SailPoint'),
response.status
response.status,
throttledRetryAfterMs(response)

@cubic-dev-ai cubic-dev-ai Bot Nov 12, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: SailPoint still loses Retry-After when a 429/503 error body cannot be read. Catch the error-body read for non-success responses and construct SailPointAuthError from the status and header anyway, so the resolver can apply the provider's backoff.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At apps/sim/lib/internal/sailpoint/client.ts, line 237:

<comment>SailPoint still loses `Retry-After` when a 429/503 error body cannot be read. Catch the error-body read for non-success responses and construct `SailPointAuthError` from the status and header anyway, so the resolver can apply the provider's backoff.</comment>

<file context>
@@ -228,7 +233,8 @@ export async function requestSailPointToken(
         getSailPointErrorMessage(data, 'Failed to authenticate with SailPoint'),
-        response.status
+        response.status,
+        throttledRetryAfterMs(response)
       )
     }
</file context>
Fix with cubic

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

This branch was previously deployed

1 inactive deployment
Preview — d014680e Deployed Nov 12, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments
@cubic-dev-ai cubic-dev-ai[bot] cubic-dev-ai[bot] left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@waleedlatif1
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.