GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat(aws-marketplace): bill a listing's LLM usage to AWS through a per-period metering tally by waleedlatif1 · Pull Request #9921 · simstudioai/sim

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 12, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator

Summary

  • Plans carry their usage pricing: platform-with-usage bills the organization's billed model usage (BYOK is model_unbilled and never billed) as llm_credits of $0.01; contract bills nothing beyond the contract. Repricing or changing what counts is a plan edit
  • Metering charges what changed in the books, not hours of rows: usage_log cost is topped up in place (recordCumulativeUsage), so each run re-sums every billing period that can still change (a period stops changing a day after it ends, since a request that outlives it records the rest in the next period) and charges the whole units above that period's tally
    • aws_marketplace_usage_tally (organization, dimension, period): cost counted, units charged, and where the count starts. Fractions wait in the tally; units only move forward
    • aws_marketplace_usage_report (license, dimension, hour), the key AWS deduplicates on: a row gathers units while its hour is open and is sent once it closes, or at once for an ended license
  • The charged license is the earliest-linked one in force (status active and not past expiresAt), so overlapping licenses never both bill and a renewal handoff needs no partial-hour logic. With none in force, a license that ended within the hour AWS allows takes the usage at the hour it ended, until that record is sent
  • aws_marketplace_license.ended_at, stamped by the read that first finds a license inactive and cleared if access returns, so a cancellation AWS reports with no entitlement (and no expiry) still gets its final flush
  • aws_marketplace_license.organization_linked_at, stamped wherever a license is linked; usage counts from it. A linked license without one (linked before it existed) is stamped by the first run that sees it
  • Sending: BatchMeterUsage by LicenseArn + CustomerAWSAccountId (no ProductCode), 25 per call, grouped per license. Success / DuplicateRecord → reported; CustomerNotSubscribed or a license-level refusal → rejected and logged; unprocessed or transient → pending. Rows AWS won't take (24h old, or last month's after 06:00 UTC on the 1st) are rejected and logged before sending; a TimestampOutOfBoundsException still raised for a batch is narrowed by sending each row alone
  • A record's quantity is fixed once it is first sent (sent_at, stamped under the organization's usage lock with the quantity re-read), since AWS keeps the first record it accepts: a retry resends the same quantity, and later usage waits for a record that can still take it
  • Cron /api/cron/aws-marketplace-meter every 15 minutes (scheduled in infra; SaaS fulfillment only runs on the hosted deployment, like aws-marketplace-reconcile)
  • Migration 0456 is additive only

Type of Change

  • New feature

Testing

  • metering.integration.ts (15, real Postgres; only the AWS metering client is mocked) + lifecycle.integration.ts link-time and ended_at coverage; marketplace integration suites 89/89, unit 10/10
  • Each guard was removed in turn and its test went red: whole-period re-sum (in-place top-ups), send only after the hour closes, send an ended license at once, stop charging once its end-hour record is sent, skip a license past expiry, stamp unmarked links, period settle window, category filter, count from link, ended_at stamp / keep / clear, callers relinking only on change, the charge-side and send-side sent_at freeze, out-of-range isolation, month-end cutoff and its grace
  • bun run lint, tsc, check:migrations, check:audits (the only local failure is the gitignored docs .content registry CI generates), drizzle-kit generate reports no drift

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@vercel

vercel Bot commented Nov 12, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Nov 12, 2026 6:17am UTC

Request Review

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@greptile-apps

greptile-apps Bot commented Nov 12, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 3/5

[Critical impact] The PR sends metered usage into AWS Marketplace billing for configured listings.

Fix the two unbilled-usage paths before merging.

Findings

  1. P1 Late costs go unbilled ▶
  2. P1 Expired licenses miss final charges ▶

Summary

Adds AWS Marketplace usage billing through per-period tallies and hourly reports.

  • AWS Marketplace listings send billed model usage as hourly credits.
  • License link and end times mark the window for Marketplace usage.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Usage["Model usage costs"] --> Sum["Re-sum eligible periods"]
  Sum --> Tally["Subtract units already counted"]
  Tally --> Report["Add units to hourly report"]
  Report --> Freeze["Freeze quantity before first send"]
  Freeze --> AWS["AWS BatchMeterUsage"]
  AWS --> Accepted["Reported"]
  AWS --> Retry["Pending retry with same quantity"]
  AWS --> Rejected["Rejected and logged"]
Loading

Reviews (3) · Last reviewed commit: "fix(aws-marketplace): fix a record's qua..." · Reviewed by Greptile

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps[bot]
greptile-apps Bot reviewed Nov 12, 2026
View reviewed changes
Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/app/api/cron/aws-marketplace-meter/route.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Nov 12, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 13 files

Confidence score: 2/5

  • route.ts is not scheduled by the Helm chart, so Marketplace usage metering never runs in standard deployments. Add a cron job for this endpoint at the intended 15-minute cadence.
  • 0456_aws_marketplace_usage_report.sql leaves existing organization-linked licenses with a NULL start time, so the metering query excludes their usage. Backfill those rows to the migration time.
  • metering.ts can drop usage during a mid-hour license handoff: the old license ends at expiry, but the new one starts at the next hour boundary. Adjust the handoff reporting window so usage between those times is included.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/billing/aws-marketplace/metering.ts">

<violation number="1" location="apps/sim/lib/billing/aws-marketplace/metering.ts:184">
P2: A mid-hour license handoff drops usage. When the lapsed license's final row covers only up to its expiry, the next license starts at the following hour boundary, so usage between expiry and that boundary is never billed. Start the successor's chain at the lapsed license's expiry instead of `last.hourStart + HOUR_MS`, or carry the exact boundary forward.</violation>
</file>

<file name="apps/sim/app/api/cron/aws-marketplace-meter/route.ts">

<violation number="1" location="apps/sim/app/api/cron/aws-marketplace-meter/route.ts:25">
P1: This handler is not scheduled by the Helm chart, so Marketplace usage metering never runs in the standard deployment. Add a `cronjobs.jobs` entry for `/api/cron/aws-marketplace-meter` with the intended 15-minute schedule.</violation>
</file>

<file name="packages/db/migrations/0456_aws_marketplace_usage_report.sql">

<violation number="1" location="packages/db/migrations/0456_aws_marketplace_usage_report.sql:19">
P1: Existing organization-linked licenses receive `NULL` here and are then excluded by the metering query, so their organization’s model usage is never billed. Backfill linked rows to the migration time while leaving unlinked rows `NULL`.</violation>
</file>

Turn on auto-fix | Re-trigger cubic

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread apps/sim/app/api/cron/aws-marketplace-meter/route.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread packages/db/migrations/0456_aws_marketplace_usage_metering.sql Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread apps/sim/lib/billing/aws-marketplace/metering.integration.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

@vercel
vercel Bot deployed to Preview November 12, 2026 03:36
View deployment
@waleedlatif1 waleedlatif1 changed the title feat(aws-marketplace): bill a listing's LLM usage to AWS hourly through a metering ledger feat(aws-marketplace): bill a listing's LLM usage to AWS through a per-period metering tally Nov 12, 2026
@vercel
vercel Bot temporarily deployed to Preview November 12, 2026 04:58 Inactive
@waleedlatif1

waleedlatif1 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile

greptile-apps[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@waleedlatif1

waleedlatif1 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

cubic-dev-ai[bot] reacted with eyes emoji
All reactions
  • 👀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps[bot]
greptile-apps Bot reviewed Nov 12, 2026
View reviewed changes
Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Nov 12, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 13 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

waleedlatif1 added 3 commits November 11, 2026 22:11
@waleedlatif1
feat(aws-marketplace): bill a listing's LLM usage to AWS hourly throu…
5e27b66
…gh a metering ledger
@waleedlatif1
fix(aws-marketplace): meter what changed in the books per billing per…
79cecd4
…iod, so in-place cost top-ups, renewals and cancellations bill exactly
@waleedlatif1
fix(aws-marketplace): fix a record's quantity once it is first sent, …
5dd3ea9
…and keep out-of-window records from failing a batch
@waleedlatif1
waleedlatif1 force-pushed the feat/aws-marketplace-metering branch from baa1879 to 5dd3ea9 Compare November 12, 2026 06:13
@waleedlatif1

waleedlatif1 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile

greptile-apps[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@waleedlatif1

waleedlatif1 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

cubic-dev-ai[bot] reacted with eyes emoji
All reactions
  • 👀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Nov 12, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 13 files

Confidence score: 3/5

  • In metering.ts, reporting an ended license as soon as it becomes inactive can miss usage that arrives during the one-hour post-end window, leaving it unbilled. Wait until that window closes before sending the final rows.
  • In packages/db/schema.ts, deleting an organization cascades to its unsent usage rows, so that usage can never be billed. Preserve those rows or prevent the cascade.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/billing/aws-marketplace/metering.ts">

<violation number="1" location="apps/sim/lib/billing/aws-marketplace/metering.ts:511">
P2: Sending an ended license's rows as soon as its status flips to `inactive` closes its record before the one-hour post-end window ends. Usage that arrives after that send is skipped, not billed, because `chargeOrganization` returns 0 once `sentAt` is set. Hold the ended license's record until the post-end window closes, or document that usage after the send is intentionally dropped.</violation>
</file>

<file name="packages/db/schema.ts">

<violation number="1" location="packages/db/schema.ts:2023">
P2: The `organizationId` FK on `aws_marketplace_usage_report` cascades on organization delete, so deleting an org drops its unsent usage rows. The license keeps its ARN but loses its link, so that owed usage is never billed to AWS. Consider a non-cascading FK, or flushing pending rows before the org is removed.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts
eq(awsMarketplaceUsageReport.status, 'pending'),
or(
lt(awsMarketplaceUsageReport.hourStart, new Date(floorHour(now))),
eq(awsMarketplaceLicense.status, 'inactive')

@cubic-dev-ai cubic-dev-ai Bot Nov 12, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Sending an ended license's rows as soon as its status flips to inactive closes its record before the one-hour post-end window ends. Usage that arrives after that send is skipped, not billed, because chargeOrganization returns 0 once sentAt is set. Hold the ended license's record until the post-end window closes, or document that usage after the send is intentionally dropped.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At apps/sim/lib/billing/aws-marketplace/metering.ts, line 511:

<comment>Sending an ended license's rows as soon as its status flips to `inactive` closes its record before the one-hour post-end window ends. Usage that arrives after that send is skipped, not billed, because `chargeOrganization` returns 0 once `sentAt` is set. Hold the ended license's record until the post-end window closes, or document that usage after the send is intentionally dropped.</comment>

<file context>
@@ -0,0 +1,560 @@
+        eq(awsMarketplaceUsageReport.status, 'pending'),
+        or(
+          lt(awsMarketplaceUsageReport.hourStart, new Date(floorHour(now))),
+          eq(awsMarketplaceLicense.status, 'inactive')
+        )
+      )
</file context>
Fix with cubic

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread packages/db/schema.ts
{
organizationId: text('organization_id')
.notNull()
.references(() => organization.id, { onDelete: 'cascade' }),

@cubic-dev-ai cubic-dev-ai Bot Nov 12, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The organizationId FK on aws_marketplace_usage_report cascades on organization delete, so deleting an org drops its unsent usage rows. The license keeps its ARN but loses its link, so that owed usage is never billed to AWS. Consider a non-cascading FK, or flushing pending rows before the org is removed.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/db/schema.ts, line 2023:

<comment>The `organizationId` FK on `aws_marketplace_usage_report` cascades on organization delete, so deleting an org drops its unsent usage rows. The license keeps its ARN but loses its link, so that owed usage is never billed to AWS. Consider a non-cascading FK, or flushing pending rows before the org is removed.</comment>

<file context>
@@ -1993,6 +2008,71 @@ export const awsMarketplaceLicense = pgTable(
+  {
+    organizationId: text('organization_id')
+      .notNull()
+      .references(() => organization.id, { onDelete: 'cascade' }),
+    dimension: text('dimension').notNull(),
+    periodStart: timestamp('period_start').notNull(),
</file context>
Fix with cubic

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vercel
vercel Bot deployed to Preview November 12, 2026 06:17
View deployment
greptile-apps[bot]
greptile-apps Bot reviewed Nov 12, 2026
View reviewed changes
Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts
and(
eq(usageLog.billingEntityType, 'organization'),
eq(usageLog.billingEntityId, organizationId),
gt(usageLog.billingPeriodEnd, new Date(now - PERIOD_SETTLE_MS))

@greptile-apps greptile-apps Bot Nov 12, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Late costs go unbilled

countNewUnits stops reading a period 24 hours after its end, but reporting-window payers can still add cost to it. AWS customers are provisioned as Enterprise customers with reporting windows. For those payers, /api/billing/update-cost omits payerSubscriptionId, so recordCumulativeUsage keeps updating the original row instead of moving later cost into a new period.

A callback or retry arriving after the cutoff adds billable cost that no future meter run reads. Keep those changes discoverable, or ensure reporting-window writes roll forward before applying this cutoff.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread apps/sim/lib/billing/aws-marketplace/metering.ts
Comment on lines +183 to +191
const inForce = licenses.find(
(license) =>
license.status === 'active' && (!license.expiresAt || license.expiresAt.getTime() > now)
)
if (inForce) return { license: inForce, hourStart: floorHour(now) }
const ended = licenses
.filter((license) => license.endedAt && license.endedAt.getTime() >= now - AFTER_END_MS)
.sort((left, right) => (right.endedAt?.getTime() ?? 0) - (left.endedAt?.getTime() ?? 0))[0]
return ended?.endedAt ? { license: ended, hourStart: floorHour(ended.endedAt.getTime()) } : null

@greptile-apps greptile-apps Bot Nov 12, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Expired licenses miss final charges

chargeTarget cannot flush an expired license while its stored status is still active. After expiresAt, the active branch rejects it, but the fallback requires endedAt. That field is only stamped after verification marks the license inactive.

A license expiring at 10:20 can gain usage after the 10:15 meter run, then have no charge target at 10:30. If no prompt event arrives, routine verification can wait a day—past AWS's one-hour final-report window. The final usage then goes unbilled. Use the known expiry as an end time when no replacement is in force, and send the final record without waiting for the status change.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

This branch was successfully deployed

1 active deployment
Preview — 5dd3ea93 Deployed Nov 12, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments
@cubic-dev-ai cubic-dev-ai[bot] cubic-dev-ai[bot] left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@waleedlatif1
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.