CI/CD for UTMStack v10 and v11. This folder contains two workflow families:

• PR checks (pr-checks.yml + _pr-reusable-*.yml) — validate every Pull Request before merge. The only gate into code on release/**, v10 and v11.
• Deployment pipelines (v10-deployment-pipeline.yml, v11-deployment-pipeline.yml, installer-release.yml) — build, publish and deploy artifacts once code is merged.

• Release policy
• PR Checks
• V10 Deployment Pipeline
• V11 Deployment Pipeline
• Installer Release
• Secrets and variables
• Approver GitHub App setup
• Reusable workflows
• How to deploy
• Troubleshooting

Hard rules:

• Direct push is forbidden on release/**, v10 and v11. PR only.
• Branch protection is enabled on those branches: PR required, status checks green (All checks passed), no force push.
• Roll-forward only. No rollbacks. If a release breaks something, ship a hotfix that bumps the version (e.g. v11.2.9 breaks → v11.2.10 fixes it). Feature flags / kill switches are fine for turning features off without a redeploy.

The team is small (3 devs + 2 part-time seniors), so the AI can approve and merge on its own in most cases. Seniors only get involved when the cost of being wrong is high.

The final tier of a PR is decided by the approver, taking the maximum across all AI prompts (see PR Checks).

When the author pushes new commits, the sticky comments are updated in-place (same comment, no stacking) and the workflow re-runs automatically. A blocked PR is never auto-closed — it stays open waiting for the fixes.

Sensitive paths for Tier 3 are identified by each prompt's own rules (see .github/ai-prompts/*.md). In the future this could be reinforced with CODEOWNERS for additional per-path gates.

The approver enables GitHub's native auto-merge only when all of the following hold:

• Target branch matches release/** (PRs to v10 / v11 stay manual so production deploys are always intentional).
• deps_failed == false.
• max_tier == 1 across every AI prompt.
• PR author is in @utmstack/administrators or @utmstack/core-developers.
• The approver GitHub App is configured (APPROVER_APP_ID + APPROVER_PRIVATE_KEY secrets present).

Auto-merge does NOT merge immediately — it queues the merge until every branch-protection requirement is satisfied. If another check fails later or a human leaves REQUEST_CHANGES, the merge stays pending.

Disabled. .github/dependabot.yml keeps updates: [] so Dependabot reads the file but creates no PRs. Dependency freshness is enforced via the go_deps check on every PR. To re-enable Dependabot, restore the previous updates: list (see git history of that file).

• hotfix/x branch from v11 → PR to v11 → same checks.
• urgent label allows fast-track: if checks pass and the AI approves, it merges without waiting for human review even when touching sensitive paths.
• Recommended (not strictly required): after the hotfix merges to v11, pull v11 into the active release/v11.x.x+1 branch (merge or cherry-pick — either works). The fix is not lost if you skip this step: git already has the hotfix in v11's history, so when release/v11.x.x+1 later merges back, git combines both lines and the fix lands automatically. Syncing early is good hygiene because it surfaces conflicts in your release branch rather than at the final merge, and it lets dev builds include the patched code immediately.

Version derivation. Dev builds are automatic: a push to release/v11.x.x queries CM DEV for the latest version of that base and increments the dev suffix (v11.x.x-dev.9 → v11.x.x-dev.10); a base that CM DEV has never seen starts at -dev.1. Release tags are not derived — the pipeline uses the GitHub release tag verbatim, with no comparison against CM PROD and no auto-bump. If a hotfix means the next patch version, pick the new tag when you create the release (e.g. v11.2.9 is already in PROD → tag v11.2.10); the build, installer, and CM registration all follow that tag.

pr-checks.yml triggers on any Pull Request whose target is:

• release/** (any release branch, v10 or v11)
• v10
• v11

PR opened / updated
        │
        ├─────────────────┬─────────────────┐
        ▼                 ▼                 │
   ┌─────────┐      ┌─────────────┐         │
   │ go_deps │      │  ai_review  │         │
   │ (repo)  │      │  (matrix    │         │
   │         │      │   per       │         │
   │         │      │   prompt)   │         │
   └────┬────┘      └──────┬──────┘         │
        │                  │                 │
        │ artifact         │ artifacts       │
        │ go-deps-result   │ ai-review-*     │
        ▼                  ▼                 │
   ┌──────────────────────────────┐         │
   │           approver           │         │
   │  - reads artifacts           │         │
   │  - decides final tier        │         │
   │  - posts sticky comments     │         │
   │  - (optional) formal review  │         │
   └──────────────┬───────────────┘         │
                  │                          │
                  ▼                          ▼
          all_checks_passed   ← single status check branch protection requires

Key decision: the producers (go_deps, ai_review) always exit green. They only upload artifacts. The approver is the single source of truth — it consolidates results, decides the tier (maximum across all AI prompts), posts comments, and passes or fails the final check.

Single job, no matrix, no change detection. Runs:

bash .github/scripts/go-deps.sh --check --discover

It discovers every go.mod in the repo (excluding vendor/, dot-directories and node_modules/) and fails if any explicit direct dependency has a newer version available. The script also detects out-of-sync go.sum files (typically caused by local replace directives in packages/) and reports them all at once.

The job uploads its stdout and exit code as the go-deps-result artifact. The approver reads it and, if exit code != 0, posts the sticky comment "Go dependencies check failed" with the script output embedded.

Expected dev fix: run bash .github/scripts/go-deps.sh --update --discover locally, commit the updated go.mod / go.sum, push.

Matrix with one job per .md under .github/ai-prompts/ (except README.md). Each job:

1. Fetches the diff via gh pr diff (same unified diff the GitHub UI shows — no need for fetch-depth: 0).
2. Calls the ThreatWinds AI /chat/completions endpoint with the prompt and the diff.
3. Validates the response against the {tier, summary, findings} schema.
4. Uploads the JSON as the ai-review-<name> artifact.

If the model's response isn't valid JSON or the tier isn't 1/2/3, the script writes a fallback with tier: 2 and a "Manual review recommended" finding (fail-safe).

Initial prompts:

• security.md — vulnerabilities introduced in the diff (injection, XSS, SSRF, hardcoded secrets, weak crypto, insecure deserialization).
• bugs.md — concrete bugs: nil derefs, races, off-by-one, unhandled errors, unclosed resources, inverted logic, out-of-context code.
• architecture.md — architectural deviations: new couplings, logic in the wrong layer, broken contracts, unsafe migrations.

Each prompt declares its own tier policy (Tier 3 covers paths critical to that dimension). See .github/ai-prompts/README.md for the full schema and tier semantics.

To scale: drop a new .md into .github/ai-prompts/. Discovered at runtime — no YAML changes needed.

Default model: silas-1.7-pro (the only model currently available on ThreatWinds). Each prompt can pin its own model in frontmatter (model: <id>).

Single job that depends on go_deps and ai_review. Steps:

1. Downloads every PR-check artifact.
2. Reads go-deps-result/exit_code.txt → determines deps_failed.
3. Reads each ai-review-*/result.json → takes the max tier as the AI verdict.
4. Sticky comments with invisible HTML markers (<!-- approver:deps -->, <!-- approver:ai -->, <!-- approver:permission -->):
   • If deps failed → upsert "Go dependencies check failed" comment with the script output. Otherwise delete it if a previous run posted one.
   • Always upsert the "AI review" comment with the final tier + findings.
   • These two are posted regardless of who opened the PR — even unauthorized contributors get useful feedback.
5. Permission check (LAST gate). Looks up github.actor against the GitHub teams administrators and core-developers of the utmstack org via API_SECRET. Notably does NOT include integration-developers. If the author is in neither team:
   • Upsert "⛔ Permission denied" comment @mentioning @utmstack/administrators.
   • Skip the formal APPROVE review (always REQUEST_CHANGES).
   • Skip auto-merge.
   • Exit 1.
6. (Optional) Formal PR review when the approver App is installed (see Approver GitHub App setup):
   • Tier 1 + deps OK + authorized → APPROVE.
   • Anything else → REQUEST_CHANGES.
7. Auto-merge — only when all of: deps OK, Tier 1, authorized, AND BASE_REF starts with release/. Calls gh pr merge --auto --<method> (default squash). This uses GitHub's native auto-merge, so the actual merge waits until every branch-protection requirement is satisfied (other checks green, no pending human reviews). PRs targeting v10 / v11 never auto-merge — those branches stay manually merged so deploys are intentional.
8. Exit code: 0 only if everything is OK; 1 if deps failed, tier ≥ 2, or author unauthorized.

When the author pushes new commits the workflow re-runs and the comments are updated in place (no stacking). The PR is never auto-closed — it stays open waiting for the author's fixes.

The architecture is designed to scale. To add, for example, a test check:

1. Create .github/workflows/_pr-reusable-<name>.yml that runs the check and uploads an artifact with the result (ideally JSON).
2. Call the reusable from pr-checks.yml as another job.
3. Add that job to the approver's needs: (and to all_checks_passed).
4. Extend approver.sh to read the new artifact and factor it into the final verdict.

To add a new AI prompt no YAML changes are needed — just drop a .md into .github/ai-prompts/.

Triggers:

• Push to v10 → deploy to v10-rc
• Push to release/v10** → deploy to v10-dev
• Release published with tag v10.* (non-prerelease) → production installer build

Main jobs:

1. setup_deployment — determines environment from the trigger.
2. build_agent — cross-compiles Linux + Windows (amd64/arm64) agents and signs the Windows binaries with signtool on the utmstack-signer self-hosted runner.
3. build_agent_manager — Docker image bundling agents + collectors.
4. build_* — microservices (aws, backend, correlation, frontend, bitdefender, mutate, office365, log-auth-proxy, soc-ai, sophos, user-auditor, web-pdf).
5. all_builds_complete — checkpoint.
6. deploy_installer_dev / deploy_installer_rc — build and upload the installer for the corresponding environment.

Note: v10 has no CM publish/schedule step — v10 deploys are installer uploads only, and there is no in-pipeline permission gate (branch protection is enforced on GitHub's side instead).

Triggers:

• Push to release/v11** → build & deploy to dev (auto-incremented version v11.x.x-dev.N), install to the dev runner, publish to CM-dev.
• Any release published with tag v11.* (prerelease or final) → build, upload the installer to the release, publish to CM-prod and schedule updates to the prod instances. Prereleases are the RC gate in practice — customers are never pointed at a pre-release build by us.

Push to release/v11.x.x
        │
        ▼
Auto-increment version (v11.x.x-dev.N)
        │
        ▼
Build all images + sign agents
        │
        ▼
Install to dev runner (utmstack-v11-dev)
        │
        ▼
Publish to CM Dev → schedule to dev instances


Publish release with tag v11.x.x (prerelease or final)
        │
        ▼
Build all images + sign agents
        │
        ▼
Generate Changelog (AI) → upload installer to the release
        │
        ▼
Publish to CM Prod → schedule to prod instances

Jobs: setup_deployment, build_agent, sign_agent_windows, sign_agent_macos, build_utmstack_collector, build_agent_manager, build_event_processor, build_backend (Java 17), build_frontend, build_user_auditor, build_web_pdf, all_builds_complete, generate_changelog (release events), build_installer_release (release events, uploads to the triggered release), deploy_installer_dev (dev events, installs to the dev runner), publish_new_version, schedule.

The pipeline has exactly two environments (dev, production); there is no rc environment in v11. "RC" in this repo means publishing a prerelease tag, which runs the production pipeline (build, installer upload, CM-prod register, prod-instance schedule) against the RC fleet.

1. Extracts the base version from the branch (release/v11.2.1 → v11.2.1).
2. Queries CM for the latest version.
3. If the base matches, bumps the dev suffix (-dev.9 → -dev.10).
4. If the base differs, starts at -dev.1.

• Community: manual — promoting the pre-release to a final released state triggers the production pipeline again, which registers the new version with CM-prod and schedules it to the prod instance list.
• Enterprise: manual with a checklist (zero crashes for 48h, no open P0 issues). The last safety net before touching large customers.

installer-release.yml has no standalone trigger — it is workflow_call-only and invoked by the deployment pipelines (deploy_installer_dev / deploy_installer_rc on v10, deploy_installer_dev / build_installer_release on v11) with version, version_major, environment, and optional prerelease / changelog inputs.

What it does, by (version_major, environment):

It authenticates private utmstack/* Go module fetches with API_SECRET (git insteadOf rewrite + GOPRIVATE).

The approver job uses a GitHub App (instead of a personal PAT) to leave formal PR reviews and enable auto-merge. Pros:

• Per-run installation token, valid for ~1 hour, auto-revoked when the job ends. No long-lived credential in the repo.
• The App acts as its own identity, so it can APPROVE PRs opened by any human contributor — including the workflow's own author (GitHub blocks self-approval when using a PAT).
• One place to audit who/what changed your branch protection state.

1. Create the App.

Go to: https://github.com/organizations/utmstack/settings/apps/new

• GitHub App name: e.g. UTMStack Approver.
• Homepage URL: any (the UTMStack repo URL is fine).
• Webhook: untick Active — no callbacks needed.
• Repository permissions:
  • Contents: Read-only
  • Pull requests: Read and write
  • Metadata: Read-only (default, can't be removed).
• Organization permissions:
  • Members: Read-only — needed for the team-membership check.
• Where can this GitHub App be installed? Only on this account.

Click Create GitHub App.

2. Get the App ID and a private key.

On the App's settings page you'll see the App ID (numeric). Save it.

Scroll to Private keys → Generate a private key. A .pem file downloads. Save the full contents (BEGIN/END lines included).

3. Install the App on the UTMStack repo.

On the App page → Install App → pick the utmstack org → choose Only select repositories → select UTMStack → Install.

4. Add the secrets to the repo.

Settings → Secrets and variables → Actions → New repository secret.

• APPROVER_APP_ID = the numeric App ID.
• APPROVER_PRIVATE_KEY = the full PEM contents of the .pem file, including the -----BEGIN/END PRIVATE KEY----- lines. Paste as-is — GitHub preserves multi-line values.

5. Optional: drop API_SECRET.

If the App has Members: Read at org level, you can stop maintaining a separate API_SECRET PAT for the permission check. The approver workflow falls back to the App token when API_SECRET is not set (API_SECRET: ${{ secrets.API_SECRET || steps.app-token.outputs.token }} in _pr-reusable-approver.yml).

API_SECRET is still used by the deployment workflows (v10-deployment-pipeline.yml, v11-deployment-pipeline.yml) for things like fetching private Go modules during installer builds — don't delete it from the repo until you confirm those workflows no longer need it.

In _pr-reusable-approver.yml:

- name: Generate approver token from GitHub App
  id: app-token
  if: ${{ env.APP_ID != '' }}
  env:
    APP_ID: ${{ secrets.APPROVER_APP_ID }}
  uses: actions/create-github-app-token@v1
  with:
    app-id: ${{ secrets.APPROVER_APP_ID }}
    private-key: ${{ secrets.APPROVER_PRIVATE_KEY }}

If the secrets aren't configured, the step is skipped, the approver runs in comments-only mode, and everything else still works (deps comment, AI review comment, status check) — just no formal review and no auto-merge.

1. Open a small, low-risk PR against release/v11.x.x (or push the workflow to a sandbox branch).
2. After the approver job runs, check the PR page:
   • The sticky <!-- approver:ai --> comment is signed by your bot account (e.g. utmstack-approver[bot]).
   • The "Files changed" tab shows a review by the same bot, marked Approved (Tier 1 + deps OK + authorized) or Changes requested.
3. If the target is release/** and Tier 1 → auto-merge is queued in the PR header ("Auto-merge enabled by … via GitHub Actions").

PR checks:

• _pr-reusable-go-deps.yml — runs go-deps.sh --check --discover at repo level and uploads go-deps-result as an artifact.
• _pr-reusable-ai-review.yml — fan-out per prompt; each job uploads ai-review-<name> as an artifact.
• _pr-reusable-approver.yml — downloads artifacts, decides verdict, posts sticky comments, optionally leaves a formal PR review.

Deployment pipelines:

• reusable-basic.yml — generic Docker builds.
• reusable-golang.yml — Go microservices.
• reusable-java.yml — Java microservices.
• reusable-node.yml — frontend / node.
• reusable-sign-agent.yml — agent signing.

Dev:

git checkout release/v10.x.x
# Make changes via PR → merge → auto-deploy to v10-dev

RC:

# PR from release/v10.x.x → v10 → merge → auto-deploy to v10-rc

Production:

# 1. Tag the commit
git tag v10.5.0
git push origin v10.5.0
# 2. On GitHub Releases, publish the tag as a NON-prerelease release.
#    The pipeline only fires on `release: types: [released]` — a bare
#    tag push does NOT trigger it.

Dev:

# Open a PR against release/v11.2.1 → checks → merge → auto-deploy
# Version auto-incremented (v11.2.1-dev.1, v11.2.1-dev.2, ...)

RC:

1. GitHub Releases → "Draft a new release".
2. New tag (e.g. v11.2.1).
3. Mark as pre-release.
4. Publish. The pipeline fires on any published v11.* release (prereleases are the norm); it builds all images, generates the AI changelog, uploads the installer to the release, publishes to CM-prod, and schedules updates to the prod instances.

Hotfix:

git checkout v11
git checkout -b hotfix/auth-bug
# fix → PR to v11 (label `urgent` if applicable) → checks → merge
# Recommended after merge: sync v11 into release/v11.x.x+1
#   git checkout release/v11.x.x+1
#   git merge origin/v11      # or cherry-pick the specific commits
#   git push

Permission denied:

• If the PR shows a "⛔ Permission denied" comment from the approver, the author is not in administrators or core-developers (see the approver step list above) — a member of one of those teams must push the fix or re-open the PR. The deployment pipelines themselves have no team gate; they rely on branch protection and the PR approver.

ai_review artifact with tier 2 fallback "Manual review recommended":

• The model didn't return valid JSON or returned an invalid tier. The approver treats it as Tier 2 (changes requested) fail-safe. Refine the prompt .md or re-run the workflow if it was transient.

go_deps fails with "Could not inspect ... run 'go mod tidy' there":

• go.sum is out of sync, typically due to local replace directives in packages/. Run go mod tidy in the affected module and commit.

The approver posts two separate comments (deps + AI):

• That's the expected behaviour when both dimensions fail. Each comment is independent and gets updated in place on subsequent runs.

The approver doesn't leave a formal review (only comments):

• The approver GitHub App is not configured. Add both APPROVER_APP_ID and APPROVER_PRIVATE_KEY secrets — see Approver GitHub App setup.

Want a senior engineer @mentioned on Tier 3:

• Edit pr-checks.yml, in the approver job set the tier3_reviewers input with comma-separated handles:

  with:
    tier3_reviewers: 'Kbayero,osmontero'

Build failures:

• Check that all required secrets are configured.
• v10: verify availability of the utmstack-signer self-hosted runner (its agent build + signtool step runs there). v11 signs on managed runners (ubuntu-latest for Windows/jsign+KMS, macos-latest for codesign+notarytool) — no self-hosted runner involved.

Version not incrementing:

• Check that CM_SERVICE_ACCOUNT_DEV / CM_SERVICE_ACCOUNT_PROD are configured and that the CM API is reachable.
• The branch name must follow release/v11.x.x.

Changelog not generated:

• Only runs on release events (v11 pipeline).
• Verify THREATWINDS_API_KEY and THREATWINDS_API_SECRET are configured.
• To test locally: export the same two secrets and run bash .github/scripts/generate-changelog.sh <current_tag> [previous_tag] from the repo root (previous tag auto-detected from git tag when omitted; output written to OUTPUT_FILE, default /tmp/changelog.md).

• Docker images are published to ghcr.io/utmstack/utmstack/*.
• Agent signing: v10 uses the utmstack-signer self-hosted runner (signtool); v11 uses GCP KMS (Windows, jsign) and Apple codesign + notarytool (macOS) on managed runners.
• Artifacts (agents, collector) have a 1-day retention.
• Dev versions: v11.x.x-dev.N (auto-incremented).
• Release versions: the published release tag (e.g. v11.2.1); RC in practice = the prerelease tag.