GeezerVison

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		May 6, 2026 9:16pm

PR Summary

Medium Risk
Changes robots.txt directives for all crawlers and specific link-preview user agents, which can materially impact indexing/crawl behavior if the disallow lists are wrong. Scope is small and isolated to SEO metadata generation.

Overview
Updates apps/sim/app/robots.ts to emit two robots rule groups: a wildcard (*) rule that allows crawling but disallows a defined set of internal/authenticated paths, and a separate rule set for link-preview bots (Twitter/Slack/Discord/etc.) with a looser disallow list so OG previews can fetch /chat/ and /form/ URLs.

Refactors the disallow lists into named constants (DISALLOWED_PATHS, LINK_PREVIEW_DISALLOWED_PATHS) and introduces a LINK_PREVIEW_BOTS allowlist to target the preview-specific behavior.

^{Reviewed by Cursor Bugbot for commit b9a1f58. Configure here.}

Greptile Summary

This PR restores a structured robots.ts by splitting what was previously a single wildcard rule into two groups: a tight wildcard disallow list for all bots, and a named group for link-preview bots with a looser list that intentionally permits /chat/ and /form/ so OG cards render correctly on social platforms.

• Wildcard rule blocks authenticated and internal surfaces (/api/, /workspace/, /chat/, /form/, /playground/, etc.) for all crawlers by default, with /blog*tag= added for tag-filtered pages.
• Link-preview bot group (Twitter, LinkedIn, Slack, Discord, Telegram, WhatsApp, Facebook, Pinterest, Reddit) drops /chat/ and /form/ from the disallow list, enabling OG metadata fetching, while still blocking authenticated paths and /playground/ — fixes from a previous revision of this PR that were already merged.

Confidence Score: 5/5

Safe to merge — the change is scoped entirely to robots.txt generation and carries no runtime risk to application behavior.

The restructuring is straightforward: one wildcard rule with a tight disallow list, one named link-preview bot group with a looser list for OG card fetching. All previously identified issues (incorrect Grok UA strings, no-op Bravebot entry, missing /playground/ and /w/ from the preview-bot disallow list) were corrected in a prior commit and are confirmed absent in the current file. Named user-agent groups take precedence over the wildcard per RFC 9309, so the routing logic is correct. No application logic, data access, or authentication paths are affected.

No files require special attention.

Important Files Changed

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Incoming Bot Request] --> B{User-Agent Match?}
    B -->|Matches LINK_PREVIEW_BOTS\nTwitterbot, LinkedInBot,\nSlackbot, Discordbot, etc.| C[Apply LINK_PREVIEW_DISALLOWED_PATHS]
    B -->|Wildcard *\nall other bots| D[Apply DISALLOWED_PATHS]
    C --> E{Path Allowed?}
    D --> F{Path Allowed?}
    C -.->|Accessible for OG cards| G["/chat/, /form/"]
    E -->|Disallowed| H["Block: /api/, /workspace/, /w/, /playground/, /resume/, /invite/, /unsubscribe/, /credential-account/, /_next/, /private/"]
    E -->|Allowed| I[Crawl Permitted]
    F -->|Disallowed| J["Block: same as preview list + /chat/, /form/, /blog*tag="]
    F -->|Allowed| K[Crawl Permitted]

_{Reviews (3): Last reviewed commit: "chore(seo): trim verbose comments in rob..." | Re-trigger Greptile}

@cursor review

@greptile

@greptile

@cursor review