GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — improvement(seo): optimize sitemaps, robots.txt, and core web vitals across sim and docs by emir-karabeg · Pull Request #4170 · simstudioai/sim

Conversation

emir-karabeg commented Apr 15, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Collaborator

Summary

Fix 6x duplicate URL bug in docs sitemap — convert to Next.js metadata convention with native hreflang and x-default
Add missing pages to sim sitemap: blog author pages, academy catalog/course pages
Remove changeFrequency/priority (Google ignores both), fix inaccurate lastModified timestamps
Consolidate 20+ redundant per-bot robots rules into single wildcard, add missing disallow paths
Extract DOCS_BASE_URL into shared constant, replacing 20+ hardcoded instances across docs app
Remove OneDollarStats analytics script and tighten CSP for improved core web vitals

Context

SEO audit found several issues: the docs sitemap generated every page 6x (once per language) without hreflang alternates, the sim sitemap was missing public pages and using new Date() as lastModified on static content (which trains Google to distrust the signal), and robots.txt had 20+ identical bot-specific rules that added noise with no effect. Additionally, hardcoded https://docs.sim.ai strings were scattered across 20+ files in the docs app with no single source of truth.

Changes

Sim sitemap (apps/sim/app/sitemap.ts)

Add blog author pages (/blog/authors/[id]) with lastModified derived from each author's latest post
Add academy pages (/academy, /academy/[courseSlug])
Fix lastModified accuracy — use real content dates for blog/models, omit for static JSON-derived pages
Remove changeFrequency and priority fields (confirmed ignored by Google)
Add empty-array guard on latestModelDate to prevent RangeError

Sim robots (apps/sim/app/robots.ts)

Replace 20+ identical per-bot rule blocks with single * wildcard
Add /form/ and /credential-account/ to disallow list
Reference image sitemap (/blog/sitemap-images.xml)
Remove deprecated host directive

Docs sitemap (apps/docs/app/sitemap.ts — new, replaces apps/docs/app/sitemap.xml/route.ts)

Convert from raw XML route handler to Next.js MetadataRoute.Sitemap convention
Use source.getLanguages() from Fumadocs to deduplicate pages by slug
Add proper alternates.languages with x-default for all 6 locales
Omit lastModified (no accurate source available without git plugin — absent is better than inaccurate)

Docs robots (apps/docs/app/robots.txt/route.ts)

Move disallow rules before allow under User-agent: *
Import DOCS_BASE_URL from shared constant

Docs URL consolidation (apps/docs/lib/urls.ts — new)

Extract DOCS_BASE_URL constant with env variable override (NEXT_PUBLIC_DOCS_URL)
Replace all 20+ hardcoded https://docs.sim.ai strings across layouts, metadata, structured data, LLM manifest, sitemap, and robots files

Core web vitals (apps/sim/)

Remove OneDollarStats analytics script and related config
Tighten Content Security Policy

Type of Change

Bug fix
Improvement

Testing

Both apps pass TypeScript type-check with no errors
Pre-commit hooks (biome lint/format) pass
Verified all sitemap URLs reference existing routes
Verified blog canonical field uses absolute URLs (enforced by Zod z.string().url())
Verified authors field is always populated (throws if empty)
Verified integration slug, model href, and course slug values match their routes
Verified zero hardcoded docs.sim.ai strings remain in docs TS/TSX files (only lib/urls.ts and static public/llms.txt)

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps[bot] reacted with thumbs up emoji

All reactions

👍 1 reaction

vercel Bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview, Comment	Apr 15, 2026 7:00pm

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

cursor Bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

PR Summary

Medium Risk
Changes sitemap/robots generation for both docs and sim, which can materially affect crawl/indexing if URLs or alternates are wrong. Also removes a third-party analytics script and updates CSP allowlists, which could break tracking or any remaining references to those domains.

Overview
Improves SEO outputs across apps/docs and apps/sim by reworking sitemap and robots generation and centralizing docs URL configuration.

In docs, replaces the custom /sitemap.xml route with a Next.js MetadataRoute.Sitemap implementation that deduplicates pages by slug and emits hreflang/x-default alternates, and simplifies robots.txt while switching all hardcoded https://docs.sim.ai references to an env-overridable DOCS_BASE_URL.

In sim, simplifies robots rules into a single wildcard block (with additional disallows) and expands the sitemap to include blog author pages and academy pages while making lastModified timestamps reflect actual content updates (dropping priority/changeFrequency). Separately removes OneDollarStats analytics and strips its domains/env var from dependencies and CSP.

^{Reviewed by Cursor Bugbot for commit c2c9ad4. Configure here.}

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vercel Bot deployed to Preview

April 15, 2026 02:50

View deployment

greptile-apps Bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

Greptile Summary

This PR addresses an SEO audit across both apps: fixes a 6x duplicate URL bug in the docs sitemap by converting to Next.js native convention with proper hreflang/x-default, adds missing blog author and academy pages to the sim sitemap, consolidates 20+ redundant per-bot robots rules into a single wildcard, extracts DOCS_BASE_URL from a hardcoded constant into a shared env-overridable variable, and removes the OneDollarStats analytics script to improve core web vitals. All three previously flagged issues (homepage new Date() on lastModified, unguarded latestModelDate empty array, and per-provider Math.max with zero-model providers) are resolved in follow-up commits.

Confidence Score: 5/5

Safe to merge — all previously flagged issues are resolved and no new blocking issues were found.

All three prior P1 concerns (homepage new Date(), latestModelDate empty-array crash, per-provider Math.max crash) are addressed in follow-up commits in this branch. The remaining code is well-structured: DOCS_BASE_URL extraction is clean, the docs sitemap correctly deduplicates with hreflang alternates, robots consolidation is accurate, and the OneDollarStats removal leaves no dangling references. Only P2 or lower findings remain.

No files require special attention.

Important Files Changed

Filename	Overview
apps/sim/app/sitemap.ts	Adds blog author pages and academy pages; all three previously flagged guard issues (homepage date, latestModelDate guard, per-provider Math.max guard) are resolved.
apps/docs/app/sitemap.ts	New file replacing sitemap.xml/route.ts; correctly deduplicates pages by slug using source.getLanguages() and adds proper hreflang alternates with x-default.
apps/docs/lib/urls.ts	New shared constant DOCS_BASE_URL with env-variable override — clean extraction replacing 20+ hardcoded strings.
apps/sim/app/robots.ts	Consolidates 20+ per-bot rule blocks into a single wildcard; adds /form/ and /credential-account/ to disallow list; references image sitemap.
apps/docs/app/robots.txt/route.ts	Correctly orders Disallow before Allow under User-agent: * and replaces hardcoded URL with DOCS_BASE_URL; revalidate=false is appropriate for a static robots file.
apps/sim/lib/core/security/csp.ts	OneDollarStats domains correctly removed from CSP directives; remaining static sources look correct.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["/sitemap.xml request"] --> B{App}
    B -->|apps/sim| C["sitemap.ts\n(MetadataRoute.Sitemap)"]
    B -->|apps/docs| D["sitemap.ts\n(MetadataRoute.Sitemap)"]

    C --> C1["Static pages\n(no lastModified on homepage)"]
    C --> C2["Blog posts\n(lastModified from content date)"]
    C --> C3["Author pages\n(latest post date per author)"]
    C --> C4["Integration pages\n(no date)"]
    C --> C5["Model/provider pages\n(Math.max guard applied)"]
    C --> C6["Academy pages\n(no date)"]

    D --> D1["source.getLanguages()"]
    D1 --> D2["Group pages by slugs.join('/')"]
    D2 --> D3["For each slug group:\nurl = defaultLanguage URL\nalternates.languages = all locale URLs\nx-default = defaultLanguage URL"]
    D3 --> D4["MetadataRoute.Sitemap entries\nwith hreflang alternates"]

    E["/robots.txt request"] --> F{App}
    F -->|apps/sim| G["robots.ts\nSingle User-agent: *\n+ disallow list + sitemap refs"]
    F -->|apps/docs| H["robots.txt/route.ts\nSingle User-agent: *\n+ DOCS_BASE_URL sitemap ref"]

_{Reviews (3): Last reviewed commit: "fix(seo): guard per-provider Math.max, c..." | Re-trigger Greptile}

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps Bot reviewed

Apr 15, 2026

View reviewed changes

Comment thread

apps/sim/app/sitemap.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

apps/sim/app/sitemap.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

cursor Bot reviewed

Apr 15, 2026

View reviewed changes

cursor Bot left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: Changelog lastModified incorrectly derived from blog posts
- Removed the incorrect lastModified from the changelog sitemap entry since it's driven by GitHub releases, not blog posts, matching the pattern used for /partners.

Or push these changes by commenting:

@cursor push 473032693b

Preview (473032693b)

diff --git a/apps/sim/app/sitemap.ts b/apps/sim/app/sitemap.ts
--- a/apps/sim/app/sitemap.ts
+++ b/apps/sim/app/sitemap.ts
@@ -37,7 +37,6 @@
     },
     {
       url: `${baseUrl}/changelog`,
-      lastModified: latestPostDate,
     },
     {
       url: `${baseUrl}/integrations`,

_{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.}

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

apps/sim/app/sitemap.ts

		url: `${baseUrl}/changelog`,
		lastModified: now,
		lastModified: latestPostDate,
		},

cursor Bot Apr 15, 2026

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changelog lastModified incorrectly derived from blog posts

Medium Severity

The /changelog entry uses latestPostDate as its lastModified, but the changelog page is driven by GitHub releases (fetched from api.github.com/repos/simstudioai/sim/releases), not blog posts. This gives search engines an inaccurate modification date that reflects the latest blog post update rather than when the changelog actually changed. Given the PR's stated goal of fixing inaccurate lastModified timestamps, this entry would be better off omitting lastModified entirely (like /partners) since no accurate source is available at sitemap-generation time.

^{Reviewed by Cursor Bugbot for commit eeac0f8. Configure here.}

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

vercel Bot temporarily deployed to Preview

April 15, 2026 02:56

Inactive

emir-karabeg changed the title ~~improvement(seo): optimize sitemaps and robots.txt across sim and docs~~ improvement(seo): optimize sitemaps, robots.txt, and core web vitals across sim and docs

Apr 15, 2026

vercel Bot deployed to Preview

April 15, 2026 18:32

View deployment

emir-karabeg added 3 commits

April 15, 2026 11:34


          improvement(seo): optimize sitemaps and robots.txt across sim and docs

2c6662c

- Add missing pages to sim sitemap: blog author pages, academy catalog and course pages
- Fix 6x duplicate URL bug in docs sitemap by deduplicating with source.getLanguages()
- Convert docs sitemap from route handler to Next.js metadata convention with native hreflang
- Add x-default hreflang alternate for docs multi-language pages
- Remove changeFrequency and priority fields (Google ignores both)
- Fix inaccurate lastModified timestamps — derive from real content dates, omit when unknown
- Consolidate 20+ redundant per-bot robots rules into single wildcard entry
- Add /form/ and /credential-account/ to sim robots disallow list
- Reference image sitemap in sim robots.txt
- Remove deprecated host directive from sim robots
- Move disallow rules before allow in docs robots for crawler compatibility
- Extract hardcoded docs baseUrl to env variable with production fallback


          fix(seo): remove homepage new Date(), guard latestModelDate empty array

ddb412e


          improvement(seo): consolidate DOCS_BASE_URL, optimize core web vitals

Extract hardcoded https://docs.sim.ai into shared DOCS_BASE_URL constant
in lib/urls.ts and replace all 20+ instances across layouts, metadata,
structured data, LLM manifest, sitemap, and robots files. Remove
OneDollarStats analytics script and tighten CSP for improved core web vitals.

emir-karabeg force-pushed the improvement/sitemap branch from d9f5f69 to 2813751 Compare

April 15, 2026 18:36


          fix: removed onedollarstats from bun lock

5798e80

vercel Bot deployed to Preview

April 15, 2026 18:41

View deployment

emir-karabeg commented Apr 15, 2026

Copy link

Copy Markdown

Collaborator Author

@greptile

greptile-apps[bot] reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

emir-karabeg commented Apr 15, 2026

Copy link

Copy Markdown

Collaborator Author

@cursor review

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps Bot reviewed

Apr 15, 2026

View reviewed changes

Comment thread

apps/sim/app/sitemap.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

cursor Bot reviewed

Apr 15, 2026

View reviewed changes

Comment thread

apps/docs/app/robots.txt/route.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.


          fix(seo): guard per-provider Math.max, consolidate docs robots to sin…

c2c9ad4

…gle wildcard

vercel Bot deployed to Preview

April 15, 2026 19:00

View deployment

emir-karabeg commented Apr 15, 2026

Copy link

Copy Markdown

Collaborator Author

@greptile

greptile-apps[bot] reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

emir-karabeg commented Apr 15, 2026

Copy link

Copy Markdown

Collaborator Author

@cursor review

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

cursor Bot reviewed

Apr 15, 2026

View reviewed changes

cursor Bot left a comment

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c2c9ad4. Configure here.}

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

emir-karabeg merged commit 5274efd into staging

Apr 15, 2026

14 checks passed

Uh oh!

There was an error while loading. Please reload this page.

emir-karabeg deleted the improvement/sitemap branch

April 15, 2026 19:13

icecrasher321 mentioned this pull request

Apr 15, 2026

v0.6.45: superagent, csp, brightdata integration, gemini response format, logs performance improvements #4187

Merged

Sg312 added a commit that referenced this pull request

Apr 15, 2026


          v0.6.45: superagent, csp, brightdata integration, gemini response for…

010435c

…mat, logs performance improvements

fix(csp): add missing analytics domains, remove unsafe-eval, fix workspace CSP gap (#4179)
fix(landing): return 404 for invalid dynamic route slugs (#4182)
improvement(seo): optimize sitemaps, robots.txt, and core web vitals across sim and docs (#4170)
fix(gemini): support structured output with tools on Gemini 3 models (#4184)
feat(brightdata): add Bright Data integration with 8 tools (#4183)
fix(mothership): fix superagent credentials (#4185)
fix(logs): close sidebar when selected log disappears from filtered list; cleanup (#4186)

waleedlatif1 mentioned this pull request

May 6, 2026

improvement(seo): restore explicit AI/search bot allow-list and add link-preview rules #4480

Merged

5 of 6 tasks

This branch was successfully deployed

1 active deployment

Preview — c2c9ad4a Deployed Apr 15, 2026 by vercel[bot]

View deployment

Reviewers

greptile-apps[bot]

greptile-apps[bot] left review comments

cursor[bot]

cursor[bot] left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

emir-karabeg commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Context

Changes

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

greptile-apps Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Apr 15, 2026

Choose a reason for hiding this comment

Changelog lastModified incorrectly derived from blog posts

Uh oh!

emir-karabeg commented Apr 15, 2026

Uh oh!

emir-karabeg commented Apr 15, 2026

Uh oh!

Uh oh!

Uh oh!

emir-karabeg commented Apr 15, 2026

Uh oh!

emir-karabeg commented Apr 15, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

emir-karabeg commented Apr 15, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

vercel Bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

cursor Bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps Bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

cursor Bot left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.