crawlscope audits sitemap URLs for indexability, metadata, structured data and internal links.

It is built by Ethos Link and used in production by Reviato.

It is designed for Rails apps and plain Ruby scripts that want:

• sitemap-driven crawling
• structured validation issues instead of free-form strings
• app-configurable rule and schema registries
• first-party rake tasks instead of a large DSL
• optional browser rendering for JavaScript-heavy pages

It works in three modes:

• as a plain Ruby library
• as a standalone CLI
• as Rails rake tasks through the included Railtie

The default rule set includes:

• indexability blockers
• metadata validation
• structured-data validation
• uniqueness checks
• content-quality checks
• internal-link checks

This README and the linked repository guides describe the checked-out code. The Ethos Link project page provides the canonical public project guide, overview and resources.

• Rails integration and tasks
• Rule catalogue and schema registries
• Server timing and diagnostic headers

Crawlscope requires Ruby 4.0 or newer. Rails is optional; the CLI and Ruby library run without it. Rails integration requires host-installed Rails 8.

Add this line to your application's Gemfile:

gem "crawlscope", require: false

require: false keeps Crawlscope and its crawl stack out of normal Rails web and job process boot. Require crawlscope at the application boundary that runs an audit, or use the task setup below.

And then execute:

bundle install

Or install it directly:

gem install crawlscope

If you want browser rendering, also add:

gem "ferrum"

crawlscope only loads Ferrum when you run in browser mode. Install a compatible Chrome or Chromium executable as well.

Validate a site from its default sitemap:

crawlscope validate --url https://example.com

Validate only specific rules:

crawlscope validate --url https://example.com --rules metadata,links

Validate structured data on one or more URLs:

crawlscope ldjson --url https://example.com/article
crawlscope ldjson --url https://example.com/a --url https://example.com/b --summary

To use a non-default sitemap, pass --sitemap:

crawlscope validate --url https://example.com --sitemap https://example.com/sitemap.xml

Child sitemap indexes are supported automatically.

Set CRAWLSCOPE_PROFILE_TOKEN through the process environment or your secret manager when the site protects diagnostic timing with X-Profile-Token. Crawlscope intentionally has no --profile-token flag because command arguments can be exposed through shell history and process listings.

Validation output is grouped for terminal scanning:

Crawlscope validation
Base URL: https://example.com
Sitemap: https://example.com/sitemap.xml
URLs: 24
Pages: 24
Status: WARNINGS
Issues: 3 total (3 warnings)

Links: Too few incoming links without nofollow (2 warnings)
  - /pricing  incoming links without nofollow: 1; minimum 2
      linked from: /
  - /features  incoming links without nofollow: 1; minimum 2
      linked from: /

Metadata: Missing title (1 warning)
  - /draft  missing <title>

Reports include all issues and related URLs. Errors appear before warnings and notices. Each related URL has a label, such as linked from, links to, or page. JSON-LD errors appear below the page URL with the type, field, and constraint. Same-origin URLs use paths; other URLs retain their scheme and host. For automation in Ruby, use result.issues and each issue's code, severity, url, and details rather than matching the report text.

Metadata length limits appear in the group heading; each row shows the measured length. The default maximums are 160 characters for a meta description and 110 characters for an Article JSON-LD headline. These are separate Crawlscope validation limits for separate fields. Server timing lists pages only when the reported duration exceeds 50 ms; aggregate statistics include all responses.

require "crawlscope"

crawl = Crawlscope::Crawl.new(
  base_url: "https://example.com",
  sitemap_path: "https://example.com/sitemap.xml",
  rules: Crawlscope::RuleRegistry.default(site_name: "Example").rules,
  schema_registry: Crawlscope::SchemaRegistry.default
)

result = crawl.call

puts result.ok?
puts result.issues.to_a.map(&:message)

Crawlscope::Crawl returns a Crawlscope::Result with:

• urls: sitemap URLs selected for validation
• pages: fetched page snapshots
• issues: structured issues with code, severity, category, url, and message
• server_timing_summary: aggregate response timing data when pages publish it

result.ok? returns false when an error is present. Warnings and notices remain available through result.issues without making the result fail.

Add gem "crawlscope", require: false to the host application's Gemfile, run bundle install, then install the task/configuration integration:

bin/rails generate crawlscope:install
bin/rails crawlscope:validate

Edit the generated Crawlscope.configure block to set base_url, sitemap_path and any custom rule/schema registries. Tasks load and apply it automatically. From runtime application code, apply the generated loader and execute the crawl explicitly:

CrawlscopeConfiguration.apply
result = Crawlscope.configuration.audit.call

See Rails integration for lazy loading, all tasks, environment overrides and structured-data URL audits.

The shared configuration defaults to HTTP rendering, the async fetch executor, 10 concurrent requests and a 20-second request timeout. Browser rendering uses the threaded executor with at most four concurrent browser fetches. Configure concurrency, timeout_seconds, renderer, fetch_executor, rule_registry and schema_registry through Crawlscope.configure when using that shared configuration; direct Crawlscope::Crawl.new callers pass their own options.

Use rule and schema registries to select or extend checks. Near-duplicate scanning is limited to 250 HTML pages by default; exact duplicate checks still run for larger crawls. Validation describes observed responses and configured rules; it does not establish Google rich-result eligibility or search ranking.

Server timing covers response timings, diagnostic profile tokens and the limits of those measurements.

1. Fork it
2. Create a branch (git checkout -b feature/my-feature)
3. Commit your changes
4. Push (git push origin feature/my-feature)
5. Open a Pull Request

Please use Conventional Commits for commit messages.

MIT License, see LICENSE.txt

Use published Ethos Gemkit for shared tooling. Keep the exact Gemfile version and workflow commit pins on the same release.

bundle install
bundle exec lefthook install
bundle exec rake verify

Tests use controlled local fixtures and need no service account. Release preparation, publication, registry setup and recovery are documented in the Gemkit release guide.

Ethos Link is a software company founded by Paulo Fidalgo. This gem is part of the open-source tooling we publish alongside our business products:

• Reviato: collect private customer feedback and analyse public reviews.
• Nuntio: prepare, review and schedule social posts.
• EthosPlaces: find places, opening hours and contact details.

Explore our open-source projects.