GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing www.scrapy.org — Scrapy — open source web scraping framework for Python

Maintained by Zyte with over 500 other contributors

The world’s most-used open source data extraction framework

Crawl sites, extract structured data, and ship it anywhere. Lean by design, extensible by nature — trusted by professionals extracting public web data at scale.

$uvaddscrapy

Read the docs

quotes — zsh

The tutorial spider, crawling quotes.toscrape.com

15+

years in production

500+

contributors

65.3k

GitHub stars

12.1k

forks

Extensions

A lean core. Everything else is a plugin away.

Scrapy handles the crawling. Battle-tested extensions from the community and Zyte add browser rendering, monitoring, anti-ban, and now direct access for AI agents.

Scrapy MCP Server

scrapy-mcp-official

New

MCP lets AI agents like Claude connect directly to your tools. This server gives them live access to running crawls — list jobs, check status, and execute Python inside the crawl process without leaving your agent.

$claude mcp add --scope user scrapy-mcp -- uvx --from scrapy-mcp-official scrapy-mcp

View on GitHub Read the docsBuilt by Zyte

Browser rendering

scrapy-playwright

Render JavaScript-heavy pages in a real browser while keeping the request/response workflow you already use.

Learn more

Monitoring

spidermon

Validate scraped data and get alerted on Slack, Discord, or email the moment a spider breaks.

Learn more

Anti-ban

scrapy-zyte-api

Automatic proxy rotation, browser fingerprinting, and ban avoidance through Zyte API — no infrastructure to run.

Learn more

Page objects

scrapy-poet

Separate extraction from crawling so each piece can be tested and reused on its own.

Learn more

View all extensions

How it works

Lean by design

Six small parts around one asynchronous engine, every one of them pluggable. Pick a part to see the code — and what it produces.

Step 03 · Spider

Spiders are the part you write. They receive responses, pull data out with CSS or XPath selectors, yield items, and follow links to discover more pages.

Extend it: Generic spiders like CrawlSpider and SitemapSpider cover the common crawl patterns for you.

Read the docs

spiders/quotes.py

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small::text").get(),
            }
        yield from response.follow_all(
            css="li.next a", callback=self.parse
        )

items.jsonl

{"text": "The world as we have created it is a
  process of our thinking.", "author": "Albert Einstein"}
{"text": "It is our choices, Harry, that show what we
  truly are.", "author": "J.K. Rowling"}
[engine] DEBUG: Crawled (200) <GET …/page/2/>
{"text": "This life is what you make it.",
  "author": "Marilyn Monroe"}

Try selectors before you write the spider

The interactive shell loads any page so you can test CSS and XPath live. Shell docs

$scrapy shell "https://quotes.toscrape.com"

The Engine drives all six asynchronously. See the full architecture

Workflow

From first spider to production

The same project that runs on your laptop scales to millions of pages — with tooling for every step along the way.

Build

Scaffold a project in one command — or hand a URL to an AI agent with Scrapy Agent Skills and get a tested spider back.

$scrapy startproject quotes

Build with AI

Run & debug

Try selectors in the interactive shell, then inspect live crawls straight from your agent with the Scrapy MCP Server.

$scrapy shell "https://quotes.toscrape.com"

Shell docs

Deploy & monitor

Push to Scrapy Cloud, schedule runs, and let spidermon alert you the moment your data drifts.

$shub deploy

Deploy

Actively maintained

Fifteen years of releases, and counting

Regular releases from a team of maintainers and 500+ contributors, with a deprecation policy that keeps upgrades boring.

v2.20.0Latest
November 2026
- Web-browser-like detection of response encodings, base URLs and meta refresh redirects, powered by frostwork
- Requests can be awaited to get their response
Release notes
v2.19.0September 2026
- New RemoteControl extension which allows inspecting and controlling a running crawl over HTTP, used by the Scrapy MCP server
- Experimental aiohttp-based download handler (now the default when running without a reactor)
Release notes
v2.18.0August 2026
- HttpxDownloadHandler now uses httpx2
- The Twisted-based HTTP/2 download handler is no longer experimental
Release notes

View full changelog Releases on GitHub

Trusted by developers in production

“If it wasn't for Scrapy, my freelancing career, and then the scraping business would have never taken off. The Scrapy framework, and especially its documentation, simplifies crawling and scraping for anyone with basic Python skills. I don't know, now there is this emotive bond with Scrapy that I've developed over the years.”

Nishant Choudhary

Founder of DataFlirt.com

“Scrapy: the gift that revolutionised web scraping and inspired me to pay it forward with Scrapoxy!"
"Scrapy is the central engine of all my projects, since its 1st release.”

Fabien Vauchelles

Creator of Scrapoxy

“Scrapy stands out as an exceptional tool for seamlessly managing open requests, making large-scale data collection efficient and reliable."
"With its remarkable scraping speed, Scrapy transforms complex web extraction tasks into swift, streamlined operations that save time and maximise output.”

Hyder Khan

Flipdish

“Scrapy is the cornerstone of web scraping with Python. Without it, scraping would be much harder."
"During my career, I've built thousands of scrapers, mostly with Scrapy. It made my life a lot easier.”

Pierluigi Vinciguerra

Co-Founder and CTO at Databoutique.com

“With over 6 years in web scraping, Scrapy has been my go-to framework for building fast, reliable, and scalable data pipelines."
"In my 6+ years of scraping experience, nothing has matched Scrapy’s balance of power, flexibility, and community support.”

Ganesh Satheendran

Turbolab Technologies

“When Google Search enforced JavaScript, our scraper broke. Struggling to build a working alternative with Puppeteer on a short notice, we fired up Scrapy, Splash, and whipped up a script. Lo and behold – it worked! Our Google benchmark was buzzing again.”

Adam

Proxyway

Want to get involved?

Discord Extract Summit

Support

Where to find help

Fifteen years of docs, answers, and a community that has probably hit your problem before.

Documentation

The tutorial, API reference, and topic guides — start here.

Open

Discord

Ask questions, share what you're building, and talk to maintainers.

Open

Knowledge Base

Guides, blog posts, and tutorials from the community and Zyte.

Browse

GitHub

Report bugs, browse good first issues, and contribute code.

Open

Start with one spider. Scale to millions of pages.

Free, open source, and running in production for over fifteen years.

Get started Star on GitHub

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.