Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

markdown crawler scraper ai html-to-markdown web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction ai-agents web-search ai-search web-data llm ai-crawler ai-scraping

• Updated Nov 12, 2026
• TypeScript

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

python crawler data automation ai mcp scraping crawling web-scraper web-scraping selectors xpath data-extraction stealth webscraping crawling-python playwright web-scraping-python ai-scraping mcp-server

• Updated Nov 11, 2026
• Python

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling web-scraping hacktoberfest web-scraping-python

• Updated Nov 10, 2026
• Python

🕵️‍♂️ Collect a dossier on a person by username from 6K websites

python cli open-source osint social-network scraping sherlock python3 cybersecurity identification infosec pentesting username information-gathering investigation reconnaissance redteam osint-framework socmint osint-python

• Updated Nov 12, 2026
• Python

Python scraper based on AI

markdown crawler web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction web-search ai-search rag web-data scraping-python web-crawlers llm ai-crawler large-language-model ai-scraping firecrawl-alternative

• Updated Oct 6, 2026
• Python

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright

• Updated Nov 12, 2026
• TypeScript

Elegant Scraper and Crawler Framework for Golang

go golang crawler scraper framework spider scraping crawling

• Updated Oct 16, 2026
• Go

Pythonic HTML Parsing for Humans™

python html http scraping requests kennethreitz beautifulsoup lxml css-selectors pyquery

• Updated Apr 16, 2024
• Python

🦊 Anti-detect browser

firefox networking scraping fingerprint webscraping playwright antidetect antidetect-browser

• Updated Nov 12, 2026
• C

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

testing chrome automation webdriver browser captcha scraping selenium navigator python3 cloudflare chromedriver anti-bot bot-detection cloudflare-bypass distil anti-detection

• Updated Jul 5, 2025
• Python

Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

nodejs javascript automation scraping web-scraping anti-bot browser-automation bot-detection puppeteer headless-browser cloudflare-bypass playwright ai-agent antidetect-browser stealth-browser

• Updated Nov 10, 2026
• JavaScript

A scalable web crawler framework for Java.

java crawler framework scraping

• Updated Dec 20, 2025
• Java

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

python crawler scraper automation web-crawler headless scraping crawling selenium pip web-scraping beautifulsoup web-crawling headless-chrome apify parsel playwright

• Updated Nov 11, 2026
• Python

List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

• Updated Nov 11, 2026
• JavaScript

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation

• Updated Jul 29, 2026
• Python

Tabula is a tool for liberating data tables trapped inside PDF files

pdf csv excel scraping tables

• Updated Mar 14, 2025
• CSS

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

automation headless scraping chromium web-scraping testing-tools fingerprinting stealth turnstile cdp puppeteer recaptcha-v3 anti-detection playwright clouflare turnstile-solver stealth-browser stealth-automation stealth-injector

• Updated Nov 8, 2026
• HTML

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

nlp crawler text-mining html-to-markdown scraping news-aggregator text-extraction web-scraping rss-feed readability tei html2text news-crawler corpus-builder corpus-tools article-extractor text-cleaning rag text-preprocessing llm

• Updated Nov 11, 2026
• Python

🚀 Free HTTP, SOCKS4, & SOCKS5 proxy list * Updated every 5 minutes * and rotating proxy API (100+ countries)

nodejs javascript scraping https-proxy socks5 proxies proxy-list proxy-api free-proxy

• Updated Nov 12, 2026

Distributed crawler powered by Headless Chrome

jquery crawler chrome scraper promise scraping crawling chromium headless-chrome puppeteer

• Updated Apr 29, 2023
• JavaScript