Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

markdown crawler scraper ai html-to-markdown web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction ai-agents web-search ai-search web-data llm ai-crawler ai-scraping

• Updated Nov 12, 2026
• TypeScript

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

python markdown open-source crawler scraper ai mcp web-crawler web-scraping data-extraction web-crawling ai-agents rag playwright llm crawl4ai-cloud

• Updated Nov 10, 2026
• Python

Create agents that monitor and act on your behalf. Your agents are standing by!

notifications agent rss scraper automation monitoring huginn feed feedgenerator webscraping

• Updated Nov 9, 2026
• Ruby

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

visualization html crawler scraper gui web spider frontend robotics parameters visual www data-collection batch-script batch-processing layman rpa visualprogramming code-free input-parameters

• Updated Nov 7, 2026
• JavaScript

👾 Fast and simple video download library and CLI tool written in Go

go golang crawler scraper downloader youtube video download tumblr bilibili qq youku iqiyi

• Updated Mar 29, 2026
• Go

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

html jquery parser scraper dom cheerio selector hacktoberfest htmlparser2 htmlparser

• Updated Nov 12, 2026
• TypeScript

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright

• Updated Nov 12, 2026
• TypeScript

Elegant Scraper and Crawler Framework for Golang

go golang crawler scraper framework spider scraping crawling

• Updated Oct 16, 2026
• Go

🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

crawler scraper downloader social-media rest-api openapi video-downloader web-scraping bilibili data-collection speech-to-text douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api tiktok-downloader model-context-protocol mcp-server

• Updated Nov 9, 2026
• Python

Turn any website into a structured API. Extract, automate, search and monitor the web.

api crawler scraper automation crawling web-scraper self-hosted web-scraping data-extraction webscraping agents browser-automation no-code web-search rpa robotic-process-automation nocode playwright

• Updated Nov 9, 2026
• TypeScript

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

python crawler scraper news crawling news-aggregator

• Updated Oct 26, 2026
• Python

📙 中华新华字典数据库。包括歇后语，成语，词语，汉字。

json data scraper json-data python3 chinese chinese-nlp chinese-characters chinese-simplified chinese-traditional json-dataset chinese-language

• Updated Dec 26, 2023
• Python

Automated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.

scraper web-scraping turnstile browser-automation perimeterx ai-agents aws-waf anti-captcha arkose cloudflare-turnstile nopecha turnstile-solver perimeterx-solver recaptcha-bypass-scraping captcha-solver-for-bots

• Updated Oct 2, 2026

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video

• Updated Jun 1, 2024
• PHP

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

python crawler scraper automation web-crawler headless scraping crawling selenium pip web-scraping beautifulsoup web-crawling headless-chrome apify parsel playwright

• Updated Nov 12, 2026
• Python

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation

• Updated Jul 29, 2026
• Python

A collection of awesome web crawler,spider in different languages

crawler scraper awesome spider web-crawler web-scraper node-crawler

• Updated Jun 16, 2024

A Chrome DevTools Protocol driver for web automation and scraping.

testing go golang scraper automation web chrome-devtools headless devtools crawling web-scraping cdp chrome-headless rod chrome-devtools-protocol devtools-protocol gorod

• Updated Aug 11, 2026
• Go

Turn any webpage into structured data using LLMs

scraper browser ai artificial-intelligence openai llama gpt browser-automation puppeteer playwright gpt-4 llm langchain

• Updated Oct 26, 2026
• TypeScript

💡 Download the complete source code of any website (including all assets). [ Javascripts, Stylesheets, Images ] using Node.js

scraper downloader assets offline-web-pages

• Updated Aug 12, 2026
• HTML