List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

• Updated Nov 11, 2026
• JavaScript

Async Python 3.6+ web scraping micro-framework based on asyncio

python crawler spider aiohttp middlewares asyncio uvloop crawling-framework asyncio-spider ruia python-ruia

• Updated Oct 30, 2026
• Python

Web Scan Lazy Tools - Python Package

security package security-audit spider web-vulnerability-scanner security-scanner python-package crawling-framework security-automation security-tools spider-framework scanner-web security-tool

• Updated Jun 30, 2020
• Python

SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.

crawler crawling crawl crawling-framework crawler-engine crawling-sites crawling-tool

• Updated Mar 18, 2025
• Python

Crawlzone is a fast asynchronous internet crawling framework for PHP.

php middleware crawler web-scraping automated-testing crawling-framework web-search

• Updated Apr 19, 2023
• PHP

Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

instagram crawler scraper youtube requests crawlers scraping-websites crawling-framework selenium-python

• Updated Oct 25, 2022
• Python

🌌 High productivity semi-automatic crawler generator 🛠️🧰

crawler crawling crawling-framework wpf-application custom-crawler

• Updated Oct 5, 2023
• C#

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

python crawler scraper vue scraping crawling python3 scrapers scraper-engine crawlers crawling-framework website-crawler scraping-framework crawler-python scraper-api crawling-engine

• Updated Aug 19, 2023
• Python

Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.

java crawler information-retrieval data-mining scraper automation framework scraping crawling selenium information-extraction crawl crawlers extract-data dynamic-webpages webspider crawling-framework selenium-crawler scraping-framework dynamic-website

• Updated Jul 7, 2022
• Java

🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

crawler scraper web-crawler headless scraping crawling selenium web-scraper web-scraping beautifulsoup webscraping web-crawling python-crawler node-crawler crawling-framework scraping-framework scraping-tool crawling-tool scraping-python crawling-python

• Updated Sep 7, 2026
• TypeScript

A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend

python crawler machine-learning database network scraping mariadb python-3 crawling-framework

• Updated Nov 1, 2026
• Python

Easily crawl news portals or blog sites using Storm Crawler.

java elasticsearch crawler storm scraping crawling vaadin crawling-framework storm-crawler

• Updated Nov 15, 2022
• Java

BFS and DFS BASED Project.A neural network-powered web crawler that intelligently extracts, classifies, and processes web content using deep learning

web-crawling crawling-framework crawling-python

• Updated Apr 1, 2026
• Python

Web crawling & scraping framework for Node.js on top of headless Chrome browser

scraper spider web-crawler headless scraping crawling web-scraping scrapy scrape scraping-websites web-crawling scrapy-crawler crawling-framework crawer spider-framework crawling-sites crawling-tool web-scraping-nodejs

• Updated Mar 15, 2026
• TypeScript

🔍 A powerful web-crawling framework, based on aiohttp.

crawler spider asyncio crawling-framework

• Updated Nov 29, 2019
• Python

An intelligent proxy server. Provide durable, real-time, high-quality proxies as a middleman or datasource server.

crawler spider proxy mitm socks5 socks-proxy mitmproxy socks4 crawling-framework spider-framework freeproxy spiderman proxy-scraper

• Updated Oct 11, 2023
• Go

Crawler written in TypeScript using ES6 generators.

crawler typescript web-crawler crawling crawl web-crawling es6-generators crawling-framework

• Updated May 3, 2021
• TypeScript

基于python协程池、用法灵活的高性能爬虫框架

python crawler asyncio crawling-framework coroutine-pool

• Updated Dec 27, 2020
• Python

A crawler program to extract all of the data and the price for symbols in the global stock exchange.

crawler crawling stock stock-market stock-price-prediction crawlers stock-data stock-prices stock-exchanges crawl-pages stock-exchange-simulator stock-prediction crawling-framework crawlspider stock-analysis stock-trading stock-exchange crawling-sites stock-exchange-platform stock-exchange-crawler

• Updated Jul 2, 2019
• PHP

Useful functions for connecting to the network in the PHP based applications.

php curl network curl-library crawl-pages curl-commands crawling-framework php-cookie php-httpstatuscode php-request curl-wrapper php-http crawler-engine crawling-sites curl-php php-network php-https curl-cookie curl-request curl-header

• Updated May 26, 2020
• PHP