🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

python crawler data automation ai mcp scraping crawling web-scraper web-scraping selectors xpath data-extraction stealth webscraping crawling-python playwright web-scraping-python ai-scraping mcp-server

• Updated Nov 11, 2026
• Python

List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

• Updated Nov 11, 2026
• JavaScript

Transform Web Content into LLM-Ready Data

crawler scraper html2markdown crawling-python llm-scraper llm-crawler crawl4ai aicrawler

• Updated Aug 17, 2026
• TypeScript

Scalable Python web scraping scripts for +40 popular domains

python crawler scraper automation spider web-crawler scraping crawling webscraper proxies web-scraping webscraping twitter-scraper datascraping python-scraper captcha-bypass antibot scraping-python crawling-python web-scraping-python

• Updated Nov 12, 2026
• Python

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

crawler tutorial crawling-python openreview

• Updated Dec 5, 2019
• Jupyter Notebook

Tracks changes to Telegram's sites, beta apps and servers

parser crawler telegram crawling crawling-python telegram-org telegram-updates

• Updated Nov 12, 2026
• Python

TLS Requests is a powerful Python library for secure HTTP requests, offering browser-like TLS client, fingerprinting, anti-bot page bypass, and high performance.

anti-bot-page anti-bot python-web-crawler python-crawler python-spider web-crawler-python web-spider python-scraper tls-client python-web-scraper cloudflare-bypass scraping-python python-web-scraping crawling-python web-scraping-python anti-bot-detection cf-clearance cloudflare-scraper web-scraping-api python-tls-client

• Updated Feb 23, 2026
• Python

BFS and DFS BASED Project.A neural network-powered web crawler that intelligently extracts, classifies, and processes web content using deep learning

web-crawler dfs-algorithm bfs-algorithm crawling-python

• Updated May 7, 2026
• Python

The script for selenium in python. Make automated testing easier! 使用json脚本驱动selenium

selenium test-automation testing-tools reptile selenium-python crawling-python crawlipt

• Updated Jun 6, 2024
• Python

A universal solution for web crawling lists. 抓取网页列表的通用解决方案

python crawler crawl reptile crawling-python crawler-python crawlist

• Updated Jun 5, 2024
• Python

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

crawler scraper user-agent scraping beautiful-soup robots-txt beautifulsoup scrapper website-scraper scrapping-python website-crawler beautifulsoup4 crawling-python iprotation

• Updated May 20, 2026
• Python

Instagram user's photos and videos downloader. Download all media files from any username. Working 2022!

python instagram crawler scraper scraping instagram-feed instagram-scraper scrap instagram-photos scraping-websites instagram-downloader scraping-tool scraping-python crawling-python

• Updated Jun 23, 2022
• Python

此项目分享安卓逆向的实战案例以及学习笔记，适合新手学习，随着作者逐渐变成大神，这个仓库也会适合大神学习~

reverse-engineering android-application crawling-python

• Updated Jul 11, 2024
• Python

Free Facebook pages MetaData Scraping Library - Unlimited Calls

python crawler facebook enrichment scraping selenium openai scrapy facebook-scraper leadgeneration facebook-scraping scraping-python crawling-python facebook-page-information facebook-apis facebook-page-scraper llm deepseek anitbots curl-cffi

• Updated Oct 12, 2026
• Python

NewsCrap adalah alat scraping berita Google berbasis Command Line Interface (CLI) yang dirancang untuk riset, investigasi, dan pengumpulan data OSINT. Dengan fitur canggih seperti rotation proxy, scheduling otomatis, dan multi-format export, alat ini memudahkan pengumpulan data berita secara efisien dan andal.

crawler scraper scraping-websites scrapping-python crawling-python osint-tool newscrap

• Updated Sep 18, 2025
• Python

Web scraping API for building AI applications.

python scraping crawling web-scraping chatbots web-crawling scraping-python crawling-python web-scraping-python

• Updated Jan 24, 2024
• Python

API to parse tibia.com content into python objects.

python python3 beautifulsoup tibia webcrawling crawling-python

• Updated Oct 19, 2026
• Python

File Crawler index files and search hard-coded credentials

elasticsearch crawler leaks crawling-python leaks-scanner

• Updated Feb 8, 2025
• Python

🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

crawler scraper web-crawler headless scraping crawling selenium web-scraper web-scraping beautifulsoup webscraping web-crawling python-crawler node-crawler crawling-framework scraping-framework scraping-tool crawling-tool scraping-python crawling-python

• Updated Sep 7, 2026
• TypeScript

项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

python api spider crawling python3 baidu spiders crawling-python baiduspider

• Updated Oct 29, 2020
• Python