A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

visualization html crawler scraper gui web spider frontend robotics parameters visual www data-collection batch-script batch-processing layman rpa visualprogramming code-free input-parameters

• Updated Nov 7, 2026
• JavaScript

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

mysql python java bigquery data pipeline etl postgresql s3 snowflake self-hosted data-engineering data-analysis mssql data-integration data-collection redshift elt data-pipeline change-data-capture

• Updated Nov 12, 2026
• Python

🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

crawler scraper downloader social-media rest-api openapi video-downloader web-scraping bilibili data-collection speech-to-text douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api tiktok-downloader model-context-protocol mcp-server

• Updated Nov 9, 2026
• Python

This GitHub repo is a powerhouse collection of APIs you can start using immediately to build everything from simple automations to full-scale applications. One of the most valuable API lists on GitHub—period. 💪

real-estate api open-source ecommerce automation social-media mcp artificial-intelligence web-scraping developer-tools awesome-list data-collection api-directory lead-generation apify public-apis api-list developer-resources ai-tools api-collection

• Updated Oct 20, 2026
• JavaScript

Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

mcp web-crawler web-scraping data-collection batch-processing content-extraction search-api claude llm-tools firecrawl model-context-protocol mcp-server firecrawl-ai javascript-rendering gemini-cli-extension

• Updated Nov 12, 2026
• JavaScript

The leader in Customer Data Infrastructure

data analytics snowplow data-collection data-pipeline product-analytics marketing-analytics snowplow-pipeline snowplow-events

• Updated Oct 19, 2026
• Scala

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

go kubernetes github-api bigquery aws data google sql etl azure gcp data-engineering data-analysis data-integration data-collection elt etl-framework cspm airbyte attack-surface-management

• Updated Nov 11, 2026
• Go

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

golang bigquery postgres clickhouse snowflake data-integration data-collection redshift data-connectors

• Updated Nov 12, 2026
• TypeScript

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

twitter-bot data-mining scraper twitter tweets twitter-api scraping web-scraping twitter-client data-collection social-network-analysis twitter-scraper twitter-data twitter-automation twitter-crawler twitter-scraping social-media-automation x-api x-twitter x-scraper

• Updated Nov 6, 2026
• Python

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

mcp scraping web-scraping data-extraction data-collection structured-data web-crawling browser-automation ai-agents web-data scraping-tools anti-bot-detection llm ai-integrations mcp-server modelcontextprotocol

• Updated Nov 12, 2026
• JavaScript

Twitter scraper for X: tweets, profiles, followers and following, no API key needed. Python library with multi-account pooling, proxy support and async, or a hosted version on Apify.

python graphql scraper social-media twitter tweets proxy python-library web-scraping data-collection x twitter-scraper scrape-tweets twitter-data twitter-search twitter-followers apify twitter-x tweet-scraper twitter-following

• Updated Oct 26, 2026
• Python

Concurrent Python made simple

python data multiprocessing concurrency parallel-computing data-engineering asyncio threading data-collection data-processing data-pipelines

• Updated Feb 4, 2025
• Python

IoT DC3 — Connect the Physical World to AI. An open-source Industrial IoT Runtime for Physical AI: 36 protocol drivers, edge-to-cloud data delivery, and an MCP tool gateway so AI agents act on devices in a safe, controllable, traceable loop.

java mqtt iot ai mcp spring-cloud modbus plc cloud-native data-collection iiot scada device-management opc-ua ai-agents edge-computing industrial-iot spring-ai model-context-protocol physical-ai

• Updated Nov 11, 2026
• Java

HTTP(S)/SOCKS5 rotating residential proxies - code examples & general information.

python proxy scraping proxy-server http-proxy https-proxy proxies web-scraping data-collection proxy-list url-scraper data-gathering socks5-proxy python-scraper proxy-rotator proxy-scraper proxy-integration ip-rotation web-scraping-api residential-proxies

• Updated Nov 11, 2026
• Java

Player Analytics plugin for Minecraft Server platforms - View player activity of your server with ease. 📆

visualization mysql statistics analytics webserver sqlite bungeecord-plugin spigot-plugin bukkit-plugin sponge-plugin data-collection hacktoberfest nukkit-plugin velocity-plugin fabric-mod

• Updated Nov 12, 2026
• Java

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

data etl data-engineering data-integration data-collection elt data-pipeline change-data-capture

• Updated Nov 12, 2026
• Rust

ODK Collect is an Android app for filling out forms. It's been used to collect billions of data points in challenging environments around the world. Contribute and make the world a better place! ✨📋✨

android java odk global-development data-collection global-health mhealth xforms social-impact mobile-data-collection

• Updated Nov 12, 2026
• Kotlin

Residential, datacenter, ISP & mobile proxies with 90M+ IPs in 200+ countries — plus Scraper & SERP APIs. Code examples in Python, Node.js, PHP, Go, Java, C#, Ruby and curl.

nodejs python golang php curl proxy mcp socks5 web-scraping data-collection sticky-sessions ai-agents serp-api scraper-api geo-targeting datacenter-proxies residential-proxies mobile-proxies isp-proxies rotating-proxies

• Updated Sep 6, 2026
• Java

When Augur reached a wall we made Aveloxis: Reliably collecting OSS Metrics Data. 40,000 repositories fully collected in 3 days is good, right?

github git linux open-source opensource research sustainability facade metrics data-visualization software data-collection data-modeling linux-foundation chaoss hacktoberfest2020 defined-metrics alt-metrics oss-governance

• Updated Aug 4, 2026
• Go

大数据采集,抽取平台,zdh_web是zdh系列服务的可视化管理平台，包含数据采集,调度,权限,审批流,私域营销等模块

data collection spark etl scheduler bigdata data-collection datapipeline pipline sparketl datax-web

• Updated Jun 7, 2026
• HTML