1004 Open source projects that are alternatives of or similar to Xcrawler

OSINT Swiss Army Knife

Elegant Scraper and Crawler Framework for Golang

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Crawly, a high-level web crawling & scraping framework for Elixir.

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

A Facebook crawler

A high performance web crawler in Elixir.

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

A web crawler (for bug hunting) that gathers more than you can imagine.

Scraping scripts for various websites.

A collection of awesome web crawler,spider in different languages

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

High performance crawler for Nodejs

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等实现，高效微信爬虫，微信公众号爬虫，历史文章，文章评论，数据更新。

pip安装的天眼查爬虫API，指定的单个/多个企业工商信息一键保存为Excel/JSON格式。A Battery-included Scraper API of Tianyancha, the best Chinese business data and investigation platform.

Universal scrapping tool, which allows you to extract data using multiple environments

dynamic crawler for web vulnerability scanner

Web crawler.

crawler framework, distributed crawler extractor

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

📰 A responsive interface of Hacker News with summaries and thumbnails.

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

JavaScript + BeautifulSoup = JSSoup

A web crawler for Go

Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok

A Laravel wrapper for CrawlerDetect - the web crawler detection library

Be nice on the web

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览

Every web site provides APIs.

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

京东口罩库存监控爬虫(非selenium)，扫码登录、查价、加购、下单、秒杀

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

多线程知乎用户爬虫，基于python3

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

Google search results crawler, get google search results that you need

👾 Fast and simple video download library and CLI tool written in Go

一个基于webmagic框架二次开发的java爬虫框架实战，已实现能爬取腾讯，搜狐，今日头条（单独集成功能）等资讯内容，配合elasticsearch框架用法，实现了自动爬虫，已投入线上生产使用。

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

An R web crawler and scraper

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

robots.txt file parsing and checking for R

磁力網站U3C3介紹以及域名更新

Get Aliexpress product details in JSON

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok数据爬取工具，支持API调用，在线批量解析及下载。

An open source webapp for scraping: towards a public service for webscraping

A web spider for shodan.io without using the Developer API.

A simple and flexible web crawler framework for java.

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!

知乎模拟登录，支持提取验证码和保存 Cookies

图片爬取下载工具，极速爬取下载 站酷https://www.zcool.com.cn/, CNU 视觉 http://www.cnu.cc/ 设计师/用户 上传的 图片/照片/插画。

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

Continuous scalable web crawler built on top of Flink and crawler-commons

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.