791 Open source projects that are alternatives of or similar to ant

A collection of awesome web crawler,spider in different languages

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok数据爬取工具，支持API调用，在线批量解析及下载。

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |

A collection of awesome web scaper, crawler.

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

一个基于webmagic框架二次开发的java爬虫框架实战，已实现能爬取腾讯，搜狐，今日头条（单独集成功能）等资讯内容，配合elasticsearch框架用法，实现了自动爬虫，已投入线上生产使用。

快速、简洁且强大的PHP爬虫框架

A Facebook crawler

OSINT Swiss Army Knife

A web spider for shodan.io without using the Developer API.

A python web scraper for public email lists.

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.

简单易用的Python爬虫框架，QQ交流群：597510560

图片爬取下载工具，极速爬取下载 站酷https://www.zcool.com.cn/, CNU 视觉 http://www.cnu.cc/ 设计师/用户 上传的 图片/照片/插画。

robots.txt file parsing and checking for R

Interactive CLI Web Crawler

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

Crawly, a high-level web crawling & scraping framework for Elixir.

An open source webapp for scraping: towards a public service for webscraping

Creating Scrapy scrapers via the Django admin interface

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Get Aliexpress product details in JSON

High performance crawler for Nodejs

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

A web crawler (for bug hunting) that gathers more than you can imagine.

Scraping scripts for various websites.

Continuous scalable web crawler built on top of Flink and crawler-commons

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Rust Web Crawler saving pages on Redis

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Grabs all of the audio files from all of the Blinkist books

A simple distributed crawler for zhihu && data analysis

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

A high performance web crawler in Elixir.

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Elegant Scraper and Crawler Framework for Golang

Un bot de Twitter que comparte cada hora un TFM hasta que Cristina Cifuentes enseñe el suyo.

Parallel Searching and Crawling Data from PTT 🚀

基于Electron爬虫程序

Araignée LeBonCoin avec Scrapy et ElasticSearch

蜘蛛纸牌 for mac

🌎 🏨 Scrape Booking.com 🏨 🌎

**[ARCHIVED]** website changes tracker 🔍

This taobao spider has been archived

Simple, easy-to-use scraper to scrape data from WordPress JSON API

Grapy, a fast high-level web crawling framework for Python 3.3 or later base on asyncio.

Twitter High level scraper for humans.

该项目是一个使用celery作为主体框架的爬虫应用，能够灵活的添加爬虫任务，并且同时运行多站点的爬虫工作，所有组件都能够原生支持规模并发和分布式，加上celery原生的分布式调用，实现大规模并发。

A spider on Dcard. Strong and speedy.

A spiders' program for weixin which made by Express & cheerio

⚡ Light weight Golang spider framework | 轻量的 Golang 爬虫框架