2266 Open source projects that are alternatives of or similar to Colly

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Crawly, a high-level web crawling & scraping framework for Elixir.

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

No description or website provided.

一个灵活、友好的爬虫框架

[Unmaintained] A simple and clean video/music/image downloader 👾

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Distributed crawler powered by Headless Chrome

Declarative web scraping

Scrapy, a fast high-level web crawling & scraping framework for Python.

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

Continuous scalable web crawler built on top of Flink and crawler-commons

High performance crawler for Nodejs

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

ProxyCrawl Python library for scraping and crawling

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

快速、简洁且强大的PHP爬虫框架

a reliable high-level web crawling & scraping framework for Node.js.

🐾 Creeper - The Next Generation Crawler Framework (Go)

A Facebook crawler

Simple but useful Python web scraping tutorial code.

Free Web Scraping Tool with Java

Universal scrapping tool, which allows you to extract data using multiple environments

Creating Scrapy scrapers via the Django admin interface

A high performance web crawler in Elixir.

HTTP API for Scrapy spiders

Scraping scripts for various websites.

Extract structured data from web sites. Web sites scraping.

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Powerful web scraping framework for Crystal

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

A web crawler (for bug hunting) that gathers more than you can imagine.

A collection of awesome web crawler,spider in different languages

OSINT Swiss Army Knife

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

A scalable web crawler framework for Java.

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

APIs for loginning some websites by using requests.

获取免费socks/https/http代理的网站集合

简单易用的Python爬虫框架，QQ交流群：597510560

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

A curated list of awesome puppeteer resources.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Digger is a powerful and flexible web crawler implemented by pure golang

微博超级话题爬虫，微博词频统计+情感分析+简单分类，新增肺炎超话爬取数据

MM131网站图片爬取 🚨

B站用户爬虫 好耶~是爬虫

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

Your will to enroll in Udemy course is here, but the money isn't? Search no more! This python program searches for your desired course in more than [insert big number here] websites, compares the last updated date, and gives you the download link of the latest one back, but you also have the choice to see the other ones as well!