1912 Open source projects that are alternatives of or similar to Gopa

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Powerful web scraping framework for Crystal

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Elegant Scraper and Crawler Framework for Golang

Continuous scalable web crawler built on top of Flink and crawler-commons

Crawly, a high-level web crawling & scraping framework for Elixir.

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

一个灵活、友好的爬虫框架

Simple but useful Python web scraping tutorial code.

Distributed crawler powered by Headless Chrome

简单易用的Python爬虫框架，QQ交流群：597510560

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

A framework for creating semi-automatic web content extractors

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

a reliable high-level web crawling & scraping framework for Node.js.

[Unmaintained] A simple and clean video/music/image downloader 👾

Free Web Scraping Tool with Java

Rust Web Crawler saving pages on Redis

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Scrapy, a fast high-level web crawling & scraping framework for Python.

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

A collection of awesome web crawler,spider in different languages

The simple, easy to use command line web crawler.

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

Declarative web scraping

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

A simple distributed crawler for zhihu && data analysis

No description or website provided.

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

dynamic crawler for web vulnerability scanner

A Laravel wrapper for CrawlerDetect - the web crawler detection library

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

京东口罩库存监控爬虫(非selenium)，扫码登录、查价、加购、下单、秒杀

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

Tutorial for web scraping / crawling with Node.js.

Web crawler.

Google search results crawler, get google search results that you need

Basic setup with random user agents and IP addresses for Python Scrapy Framework.

The complete web scraping toolkit for PHP.

👨‍👩‍👦 Social account detection and extraction in Python, e.g. for crawling/scraping.

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

A web crawler for Go

A Scrapy extension to log items coverage when the spider shuts down

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

Web Scraping Framework

A library to read a YML file with Xpath or CSS Selectors and extract data from HTML pages using them

Easily crawl news portals or blog sites using Storm Crawler.

磁力網站U3C3介紹以及域名更新

ProxyCrawl Python library for scraping and crawling