487 Open source projects that are alternatives of or similar to Spidy

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Continuous scalable web crawler built on top of Flink and crawler-commons

No description or website provided.

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Simple but useful Python web scraping tutorial code.

一个灵活、友好的爬虫框架

An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL

Crawly, a high-level web crawling & scraping framework for Elixir.

A collection of awesome web crawler,spider in different languages

Easy way to brute-force web directory.

Go process used to crawl websites

[Unmaintained] A simple and clean video/music/image downloader 👾

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

HTTP API for Scrapy spiders

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

Apache Nutch is an extensible and scalable web crawler

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

简单易用的Python爬虫框架，QQ交流群：597510560

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

Rust Web Crawler saving pages on Redis

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Scrapy, a fast high-level web crawling & scraping framework for Python.

Powerful web scraping framework for Crystal

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

네이버 뉴스 수집을 위한 도구

A simple distributed crawler for zhihu && data analysis

Declarative web scraping

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

a reliable high-level web crawling & scraping framework for Node.js.

A simple but powerful web crawler library for .NET

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

An Instagram bot developed using the Selenium Framework

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

Distributed crawler powered by Headless Chrome

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Elegant Scraper and Crawler Framework for Golang

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

🤖 Prerendering for JavaScript powered websites. Great solution for PWAs (Progressive Web Apps), SPAs (Single Page Applications), and other websites based on top of front-end JavaScript frameworks

A fluent and functional approach to querying HTML

PTT 網路版爬蟲

A Python DNS crawler to find identical domain names under different TLDs.

我的爬虫合集

🐛一只登录正方教务管理系统，爬取数据的小爬虫

Practical pinned gist which show your latest medium status 📌

Google search results crawler, get google search results that you need - php

Crawl a website and take screenshots

We will try to find your visible basic footprint from social media as much as possible - 😤 more sites is comming soon

Script to fetch webcomics and use them to create ebooks.

Simple Weibo Scraper

High performance crawler for Nodejs

SSE 50 index options crawler 上证50期权数据爬虫

TumblTwo, an Improved Fork of TumblOne, a Tumblr Downloader.

Auto Crawler Ptt Beauty Image Use Python Schedule

python requests + Django+ nodejs koa+ mysql to crawl eastmoney fund and stock data,for data analysis and visualiaztion .

Az összes számlád még egy helyen :)

A simple and flexible web crawler framework for java.

练习NLP，分析淘宝评论的项目