1011 Open source projects that are alternatives of or similar to Scrapyrt

A Facebook crawler

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

Crawly, a high-level web crawling & scraping framework for Elixir.

[Unmaintained] A simple and clean video/music/image downloader 👾

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Distributed crawler powered by Headless Chrome

Declarative web scraping

Simple but useful Python web scraping tutorial code.

No description or website provided.

Elegant Scraper and Crawler Framework for Golang

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

crawler framework, distributed crawler extractor

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

基于asyncio与aiohttp的异步协程爬虫框架 欢迎Star

基于搜狗微信搜索的微信公众号爬虫接口

ProxyCrawl Python library for scraping and crawling

An open source webapp for scraping: towards a public service for webscraping

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL

Scrapy middleware to handle javascript pages using selenium

Crawler for Nintendo Switch eShop

A crawler based on Phantom. Allows discovery of dynamic content and supports custom scrapers.

Continuous scalable web crawler built on top of Flink and crawler-commons

Extract structured data from web sites. Web sites scraping.

LinkedIn Scraper (currently working 2020)

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Araignée LeBonCoin avec Scrapy et ElasticSearch

A Scrapy extension to log items coverage when the spider shuts down

Be nice on the web

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

A collection of awesome web crawler,spider in different languages

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

A framework for creating semi-automatic web content extractors

High performance crawler for Nodejs

码云仓库链接:AJay13/ECommerceCrawlers Github 仓库链接:DropsDevopsOrg/ECommerceCrawlers 项目展示平台链接:http://wechat.doonsec.com

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

Simple Weibo Scraper

Redis-based components for Scrapy.

Crawl BookCorpus

🍭 epub generator for (light)novels (轻) 小说 epub 生成器，支持站点：轻之国度、轻小说文库

The simple, easy to use command line web crawler.

Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Az összes számlád még egy helyen :)

我的爬虫合集

An R web crawler and scraper

Crawlera middleware for Scrapy

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

快速、简洁且强大的PHP爬虫框架

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.

swiss army knife for hackers