2287 Open source projects that are alternatives of or similar to Scrapoxy

A Facebook crawler

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Crawlera middleware for Scrapy

HTTP API for Scrapy spiders

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

crawler framework, distributed crawler extractor

快速、简洁且强大的PHP爬虫框架

The Zipru scraper developed in the Advanced Web Scraping Tutorial.

Crawly, a high-level web crawling & scraping framework for Elixir.

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Abixen Platform

Distributed crawler powered by Headless Chrome

Simple but useful Python web scraping tutorial code.

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

spring-cloud-microservice-examples

swiss army knife for hackers

Declarative web scraping

use multiple proxies with Scrapy

NBoost is a scalable, search-api-boosting platform for deploying transformer models to improve the relevance of search results on different platforms (i.e. Elasticsearch)

Crawl BookCorpus

[Unmaintained] A simple and clean video/music/image downloader 👾

Life is short, *****.

Proxy-Scraper is simple Perl script for scraping proxies from multiple websites.

Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMed, Google Scholar and Sci-Hub for respective PDF files.

Scraping scripts for various websites.

Scrapy as a Web Crawler for Azure Search Samples

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

An R web crawler and scraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!

A Python Tool For google Hacking

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

OSINT Swiss Army Knife

🍭 epub generator for (light)novels (轻) 小说 epub 生成器，支持站点：轻之国度、轻小说文库

A framework for creating semi-automatic web content extractors

Crawler for Nintendo Switch eShop

A collection of awesome web crawler,spider in different languages

LinkedIn Scraper (currently working 2020)

Redis-based components for Scrapy.

基于搜狗微信搜索的微信公众号爬虫接口

我的爬虫合集

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

A high performance web crawler in Elixir.

A python web scraper for public email lists.

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Scrapy-based Crawlers for news of Taiwan

A scrapy spider that scraps times and prices from Voyages Sncf. It uses scrapyrt to provide an API interface.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

Indonesia Index News Crawler, including 10 online media

Tổng hợp script crawl dữ liệu từ các mạng xã hội & website tiếng Việt

Creating Scrapy scrapers via the Django admin interface

Some scrapy and web.py exmaples

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Python爬虫，京东自动登录，在线抢购商品

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

No description or website provided.

Simple Weibo Scraper

A multi-thread crawler framework with many builtin image crawlers provided.