1095 Open source projects that are alternatives of or similar to Awesome Web Scraper

最简单的91porn爬虫php版本

A python module for configuration of block devices

A Composer Package which installs the PhantomJS binary (Linux, Windows, Mac) into /bin of your project.

Kubernetes-native S3-like files/assets store based on CRDs and powered by MinIO

Norconex HTTP Collector is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.

Turn large Web sites into tables and charts using simple SQLs.

Library to manage your firmware configurations written in C++

run client-side Mocha tests with PhantomJS

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

yspider -- 轻量级爬虫系统

Nodejs实现的一个磁力链接爬虫 http://findit.keenwon.com (原域名http://findit.so )

Random User-Agent middleware based on fake-useragent

Simply save/load json files to/from file system in electron applications

Script for downloading Coursera.org videos and naming them.

Web 端反爬技术方案

netease-music-spider is a sipder that you can find beautiful girlfriend or handsome boyfriend.

Rapidly fast and lightweight re-implementation of SharedPreferences which stores each preference in files separately, performs disk operations via NIO with memory mapped byte buffers and works IPC (between processes). Written from scratch.

This module creates an S3 bucket suitable for receiving logs from other AWS services such as S3, CloudFront, and CloudTrail

Offline plugin (ServiceWorker, AppCache) for webpack (https://webpack.js.org/)

A web crawler (for bug hunting) that gathers more than you can imagine.

🎬 基于Pyqt5的简单电影搜索工具

Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.

A simple tool for fetching usable proxies from several websites.

QQ空间导出助手，用于备份QQ空间的说说、日志、私密日记、相册、视频、留言板、QQ好友、收藏夹、分享、最近访客为文件，便于迁移与保存

Some toolkits implements part of BT Protocol, like DHT spider.

大数据生态解决方案数据平台：基于大数据、数据平台、微服务、机器学习、商城、自动化运维、DevOps、容器部署平台、数据平台采集、数据平台存储、数据平台计算、数据平台开发、数据平台应用搭建的大数据解决方案。

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

A Container Storage Interface (CSI) Driver for DigitalOcean Block Storage

使用Scrapy采集淘宝数据，Flask展示

The modularized backend boilerplate based on Spring Boot Framework, easy to get started and add your business part.

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

create some tools use go lang.

QQZone mood spider and analysis. QQ空间多线程爬虫和数据挖掘。提供线上服务，扫码登陆即可自动爬取和分析数据，还有网易云年度报告风格的数据展示；使用docker-compose打包程序，方便部署；额外提供QQ空间抽奖小程序。

A maven plugin for installing the phantomjs binary on your system automatically.

A complete IndexedDB wrapper with SQL like syntax.

爬虫, http代理, 模拟登陆!

A generic JSON document store with sharing and synchronisation capabilities.

Python多线程下载 草榴(t66y.com) 网站【新時代的我們】和【達蓋爾的旗幟】两个板块帖子内的图片

今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish

[https://www.sofineday.com], golang项目开发脚手架,集成最佳实践(gin+gorm+go-redis+mongo+cors+jwt+json日志库zap(支持日志收集到kafka或mongo)+消息队列kafka+微信支付宝支付gopay+api加密+api反向代理+go modules依赖管理+headless爬虫chromedp+makefile+二进制压缩+livereload热加载)

A multi-host clustered implementation of the open storage specification

When you solve the problem of Baekjoon Online Judge, it automatically commits and pushes to the remote repository.

**No Longer Maintained** Official RAMCloud repo

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉

豆瓣电影/豆瓣读书 Scarpy 爬虫

Testing Using Python Demo. 使用Python测试脚本demo。

All Terrain Autonomous Quadruped

基于 scrapy-redis 的通用分布式爬虫框架

RESTful based volume management framework for GlusterFS

Docs and files for ScrapydWeb, Scrapyd, Scrapy, and other projects

Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange

Last-Statement-of-Death-Row, 人之将死，其言也善

Web file manager, disk space usage, storage search engine and file system analytics for diskover

A simple filesystem abstraction package for Laravel 4.

A Router WiFi key recovery/cracking tool with a twist.

Convert HTML to PDF thru a lambda function using PhantomJS.

Async Python 3.6+ web scraping micro-framework based on asyncio

🐞 立马理财销售统计(爬虫+页面展示)

网易云爬虫解决方案

MM131网站图片爬取 🚨