1095 Open source projects that are alternatives of or similar to Awesome Web Scraper

Scrapy-based Crawlers for news of Taiwan

A Minimalistic Wrapper for IndexedDB

Kubernetes Storage Device Management

SODA API is an open source implementation of SODA API Standards for Data and Storage Management.

Commandline tool for automated downloads of echo360 videos hosted by university

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

B站用户爬虫 好耶~是爬虫

Open source library to access USB Mass Storage devices on Android without rooting your device

The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

Android's Native key value storage system in React Native

📎 Flexible file upload and attachment library for Elixir

Possibly the best practice of Scrapy 🕷 and renting a house 🏡

Run async tasks that retry after a crash, connection loss or exception

Swifty and modern UserDefaults

Golang爬虫 爬取豆瓣电影Top250

哔哩哔哩的API调用模块

Web Crawler

High Performance, Kubernetes Native Object Storage

Store your data in others DNS revolvers cache

PHP Library for detecting CMS

Persistence layer for redux with flexible backends

微博爬虫：每天定时爬取微博热搜榜的内容，留下互联网人的记忆。

👽 OneBlog，一个简洁美观、功能强大并且自适应的Java博客

OLX Scraper in Python Scrapy

🤖 微信订阅机器人 | 🐟 微信订阅机器人之闲鱼二手商品监控

Storage Orchestration for Kubernetes

🐴 The unreliable key-value store

Hexa: The ultimate companion for Azure. Setup and deploy in seconds

Fabric for Deep Learning (FfDL, pronounced fiddle) is a Deep Learning Platform offering TensorFlow, Caffe, PyTorch etc. as a Service on Kubernetes

A Composer Package which installs the PhantomJS binary (Linux, Windows, Mac) into /bin of your project.

Kubernetes-native S3-like files/assets store based on CRDs and powered by MinIO

Norconex HTTP Collector is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.

Turn large Web sites into tables and charts using simple SQLs.

Library to manage your firmware configurations written in C++

HTTP API for Scrapy spiders

Some classic web crawler projects.一些经典的爬虫

A better way to use localStorage and sessionStorage

Zero-details, privacy-focused in-app file system.

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。

最新动态在这里【我的程序员日志】

media(audio/image/qrcode/markdown/html/svg) support web service (多媒体编辑服务, 酷炫二维码, 音频, 图片, svg, markdown, html渲染服务支持)

Some scrapy and web.py exmaples

一个浏览器端数据爬虫，做每个人的数据助手

A Burp Suite Extension that try to find all sub-domain, similar-domain and related-domain of an organization automatically! 基于流量自动收集整个企业或组织的子域名、相似域名、相关域名的burp插件

python crawler spider

一个超级轻量的百度图片爬虫

又一个 java 内容（pa）获取（chong）工具

Powerful Multi-User File Manager

Storage-agnostic streaming library for binary data transfers

API of DouYin for Humans used to Crawl Popular Videos and Musics

Web Scraper in Go, similar to BeautifulSoup

Some toolkits implements part of BT Protocol, like DHT spider.

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

create some tools use go lang.

Last-Statement-of-Death-Row, 人之将死，其言也善

Web file manager, disk space usage, storage search engine and file system analytics for diskover

A simple filesystem abstraction package for Laravel 4.

A Router WiFi key recovery/cracking tool with a twist.