563 Open source projects that are alternatives of or similar to small-spider-project

Scraping the real estate agencies for up-to-date house listings as soon as they arrive!

零基础学习python爬虫

A web crawler for Go

文章采集工具 Article collection tool

一个简单的基于DHT协议的BT磁力链接爬虫

爬虫集合

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Google search results crawler, get google search results that you need

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

bet365 bot: bet365的比赛实时比分数据、实时赔率

Inventus is a spider designed to find subdomains of a specific domain by crawling it and any subdomains it discovers.

My code for scraping financial data in Vietnam

🔞 JAVClub - 让你的大姐姐不再走丢

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案

基于Electron爬虫程序

A Penetration Testing Framework, Information gathering tool & Website Vulnerability Scanner

scrapy mysql pipeline

Araignée LeBonCoin avec Scrapy et ElasticSearch

微信公众号爬虫

Crawling the data from lazada, websosanh, compare.vn, cdiscount and cungmua with flexible configs

A Laravel wrapper for CrawlerDetect - the web crawler detection library

dynamic crawler for web vulnerability scanner

Generate an object for testing if a request is sent, request is Mikeal's request.

这是一个用Python写的小说爬虫软件

Scrapy downloader middleware that stores response HTMLs to disk.

This taobao spider has been archived

sync playlist between music platform

京东口罩库存监控爬虫(非selenium)，扫码登录、查价、加购、下单、秒杀

python爬虫项目集合

Web crawler.

LSpider 一个为被动扫描器定制的前端爬虫

蜘蛛纸牌 for mac

使用asyncio和aiohttp开发的轻量级异步协程web爬虫框架

Bilibili.com视频批量下载工具包

Grapy, a fast high-level web crawling framework for Python 3.3 or later base on asyncio.

BitTorrent DHT Protocol && DHT Spider.

A Scrapy extension to log items coverage when the spider shuts down

该项目是一个使用celery作为主体框架的爬虫应用，能够灵活的添加爬虫任务，并且同时运行多站点的爬虫工作，所有组件都能够原生支持规模并发和分布式，加上celery原生的分布式调用，实现大规模并发。

A spiders' program for weixin which made by Express & cheerio

小说精品屋是一个多平台（web、安卓app、微信小程序）、功能完善的屏幕自适应小说漫画连载系统，包含精品小说专区、轻小说专区和漫画专区。包括小说/漫画分类、小说/漫画搜索、小说/漫画排行、完本小说/漫画、小说/漫画评分、小说/漫画在线阅读、小说/漫画书架、小说/漫画阅读记录、小说下载、小说弹幕、小说/漫画自动采集/更新/纠错、小说内容自动分享到微博、邮件自动推广、链接自动推送到百度搜索引擎等功能。

一个功能全面的微信读书爬虫 wereader

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

Elegant Scraper and Crawler Framework for Golang

JavaScript + BeautifulSoup = JSSoup

Classificador de poemas do Fernando Pessoa de acordo com os seus heterônimos

⚡ Light weight Golang spider framework | 轻量的 Golang 爬虫框架

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

多线程知乎用户爬虫，基于python3

基于nodejs的网络聊天室、爬虫，vue音乐播放器，及php后台开发的管理系统等项目

A fast and powerfull dashboard (admin) finder

bing、google、baidu搜索引擎爬虫。python3.6 and scrapy

python 爬虫(amazon, confluence ...)

简单、免费、高效的百度地图poi采集和分析工具。

Your web font utility belt. It can subset web fonts. It can find unicode-ranges for you automatically. It makes julienne fries.

portia-dashboard is a visual web crawler based on scrapinghub/portia

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代理IP的爬取，针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库，支持MongoDB 4.0 使用 Python3.7（Scored IP proxy pool ,customise proxy data crawler can be added anytime）

Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data

Crawling GitHub Trending Pages every day

book.zi5.me全站kindle电子书籍爬取，按照作者书籍名分类，每本书有mobi和equb两种格式，采用分布式进行全站爬取