398 Open source projects that are alternatives of or similar to Pyspider

crawler framework, distributed crawler extractor

Polite, slim and concurrent web crawler.

以Node.js基于express以及爬虫实现的视频资源后端

JS逆向研究

码云仓库链接:AJay13/ECommerceCrawlers Github 仓库链接:DropsDevopsOrg/ECommerceCrawlers 项目展示平台链接:http://wechat.doonsec.com

DownZemAll! is a download manager for Windows, MacOS and Linux

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot

Python Lambda Chrome Automation (naming pending)

根据关键词抓取微博数据，再生成词云

稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代理IP的爬取，针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库，支持MongoDB 4.0 使用 Python3.7（Scored IP proxy pool ,customise proxy data crawler can be added anytime）

Dynamic meta tags in your AngularJS single page application

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

📢 Ptt 文章通知機器人！Notify Ptt Article in Realtime

Get (almost) original messages from google group archives. Your data is yours.

CoCrawler is a versatile web crawler built using modern tools and concurrency.

京东口罩库存监控爬虫(非selenium)，扫码登录、查价、加购、下单、秒杀

一些爬虫的代码

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Touhou Project random music video generator/player, crawling image and video from websites to generate MV.

本仓库收集整理爬虫相关资源，开发语言以Java为主

Go process used to crawl websites

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

data resource untuk NLP bahasa indonesia

Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Reconnaissance , A tool that every pentester/bughunter might wanna consider into their arsenal

This repository contains all the code I use in my YouTube tutorials.

基于 Selenium 的知乎关键词爬虫

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

Google search results crawler, get google search results that you need

To crawl all csgo skins from website.

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 ，点星支持，仅供学习参考，请勿用于商业用途，后果自负。

An Instagram bot developed using the Selenium Framework

Universal scrapping tool, which allows you to extract data using multiple environments

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户

一个获取知乎用户主页信息的多线程Python爬虫程序。

A Laravel wrapper for CrawlerDetect - the web crawler detection library

Python3 script to continuously download all images/webms of multiple 4chan thread simultaneously - without installation

Crawl instagram photos, posts and videos for download.

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok

MM131网站图片爬取 🚨

用 node.js 爬你自己的 leetcode 解题源码

微博超级话题爬虫，微博词频统计+情感分析+简单分类，新增肺炎超话爬取数据

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案

Multithreading download all HD photos / pictures from someone's Sina Weibo album.

字体混淆服务

Elegant Scraper and Crawler Framework for Golang

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

🕷️ A node crawler for github trending.

Be nice on the web

免登录下载微博图片 爬虫 Download Weibo Images without Logging-in

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Crawl some picture for fun