594 Open source projects that are alternatives of or similar to Github Spider

Crawler for LinkedIn full profiles 2019

Dynamic meta tags in your AngularJS single page application

GUI based offensive penetration testing tool (Open Source)

scrapy框架爬取51job(scrapy.Spider)，智联招聘(扒接口)，拉勾网(CrawlSpider)

네이버 뉴스 수집을 위한 도구

中国大陆大学列表爬虫

Alternative for VK Audio API

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

基于 Selenium 的知乎关键词爬虫

😱Full Automatic Amazon Distributed Spider | 亚马逊分布式四国际站采集选款产品|账号admin,密码adminadmin

一款分布式爬虫平台，帮助你更好的管理和开发爬虫。 内置一套爬虫定义规则（模版），可使用模版快速定义爬虫，也可当作框架手动开发爬虫。(兴趣使然的项目，用的不爽了就更新)

An Open Source Search Engine

用 node.js 爬你自己的 leetcode 解题源码

pigat ( Passive Intelligence Gathering Aggregation Tool ) 被动信息收集聚合工具

Netflix like full-stack application with SPA client and backend implemented in service oriented architecture

一个获取知乎用户主页信息的多线程Python爬虫程序。

直播网站数据采集

Python3 script to continuously download all images/webms of multiple 4chan thread simultaneously - without installation

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

News, full-text, and article metadata extraction in Python 3. Advanced docs:

MM131网站图片爬取 🚨

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

微博超级话题爬虫，微博词频统计+情感分析+简单分类，新增肺炎超话爬取数据

This is a sina weibo spider built by scrapy [微博爬虫/持续维护]

As you can see, a kuaishou crawler

Free proxy server, continuously crawling and providing proxies, based on Tornado and Scrapy. 免费代理服务器，基于Tornado和Scrapy，在本地搭建属于自己的代理池

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Multithreading download all HD photos / pictures from someone's Sina Weibo album.

字体混淆服务

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

A lite distributed Java spider framework :-)

Download Videos Skill Share per ID or per Class

借助Python抓取微博数据，并对抓取的数据进行情绪分析

📢 Ptt 文章通知機器人！Notify Ptt Article in Realtime

🏃 Some of the python tutorial - 《Python学习笔记》

简单易用的Python爬虫框架，QQ交流群：597510560

Scrapy crawler to collect data on the back catalog of songs listed for sale.

（已跑路）百度贴吧吧务管理工具，自动扫描帖子并处理违规帖

A php crawler that finds emails on the internets

A simple distributed crawler for zhihu && data analysis

🕷️ A node crawler for github trending.

2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infection Crawler and API

可视化爬虫自动采集平台

获取免费socks/https/http代理的网站集合

基于微博的数据挖掘与社交舆情分析

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器，分布式爬虫

Python爬虫代理IP池(proxy pool)

CoCrawler is a versatile web crawler built using modern tools and concurrency.

A Moodle downloader that downloads course content fast from Moodle (eg. lecture pdfs)

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites

APIs for loginning some websites by using requests.

用爬虫爬取小说网站上所有小说，存储到数据库中，并用爬到的数据构建自己的小说网站

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户

Web crawling framework based on asyncio.

A collection of awesome web scaper, crawler.

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Sample of using proxies to crawl baidu search results.