594 Open source projects that are alternatives of or similar to Github Spider

A simple but powerful web crawler library for .NET

Netflix like full-stack application with SPA client and backend implemented in service oriented architecture

抖音爬虫，tiktok crawler，抖音数据采集接口，抖音视频去水印，百分百成功，不需要服务器，不需要代理 IP。

😩Tool For Taobao/Tmall| 儿时玩具已经过时

爬取北大法宝网http://www.pkulaw.cn/Case/

Analysis of the characteristics of different countries

Distributed crawler powered by Headless Chrome

Crawl a website and run it through Google lighthouse

一个获取知乎用户主页信息的多线程Python爬虫程序。

Some research experiments

支援 PTT 還有 PTT2 的 PTT API

直播网站数据采集

JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816

Doujinshi downloader 绅士漫画下载

Python3 script to continuously download all images/webms of multiple 4chan thread simultaneously - without installation

A collection of awesome web crawler,spider in different languages

Automatically constructing corpus for automatic speech recognition from YouTube videos

use multiple proxies with Scrapy

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

最新动态在这里【我的程序员日志】

Crawler used to crawl papers

Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner

LinkedIn Scraper (currently working 2020)

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Crawl BookCorpus

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Html网页正文提取

爬取菜鸟教程网站并转PDF__python_crawer_by_chrome

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

MM131网站图片爬取 🚨

Open-source Enterprise Grade Search Engine Software

Verify that a request is from Google crawlers using Google's DNS verification steps

豆瓣电影/豆瓣读书 Scarpy 爬虫

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

👩 美女写真套图爬虫（一）

Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 知音漫客 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 comico webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.

Docs and files for ScrapydWeb, Scrapyd, Scrapy, and other projects

微博超级话题爬虫，微博词频统计+情感分析+简单分类，新增肺炎超话爬取数据

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.

🚀电商网站爬虫合集，淘宝京东亚马逊等

This is a sina weibo spider built by scrapy [微博爬虫/持续维护]

🎵 将可下载的网易云音乐的缓存文件转换为 MP3 文件

POOPAK - TOR Hidden Service Crawler

Build and deploy a machine learning app from scratch 🚀

As you can see, a kuaishou crawler

a reliable high-level web crawling & scraping framework for Node.js.

It covers the blockade principle of most anti-climbing strategies and corresponding solutions.👽👽👽👽（涵盖了大部分的反爬策略的封锁原理以及对应的解决方案。）

台灣股票即時爬蟲。Taiwan Stock Exchange Real Time Crawler

Free proxy server, continuously crawling and providing proxies, based on Tornado and Scrapy. 免费代理服务器，基于Tornado和Scrapy，在本地搭建属于自己的代理池

由Python编写的全异步实现的动漫之家(dmzj)漫画批量下载器（爬虫）

一只优雅的正方教务系统爬虫。

Tor website crawler (specific for Alphabay at the time)

Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMed, Google Scholar and Sci-Hub for respective PDF files.

Code for the second edition Web Scraping with Python book by Packt Publications

Crawl websites for accessibility issues from the command line.

capture pictures from website like sina, lofter, huaban and so on

新闻抓取（微信、微博、头条...）

Sina stock options crawler with CSV output 新浪上证ETF期权数据爬虫