227 Open source projects that are alternatives of or similar to Cnkispider

Scrapy-based Crawlers for news of Taiwan

基于 scrapy-redis 的通用分布式爬虫框架

慕课网python分布式爬虫源码-长期更新维护

The Zipru scraper developed in the Advanced Web Scraping Tutorial.

爬虫, http代理, 模拟登陆!

Build and deploy a machine learning app from scratch 🚀

Simple Web UI for Scrapy spider management via Scrapyd

swiss army knife for hackers

OLX Scraper in Python Scrapy

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816

A set of spiders and scrapers to extract location information from places that post their location on the internet.

Unmaintained 🐳 ☕️ 🕷 Scala crawler(spider) framework, inspired by scrapy, created by @gaocegege

百度贴吧爬虫(基于scrapy和mysql)

A scrapy spider that scraps times and prices from Voyages Sncf. It uses scrapyrt to provide an API interface.

Uno Crawler por https://douban.com

Some scrapy and web.py exmaples

python-Reptile

A python web scraper for public email lists.

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

土巴兔和谷居装修网站爬虫

今日头条科技新闻接口爬虫

[OUTDATED] scrapy spiders to crawl the financial text data 📚 📜 pertinent to train word vectors 🚀

API for scrapping common meme sites

AlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)

PTT 網路版爬蟲

Life is short, *****.

知名網站avgle下載器

scrapy专利爬虫（停止维护）

爬虫管理平台

Possibly the best practice of Scrapy 🕷 and renting a house 🏡

🕷️some python spiders with BeautifulSoup or scarpy

Parser and database to index the terpene profile of different strains of Cannabis from online databases

小密圈备份，圈子/话题/图片/文件。

TweetScraper is a simple crawler/spider for Twitter Search without using API

No description or website provided.

Some research experiments

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

Faster requests on Python 3

V2EX爬虫

Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.

Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).

A multi-thread crawler framework with many builtin image crawlers provided.

Use Scrapy crawl foreign exchange rate from BOC (Bank of China)

Code for the second edition Web Scraping with Python book by Packt Publications

基于Scrapy+Elasticsearch+Django搭建的分布式电影搜索

Python入门网络爬虫之精华版

收集各种免费的 Python 爬虫项目

🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish

Personal Data Engineering Projects

基于搜狗微信搜索的微信公众号爬虫接口

Scraping Ebay's products using Scrapy Web Crawling Framework

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

Scrapy middleware to handle javascript pages using selenium

Backend para coleta e disponibilização dos dados 📜

微博爬虫：每天定时爬取微博热搜榜的内容，留下互联网人的记忆。

lots of spider (很多爬虫）

Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner

Scrapy框架下的pixiv多功能爬虫

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis