163Music
基于Scrapy框架的网易云音乐爬虫，大致爬虫流程如下：
- 以歌手页为索引页，抓取到全部歌手；
- 从全部歌手页抓取到全部专辑；
- 通过所有专辑抓取到所有歌曲；
- 最后抓取歌曲的精彩评论。
数据保存到Mongodb数据库，保存歌曲的歌手，歌名，专辑，和热评的作者，赞数，以及作者头像url。
抓取评论者的头像url，是因为如果大家喜欢，可以将他做web端。
运行：
$ scrapy crawl music
查看详情：here
yokonsan / 163Music
Licence: other
163music spider by scrapy.
Stars: ✭ 60
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to 163Music
Fp Server
Free proxy server, continuously crawling and providing proxies, based on Tornado and Scrapy. 免费代理服务器，基于Tornado和Scrapy，在本地搭建属于自己的代理池
Stars: ✭ 154 (+156.67%)
Mutual labels: spider, scrapy
Goribot
[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。
Stars: ✭ 190 (+216.67%)
Mutual labels: spider, scrapy
Scrapingoutsourcing
ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个
Stars: ✭ 164 (+173.33%)
Mutual labels: spider, scrapy
Awesome Web Scraper
A collection of awesome web scaper, crawler.
Stars: ✭ 147 (+145%)
Mutual labels: spider, scrapy
Python3 Spider
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
Stars: ✭ 2,129 (+3448.33%)
Mutual labels: spider, scrapy
Scrapydweb
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉
Stars: ✭ 2,385 (+3875%)
Mutual labels: spider, scrapy
Feapder
feapder是一款支持分布式、批次采集、任务防丢、报警丰富的python爬虫框架
Stars: ✭ 110 (+83.33%)
Mutual labels: spider, scrapy
Spiderkeeper
admin ui for scrapy/open source scrapinghub
Stars: ✭ 2,562 (+4170%)
Mutual labels: spider, scrapy
Gerapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
Stars: ✭ 2,601 (+4235%)
Mutual labels: spider, scrapy
devsearch
A web search engine built with Python which uses TF-IDF and PageRank to sort search results.
Stars: ✭ 52 (-13.33%)
Mutual labels: spider, scrapy
Web-Iota
Iota is a web scraper which can find all of the images and links/suburls on a webpage
Stars: ✭ 60 (+0%)
Mutual labels: spider, scrapy
NScrapy
NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider
Stars: ✭ 88 (+46.67%)
Mutual labels: spider, scrapy
Marmot
💐Marmot | Web Crawler/HTTP protocol Download Package 🐭
Stars: ✭ 186 (+210%)
Mutual labels: spider, scrapy
Crawlab Lite
Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (+103.33%)
Mutual labels: spider, scrapy
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].