czy1999 / Weibo Topic Spider
Licence: mit
微博超级话题爬虫，微博词频统计+情感分析+简单分类，新增肺炎超话爬取数据
Stars: ✭ 128
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Weibo Topic Spider
Hacker News Digest
📰 A responsive interface of Hacker News with summaries and thumbnails.
Stars: ✭ 278 (+117.19%)
Mutual labels: crawler, spider, topic
Decryptlogin
APIs for loginning some websites by using requests.
Stars: ✭ 1,861 (+1353.91%)
Mutual labels: crawler, spider, weibo
Sina Weibo Album Downloader
Multithreading download all HD photos / pictures from someone's Sina Weibo album.
Stars: ✭ 125 (-2.34%)
Mutual labels: crawler, weibo
Ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
Stars: ✭ 1,366 (+967.19%)
Mutual labels: crawler, spider
Baiduspider
BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。
Stars: ✭ 105 (-17.97%)
Mutual labels: crawler, spider
Crawler Detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Stars: ✭ 1,549 (+1110.16%)
Mutual labels: crawler, spider
Crawlab Lite
Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (-4.69%)
Mutual labels: crawler, spider
Gopa Abandoned
GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）
Stars: ✭ 98 (-23.44%)
Mutual labels: crawler, spider
Skycaiji
蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统
Stars: ✭ 1,514 (+1082.81%)
Mutual labels: crawler, spider
Geziyor
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Stars: ✭ 1,246 (+873.44%)
Mutual labels: crawler, spider
Not Your Average Web Crawler
A web crawler (for bug hunting) that gathers more than you can imagine.
Stars: ✭ 107 (-16.41%)
Mutual labels: crawler, spider
Examples Of Web Crawlers
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
Stars: ✭ 10,724 (+8278.13%)
Mutual labels: crawler, spider
Crawler examples
Some classic web crawler projects.一些经典的爬虫
Stars: ✭ 74 (-42.19%)
Mutual labels: crawler, spider
weibo-topic-spyder
微博超级话题爬虫，微博词频统计+情感分析+简单分类
新增微博普通话题爬取，其中讨论和阅读数量的爬取待完善
爬取数据展示
使用方法
爬虫主文件：
微博普通话题：normal-topic-spyder.py
微博超级话题：super-topic-spyder.py
在爬虫主文件中的主函数中输入账号、密码和想要爬取的超话名称即可开始爬取，需要提前安装所需的python库和chromedriver驱动
爬取结束后数据会自动保存在当前目录下的excel文件中，每行为一个微博数据。
提示：普通的话题爬取需要添加#，例如#话题#，超级话题无需添加；一般而言，在微博中超级话题前带有钻石标志，普通话题为#话题#的格式
超级话题爬虫
使用了selenium模拟浏览器登陆进行爬取，具体话题爬取数量受微博限制，目前单个话题最大获取微博数量为8000条左右，选择了使用手机网页模式爬取，以获得最佳的爬取效果。
账号与IP数量对单个超话的爬取帮助不大，就只设置了单账号和ip模式，若需多超话同时爬取可以自行添加。
如需爬取多个超话，可以选择使用cookie登陆，最为方便
词频统计
使用了jieba库进行分词，最后对分词结果进行简单统计并且存储到txt中
情感分析
调用了百度大脑的api接口，可以自行注册获取key，平台不限调用次数，详细接口见百度大脑
其他
欢迎大家参与和完善：如有其他问题，欢迎提交issue
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].