weibo_spider
Description
Reading userid list(not nickname) from Weibo_user table in MySQL,then crawl these user's Weibo message and save messages to database(MySQL).
main.py: start py.
MysqlUtil.py: connect to MySQL and execute CRUD operations.
WeiboProducer.py: read userid list from MySQL and put userids in to the queue
WeiboConsumer.py: read userid from the queue and crawl Weibo message.
weibo_rss.sql: database sql,include table structure.
Environment
Python: 2.7.*
System: Ubuntu
MySQL: 5.5
Usage
To run main.py normaly, you need do these:
- you need to login weibo.cn(Mobile page) to get login cookie.
- copy cookies, set to variable: cookie in WeiboConsumer.py line 25.
- install MySQL and create database,tables.
- set start parameters: -t (Weiboconsumer thread numbers)(Optional)
Example:
python main.py -t 3
How to get cookie:
- open weibo.cn in Firefox or Chrome.
- open developer tools -> NetWork， find weibo.cn login request header.
- copy cookie in request header to program.
