wangy8961 / Python3 Concurrency Pics 02
爬取 www.mzitu.com 全站图片，截至目前共5162个图集，16.5万多张美女图片，使用 asyncio 和 aiohttp 实现的异步版本只需要不到2小时就能爬取完成。按日期创建图集目录，保存更合理。控制台只显示下载的进度条，详细信息保存在日志文件中。支持异常处理，不会终止爬虫程序。失败的请求，下次再执行爬虫程序时会自动下载
Stars: ✭ 275
Programming Languages
Projects that are alternatives of or similar to Python3 Concurrency Pics 02
python3-concurrency
Python3爬虫系列的理论验证，首先研究I/O模型，分别用Python实现了blocking I/O、nonblocking I/O、I/O multiplexing各模型下的TCP服务端和客户端。然后，研究同步I/O操作（依序下载、多进程并发、多线程并发）和异步I/O（asyncio）之间的效率差别
Stars: ✭ 49 (-82.18%)
Mutual labels: concurrency, progressbar, aiohttp, requests, asyncio, futures, coroutine
Python Simple Rest Client
Simple REST client for python 3.6+
Stars: ✭ 143 (-48%)
Mutual labels: asyncio, aiohttp, requests
rigor
HTTP-based DSL for for validating RESTful APIs
Stars: ✭ 65 (-76.36%)
Mutual labels: aiohttp, requests, asyncio
dvhb-hybrid
A package to mix django and asyncio in one application
Stars: ✭ 45 (-83.64%)
Mutual labels: aiohttp, asyncio
website
PySlackers website for invites and learning resources
Stars: ✭ 61 (-77.82%)
Mutual labels: aiohttp, asyncio
aiorwlock
Read/Write Lock - synchronization primitive for asyncio
Stars: ✭ 90 (-67.27%)
Mutual labels: concurrency, asyncio
pyladies-courseware
Homework/task submit and review web app · based on React and Python aiohttp
Stars: ✭ 14 (-94.91%)
Mutual labels: aiohttp, asyncio
aiohttp-jwt
aiohttp middleware and helper utils for working with JSON web token.
Stars: ✭ 70 (-74.55%)
Mutual labels: aiohttp, asyncio
aiodogstatsd
An asyncio-based client for sending metrics to StatsD with support of DogStatsD extension
Stars: ✭ 26 (-90.55%)
Mutual labels: aiohttp, asyncio
aiohttp-socks
Proxy (HTTP, SOCKS) connector for aiohttp
Stars: ✭ 147 (-46.55%)
Mutual labels: aiohttp, asyncio
glQiwiApi
The ultrarapid and multifunctional wrapper over QIWI and YooMoney
Stars: ✭ 44 (-84%)
Mutual labels: aiohttp, asyncio
feedsearch-crawler
Crawl sites for RSS, Atom, and JSON feeds.
Stars: ✭ 23 (-91.64%)
Mutual labels: aiohttp, asyncio
soabase-stages
A tiny library that makes staged/pipelined CompletableFutures much easier to create and manage
Stars: ✭ 23 (-91.64%)
Mutual labels: concurrency, futures
fbemissary
A bot framework for the Facebook Messenger platform, built on asyncio and aiohttp
Stars: ✭ 30 (-89.09%)
Mutual labels: aiohttp, asyncio
waio
Is a pretty simple and fully asynchronous framework for WhatsApp Business API written in Python 3.7 with asyncio and aiohttp.
Stars: ✭ 18 (-93.45%)
Mutual labels: aiohttp, asyncio
mux-stream
(De)multiplex asynchronous streams
Stars: ✭ 34 (-87.64%)
Mutual labels: concurrency, futures
aiohttp traversal
Traversal based router for aiohttp.web
Stars: ✭ 21 (-92.36%)
Mutual labels: aiohttp, asyncio
swift-futures
Demand-driven asynchronous programming in Swift
Stars: ✭ 32 (-88.36%)
Mutual labels: concurrency, futures
aiohttp-mako
mako template renderer for aiohttp.web
Stars: ✭ 32 (-88.36%)
Mutual labels: aiohttp, asyncio
python3-concurrency-mzitu
1. 进度条
2. 部分截图
分析爬取的过程：
3. 爬虫系列
- Python 3 爬虫｜第1章：I/O Models 阻塞/非阻塞 同步/异步
- Python 3 爬虫｜第2章：Python 并发编程
- Python 3 爬虫｜第3章：同步阻塞下载
- Python 3 爬虫｜第4章：多进程并发下载
- Python 3 爬虫｜第5章：多线程并发下载
- Python 3 爬虫｜第6章：可迭代对象 / 迭代器 / 生成器
- Python 3 爬虫｜第7章：协程 Coroutines
- Python 3 爬虫｜第8章：使用 asyncio 模块实现并发
- Python 3 爬虫｜第9章：使用 asyncio + aiohttp 并发下载
- Python 3 爬虫｜第10章：爬取少量妹子图
- Python 3 爬虫｜第11章：爬取海量妹子图
- Python 3 爬虫｜第12章：并发下载大文件 支持断点续传
4. 使用方法
此代码库中只有低速同步下载版本，协程高速版本请访问: https://madmalls.com/blog/post/python3-concurrency-pics-02/
4.1 下载代码
[[email protected] ~]# git clone https://github.com/wangy8961/python3-concurrency-pics-02.git
[cd python3-concurrency-pics-02/
4.2 准备虚拟环境
如果你的操作系统是Linux:
[-m venv venv3
[source venv3/bin/activate
Windows激活虚拟环境的命令是:venv3\Scripts\activate
4.3 安装依赖包
如果你的操作系统是Linux:
(venv3) [install -r requirements-linux.txt
如果你的操作系统是Windows（不会使用uvloop）:
(venv3) C:\Users\wangy> pip install -r requirements-win32.txt
4.4 测试
由于图片有16万多张，所以测试的时候，你可以指定只下载100个图集来对比同步下载、多线程下载和异步下载的效率区别，修改以下三个脚本中的TEST_NUM = 100
建议每次测试完，都删除相关目录：
(venv3) [rm -rf downloads/ logs/ __pycache__/
删除数据库记录：
(venv3) [[email protected] python3-concurrency-pics-02]# mongo
MongoDB shell version v3.6.6
connecting to: mongodb://127.0.0.1:27017
...
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
mzitu 0.036GB
> use mzitu
switched to db mzitu
> db.dropDatabase()
{ "dropped" : "mzitu", "ok" : 1 }
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
>
(1) 依序下载
(venv3) [root@CentOS python3-concurrency-pics-02]# python sequential.py
(2) 多线程下载
(venv3) [root@CentOS python3-concurrency-pics-02]# python threadpool.py
(3) 异步下载
(venv3) [root@CentOS python3-concurrency-pics-02]# python asynchronous.py
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].