GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → wangy8961 → python3-concurrency

wangy8961 / python3-concurrency

Licence: other
Python3爬虫系列的理论验证，首先研究I/O模型，分别用Python实现了blocking I/O、nonblocking I/O、I/O multiplexing各模型下的TCP服务端和客户端。然后，研究同步I/O操作（依序下载、多进程并发、多线程并发）和异步I/O（asyncio）之间的效率差别

Programming Languages

python
139335 projects - #7 most used programming language

Labels

concurrencyprogressbaraiohttprequestsasynciofuturescoroutinethreadpool

Projects that are alternatives of or similar to python3-concurrency

Python3 Concurrency Pics 02
爬取 www.mzitu.com 全站图片，截至目前共5162个图集，16.5万多张美女图片，使用 asyncio 和 aiohttp 实现的异步版本只需要不到2小时就能爬取完成。按日期创建图集目录，保存更合理。控制台只显示下载的进度条，详细信息保存在日志文件中。支持异常处理，不会终止爬虫程序。失败的请求，下次再执行爬虫程序时会自动下载
Stars: ✭ 275 (+461.22%)
Mutual labels:  concurrency, progressbar, aiohttp, requests, asyncio, futures, coroutine
rigor
HTTP-based DSL for for validating RESTful APIs
Stars: ✭ 65 (+32.65%)
Mutual labels:  aiohttp, requests, asyncio
Python Simple Rest Client
Simple REST client for python 3.6+
Stars: ✭ 143 (+191.84%)
Mutual labels:  aiohttp, requests, asyncio
YACLib
Yet Another Concurrency Library
Stars: ✭ 193 (+293.88%)
Mutual labels:  concurrency, coroutine
Pyrez
(ON REWRITE) An easy to use (a)sync wrapper for Hi-Rez Studios API (Paladins, Realm Royale, and Smite), written in Python. 🐍
Stars: ✭ 23 (-53.06%)
Mutual labels:  aiohttp, requests
java-red
Effective Concurrency Modules for Java
Stars: ✭ 25 (-48.98%)
Mutual labels:  concurrency, futures
Brightfutures
Write great asynchronous code in Swift using futures and promises
Stars: ✭ 1,890 (+3757.14%)
Mutual labels:  concurrency, futures
futureproof
Bulletproof concurrent.futures
Stars: ✭ 36 (-26.53%)
Mutual labels:  concurrency, futures
torequests
Async wrapper for requests / aiohttp, and some crawler toolkits. Let synchronization code enjoy the performance of asynchronous programming.
Stars: ✭ 22 (-55.1%)
Mutual labels:  aiohttp, requests
BeauRoutine
Coroutine and tweening framework for Unity3D
Stars: ✭ 88 (+79.59%)
Mutual labels:  futures, coroutine
aioneo4j
asyncio client for neo4j
Stars: ✭ 29 (-40.82%)
Mutual labels:  aiohttp, asyncio
Pyee
A port of Node.js's EventEmitter to python
Stars: ✭ 236 (+381.63%)
Mutual labels:  concurrency, asyncio
Mayhem
The Python asyncio tutorial I wish existed earlier
Stars: ✭ 184 (+275.51%)
Mutual labels:  concurrency, asyncio
universe-topology
A universal computer knowledge topology for all the programmers worldwide.
Stars: ✭ 47 (-4.08%)
Mutual labels:  aiohttp, coroutine
Smol
A small and fast async runtime for Rust
Stars: ✭ 2,206 (+4402.04%)
Mutual labels:  concurrency, futures
aiohttp-client-cache
An async persistent cache for aiohttp requests
Stars: ✭ 63 (+28.57%)
Mutual labels:  aiohttp, requests
aioScrapy
基于asyncio与aiohttp的异步协程爬虫框架 欢迎Star
Stars: ✭ 34 (-30.61%)
Mutual labels:  aiohttp, asyncio
pytest-aiohttp
pytest plugin for aiohttp support
Stars: ✭ 110 (+124.49%)
Mutual labels:  aiohttp, asyncio
yutto
🧊 一个可爱且任性的 B 站视频下载器（bilili V2）
Stars: ✭ 383 (+681.63%)
Mutual labels:  aiohttp, asyncio
Cocrawler
CoCrawler is a versatile web crawler built using modern tools and concurrency.
Stars: ✭ 148 (+202.04%)
Mutual labels:  concurrency, aiohttp
View All Similar Projects ➔

python3-concurrency

Python aiohttp BeautifulSoup4 requests pymongo progressbar2

1. 爬虫系列

2. 使用方法

2.1 Server

为防止DDoS攻击，本次测试需要在本地搭建一个HTTP测试服务器，具体方法参考 Python3爬虫系列03 (实验) - 同步阻塞下载

2.2 Client

(1) 下载代码

[root@CentOS ~]# git clone https://github.com/wangy8961/python3-concurrency.git
[root@CentOS ~]# cd python3-concurrency/

(2) 准备虚拟环境

如果你的操作系统是Linux:

[root@CentOS python3-concurrency]# python3 -m venv venv3
[root@CentOS python3-concurrency]# source venv3/bin/activate

Windows激活虚拟环境的命令是: venv3\Scripts\activate

(3) 安装依赖包

如果你的操作系统是Linux:

(venv3) [root@CentOS python3-concurrency]# pip install -r requirements-linux.txt

如果你的操作系统是Windows（不会使用uvloop）:

(venv3) C:\Users\wangy> pip install -r requirements-win32.txt

(4) 测试

依序下载：

(venv3) [root@CentOS python3-concurrency]# python sequential.py

多进程下载：

(venv3) [root@CentOS python3-concurrency]# python processpool.py

多线程下载：

(venv3) [root@CentOS python3-concurrency]# python threadpool.py

异步下载：

(venv3) [root@CentOS python3-concurrency]# python asynchronous.py
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].