Python3WebSpider / Douyin
Licence: mit
API of DouYin for Humans used to Crawl Popular Videos and Musics
Stars: ✭ 580
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Douyin
Xxl Crawler
A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）
Stars: ✭ 561 (-3.28%)
Mutual labels: crawler, spider
Filemasta
A search application to explore, discover and share online files
Stars: ✭ 571 (-1.55%)
Mutual labels: crawler, videos
Signature algorithm
各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)
Stars: ✭ 380 (-34.48%)
Mutual labels: crawler, spider
Xsrfprobe
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Stars: ✭ 532 (-8.28%)
Mutual labels: crawler, spider
Webster
a reliable high-level web crawling & scraping framework for Node.js.
Stars: ✭ 364 (-37.24%)
Mutual labels: crawler, spider
Fictiondown
小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对
Stars: ✭ 362 (-37.59%)
Mutual labels: crawler, spider
Bilili
🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器
Stars: ✭ 379 (-34.66%)
Mutual labels: crawler, spider
Freshonions Torscraper
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
Stars: ✭ 348 (-40%)
Mutual labels: crawler, spider
Awesome Crawler
A collection of awesome web crawler,spider in different languages
Stars: ✭ 4,793 (+726.38%)
Mutual labels: crawler, spider
91porn Api
🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览
Stars: ✭ 341 (-41.21%)
Mutual labels: crawler, spider
Ttbot
今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现
Stars: ✭ 338 (-41.72%)
Mutual labels: crawler, spider
Crawly
Crawly, a high-level web crawling & scraping framework for Elixir.
Stars: ✭ 440 (-24.14%)
Mutual labels: crawler, spider
Netdiscovery
NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。
Stars: ✭ 573 (-1.21%)
Mutual labels: crawler, spider
注：项目已不可用...
DouYin
API of DouYin App for humans.
Installation
pip3 install douyin
Usage
Here is the sample code:
import douyin
from douyin.structures import Topic, Music
# define file handler and specify folder
video_file_handler = douyin.handlers.VideoFileHandler(folder='./videos')
music_file_handler = douyin.handlers.MusicFileHandler(folder='./musics')
# define mongodb handler
mongo_handler = douyin.handlers.MongoHandler()
# define downloader
downloader = douyin.downloaders.VideoDownloader([mongo_handler, video_file_handler, music_file_handler])
for result in douyin.hot.trend():
for item in result.data:
# download videos of topic/music for 100 max per
downloader.download(item.videos(max=100))
then you can get:
Item <Topic: <1565818716518401, panama>>
Processing <Video: <6616517521098935565, 真香#panama>> ...
Processing <Video: <6500385230921141518, 哈哈哈哈哈>> ...
...
Processing <Video: <6479958542747962637, 👅ก่อนกินข้>> ...
Processing <Video: <6473811426107460878, 😁>> ...
0%| | 0/10 [00:00<?, ?it/s]
Processing 1-10 of files
Processing <Video: <6616517521098935565, 真香#panama>> ...
Saving <Video: <6616517521098935565, 真香#panama>> to mongodb...
Processing <Video: <6500385230921141518, 哈哈哈哈哈>> ...
Saving <Video: <6500385230921141518, 哈哈哈哈哈>> to mongodb...
Processing <Video: <6562690160868199693, 皇城相府版C哩C哩跨>> ...
....
Downloading <Video: <6580510322468064526, 第二遍 后面的小哥哥>> ...
Saved <Video: <6479958542747962637, 👅ก่อนกินข้>> to mongodb successfully
Downloading <Video: <6479958542747962637, 👅ก่อนกินข้>> ...
Saved <Video: <6473811426107460878, 😁>> to mongodb successfully
Downloading <Video: <6473811426107460878, 😁>> ...
Downloaded file to ./videos/6580510322468064526.mp4
10%|████▌ | 1/10 [00:01<00:16, 1.84s/it]
Downloaded file to ./videos/6516746291806997763.mp4
20%|█████████▏ | 2/10 [00:01<00:10, 1.33s/it]
Downloaded file to ./videos/6600742831352974596.mp4
40%|██████████████████▍ | 4/10 [00:02<00:05, 1.03it/s]
Downloaded file to ./videos/6484393014599879950.mp4
50%|███████████████████████ | 5/10 [00:02<00:04, 1.15it/s]
Downloaded file to ./videos/6616517521098935565.mp4
60%|███████████████████████████▌ | 6/10 [00:03<00:03, 1.27it/s]
Downloaded file to ./videos/6479958542747962637.mp4
70%|████████████████████████████████▏ | 7/10 [00:03<00:01, 1.68it/s]
Downloaded file to ./videos/6472305134377372941.mp4
80%|████████████████████████████████████▊ | 8/10 [00:03<00:00, 2.05it/s]
Downloaded file to ./videos/6562690160868199693.mp4
90%|█████████████████████████████████████████▍ | 9/10 [00:04<00:00, 2.27it/s]
Downloaded file to ./videos/6500385230921141518.mp4
100%|█████████████████████████████████████████████| 10/10 [00:04<00:00, 2.33it/s]
Examples
See examples
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].