dimkouv / Massivedl
Licence: gpl-3.0
Download a large list of files concurrently
Stars: ✭ 141
Projects that are alternatives of or similar to Massivedl
4chan Downloader
Python3 script to continuously download all images/webms of multiple 4chan thread simultaneously - without installation
Stars: ✭ 136 (-3.55%)
Mutual labels: downloader, download-manager
Prdownloader
PRDownloader - A file downloader library for Android with pause and resume support
Stars: ✭ 2,947 (+1990.07%)
Mutual labels: downloader, download-manager
beveldm
Cross-platform download manager
Stars: ✭ 21 (-85.11%)
Mutual labels: downloader, download-manager
wget-lua
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
Stars: ✭ 52 (-63.12%)
Mutual labels: downloader, crawling
Flutter downloader
Flutter Downloader - A plugin for creating and managing download tasks. Supports iOS and Android. Maintainer: @hnvn
Stars: ✭ 546 (+287.23%)
Mutual labels: downloader, download-manager
CocoaDownloader
An powerful download library for iOS, macOS.
Stars: ✭ 22 (-84.4%)
Mutual labels: downloader, download-manager
su-downloader3
nodejs HTTP downloader with pause/resume support and segmented downloading
Stars: ✭ 14 (-90.07%)
Mutual labels: downloader, download-manager
Nacollector
⚔ 一个采集工具箱,据说是一个用于采集各种 WEB 资源的工作站?!你可以认为这是一个框架,可拓展。淘宝、天猫、苏宁、国美 等电商平台数据采集... 一键邀请 一键打包 账号登录获取Cookie 任务多线程 下载内容管理 实时日志 dll 热更新 无边框窗体 Web App, CefSharp, WebDriver
Stars: ✭ 158 (+12.06%)
Mutual labels: downloader, download-manager
Downloader
Fast and reliable multipart downloader with asynchronous progress events for .NET applications.
Stars: ✭ 128 (-9.22%)
Mutual labels: downloader, download-manager
Downthemall
The DownThemAll! WebExtension
Stars: ✭ 512 (+263.12%)
Mutual labels: downloader, download-manager
Floatplane-Downloader
Project for automatically organizing and downloading Floatplane videos for plex.
Stars: ✭ 94 (-33.33%)
Mutual labels: downloader, download-manager
Dl
🍗 a concurrent http file downloader
Stars: ✭ 68 (-51.77%)
Mutual labels: downloader, download-manager
super-anime-downloader
A program which takes an Anime name or URL and downloads the specified range of episodes.
Stars: ✭ 26 (-81.56%)
Mutual labels: downloader, download-manager
dl
Command-line file downloader tool
Stars: ✭ 39 (-72.34%)
Mutual labels: downloader, download-manager
Xdm
Powerfull download accelerator and video downloader
Stars: ✭ 3,226 (+2187.94%)
Mutual labels: downloader, download-manager
img-cli
An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL
Stars: ✭ 15 (-89.36%)
Mutual labels: downloader, crawling
Negibox
All in one downloader 全能下载器
Stars: ✭ 335 (+137.59%)
Mutual labels: downloader, download-manager
Lulu
[Unmaintained] A simple and clean video/music/image downloader 👾
Stars: ✭ 789 (+459.57%)
Mutual labels: crawling, downloader
Docker Jdownloader
JDownloader 2 Docker Image (Multiarch) - Passed 40M Downloads
Stars: ✭ 85 (-39.72%)
Mutual labels: downloader, download-manager
Squidwarc
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
Stars: ✭ 125 (-11.35%)
Mutual labels: crawling
massivedl
Download a large list of files in parallel.
Install
# for linux 64bit
wget https://github.com/dimkouv/massivedl/releases/download/v1.2/massivedl_linux_amd64
chmod +x massivedl_linux_amd64
mv massivedl_linux_amd64 /usr/local/bin/massivedl
Usage
Create a .csv file with the downloads
filename,url
0.png,https://placehold.it/100x100
1.png,https://placehold.it/100x101
2.png,https://placehold.it/100x102
...
Assuming the file was named data.csv
we can download the files using
massivedl -p 10 -i data.csv -s 1 -o downloads
Command line parameters
-p <int> (default=10) : Maximum number of parallel requests
-s <int> (default=0) : Number of skipped lines from input csv
-i <str> : Input csv file with the list of urls
-o <str> (default='downloads') : Directory to place the downloads
Stop and continue later
You can stop and continue downloading later.
Press Ctrl+C
then you will have the following dialog.
...
Do you want to save progress? [Y/n]: yes
Progress has been saved!
Use the following command to continue downloading
massivedl --load /path/to/savedfile.save
Use Cases
With this tool I was able to download about 1.5 million images (~60GB) for a machine learning project.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].