GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → stanford-rc → Ct_gdrive

stanford-rc / Ct_gdrive

Licence: gpl-3.0
Lustre/HSM Google Drive copytool

Programming Languages

python
139335 projects - #7 most used programming language
python2
120 projects

Labels

google-drive

Projects that are alternatives of or similar to Ct gdrive

Fb2cal
Fetch Facebook Birthdays events and create an ICS file for use with calendar apps
Stars: ✭ 335 (+1993.75%)
Mutual labels:  google-drive
Filestash
🦄 A modern web client for SFTP, S3, FTP, WebDAV, Git, Minio, LDAP, CalDAV, CardDAV, Mysql, Backblaze, ...
Stars: ✭ 5,231 (+32593.75%)
Mutual labels:  google-drive
Packtpub Crawler
Download your daily free Packt Publishing eBook https://www.packtpub.com/packt/offers/free-learning
Stars: ✭ 717 (+4381.25%)
Mutual labels:  google-drive
Google Drive Upload
Bash scripts to upload files to google drive
Stars: ✭ 403 (+2418.75%)
Mutual labels:  google-drive
Uds
Unlimited Drive Storage by splitting binary files into base64
Stars: ✭ 4,204 (+26175%)
Mutual labels:  google-drive
Drive Music Player
Fully client side Music Player for Google Drive
Stars: ✭ 554 (+3362.5%)
Mutual labels:  google-drive
Stackedit
In-browser Markdown editor
Stars: ✭ 18,744 (+117050%)
Mutual labels:  google-drive
Drive Db
📊 Use Google Drive spreadsheets as a simple database
Stars: ✭ 782 (+4787.5%)
Mutual labels:  google-drive
Drive Cli
A command line interface for accessing google drive
Stars: ✭ 449 (+2706.25%)
Mutual labels:  google-drive
Rclone
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
Stars: ✭ 30,541 (+190781.25%)
Mutual labels:  google-drive
Goindex
Index your Google Drive
Stars: ✭ 418 (+2512.5%)
Mutual labels:  google-drive
Google Drive Ocamlfuse
FUSE filesystem over Google Drive
Stars: ✭ 4,447 (+27693.75%)
Mutual labels:  google-drive
Gdrivefs
An innovative FUSE wrapper for Google Drive.
Stars: ✭ 618 (+3762.5%)
Mutual labels:  google-drive
Python Aria Mirror Bot
A telegram bot for all your mirror needs
Stars: ✭ 383 (+2293.75%)
Mutual labels:  google-drive
Gdown.pl
Google Drive direct download of big files
Stars: ✭ 735 (+4493.75%)
Mutual labels:  google-drive
Jupyterlab Google Drive
Cloud storage for JupyterLab using Google Drive
Stars: ✭ 332 (+1975%)
Mutual labels:  google-drive
Raidrive.translation
RaiDrive Translation
Stars: ✭ 518 (+3137.5%)
Mutual labels:  google-drive
Packt Publishing Free Learning
Scripts that automatically claim and download free daily eBooks from https://www.packtpub.com/packt/offers/free-learning
Stars: ✭ 788 (+4825%)
Mutual labels:  google-drive
Googlesheets
Google Spreadsheets R API
Stars: ✭ 771 (+4718.75%)
Mutual labels:  google-drive
Homescripts
My Scripts for Plex / Emby with Google Drive and rclone
Stars: ✭ 652 (+3975%)
Mutual labels:  google-drive
View All Similar Projects ➔

Lustre/HSM Google Drive copytool

ct_gdrive is an open source copytool enabling secure and transparent Google Drive cloud storage tier in Lustre using Robinhood.

  • Lustre is a parallel distributed file system, generally used for high performance computing (HPC).

  • Google Drive is a Cloud-based object storage system.

  • Robinhood policy engine is used to setup and apply policies for backup and tiering.

ct_gdrive transfers files from Lustre to Google Drive, and vice versa. Used either for Backup/Disaster Recovery or as an HSM tier to expand your Lustre filesystem, ct_gdrive directly uses the Google API Client Library for Python to minimize the number of remote cloud storage API requests needed to archive/restore files using Lustre/HSM.

Each Lustre file is stored in Google Drive using its Lustre FID as the name. When archived for the first time, a file description with some info like the original file path is added. ct_gdrive supports the upload of multiple versions of a same file to Google Drive. However, managing these versions is not supported from Lustre.

ct_gdrive should be used with lhsmtool_cmd, an open source and generic Lustre/HSM agent that launches a command when an action (archive, restore) is requested by the Lustre/HSM coordinator.

Principle



Requirements and installation

Please see the HOWTO.

License

ct_gdrive is published under the GPLv3 license.

Limitations

As of today, ct_gdrive works with a single Google Drive account, but a version supporting multiple accounts is planned.

Contact info

ct_gdrive is maintained and used in production by Stanford Research Computing where we successfully pushed several millions of files and more than 1 PB to Google Drive in about a month.

Have questions? Feel free to contact us at [email protected]



Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].