All Projects → qbx2 → pscp

qbx2 / pscp

Licence: MIT license
Per-session checkpoint for boosting up your research

Programming Languages

python
139335 projects - #7 most used programming language
Jupyter Notebook
11667 projects
Makefile
30231 projects

Projects that are alternatives of or similar to pscp

save-and-resume
Checkpoint tutorial on FloydHub for Pytorch, Keras and Tensorflow.
Stars: ✭ 36 (+157.14%)
Mutual labels:  checkpoint
fw1-loggrabber
FW1-Loggrabber is a command-line tool to grab logfiles from remote Checkpoint devices using OPSEC LEA (Log Export API)
Stars: ✭ 54 (+285.71%)
Mutual labels:  checkpoint
scr
SCR caches checkpoint data in storage on the compute nodes of a Linux cluster to provide a fast, scalable checkpoint / restart capability for MPI codes.
Stars: ✭ 84 (+500%)
Mutual labels:  checkpoint
Checkpoint
Fast and simple homebrew save manager for 3DS and Switch.
Stars: ✭ 1,886 (+13371.43%)
Mutual labels:  checkpoint
Cisco2Checkpoint
Tool that assists in migrating firewall rules from Cisco to Checkpoint. Will optimize rules for you (rationalization, reuse merging, etc.).
Stars: ✭ 19 (+35.71%)
Mutual labels:  checkpoint
php-instagram-private-api-with-checkpoint-feature
This is a liam cottle instagram private library with checkpoint issue solved. Checkpoint required error solved and can be found at examples folder.
Stars: ✭ 12 (-14.29%)
Mutual labels:  checkpoint
cruise
User space POSIX-like file system in main memory
Stars: ✭ 27 (+92.86%)
Mutual labels:  checkpoint
FlinkTutorial
FlinkTutorial 专注大数据Flink流试处理技术。从基础入门、概念、原理、实战、性能调优、源码解析等内容,使用Java开发,同时含有Scala部分核心代码。欢迎关注我的博客及github。
Stars: ✭ 46 (+228.57%)
Mutual labels:  checkpoint
Deep-Compression.Pytorch
Unofficial Pytorch implementation of Deep Compression in CIFAR10
Stars: ✭ 29 (+107.14%)
Mutual labels:  checkpoint
fastfreeze
Turn-key solution to checkpoint/restore applications running in Linux containers
Stars: ✭ 68 (+385.71%)
Mutual labels:  checkpoint
tf-imagenet
TensorFlow ImageNet - Training and SOTA checkpoints
Stars: ✭ 50 (+257.14%)
Mutual labels:  checkpoint

TravisCI PyPI PyPI-License codecov

PSCP: Per-session checkpoint

PSCP is a brand-new way to make checkpoint in git repository efficiently. By logging the hash output by pscp.create(), you can git diff, git stash branch and git checkout using them whenever you want. PSCP is framework-agnostic, so can be used in any git repository.

PSCP inherits all strengths from git, which are very useful for study.

  • Recycle unchanged objects
  • Compression (zlib)
  • Show differences using git diff a..b
  • Checkout to, using git checkout
  • Create branch from, using git stash branch. This restores staged files staged and unstaged files unstaged.

Besides, PSCP does make neither git log nor git stash list messy. You won't suffer from trivial tuning commits and too many stashes.

Requirements

  • python 3.5+
  • git

Installation

pip install pscp

Getting Started

Getting Started

Frequently Asked Questions

How does it work?

How PSCP Works

Does PSCP just save 'all' files?

No, PSCP saves git-tracked files only. For details, please refer to 2.2 Git Basics - Recording Changes to the Repository.

Tracking New Files

Use git add to stage, to track new files. Untracked files will not be added to the checkpoint.

Saving environment variables or argparse results?

You're supposed to write to any file and track them (using git add). Using json built-in library, it can be done easily.

import json

To save environment variables,

import os
json.dump(fp, dict(os.environ), indent=4)

To save argparse namespace,

json.dump(fp, vars(args), indent=4)

Any other data can be saved in same way.

Command Line Tool (TODO)

python -m pscp

pscp.create(return_head_on_nothing=True, return_format='abbrev', link=True)

Create per-session checkpoint.

  • return_head_on_nothing: If True, return hash of HEAD when there are nothing changed compared to HEAD, otherwise, return None.
  • return_format
    • abbrev, short: Return abbreviated hash.
    • long: Return just unabbreviated hash.
    • ref: Return refspec. Raises exception when link == False.
  • link: See pscp.link() below.

pscp.link(hash, refspec=None)

Create reference. pscp.link() could be used to avoid pruning on garbage collection.

  • hash: The hash to be referenced.
  • refspec: If None, refs/pscp/{timestamp_ms} is used.

pscp.delete(refspec)

Delete pscp reference. Call pscp.gc() if you want.

  • refspec: Target refspec to be deleted. timestamp_ms is also allowed.

pscp.gc(prune='now')

Run git gc --prune=<prune>.

pscp.push(refspec=None, repository='origin')

Push checkpoint to the remote repository.

  • refspec: If None, all checkpoints are pushed.

pscp.fetch(refspec=None, refmap=None, repository='origin')

  • refspec: If None, all checkpoints are fetched.
  • refmap: If None, refs/pscp/*:refs/pscp/* is used.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].