All Projects → ptaoussanis → Nippy

ptaoussanis / Nippy

Licence: epl-1.0
High-performance serialization library for Clojure

Programming Languages

clojure
4091 projects

Projects that are alternatives of or similar to Nippy

Edn format
EDN reader and writer implementation in Python, using PLY (lex, yacc)
Stars: ✭ 92 (-89.02%)
Mutual labels:  serialization, edn
nason
🗜 Ultra tiny serializer / encoder with plugin-support. Useful to build binary files containing images, strings, numbers and more!
Stars: ✭ 30 (-96.42%)
Mutual labels:  serialization, compression
Qs
Quick serialization of R objects
Stars: ✭ 225 (-73.15%)
Mutual labels:  serialization, compression
Cidlib
The CIDLib general purpose C++ development environment
Stars: ✭ 179 (-78.64%)
Mutual labels:  encryption, compression
Rocketjob
Ruby's missing background and batch processing system
Stars: ✭ 258 (-69.21%)
Mutual labels:  encryption, compression
Edn Data
EDN parser and generator that works with plain JS data, with support for TS and node streams
Stars: ✭ 44 (-94.75%)
Mutual labels:  serialization, edn
NBT
A java implementation of the NBT protocol, including a way to implement custom tags.
Stars: ✭ 128 (-84.73%)
Mutual labels:  serialization, compression
7zip4powershell
Powershell module for creating and extracting 7-Zip archives
Stars: ✭ 213 (-74.58%)
Mutual labels:  encryption, compression
pakkero
Pakkero is a binary packer written in Go made for fun and educational purpose. Its main goal is to take in input a program file (elf binary, script, even appimage) and compress it, protect it from tampering and intrusion.
Stars: ✭ 143 (-82.94%)
Mutual labels:  compression, encryption
DAR
DAR - Disk ARchive
Stars: ✭ 58 (-93.08%)
Mutual labels:  compression, encryption
Cyberchef
The Cyber Swiss Army Knife - a web app for encryption, encoding, compression and data analysis
Stars: ✭ 13,674 (+1531.74%)
Mutual labels:  encryption, compression
Minizip Ng
Fork of the popular zip manipulation library found in the zlib distribution.
Stars: ✭ 750 (-10.5%)
Mutual labels:  encryption, compression
Eden
edn (extensible data notation) encoder/decoder for Elixir
Stars: ✭ 32 (-96.18%)
Mutual labels:  serialization, edn
ikeapack
Compact data serializer/packer written in Go, intended to produce a cross-language usable format.
Stars: ✭ 18 (-97.85%)
Mutual labels:  serialization, compression
Libzip
A C library for reading, creating, and modifying zip archives.
Stars: ✭ 379 (-54.77%)
Mutual labels:  encryption, compression
Peazip
Free Zip / Unzip software and Rar file extractor. Cross-platform file and archive manager. Features volume spanning, compression, authenticated encryption. Supports 7Z, 7-Zip sfx, ACE, ARJ, Brotli, BZ2, CAB, CHM, CPIO, DEB, GZ, ISO, JAR, LHA/LZH, NSIS, OOo, PAQ/LPAQ, PEA, QUAD, RAR, RPM, split, TAR, Z, ZIP, ZIPX, Zstandard.
Stars: ✭ 827 (-1.31%)
Mutual labels:  encryption, compression
Borgmatic
Simple, configuration-driven backup software for servers and workstations
Stars: ✭ 902 (+7.64%)
Mutual labels:  compression
Cryptomute
Encrypt numeric data (binary, decimal, hex) preserving it's format (eg. 16-digits card number to 16-digits encrypted number).
Stars: ✭ 24 (-97.14%)
Mutual labels:  encryption
Peergos
A p2p, secure file storage, social network and application protocol
Stars: ✭ 895 (+6.8%)
Mutual labels:  encryption
Swiftyrsa
RSA public/private key encryption in Swift
Stars: ✭ 894 (+6.68%)
Mutual labels:  encryption
Taoensso open-source

CHANGELOG | API | current Break Version:

[com.taoensso/nippy "3.1.1"] ; See CHANGELOG for details

See here if you're interested in helping support my open-source work, thanks! - Peter Taoussanis

SECURITY ADVISORY

Users of Nippy <= v2.15.0-RC1 should please upgrade ASAP due to a Remote Code Execution (RCE) vulnerability when deserializing data from an untrusted source.

See here for details, including upgrade instructions.

Nippy: the fastest serialization library for Clojure

Clojure's rich data types are awesome. And its reader allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).

Nippy is an attempt to provide a reliable, high-performance drop-in alternative to the reader. Used by the Carmine Redis client, the Faraday DynamoDB client, PigPen, Onyx and others.

Features

  • Small, uncomplicated all-Clojure library
  • Terrific performance (the fastest for Clojure that I'm aware of)
  • Comprehesive support for all standard data types
  • Easily extendable to custom data types (v2.1+)
  • Java's Serializable fallback when available (v2.5+)
  • Reader-fallback for all other types (including Clojure 1.4+ tagged literals)
  • Full test coverage for every supported type
  • Fully pluggable compression, including built-in high-performance LZ4 compressor
  • Fully pluggable encryption, including built-in high-strength AES128 enabled with a single :password [:salted "my-password"] option (v2+)
  • Utils for easy integration into 3rd-party tools/libraries (v2+)

Getting started

Add the necessary dependency to your project:

Leiningen: [com.taoensso/nippy "3.1.1"] ; or
deps.edn:   com.taoensso/nippy {:mvn/version "3.1.1"}

And setup your namespace imports:

(ns my-ns (:require [taoensso.nippy :as nippy]))

De/serializing

As an example of what it can do, let's take a look at Nippy's own reference stress data:

nippy/stress-data
=>
{:bytes     (byte-array [(byte 1) (byte 2) (byte 3)])
 :nil       nil
 :true      true
 :false     false
 :char      \ಬ
 :str-short "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"
 :str-long  (apply str (range 1000))
 :kw        :keyword
 :kw-ns     ::keyword
 :kw-long   (keyword
              (apply str "kw" (range 1000))
              (apply str "kw" (range 1000)))

 :sym       'foo
 :sym-ns    'foo/bar
 :sym-long   (symbol
               (apply str "sym" (range 1000))
               (apply str "sym" (range 1000)))

:regex     #"^(https?:)?//(www\?|\?)?"

 :queue        (-> (PersistentQueue/EMPTY) (conj :a :b :c :d :e :f :g))
 :queue-empty  (PersistentQueue/EMPTY)
 :queue-empty  (enc/queue)
 :sorted-set   (sorted-set 1 2 3 4 5)
 :sorted-map   (sorted-map :b 2 :a 1 :d 4 :c 3)

 :list         (list 1 2 3 4 5 (list 6 7 8 (list 9 10)))
 :list-quoted  '(1 2 3 4 5 (6 7 8 (9 10)))
 :list-empty   (list)
 :vector       [1 2 3 4 5 [6 7 8 [9 10]]]
 :vector-empty []
 :map          {:a 1 :b 2 :c 3 :d {:e 4 :f {:g 5 :h 6 :i 7}}}
 :map-empty    {}
 :set          #{1 2 3 4 5 #{6 7 8 #{9 10}}}
 :set-empty    #{}
 :meta         (with-meta {:a :A} {:metakey :metaval})
 :nested       [#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [] #{:a :b}}
                #{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [] #{:a :b}}
                [1 [1 2 [1 2 3 [1 2 3 4 [1 2 3 4 5]]]]]]

 :lazy-seq       (repeatedly 1000 rand)
 :lazy-seq-empty (map identity '())

 :byte         (byte 16)
 :short        (short 42)
 :integer      (int 3)
 :long         (long 3)
 :bigint       (bigint 31415926535897932384626433832795)

 :float        (float 3.14)
 :double       (double 3.14)
 :bigdec       (bigdec 3.1415926535897932384626433832795)

 :ratio        22/7
 :uuid         (java.util.UUID/randomUUID)
 :date         (java.util.Date.)

 ;;; JVM 8+
 :time-instant  (java.time.Instant/now)
 :time-duration (java.time.Duration/ofSeconds 100 100)
 :time-period   (java.time.Period/of 1 1 1)

 :objects       (object-array [1 "two" {:data "data"}])

 :stress-record (StressRecord. "data")
 :stress-type   (StressType.   "data")

 ;; Serializable
 :throwable    (Throwable. "Yolo")
 :exception    (try (/ 1 0) (catch Exception e e))
 :ex-info      (ex-info "ExInfo" {:data "data"})}

Serialize it:

(def frozen-stress-data (nippy/freeze nippy/stress-data))
=> #<byte[] [B@3253bcf3>

Deserialize it:

(nippy/thaw frozen-stress-data)
=> {:bytes        (byte-array [(byte 1) (byte 2) (byte 3)])
    :nil          nil
    :boolean      true
    <...> }

Couldn't be simpler!

See also the lower-level freeze-to-out! and thaw-from-in! fns for operating on DataOutput and DataInput types directly.

Encryption (v2+)

Nippy also gives you dead simple data encryption. Add a single option to your usual freeze/thaw calls like so:

(nippy/freeze nippy/stress-data {:password [:salted "my-password"]}) ; Encrypt
(nippy/thaw   <encrypted-data>  {:password [:salted "my-password"]}) ; Decrypt

There's two default forms of encryption on offer: :salted and :cached. Each of these makes carefully-chosen trade-offs and is suited to one of two common use cases. See the aes128-encryptor API docs for a detailed explanation of why/when you'd want one or the other.

Custom types (v2.1+)

(defrecord MyType [data])

(nippy/extend-freeze MyType :my-type/foo ; A unique (namespaced) type identifier
  [x data-output]
  (.writeUTF data-output (:data x)))

(nippy/extend-thaw :my-type/foo ; Same type id
  [data-input]
  (MyType. (.readUTF data-input)))

(nippy/thaw (nippy/freeze (MyType. "Joe"))) => #taoensso.nippy.MyType{:data "Joe"}

Performance

Nippy is currently the fastest serialization library for Clojure that I'm aware of, and offers roundtrip times between ~10x and ~15x faster than Clojure's tools.reader.edn, with a ~40% smaller output size.

benchmarks-png

Detailed benchmark info is available on Google Docs.

Contacting me / contributions

Please use the project's GitHub issues page for all questions, ideas, etc. Pull requests welcome. See the project's GitHub contributors page for a list of contributors.

Otherwise, you can reach me at Taoensso.com. Happy hacking!

- Peter Taoussanis

License

Distributed under the EPL v1.0 (same as Clojure).
Copyright © 2012-2020 Peter Taoussanis.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].