All Projects → explosion → Srsly

explosion / Srsly

Licence: mit
🦉 Modern high-performance serialization utilities for Python (JSON, MessagePack, Pickle)

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Srsly

Mashumaro
Fast and well tested serialization framework on top of dataclasses
Stars: ✭ 208 (+10.05%)
Mutual labels:  json, yaml, msgpack, serialization
Chronicle Wire
A Java Serialisation Library that supports multiple formats
Stars: ✭ 204 (+7.94%)
Mutual labels:  json, yaml, serialization
Rapidyaml
Rapid YAML - a library to parse and emit YAML, and do it fast.
Stars: ✭ 183 (-3.17%)
Mutual labels:  json, yaml, serialization
Airframe
Essential Building Blocks for Scala
Stars: ✭ 442 (+133.86%)
Mutual labels:  json, msgpack, serialization
Thorsserializer
C++ Serialization library for JSON
Stars: ✭ 241 (+27.51%)
Mutual labels:  json, yaml, serialization
Remarshal
Convert between CBOR, JSON, MessagePack, TOML, and YAML
Stars: ✭ 421 (+122.75%)
Mutual labels:  json, yaml, msgpack
Datafiles
A file-based ORM for Python dataclasses.
Stars: ✭ 113 (-40.21%)
Mutual labels:  json, yaml, serialization
Gelatin
Transform text files to XML, JSON, or YAML
Stars: ✭ 150 (-20.63%)
Mutual labels:  json, yaml
Dhallj
Dhall for Java
Stars: ✭ 154 (-18.52%)
Mutual labels:  json, yaml
Feedparser
feedparser gem - (universal) web feed parser and normalizer (XML w/ Atom or RSS, JSON Feed, HTML w/ Microformats e.g. h-entry/h-feed or Feed.HTML, Feed.TXT w/ YAML, JSON or INI & Markdown, etc.)
Stars: ✭ 156 (-17.46%)
Mutual labels:  json, yaml
Js Yaml Front Matter
Parses yaml or json from the beginning of a string or file
Stars: ✭ 165 (-12.7%)
Mutual labels:  json, yaml
Configurate
A simple configuration library for Java applications providing a node structure, a variety of formats, and tools for transformation
Stars: ✭ 148 (-21.69%)
Mutual labels:  json, yaml
Daw json link
Static JSON parsing in C++
Stars: ✭ 146 (-22.75%)
Mutual labels:  json, serialization
Orjson
Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy
Stars: ✭ 2,595 (+1273.02%)
Mutual labels:  json, serialization
Fig
A minimalist Go configuration library
Stars: ✭ 142 (-24.87%)
Mutual labels:  json, yaml
Simple Settings
A simple way to manage your project settings.
Stars: ✭ 165 (-12.7%)
Mutual labels:  json, yaml
Autocser
AutoCSer is a high-performance RPC framework. AutoCSer 是一个以高效率为目标向导的整体开发框架。主要包括 TCP 接口服务框架、TCP 函数服务框架、远程表达式链组件、前后端一体 WEB 视图框架、ORM 内存索引缓存框架、日志流内存数据库缓存组件、消息队列组件、二进制 / JSON / XML 数据序列化 等一系列无缝集成的高性能组件。
Stars: ✭ 140 (-25.93%)
Mutual labels:  json, serialization
Xresloader
跨平台Excel导表工具(Excel=>protobuf/msgpack/lua/javascript/json/xml)
Stars: ✭ 161 (-14.81%)
Mutual labels:  json, msgpack
Autoserver
Create a full-featured REST/GraphQL API from a configuration file
Stars: ✭ 188 (-0.53%)
Mutual labels:  json, yaml
Related
Nested Object Models in Python with dictionary, YAML, and JSON transformation support
Stars: ✭ 169 (-10.58%)
Mutual labels:  json, yaml

srsly: Modern high-performance serialization utilities for Python

This package bundles some of the best Python serialization libraries into one standalone package, with a high-level API that makes it easy to write code that's correct across platforms and Pythons. This allows us to provide all the serialization utilities we need in a single binary wheel. Currently supports JSON, JSONL, MessagePack, Pickle and YAML.

Azure Pipelines PyPi conda GitHub Python wheels

Motivation

Serialization is hard, especially across Python versions and multiple platforms. After dealing with many subtle bugs over the years (encodings, locales, large files) our libraries like spaCy and Prodigy have steadily grown a number of utility functions to wrap the multiple serialization formats we need to support (especially json, msgpack and pickle). These wrapping functions ended up duplicated across our codebases, so we wanted to put them in one place.

At the same time, we noticed that having a lot of small dependencies was making maintenance harder, and making installation slower. To solve this, we've made srsly standalone, by including the component packages directly within it. This way we can provide all the serialization utilities we need in a single binary wheel.

srsly currently includes forks of the following packages:

Installation

⚠️ Note that v2.x is only compatible with Python 3.6+. For 2.7+ compatibility, use v1.x.

srsly can be installed from pip. Before installing, make sure that your pip, setuptools and wheel are up to date.

pip install -U pip setuptools wheel
pip install srsly

Or from conda via conda-forge:

conda install -c conda-forge srsly

Alternatively, you can also compile the library from source. You'll need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler (XCode command-line tools on macOS / OS X or Visual C++ build tools on Windows), pip, virtualenv and git installed.

pip install -r requirements.txt  # install development dependencies
python setup.py build_ext --inplace  # compile the library

API

JSON

📦 The underlying module is exposed via srsly.ujson. However, we normally interact with it via the utility functions only.

function srsly.json_dumps

Serialize an object to a JSON string. Falls back to json if sort_keys=True is used (until it's fixed in ujson).

data = {"foo": "bar", "baz": 123}
json_string = srsly.json_dumps(data)
Argument Type Description
data - The JSON-serializable data to output.
indent int Number of spaces used to indent JSON. Defaults to 0.
sort_keys bool Sort dictionary keys. Defaults to False.
RETURNS str The serialized string.

function srsly.json_loads

Deserialize unicode or bytes to a Python object.

data = '{"foo": "bar", "baz": 123}'
obj = srsly.json_loads(data)
Argument Type Description
data str / bytes The data to deserialize.
RETURNS - The deserialized Python object.

function srsly.write_json

Create a JSON file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_json("/path/to/file.json", data)
Argument Type Description
path str / Path The file path or "-" to write to stdout.
data - The JSON-serializable data to output.
indent int Number of spaces used to indent JSON. Defaults to 2.

function srsly.read_json

Load JSON from a file or standard input.

data = srsly.read_json("/path/to/file.json")
Argument Type Description
path str / Path The file path or "-" to read from stdin.
RETURNS dict / list The loaded JSON content.

function srsly.write_gzip_json

Create a gzipped JSON file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_gzip_json("/path/to/file.json.gz", data)
Argument Type Description
path str / Path The file path.
data - The JSON-serializable data to output.
indent int Number of spaces used to indent JSON. Defaults to 2.

function srsly.read_gzip_json

Load gzipped JSON from a file.

data = srsly.read_gzip_json("/path/to/file.json.gz")
Argument Type Description
path str / Path The file path.
RETURNS dict / list The loaded JSON content.

function srsly.write_jsonl

Create a JSONL file (newline-delimited JSON) and dump contents line by line, or write to standard output.

data = [{"foo": "bar"}, {"baz": 123}]
srsly.write_jsonl("/path/to/file.jsonl", data)
Argument Type Description
path str / Path The file path or "-" to write to stdout.
lines iterable The JSON-serializable lines.
append bool Append to an existing file. Will open it in "a" mode and insert a newline before writing lines. Defaults to False.
append_new_line bool Defines whether a new line should first be written when appending to an existing file. Defaults to True.

function srsly.read_jsonl

Read a JSONL file (newline-delimited JSON) or from JSONL data from standard input and yield contents line by line. Blank lines will always be skipped.

data = srsly.read_jsonl("/path/to/file.jsonl")
Argument Type Description
path str / Path The file path or "-" to read from stdin.
skip bool Skip broken lines and don't raise ValueError. Defaults to False.
YIELDS - The loaded JSON contents of each line.

function srsly.is_json_serializable

Check if a Python object is JSON-serializable.

assert srsly.is_json_serializable({"hello": "world"}) is True
assert srsly.is_json_serializable(lambda x: x) is False
Argument Type Description
obj - The object to check.
RETURNS bool Whether the object is JSON-serializable.

msgpack

📦 The underlying module is exposed via srsly.msgpack. However, we normally interact with it via the utility functions only.

function srsly.msgpack_dumps

Serialize an object to a msgpack byte string.

data = {"foo": "bar", "baz": 123}
msg = srsly.msgpack_dumps(data)
Argument Type Description
data - The data to serialize.
RETURNS bytes The serialized bytes.

function srsly.msgpack_loads

Deserialize msgpack bytes to a Python object.

msg = b"\x82\xa3foo\xa3bar\xa3baz{"
data = srsly.msgpack_loads(msg)
Argument Type Description
data bytes The data to deserialize.
use_list bool Don't use tuples instead of lists. Can make deserialization slower. Defaults to True.
RETURNS - The deserialized Python object.

function srsly.write_msgpack

Create a msgpack file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_msgpack("/path/to/file.msg", data)
Argument Type Description
path str / Path The file path.
data - The data to serialize.

function srsly.read_msgpack

Load a msgpack file.

data = srsly.read_msgpack("/path/to/file.msg")
Argument Type Description
path str / Path The file path.
use_list bool Don't use tuples instead of lists. Can make deserialization slower. Defaults to True.
RETURNS - The loaded and deserialized content.

pickle

📦 The underlying module is exposed via srsly.cloudpickle. However, we normally interact with it via the utility functions only.

function srsly.pickle_dumps

Serialize a Python object with pickle.

data = {"foo": "bar", "baz": 123}
pickled_data = srsly.pickle_dumps(data)
Argument Type Description
data - The object to serialize.
protocol int Protocol to use. -1 for highest. Defaults to None.
RETURNS bytes The serialized object.

function srsly.pickle_loads

Deserialize bytes with pickle.

pickled_data = b"\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00}\x94(\x8c\x03foo\x94\x8c\x03bar\x94\x8c\x03baz\x94K{u."
data = srsly.pickle_loads(pickled_data)
Argument Type Description
data bytes The data to deserialize.
RETURNS - The deserialized Python object.

YAML

📦 The underlying module is exposed via srsly.ruamel_yaml. However, we normally interact with it via the utility functions only.

function srsly.yaml_dumps

Serialize an object to a YAML string. See the ruamel.yaml docs for details on the indentation format.

data = {"foo": "bar", "baz": 123}
yaml_string = srsly.yaml_dumps(data)
Argument Type Description
data - The JSON-serializable data to output.
indent_mapping int Mapping indentation. Defaults to 2.
indent_sequence int Sequence indentation. Defaults to 4.
indent_offset int Indentation offset. Defaults to 2.
sort_keys bool Sort dictionary keys. Defaults to False.
RETURNS str The serialized string.

function srsly.yaml_loads

Deserialize unicode or a file object to a Python object.

data = 'foo: bar\nbaz: 123'
obj = srsly.yaml_loads(data)
Argument Type Description
data str / file The data to deserialize.
RETURNS - The deserialized Python object.

function srsly.write_yaml

Create a YAML file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_yaml("/path/to/file.yml", data)
Argument Type Description
path str / Path The file path or "-" to write to stdout.
data - The JSON-serializable data to output.
indent_mapping int Mapping indentation. Defaults to 2.
indent_sequence int Sequence indentation. Defaults to 4.
indent_offset int Indentation offset. Defaults to 2.
sort_keys bool Sort dictionary keys. Defaults to False.

function srsly.read_yaml

Load YAML from a file or standard input.

data = srsly.read_yaml("/path/to/file.yml")
Argument Type Description
path str / Path The file path or "-" to read from stdin.
RETURNS dict / list The loaded YAML content.

function srsly.is_yaml_serializable

Check if a Python object is YAML-serializable.

assert srsly.is_yaml_serializable({"hello": "world"}) is True
assert srsly.is_yaml_serializable(lambda x: x) is False
Argument Type Description
obj - The object to check.
RETURNS bool Whether the object is YAML-serializable.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].