All Projects → bcicen → Jstream

bcicen / Jstream

Licence: mit
Streaming JSON parser for Go

Programming Languages

go
31211 projects - #10 most used programming language

Projects that are alternatives of or similar to Jstream

Web Database Analytics
Web scrapping and related analytics using Python tools
Stars: ✭ 175 (-59.02%)
Mutual labels:  json, json-parser
Thorsserializer
C++ Serialization library for JSON
Stars: ✭ 241 (-43.56%)
Mutual labels:  json, json-parser
Jsons
🐍 A Python lib for (de)serializing Python objects to/from JSON
Stars: ✭ 178 (-58.31%)
Mutual labels:  json, json-parser
Jsonparser
One of the fastest alternative JSON parser for Go that does not require schema
Stars: ✭ 4,323 (+912.41%)
Mutual labels:  json, json-parser
Easyjson
Fast JSON serializer for golang.
Stars: ✭ 3,512 (+722.48%)
Mutual labels:  json, json-parser
Helios
A purely functional JSON library for Kotlin built on Λrrow
Stars: ✭ 157 (-63.23%)
Mutual labels:  json, json-parser
Json Dry
🌞 JSON-dry allows you to serialize & revive objects containing circular references, dates, regexes, class instances,...
Stars: ✭ 214 (-49.88%)
Mutual labels:  json, json-parser
Dynamodb Json
DynamoDB json util to load and dump strings of Dynamodb json format to python object and vise-versa
Stars: ✭ 114 (-73.3%)
Mutual labels:  json, json-parser
Mojojson
A simple and fast JSON parser.
Stars: ✭ 271 (-36.53%)
Mutual labels:  json, json-parser
Oj
Optimized JSON
Stars: ✭ 2,824 (+561.36%)
Mutual labels:  json, json-parser
Poison
An incredibly fast, pure Elixir JSON library
Stars: ✭ 1,898 (+344.5%)
Mutual labels:  json, json-parser
Jsoncons
A C++, header-only library for constructing JSON and JSON-like data formats, with JSON Pointer, JSON Patch, JSON Schema, JSONPath, JMESPath, CSV, MessagePack, CBOR, BSON, UBJSON
Stars: ✭ 400 (-6.32%)
Mutual labels:  json, json-parser
Daw json link
Static JSON parsing in C++
Stars: ✭ 146 (-65.81%)
Mutual labels:  json, json-parser
Json To Ast
JSON AST parser
Stars: ✭ 161 (-62.3%)
Mutual labels:  json, json-parser
Spotify Json
Fast and nice to use C++ JSON library.
Stars: ✭ 145 (-66.04%)
Mutual labels:  json, json-parser
Simdjson
Parsing gigabytes of JSON per second
Stars: ✭ 15,115 (+3439.81%)
Mutual labels:  json, json-parser
Simplejson
simplejson is a simple, fast, extensible JSON encoder/decoder for Python
Stars: ✭ 1,372 (+221.31%)
Mutual labels:  json, json-parser
Gitlogg
💾 🧮 🤯 Parse the 'git log' of multiple repos to 'JSON'
Stars: ✭ 102 (-76.11%)
Mutual labels:  json, json-parser
Whc model
iOS平台高效转换引擎json->model,model->json,model->Dictionary,支持模型类继承其他模型类,支持指定路径转换,不区分json的key和模型属性名称大小写,自动处理json中null
Stars: ✭ 244 (-42.86%)
Mutual labels:  json, json-parser
Bad json parsers
Exposing problems in json parsers of several programming languages.
Stars: ✭ 351 (-17.8%)
Mutual labels:  json, json-parser

jstream

GoDoc

jstream is a streaming JSON parser and value extraction library for Go.

Unlike most JSON parsers, jstream is document position- and depth-aware -- this enables the extraction of values at a specified depth, eliminating the overhead of allocating encompassing arrays or objects; e.g:

Using the below example document: jstream

we can choose to extract and act only the objects within the top-level array:

f, _ := os.Open("input.json")
decoder := jstream.NewDecoder(f, 1) // extract JSON values at a depth level of 1
for mv := range decoder.Stream() {
  fmt.Printf("%v\n ", mv.Value)
}

output:

map[desc:RGB colors:[red green blue]]
map[desc:CMYK colors:[cyan magenta yellow black]]

likewise, increasing depth level to 3 yields:

red
green
blue
cyan
magenta
yellow
black

optionally, kev:value pairs can be emitted as an individual struct:

decoder := jstream.NewDecoder(f, 2).EmitKV() // enable KV streaming at a depth level of 2
jstream.KV{desc RGB}
jstream.KV{colors [red green blue]}
jstream.KV{desc CMYK}
jstream.KV{colors [cyan magenta yellow black]}

Installing

go get github.com/bcicen/jstream

Commandline

jstream comes with a cli tool for quick viewing of parsed values from JSON input:

jstream -d 1 < input.json
{"colors":["red","green","blue"],"desc":"RGB"}
{"colors":["cyan","magenta","yellow","black"],"desc":"CMYK"}

detailed output with -v option:

cat input.json | jstream -v -d -1

depth	start	end	type   | value
2	018	023	string | "RGB"
3	041	046	string | "red"
3	048	055	string | "green"
3	057	063	string | "blue"
2	039	065	array  | ["red","green","blue"]
1	004	069	object | {"colors":["red","green","blue"],"desc":"RGB"}
2	087	093	string | "CMYK"
3	111	117	string | "cyan"
3	119	128	string | "magenta"
3	130	138	string | "yellow"
3	140	147	string | "black"
2	109	149	array  | ["cyan","magenta","yellow","black"]
1	073	153	object | {"colors":["cyan","magenta","yellow","black"],"desc":"CMYK"}
0	000	155	array  | [{"colors":["red","green","blue"],"desc":"RGB"},{"colors":["cyan","magenta","yellow","black"],"desc":"CMYK"}]

Options

Opt Description
-d <n> emit values at depth n. if n < 0, all values will be emitted
-kv output inner key value pairs as newly formed objects
-v output depth and offset details for each value
-h display help dialog

Benchmarks

Obligatory benchmarks performed on files with arrays of objects, where the decoded objects are to be extracted.

Two file sizes are used -- regular (1.6mb, 1000 objects) and large (128mb, 100000 objects)

input size lib MB/s Allocated
regular standard 97 3.6MB
regular jstream 175 2.1MB
large standard 92 305MB
large jstream 404 69MB

In a real world scenario, including initialization and reader overhead from varying blob sizes, performance can be expected as below: jstream

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].