All Projects → JuliaIO → Jld.jl

JuliaIO / Jld.jl

Licence: mit
Saving and loading julia variables while preserving native types

Programming Languages

julia
2034 projects

Projects that are alternatives of or similar to Jld.jl

Hprose Delphi
Hprose is a cross-language RPC. This project is Hprose 2.0 for Delphi and FreePascal
Stars: ✭ 100 (-46.81%)
Mutual labels:  serializer
Structomap
Easily and dynamically generate maps from Go static structures
Stars: ✭ 122 (-35.11%)
Mutual labels:  serializer
Django Simple Serializer
serialize django data in a simple way
Stars: ✭ 157 (-16.49%)
Mutual labels:  serializer
Rdflib
RDFLib is a Python library for working with RDF, a simple yet powerful language for representing information.
Stars: ✭ 1,584 (+742.55%)
Mutual labels:  serializer
Jsonapi.rb
Lightweight, simple and maintained JSON:API support for your next Ruby HTTP API.
Stars: ✭ 116 (-38.3%)
Mutual labels:  serializer
Swiftmsgpack
💬 Fast & Lightweight MsgPack Serializer & Deserializer for Swift
Stars: ✭ 128 (-31.91%)
Mutual labels:  serializer
Go
A high-performance 100% compatible drop-in replacement of "encoding/json"
Stars: ✭ 10,248 (+5351.06%)
Mutual labels:  serializer
Zeroformatter
Infinitely Fast Deserializer for .NET, .NET Core and Unity.
Stars: ✭ 2,216 (+1078.72%)
Mutual labels:  serializer
Lora Serialization
LoraWAN serialization/deserialization library for The Things Network
Stars: ✭ 120 (-36.17%)
Mutual labels:  serializer
Djangorestframework Dataclasses
Dataclasses serializer for Django REST framework
Stars: ✭ 138 (-26.6%)
Mutual labels:  serializer
Block Content To React
React component for transforming Sanity block content to React components
Stars: ✭ 111 (-40.96%)
Mutual labels:  serializer
Bssom.net
A small, high performance, powerful serializer using bssom binary protocol
Stars: ✭ 117 (-37.77%)
Mutual labels:  serializer
Hprose Js
Hprose is a cross-language RPC. This project is Hprose 2.0 RPC for JavaScript
Stars: ✭ 133 (-29.26%)
Mutual labels:  serializer
Seriality
Seriality is a library for serializing and de-serializing all the Solidity types in a very efficient way which mostly written in solidity-assembly
Stars: ✭ 105 (-44.15%)
Mutual labels:  serializer
Serializer
With the Serializer component it's possible to handle serializing data structures, including object graphs, into array structures or other formats like XML and JSON. It can also handle deserializing XML and JSON back to object graphs.
Stars: ✭ 2,021 (+975%)
Mutual labels:  serializer
Java
jsoniter (json-iterator) is fast and flexible JSON parser available in Java and Go
Stars: ✭ 1,308 (+595.74%)
Mutual labels:  serializer
Adonis Bumblebee
Api Transformer for AdonisJs Framework
Stars: ✭ 125 (-33.51%)
Mutual labels:  serializer
Activej
ActiveJ is an alternative Java platform built from the ground up. ActiveJ redefines web, high load, and cloud programming in Java, featuring ultimate performance and scalability!
Stars: ✭ 183 (-2.66%)
Mutual labels:  serializer
Hprose Php
Hprose is a cross-language RPC. This project is Hprose 3.0 for PHP
Stars: ✭ 1,952 (+938.3%)
Mutual labels:  serializer
Flatsharp
Fast, idiomatic C# implementation of Flatbuffers
Stars: ✭ 133 (-29.26%)
Mutual labels:  serializer

JLD.jl

Build Status

Save and load variables in Julia Data format (JLD)

JLD, for which files conventionally have the extension .jld, is a widely-used format for data storage with the Julia programming language. JLD is a specific "dialect" of HDF5, a cross-platform, multi-language data storage format most frequently used for scientific data. By comparison with "plain" HDF5, JLD files automatically add attributes and naming conventions to preserve type information for each object.

For lossless storage of arbitrary Julia objects, the only other complete solution appears to be Julia's serializer, which can be accessed via the serialize and deserialize commands. However, because the serializer is also used for inter-process communication, long-term backwards compatibility is currently uncertain. (The JLDArchives repository exists to test compatibility of older JLD file formats.) If you choose to save data using the serializer, please use the file extension .jls to distinguish the files from .jld files.

Note: You should only read JLD files from trusted sources, as JLD files are capable of executing arbitrary code when read in.

Installation

Within Julia, use the package manager:

Pkg.add("JLD")

Currently this also requires the HDF5 package.

Quickstart

To use the JLD module, begin your code with

using JLD

If you just want to save a few variables and don't care to use the more advanced features, then a simple syntax is:

t = 15
z = [1,3]
save("/tmp/myfile.jld", "t", t, "arr", z)

Here we're explicitly saving t and z as "t" and "arr" within myfile.jld. You can alternatively pass save a dictionary; the keys must be strings and are saved as the variable names of their values within the JLD file. You can read these variables back in with

d = load("/tmp/myfile.jld")

which reads the entire file into a returned dictionary d. Or you can be more specific and just request particular variables of interest. For example, z = load("/tmp/myfile.jld", "arr") will return the value of arr from the file and assign it back to z.

JLD uses the FileIO package to provide a generic interface to save and load files. However this means that the user needs to explicitly request for the JLD format to be used while saving a new file.

save("/tmp/foo","bar",0.0) # ambiguous
save("/tmp/foo.jld","bar",0.0) # JLD format is inferred from the file extension
using FileIO; save(File(format"JLD","/tmp/foo"),"bar",0.0) # JLD format explicitly requested using FileIO

This problem is not encountered while loading a JLD file because FileIO can use magic bytes at the beginning of the file to infer its data format.

There are also convenience macros @save and @load that work on the variables themselves. @save "/tmp/myfile.jld" t z will create a file with just t and z; if you don't mention any variables, then it saves all the variables in the current module. Conversely, @load will pop the saved variables directly into the global workspace of the current module. However, keep in mind that these macros have significant limitations: for example, you can't use @load inside a function, there are constraints on using string interpolation inside filenames, etc. These limitations stem from the fact that Julia compiles functions to machine code before evaluation, so you cannot introduce new variables at runtime or evaluate expressions in other workspaces. The save and load functions do not have these limitations, and are therefore recommended as being considerably more robust, at the cost of some slight reduction in convenience.

More fine-grained control can be obtained using functional syntax:

jldopen("mydata.jld", "w") do file
    write(file, "A", A)  # alternatively, say "@write file A"
end

c = jldopen("mydata.jld", "r") do file
    read(file, "A")
end

This allows you to add variables as they are generated to an open JLD file. You don't have to use the do syntax (file = jldopen("mydata.jld", "w") works just fine), but an advantage is that it will automatically close the file (close(file)) for you, even in cases of error.

Julia's high-level wrapper, providing a dictionary-like interface, may also be of interest:

using JLD, HDF5

jldopen("test.jld", "w") do file
    g = create_group(file, "mygroup") # create a group
    g["dset1"] = 3.2              # create a scalar dataset inside the group
    g["dest2"] = rand(2,2)
end

Note that the features of HDF5 generally can also be used on JLD files.

Types and their definitions

You can save objects that have user-defined type; in a fresh Julia session, before loading those objects these types need to be defined. If no definition is available, the JLD module will automatically create the types for you. However, it's important to note that MyType, defined automatically by JLD, is not the same MyType as defined in an external module---in particular, module functions will not work for types defined by JLD. To ensure that loaded types have the full suite of behaviors provided by their definition in external modules, you should ensure that such modules are available before reading such variables from a .jld file.

To ensure automatic loading of modules, use addrequire to specify any dependencies. For example, suppose you have a file "MyTypes.jl" somewhere on your default LOAD_PATH, defined this way:

module MyTypes

export MyType

struct MyType
    value::Int
end

end

and you have an object x of type MyType. Then save x in the following way:

jldopen("somedata.jld", "w") do file
    addrequire(file, MyTypes)
    write(file, "x", x)
end

This will cause "MyTypes.jl" to be loaded automatically whenever "somedata.jld" is opened.

If you have performance problems...

Please see the complete documentation, particularly the section about custom serializers.

Complete documentation

More extensive documentation, including information about the JLD format conventions, can be found in the doc directory.

The test directory contains a number of test scripts that also demonstrate usage.

Credits

  • Simon Kornblith and Tim Holy (co-maintainers and primary authors)

  • Tom Short contributed to string->type conversion

  • Thanks also to the users who have reported bugs and tested fixes

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].