GRIB2 Golang parser application and library
Parser and library for the grib2 file format for meteorological data representations.
For a full presentation of the library, see https://github.com/nilsmagnus/what-is-griblib
Usage
Install by typing
go get -u github.com/nilsmagnus/grib
Library Usage:
Have a look at 'main.go' for main usage:
gribfile, err := os.Open("somegrib2file.grib2")
if err != nil {
log.Fatalf("Could not open test-file %v", err)
}
messages, err := griblib.ReadMessages(gribfile)
for _, message := range messages {
// do your thing
}
Read first n messages in a grib-file:
gribfile, err := os.Open("somegrib2file.grib2")
if err != nil { log.Fatalf("Could not open test-file %v", err) }
messages, err := griblib.ReadNMessages(gribfile, 2)
for _, message := range messages {
// do your thing with the n first messages
}
Application Usage:
$ grib -h
Usage of grib:
-category int
Filters on Category within discipline. -1 means all categories (default -1)
-dataExport
Export data values. (default true)
-discipline int
Filters on Discipline. -1 means all disciplines (default -1)
-export int
Export format. Valid types are 0 (none) 1(print discipline names) 2(print categories) 3(json)
-file string
Grib filepath
-latMax int
Maximum latitude multiplied with 100000. (default 36000000)
-latMin int
Minimum latitude multiplied with 100000.
-longMax int
Maximum longitude multiplied with 100000. (default 9000000)
-longMin int
Minimum longitude multiplied with 100000. (default -9000000)
-maxmsg int
Maximum number of messages to parse. Does not work in combination with filters. (default 2147483647)
-operation string
Operation. Valid values: 'parse', 'reduce'. (default "parse")
-reducefile string
Destination for reduced file. (default "reduced.grib2")
Examples:
Reduce input file to default output-file with discipline 0 (Meteorology):
grib -operation reduce -file testdata/reduced.grib2 -discipline 0
Filter on area on size of norway+sweden, output to json:
grib -file testdata/gfs.t00z.pgrb2.2p50.f003 -latMin 57000000 -latMax 71000000 -longMin 4400000 -longMax 32000000 -export 3
Filter on temperature only:
grib -file testdata/gfs.t00z.pgrb2.2p50.f003 -discipline 0 -category 0
What works?
- basic binary parsing of GRIB2 GFS files from NOAA
- implemented only "Grid point data - complex packing and spatial differencing"
- Parsing Data3 type
- Parsing Data0 type (Thanks to Cyrille Meichel)
Development
Dependencies
grib itself has no dependencies and wants to stay that way to keep it simple. Therefore, there is no dep/glide and no dependency-configuration-files.
Build
Grib uses make to build. You probably need go-lint installed in order to build.
To build, simply type
$ make
to test:
$ make test
TODOs
- Support different types of grids, not only grid0
- Support different types of products, not only product0
- Tests for reduction
- Tests for reading all sections
Contributors
- "analogic", https://github.com/analogic
- Cyrille Meichel, https://github.com/landru29
- Nils Larsgård, https://github.com/nilsmagnus
- measuredweighed, https://github.com/measuredweighed
Help appreciated
Feel free to fork and submit pull requests or simply create issues for improvements :)
Performance
to run a performance-test go to griblib/gribtest and run make benchmark.
CPU
Sample output:
go version go1.11.2 linux/amd64
go test -bench=. -benchmem -memprofile memprofile.out -cpuprofile profile.out
category number 0,parameter number 0,surface type 1, surface value 0 max: 329.500000 min: 197.900000
goos: linux
goarch: amd64
pkg: github.com/nilsmagnus/grib/griblib/gribtest
BenchmarkReadMessages-4 100 11681028 ns/op 6587281 B/op 9344 allocs/op
PASS
ok github.com/nilsmagnus/grib/griblib/gribtest 5.456s
go tool pprof -top profile.out
File: gribtest.test
Type: cpu
Time: Feb 9, 2019 at 6:44pm (CET)
Duration: 5.41s, Total samples = 5.34s (98.77%)
Showing nodes accounting for 5.12s, 95.88% of 5.34s total
Dropped 69 nodes (cum <= 0.03s)
flat flat% sum% cum cum%
1.58s 29.59% 29.59% 1.78s 33.33% github.com/nilsmagnus/grib/griblib.(*BitReader).readBit
1.07s 20.04% 49.63% 2.85s 53.37% github.com/nilsmagnus/grib/griblib.(*BitReader).readUint
0.53s 9.93% 59.55% 0.53s 9.93% runtime.memclrNoHeapPointers
0.36s 6.74% 66.29% 0.85s 15.92% encoding/binary.(*decoder).value
0.15s 2.81% 69.10% 2.78s 52.06% github.com/nilsmagnus/grib/griblib.(*BitReader).readIntsBlock
0.15s 2.81% 71.91% 0.15s 2.81% github.com/nilsmagnus/grib/griblib.(*Data3).applySpacialDifferencing
0.14s 2.62% 74.53% 0.26s 4.87% reflect.Value.Index
0.13s 2.43% 76.97% 0.20s 3.75% bytes.(*Buffer).ReadByte
0.11s 2.06% 79.03% 0.28s 5.24% github.com/nilsmagnus/grib/griblib.(*Data2).scaleValues
0.11s 2.06% 81.09% 0.19s 3.56% reflect.Value.SetUint
0.08s 1.50% 82.58% 0.08s 1.50% github.com/nilsmagnus/grib/griblib.Data0.scaleFunc.func1
0.08s 1.50% 84.08% 0.08s 1.50% reflect.(*rtype).Kind (inline)
0.07s 1.31% 85.39% 0.07s 1.31% bytes.(*Buffer).empty (inline)
0.06s 1.12% 86.52% 0.68s 12.73% runtime.mallocgc
0.06s 1.12% 87.64% 0.06s 1.12% runtime.memmove
0.05s 0.94% 88.58% 0.05s 0.94% reflect.flag.mustBeAssignable
0.05s 0.94% 89.51% 0.05s 0.94% runtime.nextFreeFast (inline)
0.04s 0.75% 90.26% 0.04s 0.75% encoding/binary.(*decoder).uint8 (inline)
- As you can see from the sample output, it is the readBit and readUint that takes most of the time. If anyone know how to optimize these functions further, please let me know :)
- Performance can be better if you only want to read certain kinds of messages. The filter that is implemented will filter only after reading all the messages. (help appreciated :) )
Memory
go tool pprof -top memprofile.out
File: gribtest.test
Type: alloc_space
Time: Feb 12, 2019 at 8:19pm (CET)
Showing nodes accounting for 3.39GB, 99.31% of 3.41GB total
Dropped 43 nodes (cum <= 0.02GB)
flat flat% sum% cum cum%
1.52GB 44.71% 44.71% 2.05GB 60.09% github.com/nilsmagnus/grib/griblib.(*Data2).extractData
0.76GB 22.19% 66.90% 0.76GB 22.19% github.com/nilsmagnus/grib/griblib.(*Data2).scaleValues
0.48GB 14.09% 80.99% 0.48GB 14.09% github.com/nilsmagnus/grib/griblib.(*BitReader).readIntsBlock
0.26GB 7.68% 88.67% 0.34GB 10.01% github.com/nilsmagnus/grib/griblib.(*Data2).extractBitGroupParameters
0.08GB 2.38% 91.05% 0.08GB 2.38% github.com/nilsmagnus/grib/griblib.(*BitReader).readUintsBlock
0.06GB 1.90% 92.95% 3.34GB 97.93% github.com/nilsmagnus/grib/griblib.readMessage
0.06GB 1.76% 94.70% 0.06GB 1.77% encoding/binary.Read
0.06GB 1.70% 96.40% 3.40GB 99.64% github.com/nilsmagnus/grib/griblib.ReadMessage
0.06GB 1.62% 98.02% 0.06GB 1.62% github.com/nilsmagnus/grib/griblib.makeBitReader
0.04GB 1.29% 99.31% 0.04GB 1.29% github.com/nilsmagnus/grib/griblib.(*bitGroupParameter).zeroGroup (inline)
- extractData is the most memory-hungry function. It could probably be more efficient, but is now optimized 80%.
- scaleValues could possibly be optimized, but is much better now than previous versions.
Grib Documentation
Grib specification:
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/
Documentation from noaa.gov :
http://www.nco.ncep.noaa.gov/pmb/docs/on388/
Daily grib2 files from NOAA can be found at