GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → coolbutuseless → foist

coolbutuseless / foist

Licence: MIT license
Fast Output of Images

Programming Languages

C++
36643 projects - #6 most used programming language
r
7636 projects
c
50402 projects - #5 most used programming language

FOIst - Fast Output of Images


foist is a very fast way to output a matrix or array to a lossless, uncompressed image file.


foist can write lossless grey image files ~5x faster than the png library.

foist can write lossless RGB image files ~7x faster than the png library.

foist allocates a lot less memory than other methods.


  • foist supports writing lossless images in NETPBM, PNG and GIF formats.
  • foist is fast because it uses Rcpp to quickly scale, manipulate and re-order data for image output.
  • foist can be wicked fast if data-ordering is ignored. The price paid for this speed is that the image will appear transposed in the output.

What’s in the box

  • write_pnm() - NETPBM format RGB, grey and indexed colour palette images.
  • write_png() - PNG format RGB, grey and indexed colour palette images.
  • write_gif() - GIF format grey and indexed colour palette images.
  • vir The 5 palettes from viridis.

This package would not be possible without:

  • Rcpp - The easiest way to get fast C/C++ code into R.
  • viridis - Wonderful palettes originally from matplotlib.
  • NETPBM - A 30-year-old uncompressed image format for full-colour images.
  • PNG - A 20-year old image format for full-colour and indexed-colour images.
  • GIF - A 30-year old image format for indexed-colour images.

Technical Notes

  • foist contains a bespoke, minimalist PNG encoder written in C++
    • Written so the package has complete control over the image output.
    • There is no lossless compression enabled in this PNG encoder i.e. only uncompressed DEFLATE blocks are used (see https://datatracker.ietf.org/doc/rfc1951 Sect 3.2.4).
    • The IDAT and ZLIB/DEFLATE blocks are output in sync (one-DEFLATE-block-per-IDAT-chunk) as this made the PNG implementation much simpler.
    • The encoder uses Mark Adler’s adler32.c code from zlib Copyright (C) 1995-2011, 2016 Mark Adler.
    • A SIMD version of adler32() was included but not enabled by default. The speed gains weren’t significant enough for the machine imcompatibility headaches it would introduce.
    • crc32 implementation is a very fast slice-by-16 implementation by Stephan Brumme. This is noticeably much faster than the slice-by-4 crc32 that comes with the standard zlib library.
  • foist contains a bespoke, minimalist GIF encoder written in C++
    • Written so the package has complete control over the image output.
    • Writes uncompressed GIFs only (No LZW compression is included).
    • Only 128 colours/image are possible with this GIF encoder - this limitiation greatly reduces the complexity of the code.
  • Because PNG data also needs CRC32 and ADLER32 checksumming it is generally slower than GIF/PGM/PPM output.
  • However, writing a matrix with a palette will be faster in GIF/PNG as it has direct support for indexed colours, whereas for a NETPBM PPM file the intensity values need to be explicitly mapped to an RGB triplet and then written out in full.
  • All my benchmark timings are on a machine with an SSD.

Installation

You can install the package from GitHub with:

# install.packages("remotes")
remotes::install_github("coolbutuseless/foist")

Setup data

  • dbl_mat - A 2D numeric matrix for output to a grey image. All values in range [0, 1]
  • dbl_arr - A 3D numeric array for output to an RGB image. All values in range [0, 1]
ncol    <- 256
nrow    <- 160
int_vec <- seq(nrow * ncol) %% 254L
int_mat <- matrix(int_vec, nrow = nrow, ncol = ncol, byrow = TRUE)
dbl_mat <- int_mat/255

# A non-boring RGB array/image
r       <- dbl_mat
g       <- matrix(rep(seq(0, 255, length.out = nrow)/255, each = ncol), nrow, ncol, byrow = TRUE)
b       <- dbl_mat[, rev(seq(ncol(dbl_mat)))  ]
dbl_arr <- array(c(r, g, b), dim = c(nrow, ncol, 3))

Save a 2D matrix as a grey image

write_png() and write_pnm() will save a 2D numeric matrix as a grey image.

  • The matrix values must be in the range [0, 1].
  • Use the intensity_factor argument to scale image values on-the-fly as they are written to file.
write_pnm(dbl_mat, "man/figures/col-0-n.pgm")                               # PGM
write_pnm(dbl_mat, "man/figures/col-0-i.pgm", invert = TRUE)                # PGM
write_png(dbl_mat, "man/figures/col-0-f.png", flipy = TRUE)                 # PNG
write_gif(dbl_mat, "man/figures/col-0-t.gif", convert_to_row_major = FALSE) # GIF

Save a 3D array as an RGB image

write_png() and write_pnm() will save a 3D numeric array as an RGB image.

  • Array dimensions must be NxMx3 where the 3 colour planes correspond to the third dimension of the array.
  • The matrix values must be in the range [0, 1].
  • Use the intensity_factor argument to scale image values on-the-fly as they are written to file.
write_pnm(dbl_arr, filename = "man/figures/col-1-n.ppm")                                # NETPBM PPM
write_pnm(dbl_arr, filename = "man/figures/col-1-i.ppm", invert = TRUE)                 # NETPBM PPM
write_png(dbl_arr, filename = "man/figures/col-1-f.png", flipy = TRUE)                  # PNG
write_png(dbl_arr, filename = "man/figures/col-1-t.png", convert_to_row_major = FALSE)  # PNG

Save a matrix to an RGB image using a palette lookup

write_png() and write_pnm() will save a 2D numeric matrix as an RGB image if also supplied with a colour palette.

  • A palette must be an integer matrix with dimensions N x 3
    • N is the number of colours in the palette
    • 2 <= N <= 256
  • Values in the palette must be in the range [0, 255].
  • The matrix values must initially be in the range [0, 1].
  • Pixel values in the matrix are first scaled into the range [0, N] and are then mapped to one of the RGB colours in the palette.

foist includes the 5 palettes from viridis as vir$magma etc.

write_pnm(dbl_mat,                    "man/figures/col-0.pgm")  # NETPBM PGM
write_pnm(dbl_mat, pal = vir$magma  , "man/figures/col-3.ppm")  # NETPBM PPM
write_png(dbl_mat, pal = vir$inferno, "man/figures/col-4.png")  # PNG
write_png(dbl_mat, pal = vir$plasma , "man/figures/col-5.png")  # PNG
write_gif(dbl_mat, pal = vir$viridis, "man/figures/col-6.gif")  # GIF
write_gif(dbl_mat, pal = vir$cividis, "man/figures/col-7.gif")  # GIF

Manipulate palettes

Some visual effects can be created by keeping the same data, but manipulating the palette of a sequence of image outputs.

Benchmark: Saving a matrix as a grey image

The following benchmark compares the time to output of a grey image using:

  • foist::write_pnm() in both row-major and column-major ordering
  • foist::write_png() in both row-major and column-major ordering
  • foist::write_gif() in both row-major and column-major ordering
  • png::writePNG()
  • caTools::write.gif()

As can be seen in the benchmark using flipy = TRUE or invert = TRUE have almost no speed penalty.

expression min median itr/sec mem_alloc
foist::write_pnm(dbl_mat, tmp) 2.9ms 3.85ms 231 2.49KB
foist::write_pnm(dbl_mat, tmp, convert_to_row_major = FALSE) 1.9ms 2.25ms 395 2.49KB
foist::write_gif(dbl_mat, tmp) 2.79ms 3.33ms 276 2.49KB
foist::write_gif(dbl_mat, tmp, convert_to_row_major = FALSE) 1.86ms 2.25ms 403 2.49KB
foist::write_png(dbl_mat, tmp) 3.21ms 3.71ms 248 2.49KB
foist::write_png(dbl_mat, tmp, convert_to_row_major = FALSE) 2.25ms 2.53ms 368 2.49KB
foist::write_png(dbl_mat, tmp, convert_to_row_major = FALSE, flipy = TRUE, invert = TRUE) 2.21ms 2.57ms 358 2.49KB
png::writePNG(dbl_mat, tmp) 12.64ms 14.24ms 69 670.81KB
caTools::write.gif(dbl_mat, tmp) 27.43ms 31.96ms 31 35.18MB

Benchmark results

#> Loading required namespace: tidyr

Benchmark: Saving an RGB image

The following benchmark compares the time to output a colour image using:

  • foist::write_pnm() saving a 3D array in both row-major and column-major ordering
  • foist::write_png() saving a 3D array in both row-major and column-major ordering
  • foist::write_png() saving a 2D matrix with an indexed colour palette
  • foist::write_gif() saving a 2D matrix with an indexed colour palette
  • png::writePNG() saving a 3D array
expression min median itr/sec mem_alloc
foist::write_pnm(dbl_arr, tmp) 17.7ms 20.36ms 48 2.49KB
foist::write_pnm(dbl_arr, tmp, convert_to_row_major = FALSE) 4.72ms 5.35ms 167 2.49KB
foist::write_png(dbl_arr, tmp) 18.93ms 20.68ms 48 2.49KB
foist::write_png(dbl_arr, tmp, convert_to_row_major = FALSE) 5.94ms 6.51ms 140 2.49KB
foist::write_png(dbl_mat, tmp, convert_to_row_major = FALSE, pal = foist::vir$magma) 2.08ms 2.5ms 370 2.49KB
foist::write_gif(dbl_mat, tmp, convert_to_row_major = FALSE, pal = foist::vir$magma) 1.9ms 2.23ms 340 2.49KB
png::writePNG(dbl_arr, tmp) 46.57ms 50.49ms 20 1.88MB

Benchmark results

Benchmark: Saving an RGB image vs JPEG

The following benchmark compares the time to output a colour image using:

  • foist::write_png() saving a 3D array in both row-major and column-major ordering
expression min median itr/sec mem_alloc
foist::write_png(dbl_arr, tmp) 19.41ms 21.93ms 44 2.49KB
foist::write_png(dbl_arr, tmp, convert_to_row_major = FALSE) 5.98ms 6.93ms 76 2.49KB
jpeg::writeJPEG(dbl_arr, tmp) 26.99ms 31.08ms 32 1.9MB

Benchmark results

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].