Clash of the Lambdas

Microbenchmarking Stream APIs of the VM-based, multiparadigm languages: Java 8, Scala, C#, F# on Windows and on Linux.

The results are discussed in our ICOOOLPS'14 workshop paper:

http://arxiv.org/abs/1406.6631

To run the benchmarking suite as is, you will need a system with approximately 3GB of free space for heap allocation. Regarding execution time, a run on a single platform takes approximately 30-40 minutes on an Intel Core i5.

• Java 8 (jvm / jmh)
  • sequential
  • parallel
• Scala (jvm / jmh)
  • sequential
  • parallel
  • optimized sequential/parallel with ScalaBlitz
• C# (clr & mono / Lambda Microbenchmarking¹)
  • sequential
  • parallel
  • optimized sequential/parallel with LinqOptimizer
• F# (clr & mono / Lambda Microbenchmarking¹)
  • sequential
  • parallel
  • optimized sequential/parallel with LinqOptimizer

Setting up on Windows

• Install JDK8. Create the JAVA_HOME env variable (for maven to work) and add to Path the JAVA_HOME\bin directory
• Install Gnu Make for Windows and put the bin directory to PATH
• Extract Maven and put the bin directory to PATH
• Install the C#, F# compilers (if you have Visual Studio 2013 you already have them)
• Update the paths in Makefile

Setting up on Ubuntu

• Install/Extract JDK8 and update PATH
• sudo apt-get install make
• sudo apt-get install maven
• Follow the instructions on fsharp.org

Running the microbenchmarks

(via a console on Linux o via a Powershell console on Windows)

• Compile all tests with make
• Clean with make clean
• Compile a specific suite with make {java, csharp, fsharp, scala}
• Run microbenchmarks with make bench-{java, csharp, fsharp, scala} (e.g., make bench-java).

Processing the results

If you want to process the unified results we offer a gawk script for linux (that also makes use of the dos2unix command to convert windows result file encoding).

• Run the benchmark suite with on both windows and linux with make bench > results.{windows, linux} (any filename), or use it with tee to view the results as it runs with make bench | tee results.{windows, linux}
• See the unified results for all languages, platforms and benchmarks (the comma separated values are benchmark, mean, mean error, sdev) by running ./process results.linux results.windows > results.processed (make the script executable first).

Team

@biboudis, @NickPalladinos and Y. Smaragdakis

Footnotes:

1. A utility that was factored out as a seperate project.