karma-benchmark

A Karma plugin to run Benchmark.js over multiple browsers with CI compatible output.

Table of Contents

• 🌩 Installation
• 🕵🏾‍♀️ Reporters
• ⚖️ Configuration
• 👩🏻‍🔬 Writing Benchmarks
• 🙋🏾‍♂️ Getting Help
• 👀 Other Projects
• 🤓 Author

🌩 Installation

npm install --save-dev benchmark karma-benchmark

🕵🏾‍♀️ Reporters

• karma-benchmark-json-reporter by @etpinard
• karma-benchmark-plotly-reporter by @etpinard
• karma-benchmarkjs-reporter by @FormidableLabs
• karma-benchmark-reporter by @lazd

⚖️ Configuration

In karma.conf.js, add 'benchmark' to the list of frameworks:

module.exports = config => {
  config.set({
    autoWatch: false,
    browsers: ["Chrome"],
    concurrency: 1,
    files: ["bench/**/*.bench.js"],
    frameworks: ["benchmark"],
    singleRun: true
  });
};

Terminal Reporting

Now let's add karma-benchmarkjs-reporter by @FormidableLabs to report results to the Terminal:

npm install --save-dev karma-benchmarkjs-reporter

In karma.conf.js, add 'benchmark' to the list of reporters:

module.exports = (config) => {
  config.set({
    autoWatch: false,
    browsers: ['Chrome'],
    concurrency: 1,
    files: ['bench/**/*.bench.js'],
    frameworks: ['benchmark'],
+   reporters: ['benchmark'],
    singleRun: true
  });
};

JUnit Reporting

To feed our data into Continuous Integration, we can use the karma-junit-reporter.

npm install --save-dev karma-junit-reporter

In karma.conf.js, add junit to the list of reporters and configure the reporter accordingly:

module.exports = (config) => {
  config.set({
    autoWatch: false,
    browsers: ['Chrome'],
    concurrency: 1,
    files: ['bench/**/*.bench.js'],
    frameworks: ['benchmark'],
+   junitReporter: {
+     outputDir: 'reports',
+     outputFile: 'benchmark.xml'
+   },
-   reporters: ['benchmark'],
+   reporters: ['benchmark', 'junit'],
    singleRun: true
  });
};

Data Visualisation Reporting

With a free plot.ly account, we can generate visualisations using the karma-benchmark-plotly-reporter by @etpinard.

npm install --save-dev karma-benchmark-plotly-reporter

In karma.conf.js, add benchmark-plotly to the list of reporters and configure the reporter accordingly:

module.exports = (config) => {
  config.set({
    autoWatch: false,
+   benchmarkPlotlyReporter: {
+     username: '<your username>',
+     apiKey: '<your api key>',
+     cloudFilename: 'plotly-example',
+     imageFilename: 'plotly-example.png'
+   },
    browsers: ['Chrome'],
    concurrency: 1,
    files: ['bench/**/*.bench.js'],
    frameworks: ['benchmark'],
    junitReporter: {
      outputDir: 'reports',
      outputFile: 'benchmark.xml'
    },
-   reporters: ['benchmark', 'junit'],
+   reporters: ['benchmark', 'benchmark-plotly', 'junit'],
    singleRun: true
  });
};

👩🏻‍🔬 Writing Benchmarks

Benchmarks can be written by using the original Benchmark.js API, but a wrapper API is also provided by karma-benchmark in the form of the suite and benchmark globals. The karma-benchmark API aims to make the process of writing Benchmarks feel familiar to users of Jasmine or Jest.

In this example, a suite is defined that pits _.each against the native Array.forEach method:

suite("Array iteration", () => {
  benchmark("_.each", () => {
    _.each([1, 2, 3], el => {
      return el;
    });
  });

  benchmark("native forEach", () => {
    [1, 2, 3].forEach(el => {
      return el;
    });
  });
});

Suite options

Suite options are the same as in Benchmark.js with one exception: onStart and onComplete can be set at the suite level.

See the Benchmark.js Suite constructor API docs for a full list of options.

suite(
  "Array iteration",
  () => {
    benchmark("_.each", () => {
      _.each(this.list, number => {
        return number;
      });
    });

    benchmark("native forEach", () => {
      this.list.forEach(number => {
        return number;
      });
    });
  },
  {
    onCycle(event) {
      var suite = this;
      var benchmark = event.target;
      console.log("Cycle completed for " + suite.name + ": " + benchmark.name);
    },
    onStart() {
      this.list = [5, 4, 3];
    },
    onComplete() {
      this.list = null;
    }
  }
);

Benchmark options

Benchmark options are the same as in Benchmark.js. If setup and teardown are passed to benchmark(), they will override setup and teardown from the suite. Pass null or undefined to remove them.

See the Benchmark.js Benchmark constructor API docs for a full list of options.

suite("Iteration", () => {
  benchmark(
    "_.each with array",
    () => {
      _.each(this.list, number => {
        return number;
      });
    },
    {
      setup() {
        this.list = ["a", "b", "c"];
      },
      teardown() {
        delete this.list;
      }
    }
  );

  benchmark(
    "_.each with object",
    () => {
      _.each(this.list, number => {
        return number;
      });
    },
    {
      setup() {
        this.list = { 0: "a", 1: "b", 2: "c" };
      },
      teardown() {
        delete this.list;
      }
    }
  );
});

Running only a specific benchmark or suite

To run only a specific benchmark, use benchmark.only() or bbenchmark() instead of benchmark():

benchmark.only(() => {
  // Only this benchmark will run
  // bbenchmark() does the same thing
});

benchmark(() => {
  // This benchmark won't run
});

The same applies to suites with suite.only() and ssuite().

Skipping benchmarks & suites

To skip a benchmark, use benchmark.skip() or xbenchmark() instead of benchmark():

benchmark.skip(() => {
  // This benchmark won't run
  // xbenchmark() does the same thing
});

benchmark(() => {
  // This and all other benchmarks will run
});

The same applies to suites with suite.skip() and xsuite().

🙋🏾‍♂️ Getting Help

Get help with issues by creating a Bug Report or discuss ideas by opening a Feature Request.

👀 Other Projects

If you find my Open Source projects useful, please share them ❤️

• eslint-formatter-git-log
  ESLint Formatter featuring Git Author, Date, and Hash
• eslint-plugin-move-files
  Move and rename files while keeping imports up to date
• eslint-plugin-prefer-arrow-functions
  Convert functions to arrow functions
• ImageOptim-CLI
  Automates ImageOptim, ImageAlpha, and JPEGmini for Mac to make batch optimisation of images part of your automated build process.
• Jasmine-Matchers
  Write Beautiful Specs with Custom Matchers
• self-help
  Interactive Q&A Guides for Web and the Command Line
• syncpack
  Manage multiple package.json files, such as in Lerna Monorepos and Yarn Workspaces

🤓 Author

I'm Jamie Mason from Leeds in England, I began Web Design and Development in 1999 and have been Contracting and offering Consultancy as Fold Left Ltd since 2012. Who I've worked with includes Sky Sports, Sky Bet, Sky Poker, The Premier League, William Hill, Shell, Betfair, and Football Clubs including Leeds United, Spurs, West Ham, Arsenal, and more.