RobinMalfait / Lazy Collections

Lazy Collections

Fast and lazy collection operations.

Working with methods like .map(), .filter() and .reduce() is nice, however they create new arrays and everything is eagerly done before going to the next step.

This is where lazy collections come in, under the hood we use iterators and async iterators so that your data flows like a stream to have the optimal speed.

All functions should work with both iterator and asyncIterator, if one of the functions uses an asyncIterator (for example when you introduce delay(100)), don't forget to await the result!

const program = pipe(
  map(x => x * 2),
  filter(x => x % 4 === 0),
  filter(x => x % 100 === 0),
  filter(x => x % 400 === 0),
  toArray()
);

program(range(0, 1000000));

Table of Contents

• Benchmark
• API
  • Composing functions
    • compose
    • pipe
  • Known array functions
    • concat
    • every
    • filter
    • find
    • findIndex
    • map
    • reduce
    • reverse
    • some
  • Math / Statistics
    • average
    • max
    • min
    • sum
  • Utilities
    • chunk
    • compact
    • delay
    • flatten
    • generate
    • groupBy
    • head
    • partition
    • range
    • skip
    • slice
    • take
    • takeWhile
    • tap
    • toArray
    • unique
    • where
    • zip

Benchmark

⚠️ This is not a scientific benchmark, there are flaws with this. This is just meant to showcase the power of lazy-collections.

Memory data collected using: http://nodejs.org/api/process.html#process_process_memoryusage

import {
  pipe,
  range,
  filter,
  takeWhile,
  slice,
  toArray,
} from 'lazy-collections';

// Lazy example
const program = pipe(
  range(0, 10_000_000),
  filter(x => x % 100 === 0),
  filter(x => x % 4 === 0),
  filter(x => x % 400 === 0),
  takeWhile(x => x < 1_000),
  slice(0, 1_000),
  toArray()
);

program(); // [ 0, 400, 800 ]

// Eager example
function program() {
  return (
    // Equivalent of the range()
    [...new Array(10_000_000).keys()]
      .filter(x => x % 100 === 0)
      .filter(x => x % 4 === 0)
      .filter(x => x % 400 === 0)

      // Equivalent of the takeWhile
      .reduce((acc, current) => {
        return current < 1_000 ? (acc.push(current), acc) : acc;
      }, [])
      .slice(0, 1_000)
  );
}

program(); // [ 0, 400, 800 ]

This is actually a stupid non-real-world example. However, it is way more efficient at doing things. That said, yes you can optimize the eager example way more if you want to. You can combine the filter / reduce / .... However, what I want to achieve is that we can have separated logic in different filter or map steps without thinking about performance bottlenecks.

API

Composing functions

compose

Table of contents

We can use compose to compose functions together and return a new function which combines all other functions.

import { compose } from 'lazy-collections';

// Create a program (or a combination of functions)
const program = compose(fn1, fn2, fn3);

program();
// fn1(fn2(fn3()))

pipe

Table of contents

We can use pipe to compose functions together and return a new function which combines all other functions.

The difference between pipe and compose is the order of execution of the functions.

import { pipe } from 'lazy-collections';

// Create a program (or a combination of functions)
const program = pipe(fn1, fn2, fn3);

program();
// fn3(fn2(fn1()))

Known array functions

concat

Table of contents

Concat multiple iterators or arrays into a single iterator.

import { pipe, concat, toArray } from 'lazy-collections';

const program = pipe(
  concat([0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10]),
  toArray()
);

program();
// [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]

every

Table of contents

Should return true if all values match the predicate.

import { pipe, every } from 'lazy-collections';

const program = pipe(every(x => x === 2));

program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// false

filter

Table of contents

Filter out values that do not meet the condition.

import { pipe, filter, toArray } from 'lazy-collections';

const program = pipe(
  filter(x => x % 2 === 0),
  toArray()
);

program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// [ 2, 4, 6, 8, 10 ]

find

Table of contents

Find a value based on the given predicate.

import { pipe, find } from 'lazy-collections';

const program = pipe(find(x => x === 2));

program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// 2

findIndex

Table of contents

Find an index based on the given predicate.

import { pipe, findIndex } from 'lazy-collections';

const program = pipe(findIndex(x => x === 2));

program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// 2

map

Table of contents

Map a value from A to B.

import { pipe, map, toArray } from 'lazy-collections';

const program = pipe(
  map(x => x * 2),
  toArray()
);

program([1, 2, 3]);
// [ 2, 4, 6 ]

reduce

Table of contents

Reduce the data to a single value.

import { pipe, reduce } from 'lazy-collections';

const program = pipe(reduce((total, current) => total + current, 0));

program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// 55

reverse

Table of contents

Reverses the iterator.

note: This is currently very slow because it has to go through the full iterator first!

import { pipe, reverse, toArray } from 'lazy-collections';

const program = pipe(range(0, 5), reverse(), toArray());

program();
// [ 5, 4, 3, 2, 1, 0 ]

some

Table of contents

Should return true if some of the values match the predicate.

import { pipe, some } from 'lazy-collections';

const program = pipe(some(x => x === 2));

program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// true

Math / Statistics

average

Table of contents

Alias: mean

Gets the average of number of values.

import { pipe, average, toArray } from 'lazy-collections';

const program = pipe(average());

program([6, 7, 8, 9, 10]);
// 8

max

Table of contents

Find the maximum value of the given list

import { pipe, range, max } from 'lazy-collections';

const program = pipe(range(0, 5), max());

program();
// 5

min

Table of contents

Find the minimum value of the given list

import { pipe, range, min } from 'lazy-collections';

const program = pipe(range(5, 10), min());

program();
// 5

sum

Table of contents

Should sum an array or iterator.

import { pipe, sum } from 'lazy-collections';

const program = pipe(sum());

program([1, 1, 2, 3, 2, 4, 5]);
// 18

Utilities

chunk

Table of contents

Chunk the data into pieces of a certain size.

import { pipe, chunk, toArray } from 'lazy-collections';

const program = pipe(chunk(3), toArray());

program([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
// [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ], [ 10 ] ];

compact

Table of contents

Filters out all falsey values.

import { pipe, compact, toArray } from 'lazy-collections';

const program = pipe(compact(), toArray());

program([0, 1, true, false, null, undefined, '', 'test', NaN]);
// [ 1, true, 'test' ];

delay

Table of contents

Will make he whole program async. It will add a delay of x milliseconds when an item goes through the stream.

import { pipe, range, delay, map, toArray } from 'lazy-collections';

const program = pipe(
  range(0, 4),
  delay(5000), // 5 seconds
  map(() => new Date().toLocaleTimeString()),
  toArray()
);

await program();
// [ '10:00:00', '10:00:05', '10:00:10', '10:00:15', '10:00:20' ];

flatten

Table of contents

By default we will flatten recursively deep.

import { pipe, flatten, toArray } from 'lazy-collections';

const program = pipe(flatten(), toArray());

program([1, 2, 3, [4, 5, 6, [7, 8], 9, 10]]);
// [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]

But you can also just flatten shallowly

import { pipe, flatten, toArray } from 'lazy-collections';

const program = pipe(flatten({ shallow: true }), toArray());

program([1, 2, 3, [4, 5, 6, [7, 8], 9, 10]]);
// [ 1, 2, 3, 4, 5, 6, [ 7, 8 ], 9, 10 ]

generate

Table of contents

Generate accepts a function that function will be called over and over again. Don't forget to combine this with a function that ensures that the data stream will end. For example, you can use take, takeWhile or slice.

import { pipe, generate, take, toArray } from 'lazy-collections';

const program = pipe(generate(Math.random), take(3), toArray());

program();
// [ 0.7495421596380878, 0.09819118640607383, 0.2453718461872143 ]

groupBy

Table of contents

Groups the iterator to an object, using the keySelector function.

import { pipe, groupBy, range } from 'lazy-collections';

// A function that will map the value to the nearest multitude. In this example
// we will map values to the nearest multitude of 5. So that we can group by
// this value.
function snap(multitude: number, value: number) {
  return Math.ceil(value / multitude) * multitude;
}

const program = pipe(
  range(0, 10),
  groupBy((x: number) => snap(5, x))
);

program();
// {
//   0: [0],
//   5: [1, 2, 3, 4, 5],
//   10: [6, 7, 8, 9, 10],
// }

head

Table of contents

Alias: first

Gets the first value of the array / iterator. Returns undefined if there is no value.

import { pipe, chunk, toArray } from 'lazy-collections';

const program = pipe(head());

program([6, 7, 8, 9, 10]);
// 6

partition

Table of contents

Partition data into 2 groups based on the predicate.

import { pipe, partition, range, toArray } from 'lazy-collections';

const program = pipe(
  range(1, 4),
  partition(x => x % 2 !== 0),
  toArray()
);

program();
// [ [ 1, 3 ], [ 2, 4 ] ]

range

Table of contents

Create a range of data using a lowerbound, upperbound and step. The step is optional and defaults to 1.

import { pipe, range, toArray } from 'lazy-collections';

const program = pipe(range(5, 20, 5), toArray());

program();
// [ 5, 10, 15, 20 ]

skip

Table of contents

Allows you to skip X values of the input.

import { pipe, range, skip, toArray } from 'lazy-collections';

const program = pipe(range(0, 10), skip(3), toArray());

program();
// [ 4, 5, 6, 7, 8, 9, 10 ]

slice

Table of contents

Slice a certain portion from your data set. It accepts a start index and an end index.

import { pipe, range, slice, toArray } from 'lazy-collections';

const program = pipe(range(0, 10), slice(3, 5), toArray());

program();
// [ 3, 4, 5 ]

// Without the slice this would have generated
// [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]

take

Table of contents

Allows you to take X values of the input.

import { pipe, range, take, toArray } from 'lazy-collections';

const program = pipe(range(0, 10), take(3), toArray());

program();
// [ 1, 2, 3 ]

takeWhile

Table of contents

This is similar to take, but instead of a number as a value it takes a function as a condition.

import { pipe, range, takeWhile, toArray } from 'lazy-collections';

const program = pipe(
  range(0, 10),
  takeWhile(x => x < 5),
  toArray()
);

program();
// [ 0, 1, 2, 3, 4 ]

tap

Table of contents

Allows you to tap into the stream, this way you can intercept each value.

import { pipe, range, tap, toArray } from 'lazy-collections';

const program = pipe(
  range(0, 5),
  tap(x => {
    console.log('x:', x);
  }),
  toArray()
);

program();
// x: 0
// x: 1
// x: 2
// x: 3
// x: 4
// x: 5
// [ 0, 1, 2, 3, 4, 5 ]

toArray

Table of contents

Converts an array or an iterator to an actual array.

import { pipe, range, toArray } from 'lazy-collections';

const program = pipe(range(0, 10), toArray());

program();
// [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]

unique

Table of contents

Make your data unique.

import { pipe, unique, toArray } from 'lazy-collections';

const program = pipe(unique(), toArray());

program([1, 1, 2, 3, 2, 4, 5]);
// [ 1, 2, 3, 4, 5 ]

where

Table of contents

Filter out values based on the given properties.

import { pipe, where, range, map, where, toArray } from 'lazy-collections';

const program = pipe(
  range(15, 20),
  map(age => ({ age })),
  where({ age: 18 }),
  toArray()
);

program();
// [ { age: 18 } ]

zip

Table of contents

Zips multiple arrays / iterators together.

import { pipe, zip, toArray } from 'lazy-collections';

const program = pipe(zip(), toArray());

program([
  [0, 1, 2],
  ['A', 'B', 'C'],
]);
// [ [ 0, 'A' ], [ 1, 'B' ], [ 2, 'C' ] ]