All Projects → oliyh → Superlifter

oliyh / Superlifter

Licence: epl-1.0
A DataLoader for Clojure/script

Programming Languages

clojure
4091 projects
clojurescript
191 projects

Projects that are alternatives of or similar to Superlifter

Hotchocolate
Welcome to the home of the Hot Chocolate GraphQL server for .NET, the Strawberry Shake GraphQL client for .NET and Banana Cake Pop the awesome Monaco based GraphQL IDE.
Stars: ✭ 3,009 (+2850%)
Mutual labels:  graphql, dataloader
Graphql Dataloader Boilerplate
Very simple boilerplate using GraphQL and DataLoader
Stars: ✭ 405 (+297.06%)
Mutual labels:  graphql, dataloader
Redux Requests
Declarative AJAX requests and automatic network state management for single-page applications
Stars: ✭ 330 (+223.53%)
Mutual labels:  graphql, fetch
Use Http
🐶 React hook for making isomorphic http requests
Stars: ✭ 2,066 (+1925.49%)
Mutual labels:  graphql, fetch
Batch Loader
⚡️ Powerful tool for avoiding N+1 DB or HTTP queries
Stars: ✭ 812 (+696.08%)
Mutual labels:  graphql, dataloader
Dataloader Php
DataLoaderPhp is a generic utility to be used as part of your application's data fetching layer to provide a simplified and consistent API over various remote data sources such as databases or web services via batching and caching.
Stars: ✭ 160 (+56.86%)
Mutual labels:  graphql, dataloader
React Query
⚛️ Hooks for fetching, caching and updating asynchronous data in React
Stars: ✭ 24,427 (+23848.04%)
Mutual labels:  graphql, fetch
Java Dataloader
A Java 8 port of Facebook DataLoader
Stars: ✭ 367 (+259.8%)
Mutual labels:  graphql, dataloader
Dataloader
Implementation of Facebook's DataLoader in Golang
Stars: ✭ 703 (+589.22%)
Mutual labels:  graphql, dataloader
Gqlgen
go generate based graphql server library
Stars: ✭ 6,880 (+6645.1%)
Mutual labels:  graphql, dataloader
Dataloader
DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
Stars: ✭ 11,040 (+10723.53%)
Mutual labels:  graphql, dataloader
Typeorm Graphql Loader
A query builder to easily resolve nested fields and relations for TypeORM-based GraphQL servers
Stars: ✭ 47 (-53.92%)
Mutual labels:  graphql, dataloader
Nestjs Example
NestJS example with GraphQL, Schema-Stitching, Dataloader, GraphQL Upload, RabbitMQ, Redis, Scalable Websocket and JWT authentication
Stars: ✭ 111 (+8.82%)
Mutual labels:  graphql, dataloader
Fetchql
GraphQL client with Fetch
Stars: ✭ 178 (+74.51%)
Mutual labels:  graphql, fetch
Apollo Fetch
🐶 Lightweight GraphQL client that supports middleware and afterware
Stars: ✭ 581 (+469.61%)
Mutual labels:  graphql, fetch
Graphql Serverless
Sample project to guide the use of GraphQL and Serverless Architecture.
Stars: ✭ 28 (-72.55%)
Mutual labels:  graphql, dataloader
Type Graphql Dataloader
TypeGraphQL + DataLoader + TypeORM made easy
Stars: ✭ 73 (-28.43%)
Mutual labels:  graphql, dataloader
Miraql
GraphQL performance monitoring & error-handling tool
Stars: ✭ 97 (-4.9%)
Mutual labels:  graphql
Hype Beats
Real-time Collaborative Beatbox with React & GraphQL
Stars: ✭ 100 (-1.96%)
Mutual labels:  graphql
Redux Cnode
react+react-router+redux+es6+antd-mobile+webpack版本的Cnode
Stars: ✭ 96 (-5.88%)
Mutual labels:  fetch

superlifter

Superlifter is an implementation of DataLoader for Clojure.

To quote from the DataLoader readme:

DataLoader allows you to decouple unrelated parts of your application without sacrificing the performance of batch data-loading. While the loader presents an API that loads individual values, all concurrent requests will be coalesced and presented to your batch loading function. This allows your application to safely distribute data fetching requirements throughout your application and maintain minimal outgoing data requests.

Superlifter uses Urania, a remote data access library for Clojure/script inspired by Haxl which in turn inspired DataLoader. Urania allows batching of similar fetches and deduplication via caching of identical fetches.

Superlifter adds smooth integration with libraries like lacinia, where GraphQL resolvers are run independently and must return data (or promises of data), leading to 1+n problems which can otherwise only be resolved by prefetching which complicates code.

The aim of superlifter is to provide a way of combining fetches delineated by time buckets, thresholds or explicit trigger rather than by node resolution.

As the underlying fetches are performed by Urania, knowledge of this library is required (it's very simple, though!).

Superlifter provides the following features:

  • Fast, simple implementation of DataLoader pattern
  • Bucketing by time or by queue size
  • Asynchronous fetching
  • Batching of fetches
  • Shared cache for all fetches in a session
    • Guarantees consistent results
    • Avoids duplicating work
  • Access to the cache allows longer-term persistence

Clojars Project

Vanilla usage

Start a superlifter as follows:

(require '[superlifter.api :as s])
(require '[urania.core :as u])

(def context (s/start! {:buckets {:default {:triggers {}}}}))

This superlifter has no triggers, and must be fetched manually. Other kinds of trigger include queue-size and interval (like DataLoader), detailed below. Remember to call (s/stop! context) when you have finished using it.

You can enqueue items for fetching:

(s/with-superlifter context
  (def hello-promise (s/enqueue! (u/value "Hello world"))))

When the fetch is triggered the promises will be delivered.

Triggering fetches

Regardless of the trigger used, you can always manually trigger a fetch of whatever is currently in the queue using (s/with-superlifter context (s/fetch!)). This returns a promise which is delivered when all the fetches in the queue are complete, containing the results of all the fetches.

On demand

In the example above no triggers were specified. Fetches will only happen when you call (s/with-superlifter context (s/fetch!)).

Queue size trigger

You can specify that the queue is fetched when the queue reaches a certain size. You can configure this to e.g. 10 using the following options:

{:triggers {:queue-size {:threshold 10}}}

Elastic trigger

You can specify that the queue is fetched when the queue size exceeds the threshold. The threshold can be updated dynamically and snaps back to zero when a fetch is performed, in contrast to the queue size trigger which remains at a fixed size. The trigger can be specified as follows:

{:triggers {:elastic {:threshold 0}}}

Interval trigger

You can specify that the queue is fetched every e.g. 100ms using the following options:

{:triggers {:interval {:interval 100}}}

This will give batching by time in a similar fashion to DataLoader.

Debounced trigger

You can specify that the queue is fetched when no items have been added within the last e.g. 100ms with these options

{:triggers {:debounced {:interval 100}}}

Your own trigger

You can register your own kind of trigger by participating the in s/start-trigger! multimethod, so you can listen for other kinds of events that might let you know when it's a good time to perform the fetch. See the interval trigger implementation for inspiration.

Trigger combinations

You can supply any number of triggers which will all run concurrently and the queue will be fetched when any one condition is met.

{:triggers {:queue-size {:threshold 10}
            :interval {:interval 100}}}

It is recommended that a :queue-size trigger is always used in combination with an :interval or debounced trigger in order to avoid hanging when you have e.g. a queue size of 5 but only four muses are enqueued within it.

Lacinia usage

Given the following schema in lacinia:

(def schema
 {:objects {:PetDetails {:fields {:name {:type 'String}
                                  :age {:type 'Int}}}
            :Pet {:fields {:id {:type 'String}
                           :details {:type :PetDetails
                                     :resolve resolve-pet-details}}}}
  :queries {:pets
            {:type '(list :Pet)
             :resolve resolve-pets}}})

Where the resolvers are as follows:

(defn- resolve-pets [context args parent]
  (let [ids (keys (:db context))]
    (map (fn [id] {:id id}) ids)))

;; invoked n times, once for every id from the parent resolver
(defn- resolve-pet-details [context args {:keys [id]}]
  (get-in (:db context) id))

We can rewrite this using superlifter (see the example code for full context):

;; superlifter.lacinia has a different `with-superlifter` macro
;; to help you return a lacinia promise
(require '[superlifter.lacinia :refer [with-superlifter]])
(require '[clojure.tools.logging :as log])

;; def-fetcher - a convenience macro like defrecord for things which cannot be combined
(s/def-fetcher FetchPets []
  (fn [_this env]
    (map (fn [id] {:id id}) (keys (:db env)))))

;; def-superfetcher - a convenience macro like defrecord for combinable things
(s/def-superfetcher FetchPet [id]
  (fn [many env]
    (log/info "Combining request for" (count many) "pets")
    (map (:db env) (map :id many))))

(defn- resolve-pets [context _args _parent]
  (with-superlifter context
    (-> (s/enqueue! (->FetchPets))
        (s/update-trigger! :pet-details :elastic
                           (fn [trigger-opts pet-ids]
                             (update trigger-opts :threshold + (count pet-ids)))))))

(defn- resolve-pet-details [context _args {:keys [id]}]
  (with-superlifter context
    (s/enqueue! :pet-details (->FetchPet id))))

It's usual to start a Superlifter before each query and stop it afterwards. There is an inject-superlifter interceptor which will help you do this:

(require '[com.walmartlabs.lacinia.pedestal :as lacinia])
(require '[com.walmartlabs.lacinia.schema :as schema])
(require '[superlifter.lacinia :refer [inject-superlifter]])

(def pet-db (atom {"abc-123" {:name "Lyra"
                              :age 11}
                   "def-234" {:name "Pantalaimon"
                              :age 11}
                   "ghi-345" {:name "Iorek"
                              :age 41}}))

(def lacinia-opts {:graphiql true})

(def superlifter-args
  {:buckets {:default {:triggers {:queue-size {:threshold 1}}}
             :pet-details {:triggers {:elastic {:threshold 0}}}}
   :urania-opts {:env {:db @pet-db}}})

(def service
  (lacinia/service-map
   (fn [] (schema/compile schema))
   (assoc lacinia-opts
          :interceptors (into [(inject-superlifter superlifter-args)]
                              (lacinia/default-interceptors (fn [] (schema/compile schema)) lacinia-opts)))))

Development

cider-jack-in-clj&cljs and open the test page http://localhost:9500/figwheel-extra-main/auto-testing

Build

CircleCI

License

Copyright © 2019 oliyh

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].