grammarly / Perseverance

• Perseverance [[https://circleci.com/gh/grammarly/perseverance][https://circleci.com/gh/grammarly/perseverance.svg?style=svg]]

  Perseverance is a flexible retried operations library inspired by the Common Lisp's condition system. It decouples the logic of marking something as retriable from the decision on how it should be retried.

** Why?

There already exist Clojure libraries for retrying operations: [[https://github.com/liwp/again][again]], [[https://github.com/joegallo/robert-bruce][robert-bruce]]. These libraries require you to specify the retry strategy in the same place where something needs to be retried. Such urgency results in high-level decisions being made inside low-level code.

Should a function downloading a file automatically retry on failure? If it doesn't, the higher-level code cannot choose to retry this particular failed case. If it does retry automatically, how many attempts should it make? What if the higher-level code needs the function to fail fast and then try something else instead?

In many cases, the lower-level code has enough knowledge of how to do something (i.e., retry an operation) but doesn't know what to do (retry or not, with which delay). The high-level code knows what it wants, but doesn't know how to achieve it. Perseverance bridges these levels by separating the how's from what's.

The following code demonstrates a scenario where unreliable functions are reinforced with Perseverance:

#+BEGIN_SRC clojure (require '[perseverance.core :as p])

;; Fake function that returns a list of files but fails the first three times. (let [cnt (atom 0)] (defn list-s3-files [] (when (< @cnt 3) (swap! cnt inc) (throw (RuntimeException. "Failed to connect to S3."))) (range 10)))

;; Fake function that imitates downloading a file with 50/50 probability. (defn download-one-file [x] (if (> (rand) 0.5) (println (format "File #%d downloaded." x)) (throw (java.io.IOException. "Failed to download a file."))))

;; Let's wrap the previous function in retriable. (defn download-one-file-safe [x] (p/retriable {} (download-one-file x)))

;; Now to a function that downloads all files. (defn download-all-files [] (let [files (p/retriable {:catch [RuntimeException] :tag ::list-files} (list-s3-files))] (mapv download-one-file-safe files)))

;; Let's call it and see what happens. (download-all-files) ;; Unhandled java.lang.RuntimeException: Failed to connect to S3.

;; Bam! The exception still happened. It's because we haven't established ;; the retry context.

(p/retry {} (download-all-files))

;; java.lang.RuntimeException: Failed to connect to S3., retrying in 0.5 seconds... ;; java.lang.RuntimeException: Failed to connect to S3., retrying in 0.5 seconds... ;; java.lang.RuntimeException: Failed to connect to S3., retrying in 0.5 seconds... ;; java.io.IOException: Failed to download a file., retrying in 0.5 seconds... ;; File #0 downloaded. ;; File #1 downloaded. ;; java.io.IOException: Failed to download a file., retrying in 0.5 seconds... ;; File #2 downloaded. ;; File #3 downloaded. ;; java.io.IOException: Failed to download a file., retrying in 0.5 seconds... ;; java.io.IOException: Failed to download a file., retrying in 0.5 seconds... ;; File #4 downloaded. ;; ...

;; The call eventually succeeds! #+END_SRC

** Usage

Add this line to the list of your dependencies:

[[https://clojars.org/com.grammarly/perseverance][https://clojars.org/com.grammarly/perseverance/latest-version.svg]]

=perseverance.core= is the only namespace. It exposes two main macros: =retriable= and =retry=. The former is used to mark a piece of code as suitable for retrying. The arguments are =[options-map & body]=.

#+BEGIN_SRC clojure (retriable {:catch [RuntimeException] :tag ::list-files ;; :ex-wrapper #(ex-info "My wrapped exception". {:e %, :foo :bar}) } (list-s3-files)) #+END_SRC

Options map supports the following keys (all of them are optional):

• =:catch= --- should be a list of Exception classes that are going to be caught by =retriable=. The default value is =[java.io.IOException]=. Perseverance doesn't catch all exceptions intentionally to avoid retrying the errors that aren't IO-related, which would circumvent the proper error handling in your program. Yet you can always provide =:catch [Exception]= if you are sure that any potential exception inside is retriable.
• =:tag= --- attaches a keyword tag to the exceptions caught so that the outer =retry= macro can more accurately specify what it wants to retry.
• =:ex-wrapper= --- function that is called on the originally caught exception and should return a wrapped exception. This can be used for even more specific control of how each retriable block should be retried. If this option is present, =:tag= is ignored.

The =retry= macro has the same signature: =[options-map & body]=.

#+BEGIN_SRC clojure (retry {:strategy (constant-retry-strategy 100) :selector ::list-files ;; :selector #(and (instance? ExceptionInfo %) ;; (= (:foo (ex-data %)) :bar)) } (download-all-files)) #+END_SRC

The options-map can have the following keys:

• =:strategy= --- specifies the delay before each attempt and how many attempts at most should be taken. If not specified, a progressive strategy with default settings will be used. See [[#strategies][Strategies]] for details.
• =:selector= --- can be either a keyword or a predicate function. If it's a keyword, this =retry= block will control only the =retriable='s with the same tag. If it's a function, it will be called on the wrapped exception from =retriable=, and if that returns true, the retry will be performed. If =:selector= is not specified, the =retry= block will handle any underlying =retriable= error, no matter which tags it has.
• =:log-fn= --- function of =[wrapped-ex attempt delay]= called every time a retry is performed. By default, it prints the message to stdout. You can override the function with custom logging (or just silence it with a NOP).

With the help of selectors, you can nest =retry= blocks to specify different retry strategies for different retriable cases:

#+BEGIN_SRC clojure (retry {:strategy (constant-retry-strategy 500)} ;; Catches everything. (retry {:strategy (progressive-retry-strategy :initial-delay 2000, :max-delay 10000) :selector ::list-files} (download-all-files))) #+END_SRC

*** Strategies

Perseverance ships with two strategies (or, more specifically, strategy
constructors):

=constant-retry-strategy= takes a delay and returns the same delay on each
attempt. If =max-count= is provided, the strategy starts returning =nil= after
the number of attempts reaches =max-count=. Perseverance treats =nil= from a
strategy as a signal to stop retrying the operation.

=progressive-retry-strategy= is a fancy variation of exponential backoff
algorithm. It starts with =initial-delay= and returns it =stable-length=
times. Then for each next attempt, the delay is multiplied by =multiplier=
but cannot reach more than =max-delay=. After =max-count= attempts (if
provided), the strategy starts returning =nil=. For example, for this
strategy:

#+BEGIN_SRC clojure

(progressive-retry-strategy :initial-delay 1000, :stable-length 4, :multiplier 2, :max-delay 10000) #+END_SRC

the delays will be:

: 1000, 1000, 1000, 1000, 2000, 4000, 8000, 10000, 10000, 10000...

You can write custom strategies too. A strategy is a function that takes the
attempt number and returns a delay in milliseconds (or =nil= if retry
shouldn't be made). Attempts start from =1=, not zero.

** Drawbacks and considerations

Like any stack-based error-handling mechanism, Perseverance is susceptible to mistakes when used with multi-threaded, asynchronous, or lazily evaluated code. Perseverance is implemented on top of try/catch and Clojure's dynamic variables; so, you should be especially careful that the code inside =retriable= and =retry= doesn't escape the dynamic scope. Lately, some of the concurrency primitives (i.e., =future= and core.async's =go= blocks) started forwarding the dynamic bindings into their threads, but laziness still causes problems.

Taking away all the strategies and dynamic fanciness, Perseverance is just a dumb retrier. This is OK for requests that don't impact the other side of the communication, or if the actions are idempotent. But if you are making a call that must succeed only once, or you have to be sure that the retries don't make the outage in the system even worse, you might want to use a more sophisticated fault tolerance mechanism.

** License

© Copyright 2016 Grammarly, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.