attr-gather
A gem for creating workflows that "enhance" entities with extra attributes. At a high level, attr-gather provides a process to fetch information from many data sources (such as third party APIs, legacy databases, etc.) in a fully parallelized fashion.
Usage
Defining your workflow
# define a workflow
class EnhanceProfile
include Attr::Gather::Workflow
# contains all the task implementations
container TasksContainer
# filter out invalid data using a Dry::Validation::Contract
# anything that doesn't match this schema will be filtered out
filter_with_contract do
params do
required(:user_id).filled(:integer)
optional(:user).hash do
optional(:name).filled(:string)
optional(:email).filled(:string)
optional(:gravatar).filled(:string)
optional(:email_info).hash do
optional(:deliverable).filled(:bool?)
optional(:free).filled(:bool?)
end
end
end
end
# each task returns a hash of data that will be merged into the result
task :fetch_post do |t|
t.depends_on = []
end
# will run in parallel
task :fetch_user do |t|
t.depends_on = [:fetch_post]
end
# will run in parallel
task :fetch_email_info do |t|
t.depends_on = [:fetch_user]
end
endDefining some tasks
class PostFetcher
def call(attrs)
res = HTTP.get("https://jsonplaceholder.typicode.com/posts/#{attrs[:id]}")
post = JSON.parse(res.to_s, symbolize_names: true)
{ title: post[:title], user_id: post[:userId], body: post[:body] }
end
endclass UserFetcher
# will have access to the PostFetcher attributes here
def call(attrs)
res = HTTP.get("https://jsonplaceholder.typicode.com/users/#{attrs[:user_id]}")
user = JSON.parse(res.to_s, symbolize_names: true)
{ user: { name: user[:name], email: user[:email] } }
end
endclass EmailInfoFetcher
# will have access to the PostFetcher attributes here
def call(user:)
res = HTTP.timeout(3).get("https://api.trumail.io/v2/lookups/json?email=#{user[:email]}")
info = JSON.parse(res.to_s, symbolize_names: true)
# will deep merge with the final result
{ user: { email_info: { deliverable: info[:deliverable], free: info[:free] } } }
end
endRegistering your tasks
class MyContainer
extend Dry::Container::Mixin
register :fetch_post, PostFetcher
register :fetch_user, UserFetcher
register :fetch_email_info, EmailInfoFetcher
endRun it!
enhancer = EnhanceUserProfile.new
enhancer.call(id: 12).value!And this is the result...
{
:id => 12,
:user_id => 2,
:user => {
:email => "[email protected]",
:name => "Ervin Howell",
:email_info => { :deliverable => true, :free => true },
:gravatar => "https://www.gravatar.com/avatar/241af7d19a0a7438794aef21e4e19b79"
}
}You can even preview it as an SVG!
enhancer.to_dot(preview: true) # requires graphviz (brew install graphviz)Features
- Offers DSL for defining workflows and merging the results from each task
- Execution engine optimally parallelizes the execution of the workflow dependency graph using concurrent-ruby Promises
- Very easy to unit test
- Ability to filter out bad/junky data using dry-validation contracts
What are the main difference between this Ruby project and similar ones?
- Operates on a single entity rather than a list, so easily adoptable in existing systems
- Focuses on the "fetching" and filtering of data solely, and not transformation or storage
- Focuses on having a clean PORO interface to make testing simple
- Provides a declarative interface for merging results from many sources (APIs, legacy databases, etc.) which allows for prioritization
Links
Examples
 |
|---|
| Example of workflow that enhances a blog post |
Installation
Add this line to your application's Gemfile:
gem 'attr-gather'And then execute:
$ bundle
Or install it yourself as:
$ gem install attr-gather
Development
After checking out the repo, run bin/setup to install dependencies. Then, run
rake spec to run the tests. You can also run bin/console for an interactive
prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install. To
release a new version, update the version number in version.rb, and then run
bundle exec rake release, which will create a git tag for the version, push
git commits and tags, and push the .gem file to
rubygems.org.
Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/ianks/attr-gather. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
License
The gem is available as open source under the terms of the MIT License.
Code of Conduct
Everyone interacting in the Attr::Gather project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.