All Projects → reidmorrison → rocketjob

reidmorrison / rocketjob

Licence: Apache-2.0 license
Ruby's missing background and batch processing system

Programming Languages

ruby
36898 projects - #4 most used programming language

Projects that are alternatives of or similar to rocketjob

Rocketjob
Ruby's missing background and batch processing system
Stars: ✭ 258 (-8.19%)
Mutual labels:  sidekiq, mongoid, compression, batch, batch-processing
taskinator
A simple orchestration library for running complex processes or workflows in Ruby
Stars: ✭ 25 (-91.1%)
Mutual labels:  sidekiq, resque, delayed-jobs
BAT FFMPEG
Batch script files for FFMPEG (Microsoft Windows and DOS, OS/2 🦄)
Stars: ✭ 104 (-62.99%)
Mutual labels:  batch, batch-file
spring-batch-rest
REST API for Spring Batch using Spring Boot 2.2
Stars: ✭ 85 (-69.75%)
Mutual labels:  batch, batch-processing
jobqueue
Jobqueue manages running and scheduling jobs (think Sidekiq or Resque for Go).
Stars: ✭ 37 (-86.83%)
Mutual labels:  sidekiq, resque
sidekiq-merger
Merge Sidekiq jobs
Stars: ✭ 49 (-82.56%)
Mutual labels:  sidekiq, batch
Sidekiq Batch
Sidekiq Batch Jobs Implementation
Stars: ✭ 233 (-17.08%)
Mutual labels:  sidekiq, batch
EverythingPortable
EverythingPortable
Stars: ✭ 59 (-79%)
Mutual labels:  batch, batch-file
Faast.js
Serverless batch computing made simple.
Stars: ✭ 1,323 (+370.82%)
Mutual labels:  batch, batch-processing
goroutines
It is an efficient, flexible, and lightweight goroutine pool. It provides an easy way to deal with concurrent tasks with limited resource.
Stars: ✭ 88 (-68.68%)
Mutual labels:  batch, batch-processing
corb2
MarkLogic tool for processing and reporting on content, enhanced from the original CoRB
Stars: ✭ 18 (-93.59%)
Mutual labels:  batch-job, batch-processing
rack-cargo
🚚 Batch requests for Rack apps (works with Rails, Sinatra, etc)
Stars: ✭ 17 (-93.95%)
Mutual labels:  batch, batch-processing
Excel Parser Processor
Simply does the tedious, repetitive operations for all rows of excel files step by step and reports after the job is done. It can download files from URL(s) in a column of Excel files. If a new filename is provided at column B it will rename the file before saving. It will even create sub folders if column C is full with a valid folder name.
Stars: ✭ 177 (-37.01%)
Mutual labels:  batch, batch-processing
Batch-Antivirus
Batch Antivirus, a powerful antivirus suite written in batch with real-time protection and heuristical scanning.
Stars: ✭ 26 (-90.75%)
Mutual labels:  batch, batch-file
Asakusafw
Asakusa Framework
Stars: ✭ 114 (-59.43%)
Mutual labels:  batch, batch-processing
aws-batch-example
Example use of AWS batch
Stars: ✭ 96 (-65.84%)
Mutual labels:  batch, batch-processing
Easy Batch
The simple, stupid batch framework for Java
Stars: ✭ 493 (+75.44%)
Mutual labels:  batch, batch-processing
Mkvtoolnix Batch
Windows Batch script to automate batch processing using mkvtoolnix.
Stars: ✭ 42 (-85.05%)
Mutual labels:  batch, batch-processing
python-batch-runner
A tiny framework for building batch applications as a collection of tasks in a workflow.
Stars: ✭ 22 (-92.17%)
Mutual labels:  batch, batch-processing
Windows-10-tweaks
This repo contains multiple scripts to optimize windows 10
Stars: ✭ 37 (-86.83%)
Mutual labels:  batch, batch-file

Rocket Job

Gem Version Downloads License

Ruby's missing batch system

Checkout https://rocketjob.io/

Rocket Job

Documentation

Support

Rocket Job v6

  • Support for Ruby v3 and Rails 6.
  • Major enhancements in Batch job support:
    • Direct built-in Tabular support for all input and output categories.
    • Multiple output file support, each with its own settings for:
      • Compression
        • GZip, Zip, BZip2 (Chunked for much faster loading into Apache Spark).
      • Encryption
        • PGP, Symmetric Encryption.
      • File format
        • CSV, PSV, JSON, Fixed Format, xlsx.
  • Significant error handling improvements, especially around throttle failures that used to result in "hanging" jobs.
  • Support AWS DocumentDB in addition to MongoDB as the data store.
  • Removed use of Symbols to meet Symbol deprecation in MongoDB and Mongoid.

Upgrading to Rocket Job v6

The following plugins have been deprecated and are no longer loaded by default.

  • RocketJob::Batch::Tabular::Input
  • RocketJob::Batch::Tabular::Output

If your code relies on these plugins and you still want to upgrade to Rocket Job v6, add the following require statement to any jobs that still use them:

require "rocket_job/batch/tabular"

It is important to migrate away from these plugins, since they will be removed in a future release.

Scheduled Jobs

For any scheduled jobs that include the RocketJob::Plugins::Cron plugin, the default behavior has changed so that the scheduled job instance is created immediately after the currently scheduled instance starts.

To maintain the old behavior of creating the job when it fails, aborts, or completes, add the following line to each of the applicable jobs:

self.cron_after_start = false

Additionally, scheduled jobs will now prevent a new one from being created when another scheduled instance of the same job is already queued, or running with the same cron_schedule.

To maintain the old behavior of allowing multiple instances with the same cron schedule, add the following line to each of the applicable jobs:

self.cron_singleton = false
Singleton

Since Scheduled jobs now implement their own singleton logic, remove the singleton plugin from any scheduled jobs.

Upgrading Batch Jobs to Rocket Job v6

Rocket Job v6 replaces the array of symbol type for input_categories and output_categories with an array of RocketJob::Category::Input and RocketJob::Category::Output.

Jobs that added or modified the input or output categories need to be upgraded. For example:

class MyJob < RocketJob::Job
  include RocketJob::Batch
  
  self.output_categories = [:main, :errors, :ignored]
end

Needs to be changed to:

class MyJob < RocketJob::Job
  include RocketJob::Batch
  
  output_category name: :main
  output_category name: :errors
  output_category name: :ignored
end
slice_size, encrypt, compress

These fields have been removed from the job itself:

class MyJob < RocketJob::Job
  include RocketJob::Batch

  self.slice_size = 1_000
  self.encrypt    = true
  self.compress   = true
end

They are now specified on the input_category as follows:

  • slice_size just moves under input_category.
  • encrypt becomes an option to serializer.
  • compress is now the default for all batch jobs so is not needed.

If the serializer is set to encrypt then it is automatically compressed.

class MyJob < RocketJob::Job
  include RocketJob::Batch

  input_category slice_size: 1_000, serializer: :encrypt
end
collect_output, collect_nil_output

The following fields have been moved from the job itself:

class MyJob < RocketJob::Job
  include RocketJob::Batch
  
  self.collect_output     = true
  self.collect_nil_output = true
end

Into the corresponding output_category:

  • collect_output no longer has any meaning. Output is collected anytime an output_category is defined.
  • collect_nil_output is now the option nils on the output_category. It defaults to falseso that by default anyniloutput from theperform` method is not collected.
class MyJob < RocketJob::Job
  include RocketJob::Batch

  output_category nils: true
end
name

For both input_category and output_category, when the name argument is not supplied it defaults to :main.

For Example:

class MyJob < RocketJob::Job
  include RocketJob::Batch

  input_category name: :main, serializer: :encrypt
  output_category name: :main
end

Is the same as:

class MyJob < RocketJob::Job
  include RocketJob::Batch

  input_category serializer: :encrypt
  output_category
end
Existing and inflight jobs

When migrating to Rocket Job 6, it is recommended to load every job and then save it back again as part of the deployment. When the job loads it will automatically convert itself from the old schema to the new v6 schema.

In flight jobs should not be affected, other than it is important to shutdown all running batch servers before running any new instances.

Rocket Job v4

Rocket Job Pro is now fully open source and included in Rocket Job under the Apache License.

The RocketJob::Batch plugin now adds batch processing capabilities to break up a single task into many concurrent workers processing slices of the entire job at the same time.

Example:

class MyJob < RocketJob::Job
  include RocketJob::Batch
  
  self.description         = "Reverse names"
  self.destroy_on_complete = false
  
  # Collect the output for this job in the default output category: `:main`
  output_category

  # Method to call by all available workers at the same time.
  # Reverse the characters for each line: 
  def perform(line)
    line.reverse
  end
end

Upload a file for processing, for example names.csv which could contain:

jack
jane
bill
john
blake
chris
dave
marc

To queue the above job for processing:

job = MyJob.new
job.upload('names.csv')
job.save!

Once the job has completed, download the results into a file:

job.download('names_reversed.csv')

Contributing to the documentation

To contribute to the documentation it is as easy as forking the repository and then editing the markdown pages directly via the github web interface.

For more complex documentation changes checkout the source code locally.

Local checkout

  • Fork the repository in github.
  • Checkout your fork of the source code locally.
  • Install Jekyll
    cd docs
    bundle update
  • Run Jekyll web server:
    jekyll serve
  • Open a web browser to view the local documentation: http://127.0.0.1:4000
  • Edit the files in the /docs folder.
  • Refresh the page to see the changes.

Once the changes are complete, submit a github pull request.

Upgrading to V3

V3 replaces MongoMapper with Mongoid which supports the latest MongoDB Ruby client driver.

Upgrading Mongo Config file

Replace mongo.yml with mongoid.yml.

Start with the sample mongoid.yml.

For more information on the new Mongoid config file.

Note: The rocketjob and rocketjob_slices clients in the above mongoid.yml file are required.

Other changes

  • Arguments are no longer supported, use fields for defining all named arguments for a job.

  • Replace usages of rocket_job do to set default values:

  rocket_job do |job|
    job.priority = 25
  end

With:

  self.priority = 25
  • Replace key with field when adding attributes to a job:
  key :inquiry_defaults, Hash

With:

  field :inquiry_defaults, type: Hash, default: {}
  • Replace usage of public_rocket_job_properties with the user_editable option:
field :priority, type: Integer, default: 50, user_editable: true

Ruby Support

Rocket Job is tested and supported on the following Ruby platforms:

  • Ruby 2.1, 2.2, 2.3, 2.4, and above
  • JRuby 9.0.5 and above

Dependencies

  • MongoDB
    • Persists job information.
    • Version 2.7 or greater.
  • Semantic Logger
    • Highly concurrent scalable logging.

Versioning

This project uses Semantic Versioning.

Author

Reid Morrison

Contributors

Contributors

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].