EtlFlow

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

Documentation

Library Documentation https://tharwaninitin.github.io/etlflow/site/docs

Examples

You can use this library in different ways mentioned below.

• Core Module:
  Using this module you can use features of Step API into your project.
• Spark Module (spark steps):
  Using this addon module along with core you can use Apache Spark steps into your project.
• GCP Module (spark steps):
  Using this addon module along with core you can use GCP steps into your project.

Modules Dependency Graph

Scala Version Compatibility Matrix

Requirements and Installation

This project is compiled with scala versions 2.12.15, 2.13.8, 3.1.1 Available via maven central. Add the latest release as a dependency to your project

SBT

libraryDependencies += "com.github.tharwaninitin" %% "etlflow-core" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-spark" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-cloud" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-http" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-redis" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-aws" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-gcp" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-email" % "x.x.x"

Maven

<dependency>
    <groupId>com.github.tharwaninitin</groupId>
    <artifactId>etlflow-core_2.12</artifactId>
    <version>x.x.x</version>
</dependency>

Contributions

Please feel free to add issues to report any bugs or to propose new features.