NAME

ETL::Yertl - ETL with a Shell

VERSION

version 0.044

STATUS

SYNOPSIS

### On a shell...
# Convert file to Yertl's format
$ yfrom csv file.csv >work.yml
$ yfrom json file.json >work.yml

# Convert file to output format
$ yto csv work.yml
$ yto json work.yml

# Parse HTTP logs into documents
$ ygrok '%{LOG.HTTP_COMMON}' httpd.log

# Read data from a database
$ ysql db_name 'SELECT * FROM employee'

# Write data to a database
$ ysql db_name 'INSERT INTO employee ( id, name ) VALUES ( $.id, $.name )'

### In Perl...
use ETL::Yertl;

# Give everyone a 5% raise
my $xform = file( '<', 'employees.yaml' )
          | transform( sub { $_->{salary} *= 1.05 } )
          >> stdout;
$xform->run;

DESCRIPTION

ETL::Yertl is an ETL (Extract, Transform, Load) for shells. It is designed to accept data from multiple formats (CSV, JSON), manipulate them using simple tools, and then convert them to an output format.

Yertl will have tools for:

Extracting data from databases (MySQL, Postgres, MongoDB)
Loading data into databases
Extracting data from web services
Writing data to web services
Distributing data through messaging APIs (ZeroMQ)

SUBROUTINES

stdin

my $stdin = stdin( %args );

Get a ETL::Yertl::FormatStream object for standard input. %args is a list of key/value pairs passed to "new" in ETL::Yertl::FormatStream. Useful keys are:

format

Specify the format that standard input is. Defaults to yaml or the value of YERTL_FORMAT (see "get_default" in ETL::Yertl::Format.

stdout

my $stdout = stdout( %args );

Get a ETL::Yertl::FormatStream object for standard output. %args is a list of key/value pairs passed to "new" in ETL::Yertl::FormatStream. Useful keys are:

format

Specify the format that standard input is. Defaults to yaml or the value of YERTL_FORMAT (see "get_default" in ETL::Yertl::Format.
autoflush

Immediately write documents to standard output to improve responsiveness, instead of queuing them for later writes for efficiency. This defaults to 1 (on). Set it to 0 to turn autoflush off.

transform

my $xform = transform( sub { ... } );
my $xform = transform( 'Local::Transform::Class' => @args );

Create a new ETL::Yertl::Transform object, passing in either a subref to transform documents, or a class to instantiate and arguments to pass to its constructor.

The subref will receive two arguments: The ETL::Yertl::Transform object and the document to transform. $_ will also be set to the document to transform. The subref should return the transformed document (either a new document, or the existing document after being modified).

If given a transform class, it should inherit from ETL::Yertl::Transform. The class will be loaded and an object instantiated using the @args.

file

my $stream = file( $mode, $path, %args );

Create a ETL::Yertl::FormatStream object for the given $path. $mode should be one of < for reading or > for writing. %args are additional arguments to pass to the ETL::Yertl::FormatStream constructor. Useful keys are:

format

Specify the format that standard input is. Defaults to yaml or the value of YERTL_FORMAT (see "get_default" in ETL::Yertl::Format.

yq

my $xform = yq( $filter );

Create a ETL::Yertl::Transform::Yq object with the given filter. See "SYNTAX" in yq for full filter syntax.

loop

my $loop = loop();

Get the IO::Async::Loop singleton. Use this to add other IO::Async objects to a larger program. This is not needed for simple Yertl streams, and is mostly used internally.

This is not exported by default. You can import it using use ETL::Yertl 'loop'.

AUTHOR

Doug Bell [email protected]

CONTRIBUTORS

James E Keenan [email protected]
Luke Triantafyllidis [email protected]

COPYRIGHT AND LICENSE

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

preaction / ETL-Yertl

Programming Languages