pypely

Make your data processing easy - build pipelines in a functional manner. In general this package will not make your code faster or necessarily make you write less code. The purpose of this package is to make you think differently about data processing.

You are encouraged to write your data processing step by step - each step being a function. By naming each step with great awareness and chaining them together you will receive a consise and descriptive scheme of the process. This should give you and your colleagues a nice overview on how the process is structured and makes it easy to understand. Addtionally you can test every small step easily.

Installation

pip install pypely

Usage

Use pypely to chain functions and structure your data processing code in a readable way.

use_pypely = pipeline(
    open_favourite_ide,
    create_new_conda_environment,
    activate_environment,
    install_pypely,
    have_fun_building_pipelines 
)

use_pypely() # -> 🥳

If you want to learn more check out the following links.

Links

• Documentation
• Examples
• Contributing

Why functional?

Functional programming is a data driven approach to building software - so let's move data to the center of our thinking when building data processing pipelines. To illustrate the idea a little more two analogies will be used.

Railway

The railway analogy used by Scott Wlaschin in this talk is a good way of looking at functional programming. With pypely you can easily build a route from start to finish without caring about the stops in between. 🚂

In this analogy you should translate:

• railway stop to intermediate result
• railway to tranformative function

Git

git branching might be an even easier analogy:

Our every day work is managed by git and hopefully you don't need to care about special commit hashes etc.. "Shouldn't it be the same for intermediate results in data processing?" 🤔 - "I guess I just care about raw data and processing results".

In this analogy you should translate:

• git commit to intermediate result
• you writing & commiting code to tranformative function

Cites by smart people (Who use functional programming)

"Design is separating into things that can be composed." - Rich Hickey

What can I use this for?

This may be the main question that should be answered. This library focuses on structuring data processing, so consider it for dataframes operations. There are two libraries that need to be mentioned:

• pandas
• pyspark

But ☝️.. if you want to build your whole application in a functional style, pypely provides you with the basics for this. So get creative 🤩

Contributing

If you want to contribute:

1. Woohoo! 🥳
2. Please check out the contribution guide.
3. See the issues to find a contribution possibility or create one to tell your plan and start a discussion.