coiled-resources
Notebooks that support content like blogs and videos.
Project goals
- Make it easy for you to reproduce any computations covered in blogs or videos
- Show best practices for organizing a repo with hundreds of notebooks & tens of environments
Here are some notable blog posts that are backed by notebooks in this repo:
| Blog post | Notebook link |
|---|---|
| Speed up pandas query 10x | Notebook |
| Convert Dask DataFrame to pandas DataFrame | Notebook |
| Convert Parquet to CSV | Notebook |
Repo organization
This repo contains notebooks that are used in blogs and other content. The notebooks are cleanly organized, so you can easily find the notebook that corresponds to a blog post. For example, the blogs/save-numpy-dask-array-to-zarr.ipynb notebook corresponds with the coiled.io/blog/save-numpy-dask-array-to-zarr/ blog post. Notice how the notebook name aligns with the blog post URL.
The instructions for creating an environment to run each notebook are at the top of every notebook. The following setup instruction will work for most of the notebooks.
Setting up your machine
You can install the dependencies on your local machine to run these notebooks by creating a conda environment:
conda env create -f envs/crt-004.yml
crt stands for coiled-runtime, which pins a set of Dask runtime dependencies that are known to happily coexist.
Activate the environment with conda activate crt-004.
Open the project in your browser with jupyter lab.
Create Coiled software environments
To a Coiled software environment that matches you local environment, run a command like this: coiled env create -n crt-004 --conda envs/crt-004.yml.
Your Coiled sofware environment should always match your local environment exactly.
Here's how to create a cluster that uses the coiled-runtime software environment: cluster = coiled.Cluster(name="powers-crt-004", software="crt-004", n_workers=5).
Notebooks
Some of the notebooks are designed to run locally and others run on cloud machines via Coiled.
You can follow the Coiled getting started guide to get your machine setup. Coiled gives you some free credits, so you can easily try out the platform.
Some notebooks in this repo require conda environments with additional customization. You can find environment.yml files to build those environments in the respective directories.
Contributing
We welcome community contributions, especially MCVE analyses that others will find useful.
Feel free to create an issue and we'll be happy to brainstorm contributions.