All Projects → scikit-hep → Awkward 1.0

scikit-hep / Awkward 1.0

Licence: bsd-3-clause
Manipulate JSON-like data with NumPy-like idioms.

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Awkward 1.0

data-analysis-using-python
Data Analysis Using Python: A Beginner’s Guide Featuring NYC Open Data
Stars: ✭ 81 (-60.1%)
Mutual labels:  numpy, pandas, data-analysis
Pytablewriter
pytablewriter is a Python library to write a table in various formats: CSV / Elasticsearch / HTML / JavaScript / JSON / LaTeX / LDJSON / LTSV / Markdown / MediaWiki / NumPy / Excel / Pandas / Python / reStructuredText / SQLite / TOML / TSV.
Stars: ✭ 422 (+107.88%)
Mutual labels:  json, pandas, numpy
visions
Type System for Data Analysis in Python
Stars: ✭ 136 (-33%)
Mutual labels:  numpy, pandas, data-analysis
Datscan
DatScan is an initiative to build an open-source CMS that will have the capability to solve any problem using data Analysis just with the help of various modules and a vast standardized module library
Stars: ✭ 13 (-93.6%)
Mutual labels:  numpy, pandas, data-analysis
100 Pandas Puzzles
100 data puzzles for pandas, ranging from short and simple to super tricky (60% complete)
Stars: ✭ 1,382 (+580.79%)
Mutual labels:  data-analysis, pandas, numpy
Data-Analyst-Nanodegree
Kai Sheng Teh - Udacity Data Analyst Nanodegree
Stars: ✭ 42 (-79.31%)
Mutual labels:  numpy, pandas, data-analysis
Ai Learn
人工智能学习路线图,整理近200个实战案例与项目,免费提供配套教材,零基础入门,就业实战!包括:Python,数学,机器学习,数据分析,深度学习,计算机视觉,自然语言处理,PyTorch tensorflow machine-learning,deep-learning data-analysis data-mining mathematics data-science artificial-intelligence python tensorflow tensorflow2 caffe keras pytorch algorithm numpy pandas matplotlib seaborn nlp cv等热门领域
Stars: ✭ 4,387 (+2061.08%)
Mutual labels:  data-analysis, pandas, numpy
Data-Science-Resources
A guide to getting started with Data Science and ML.
Stars: ✭ 17 (-91.63%)
Mutual labels:  numpy, pandas, data-analysis
Mlcourse.ai
Open Machine Learning Course
Stars: ✭ 7,963 (+3822.66%)
Mutual labels:  data-analysis, pandas, numpy
Data Forge Ts
The JavaScript data transformation and analysis toolkit inspired by Pandas and LINQ.
Stars: ✭ 967 (+376.35%)
Mutual labels:  json, data-analysis, pandas
Udacity-Data-Analyst-Nanodegree
Repository for the projects needed to complete the Data Analyst Nanodegree.
Stars: ✭ 31 (-84.73%)
Mutual labels:  numpy, pandas, data-analysis
Data Science Notebook
📖 每一个伟大的思想和行动都有一个微不足道的开始
Stars: ✭ 196 (-3.45%)
Mutual labels:  data-analysis, pandas, numpy
Data Science Hacks
Data Science Hacks consists of tips, tricks to help you become a better data scientist. Data science hacks are for all - beginner to advanced. Data science hacks consist of python, jupyter notebook, pandas hacks and so on.
Stars: ✭ 273 (+34.48%)
Mutual labels:  data-analysis, pandas, numpy
Pyda 2e Zh
📖 [译] 利用 Python 进行数据分析 · 第 2 版
Stars: ✭ 866 (+326.6%)
Mutual labels:  data-analysis, pandas, numpy
Seaborn Tutorial
This repository is my attempt to help Data Science aspirants gain necessary Data Visualization skills required to progress in their career. It includes all the types of plot offered by Seaborn, applied on random datasets.
Stars: ✭ 114 (-43.84%)
Mutual labels:  data-analysis, pandas, numpy
Data Analysis
主要是爬虫与数据分析项目总结,外加建模与机器学习,模型的评估。
Stars: ✭ 142 (-30.05%)
Mutual labels:  data-analysis, pandas, numpy
Pandas Datareader
Extract data from a wide range of Internet sources into a pandas DataFrame.
Stars: ✭ 2,183 (+975.37%)
Mutual labels:  data-analysis, pandas
Py
Repository to store sample python programs for python learning
Stars: ✭ 4,154 (+1946.31%)
Mutual labels:  pandas, numpy
Ta
Technical Analysis Library using Pandas and Numpy
Stars: ✭ 2,649 (+1204.93%)
Mutual labels:  pandas, numpy
Panthera
Data-frames & arrays on Clojure
Stars: ✭ 168 (-17.24%)
Mutual labels:  pandas, numpy

PyPI version Conda-Forge Python 2.7,3.5‒3.9 BSD-3 Clause License Continuous integration tests

Scikit-HEP NSF-1836650 DOI Documentation Gitter

Awkward Array is a library for nested, variable-sized data, including arbitrary-length lists, records, mixed types, and missing data, using NumPy-like idioms.

Arrays are dynamically typed, but operations on them are compiled and fast. Their behavior coincides with NumPy when array dimensions are regular and generalizes when they're not.

Motivating example

Given an array of objects with x, y fields and variable-length nested lists like

array = ak.Array([
    [{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}],
    [],
    [{"x": 4.4, "y": [1, 2, 3, 4]}, {"x": 5.5, "y": [1, 2, 3, 4, 5]}]
])

the following slices out the y values, drops the first element from each inner list, and runs NumPy's np.square function on everything that is left:

output = np.square(array["y", ..., 1:])

The result is

[
    [[], [4], [4, 9]],
    [],
    [[4, 9, 16], [4, 9, 16, 25]]
]

The equivalent using only Python is

output = []
for sublist in array:
    tmp1 = []
    for record in sublist:
        tmp2 = []
        for number in record["y"][1:]:
            tmp2.append(np.square(number))
        tmp1.append(tmp2)
    output.append(tmp1)

Not only is the expression using Awkward Arrays more concise, using idioms familiar from NumPy, but it's much faster and uses less memory.

For a similar problem 10 million times larger than the one above (on a single-threaded 2.2 GHz processor),

  • the Awkward Array one-liner takes 4.6 seconds to run and uses 2.1 GB of memory,
  • the equivalent using Python lists and dicts takes 138 seconds to run and uses 22 GB of memory.

Speed and memory factors in the double digits are common because we're replacing Python's dynamically typed, pointer-chasing virtual machine with type-specialized, precompiled routines on contiguous data. (In other words, for the same reasons as NumPy.) Even higher speedups are possible when Awkward Array is paired with Numba.

Our presentation at SciPy 2020 provides a good introduction, showing how to use these arrays in a real analysis.

Installation

Awkward Array can be installed from PyPI using pip:

pip install awkward

You will likely get a precompiled binary (wheel), depending on your operating system and Python version. If not, pip attempts to compile from source (which requires a C++ compiler, make, and CMake).

Awkward Array is also available using conda, which always installs a binary:

conda install -c conda-forge awkward

If you have already added conda-forge as a channel, the -c conda-forge is unnecessary. Adding the channel is recommended because it ensures that all of your packages use compatible versions:

conda config --add channels conda-forge
conda update --all

Getting help

How-to tutorials

Python API reference

C++ API reference

Installation for developers

Be sure to clone this repository recursively to get the header-only C++ dependencies.

git clone --recursive https://github.com/scikit-hep/awkward-1.0.git

Also be aware that the default branch is named main, not master, which could be important for pull requests from forks.

You can install it on your system with pip, which uses exactly the same procedure as deployment. This is recommended if you do not expect to change the code.

pip install .[test,dev]

Or you can build it locally for incremental development. The following reuses a local directory so that you only recompile what you've changed. This is recommended if you do expect to change the code.

python localbuild.py --pytest tests

The --pytest tests runs the integration tests from the tests directory (drop it to build only).

For more fine-grained testing, we also have tests of the low-level kernels, which can be invoked with

python dev/generate-tests.py
python -m pytest -vv -rs tests-spec
python -m pytest -vv -rs tests-cpu-kernels

Furthermore, if you have an Nvidia GPU, you can build and locally install the experimental CUDA plug-in with

pip uninstall -y awkward-cuda-kernels
python dev/generate-cuda.py
./cuda-build.sh --install

The --install does a local pip install on your system, which is the only way to use it. You can run its tests with

python dev/generate-tests.py
python -m pytest -vv -rs tests-cuda-kernels
python -m pytest -vv -rs tests-cuda

Using Awkward Array as a dependency

Python projects can simply import awkward.

C++ projects can link against the shared libraries libawkward-cpu-kernels.so and libawkward.so or their static library equivalents. These libraries are shipped, along with the include files, as part of pip's installation.

Roadmap

Since version 0.4.0, Awkward Array has been compiled with versions of pybind11 that have ABI version 4. Any other Python extension built with pybind11 ABI version 4 can consume and produce Awkward Arrays in the way described here.

The table below indicates when interface-breaking changes are planned; each is discussed in pull requests and issues. It doesn't include new additions that don't interfere with old behavior or corrections to bugs (i.e. behaviors that were never intended and fixed immediately). Each deprecated feature is announced by a FutureWarning that indicates the target removal version/date.

See release history for a detailed changelog of past releases. See projects for planning and prioritizing future fixes and features.

Version number Target/actual date Deprecated features removed in this version
1.0.0 2020-12-05 Broadcasting NumPy ufuncs through records (#457), lazy_cache="attach" option in ak.from_parquet (#576).
1.1.0 2021-02-09 Removed ak.to_arrayset/ak.from_arrayset in favor of ak.to_buffers/ak.from_buffers (#592).
1.2.0 2021-04-01
1.3.0 2021-06-01
1.4.0 2021-08-01
1.5.0 2021-10-01
1.6.0 2021-12-01
1.7.0 2022-02-01

Voting for fixes/features

You can vote for issues by adding a "thumbs up" (👍) using the "smile/pick your reaction" menu on the top-right of the issue. Issues with the most "thumbs ups" will be prioritized, though this isn't the only consideration determining when an issue gets addressed. (An estimate of how long it will take is also important.)

See the prioritized list of open issues.

Papers and talks about Awkward Array

Acknowledgements

Support for this work was provided by NSF cooperative agreement OAC-1836650 (IRIS-HEP), grant OAC-1450377 (DIANA/HEP) and PHY-1520942 (US-CMS LHC Ops).

We also thank Erez Shinan and the developers of the Lark standalone parser, which is used to parse type strings as type objects.

Thanks especially to the gracious help of Awkward Array contributors (including the original repository).


Jim Pivarski

💻 📖 🚇 🚧

Ianna Osborne

💻

Pratyush Das

💻

Anish Biswas

💻

glass-ships

💻 ⚠️

Henry Schreiner

💻 🚇

Nicholas Smith

💻 ⚠️

Lindsey Gray

💻 ⚠️

Ellipse0934

⚠️

Dmitry Kalinkin

🚇

Charles Escott

💻

Mason Proffitt

💻

Michael Hedges

💻

Jonas Rembser

💻

Jaydeep Nandi

💻

benkrikler

💻

bfis

💻

Doug Davis

💻

Joosep Pata

🤔

Martin Durant

🤔

Gordon Watts

🤔

Nikolai Hartmann

💻

Simon Perkins

💻

.hard

💻 ⚠️

💻: code, 📖: documentation, 🚇: infrastructure, 🚧: maintainance, ⚠: tests and feedback, 🤔: foundational ideas.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].