All Projects → johnkerl → Miller

johnkerl / Miller

Licence: other
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Programming Languages

50402 projects - #5 most used programming language
31211 projects - #10 most used programming language
36898 projects - #4 most used programming language
139335 projects - #7 most used programming language
77523 projects
Vim Script
2826 projects

Projects that are alternatives of or similar to Miller

A terminal spreadsheet multitool for discovering and arranging data
Stars: ✭ 4,606 (-0.58%)
Mutual labels:  json, csv, tsv, devops-tools, tabular-data, unix-toolkit
Tsv Utils
eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.
Stars: ✭ 1,215 (-73.78%)
Mutual labels:  command-line, csv, tsv, statistics, tabular-data
🧚 pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.
Stars: ✭ 248 (-94.65%)
Mutual labels:  json, csv, tsv, data-processing
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Stars: ✭ 335 (-92.77%)
Mutual labels:  command-line, json, data-processing
Like Awk but with SQL and table joins
Stars: ✭ 263 (-94.32%)
Mutual labels:  json, csv, tsv
swiss-army knife for data
Stars: ✭ 275 (-94.06%)
Mutual labels:  json, csv, tsv
pytablewriter is a Python library to write a table in various formats: CSV / Elasticsearch / HTML / JavaScript / JSON / LaTeX / LDJSON / LTSV / Markdown / MediaWiki / NumPy / Excel / Pandas / Python / reStructuredText / SQLite / TOML / TSV.
Stars: ✭ 422 (-90.89%)
Mutual labels:  json, csv, tsv
Csv Parser
A modern C++ library for reading, writing, and analyzing CSV (and similar) files.
Stars: ✭ 359 (-92.25%)
Mutual labels:  json, csv, statistics
A cross-platform, efficient and practical CSV/TSV toolkit in Golang
Stars: ✭ 566 (-87.78%)
Mutual labels:  command-line, csv, tsv
Data Forge Ts
The JavaScript data transformation and analysis toolkit inspired by Pandas and LINQ.
Stars: ✭ 967 (-79.13%)
Mutual labels:  json, csv, data-cleaning
Structured Text Tools
A list of command line tools for manipulating structured text data
Stars: ✭ 6,180 (+33.39%)
Mutual labels:  json, csv, tsv
Faster Than Csv
Faster CSV on Python 3
Stars: ✭ 52 (-98.88%)
Mutual labels:  csv, tsv, tabular-data
Detects tabular data (spreadsheets, dsv or json, 20+ different formats) and emits normalized objects.
Stars: ✭ 34 (-99.27%)
Mutual labels:  tsv, csv, tabular-data
A CLI tool to convert CSV / Excel / HTML / JSON / Jupyter Notebook / LDJSON / LTSV / Markdown / SQLite / SSV / TSV / Google-Sheets to a SQLite database file.
Stars: ✭ 601 (-87.03%)
Mutual labels:  json, csv, tsv
q - Run SQL directly on CSV or TSV files
Stars: ✭ 8,809 (+90.14%)
Mutual labels:  command-line, csv, tsv
csvreader library / gem - read tabular data in the comma-separated values (csv) format the right way (uses best practices out-of-the-box with zero-configuration)
Stars: ✭ 169 (-96.35%)
Mutual labels:  json, csv, tabular-data
Advanced yet easy to use escaping library for Java
Stars: ✭ 207 (-95.53%)
Mutual labels:  json, csv
Bulk port forwarding Kubernetes services for local development.
Stars: ✭ 2,713 (-41.44%)
Mutual labels:  devops, devops-tools
Vscode Data Preview
Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files
Stars: ✭ 245 (-94.71%)
Mutual labels:  json, csv
Chronicle Wire
A Java Serialisation Library that supports multiple formats
Stars: ✭ 204 (-95.6%)
Mutual labels:  json, csv

Take the 2021 Miller User Survey!

What is Miller?

Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, tabular JSON and positionally-indexed.

What can Miller do for me?

With Miller, you get to use named fields without needing to count positional indices, using familiar formats such as CSV, TSV, JSON, and positionally-indexed. Then, on the fly, you can add new fields which are functions of existing fields, drop fields, sort, aggregate statistically, pretty-print, and more.


  • Miller operates on key-value-pair data while the familiar Unix tools operate on integer-indexed fields: if the natural data structure for the latter is the array, then Miller's natural data structure is the insertion-ordered hash map.

  • Miller handles a variety of data formats, including but not limited to the familiar CSV, TSV, and JSON. (Miller can handle positionally-indexed data too!)

In the above image you can see how Miller embraces the common themes of key-value-pair data in a variety of data formats.

Getting started

More documentation links


There's a good chance you can get Miller pre-built for your system:

Ubuntu Ubuntu 16.04 LTS Fedora Debian Gentoo

Pro-Linux Arch Linux


Anaconda Homebrew/MacOSX MacPorts/MacOSX Chocolatey

OS Installation command
Linux yum install miller
apt-get install miller
Mac brew install miller
port install miller
Windows choco install miller

See also building from source.

Build status

Go-port multi-platform build status

Building from source

  • make: takes just a few seconds and produces the Miller executable, which is ./mlr (or .\mlr.exe on Windows).
  • make check runs tests.
  • make install installs executable /usr/local/bin/mlr and manual page /usr/local/share/man/man1/mlr.1 (so you can do man mlr).
    • You can instead do ./configure --prefix=/some/install/path followed by make install if you want to install somewhere other than /usr/local.
  • See also the doc page on building from source.
  • For more developer information please see


License: BSD2



Thanks to all the fine people who help make Miller better by contributing commits/PRs! (Coming soon -- there will be an equally good way to honor all the fine people who contribute through issues and feature requests!)


  • Miller is multi-purpose: it's useful for data cleaning, data reduction, statistical reporting, devops, system administration, log-file processing, format conversion, and database-query post-processing.

  • You can use Miller to snarf and munge log-file data, including selecting out relevant substreams, then produce CSV format and load that into all-in-memory/data-frame utilities for further statistical and/or graphical processing.

  • Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data. While you can do basic statistics entirely in Miller, its streaming-data feature and single-pass algorithms enable you to reduce very large data sets.

  • Miller complements SQL databases: you can slice, dice, and reformat data on the client side on its way into or out of a database. You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.

  • Miller also goes beyond the classic Unix tools by stepping fully into our modern, no-SQL world: its essential record-heterogeneity property allows Miller to operate on data where records with different schema (field names) are interleaved.

  • Miller is streaming: most operations need only a single record in memory at a time, rather than ingesting all input before producing any output. For those operations which require deeper retention (sort, tac, stats1), Miller retains only as much data as needed. This means that whenever functionally possible, you can operate on files which are larger than your system’s available RAM, and you can use Miller in tail -f contexts.

  • Miller is pipe-friendly and interoperates with the Unix toolkit.

  • Miller's I/O formats include tabular pretty-printing, positionally indexed (Unix-toolkit style), CSV, JSON, and others.

  • Miller does conversion between formats.

  • Miller's processing is format-aware: e.g. CSV sort and tac keep header lines first.

  • Miller has high-throughput performance on par with the Unix toolkit.

  • Miller is written in portable, modern Go, with zero runtime dependencies. You can download or compile a single binary, scp it to a faraway machine, and expect it to work.

What people are saying about Miller

Today I discovered Miller—it's like jq but for CSV:

Also, "Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data." @GreatBlueC @nfmcclure

— Adrien Trouillaud (@adrienjt) September 24, 2020

Underappreciated swiss-army command-line chainsaw.

"Miller is like awk, sed, cut, join, and sort for [...] CSV, TSV, and [...] JSON."

— Dirk Eddelbuettel (@eddelbuettel) February 28, 2017

Miller looks like a great command line tool for working with CSV data. Sed, awk, cut, join all rolled into one:

— Mike Loukides (@mikeloukides) August 16, 2015

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV: - handy tool!

— Ilya Grigorik (@igrigorik) August 22, 2015

Btw, I think Miller is the best CLI tool to deal with CSV. I used to use this when I need to preprocess too big CSVs to load into R (now we have vroom, so such cases might be rare, though...)

— Hiroaki Yutani (@yutannihilat_en) April 21, 2020

Miller: a *format-aware* data munging tool By @__jo_ker__ to overcome limitations with *line-aware* workshorses like awk, sed et al

The project website is a fantastic example of good software documentation!!

— Donny Daniel (@dnnydnl) September 9, 2018

Holy holly data swiss army knife batman! How did noone suggest Miller for solving database cleaning / ETL issues to me before

Congrats to @__jo_ker__ for amazingly intuitive tool for critical data management tasks!#DataScienceandLaw #ComputationalLaw

— James Miller (@japanlawprof) June 12, 2018

🤯@__jo_ker__'s Miller easily reads, transforms, + writes all sorts of tabular data. It's standalone, fast, and built for streaming data (operating on one line at a time, so you can work on files larger than memory).

And the docs are dream. I've been reading them all morning!

— Benjamin Wolfe (he/him) (@BenjaminWolfe) September 9, 2021
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].