An Introduction to DataFrames

Bogumił Kamiński, November 2020, 2020

The tutorial is for DataFrames 0.22.1

A brief introduction to basic usage of DataFrames.

The tutorial contains a specification of the project environment version under which it should be run. In order to prepare this environment, before using the tutorial notebooks, while in the project folder run the following command in the command line:

julia -e 'using Pkg; Pkg.activate("."); Pkg.instantiate()'

Tested under Julia 1.5.3. The project dependencies are the following:

  [69666777] Arrow v1.0.1
  [6e4b80f9] BenchmarkTools v0.5.0
  [336ed68f] CSV v0.8.2
  [324d7699] CategoricalArrays v0.9.0
  [944b1d66] CodecZlib v0.7.0
  [a93c6f00] DataFrames v0.22.1
  [1313f7d8] DataFramesMeta v0.6.0
  [5789e2e9] FileIO v1.4.4
  [da1fdf0e] FreqTables v0.4.2
  [7073ff75] IJulia v1.23.0
  [babc3d20] JDF v0.2.20
  [9da8a3cd] JLSO v2.4.0
  [b9914132] JSONTables v1.0.0
  [86f7a689] NamedArrays v0.9.4
  [b98c9c47] Pipe v1.3.0
  [2dfb63ee] PooledArrays v0.5.3
  [f3b207a7] StatsPlots v0.14.17
  [bd369af6] Tables v1.2.1
  [a5390f91] ZipFile v0.9.3
  [9a3f8284] Random
  [10745b16] Statistics

I will try to keep the material up to date as the packages evolve.

This tutorial covers DataFrames and CategoricalArrays, as they constitute the core of DataFrames along with selected file reading and writing packages.

In the last extras part mentions selected functionalities of selected useful packages that I find useful for data manipulation, currently those are: FreqTables, DataFramesMeta (pending its update to support DataFrames.jl 0.22 release), StatsPlots.

Setting up Jupyter Notebook for work with DataFrames.jl

By default Jupyter Notebook will limit the number of rows and columns when displaying a data frame to roughly fit the screen size (like in the REPL).

You can override this behavior by setting ENV["COLUMNS"] or ENV["LINES"] variables to hold the maximum width and height of output in characters respectively when running a notebook. Alternatively you can add the following entry "COLUMNS": "1000", "LINES": "100" to "env" variable in your Jupyter kernel file. See here for information about location and specification of Jupyter kernels.

TOC

Changelog:

Core functions summary

1. Constructors: DataFrame, DataFrame!, Tables.rowtable, Tables.columntable, Matrix, eachcol, eachrow, Tables.namedtupleiterator, empty, empty!
2. Getting summary: size, nrow, ncol, describe, names, eltypes, first, last, getindex, setindex!, @view, isapprox
3. Handling missing: missing (singleton instance of Missing), ismissing, nonmissingtype, skipmissing, replace, replace!, coalesce, allowmissing, disallowmissing, allowmissing!, completecases, dropmissing, dropmissing!, disallowmissing, disallowmissing!, passmissing
4. Loading and saving: CSV (package), CSVFiles (package), Serialization (module), CSV.read, CSV.write, save, load, serialize, deserialize, Arrow.write, Arrow.Table (from Arrow.jl package), JSONTables (package), arraytable, objecttable, jsontable, CodecZlib (module), GzipCompressorStream, GzipDecompressorStream, JDF.jl (package), JDF.savejdf, JDF.loadjdf, JLSO.jl (package), JLSO.save, JLSO.load, ZipFile.jl (package), ZipFile.reader, ZipFile.writer, ZipFile.addfile
5. Working with columns: rename, rename!, hcat, insertcols!, categorical!, columnindex, hasproperty, select, select!, transform, transform!, combine, Not, All, Between, ByRow, AsTable
6. Working with rows: sort!, sort, issorted, append!, vcat, push!, view, filter, filter!, delete!, unique, nonunique, unique!, repeat, parent, parentindices, flatten, @pipe (from Pipe package), only
7. Working with categorical: categorical, cut, isordered, ordered!, levels, unique, levels!, droplevels!, get, recode, recode!
8. Joining: innerjoin, leftjoin, rightjoin, outerjoin, semijoin, antijoin, crossjoin
9. Reshaping: stack, unstack
10. Transforming: groupby, mapcols, parent, groupcols, valuecols, groupindices, keys (for GroupedDataFrame), combine, select, select!, transform, transform!, @pipe (from Pipe package)
11. Extras:
    • FreqTables: freqtable, prop, Name
    • DataFramesMeta: @with, @where, @select, @transform, @orderby, @linq, @by, @combine, @eachrow, @newcol, ^, cols
    • StatsPlots: @df, plot, density, histogram,boxplot, violin