COVID-19 Mobility .pdf to .tsv

Google has now released the data in .csv files

If you need to parse the old .pdf files I recommend this implementation instead. This repo was an early implementation but is less robust, less up to date, and probably less accurate.

Download the .tsv data.

This script attempts to digitize the COVID-19 Mobility Reports produced by Google.

The start date is currently hardcoded to 2020-02-16, and the total number of days defaults to 43.

If the PDFs are re-downloaded and re-processed after they are updated, there's a good chance this script won't work, or (at least) the dates in the header will be incorrect.

To download the .pdf files, run the following command:

$ mkdir pdf && cd pdf && xargs -n 1 curl -O < ../urls.txt

To convert all the .pdf files to .tsv files, first make sure your python environment is ready:

• Python 3
• Python packages: numpy
• pdftocairo and pdftotext: sudo apt install poppler-utils

Then run the following commands to create an output folder and pipe all the data to the folder.

$ mkdir tsv
$ for f in pdf/*; do echo $f; python covid-pdf-to-tsv.py $f > tsv/`basename $f`.tsv; done

Operating principle

This script uses pdftocairo to convert to Postscript, and a regular expression to identify polylines. Polylines and ticks in different regions are categorized appropriately.

We use pdftotext to extract all the place names. The place names in the Postscript output are too obfuscated by layout and style commands for the extraction to work correctly.

Accuracy

Please visually confirm that this works for you. From examining a few hundred datapoints manually, there is less than a 0.5% error compared to the printed numbers. But there may be some edge cases where the extraction failed.