Open Traffic v2 platform

Contents

• Introduction
• Platform architecture
  • Overview diagram and description of components
  • More detailed component diagram
• System components
  • Basemap Generator
    • OSMLR
    • Valhalla routing graph tiles with OSMLR segment associations
  • Reporter
  • Datastore
  • Data Privacy and Competitive Concerns
  • Analyst User Interface
  • Routing engine
  • Stand-alone traffic maps

Introduction

Open Traffic is a global data platform to process anonymous positions of vehicles and smartphones into real-time and historical traffic statistics. We're building this in the open, using fully open-source software, with involvement from a growing list of partners.

After a successful proof-of-concept by the World Bank, Grab, and Conveyal (known as OTv1), Mapzen is working with the partners to build out a new version of the platform that will scale to have global coverage (known as OTv2). For more information on the partnerships behind Open Traffic and the OTv2 platform, see:

• The World Bank Launches New Open Transport Partnership to Improve Transportation through Open Data press release from the World Bank
• Open Traffic Data to Revolutionize Transport story from the World Bank
• World Bank + Mapzen + partners = open traffic data on the Mapzen blog

The following diagrams and documentation describe OTv2. For more information on OTv1, see the opentraffic/architecture repository.

Platform architecture

Overview diagram and description of components

The OTv2 platform is built of both distributed and centralized services:

From left to right the components in this diagram are:

1. Open Traffic Basemap Producer is a centralized service. On a regular basis, it ingests OpenStreetMap data and outputs the OSMLR segments against which traffic statistics are matched, reported, stored, and displayed.
2. Open Traffic Reporter is run by each organization that contributes probe data to Open Traffic. Each Reporter instance ingests GPS location streams, map-matches those locations against OSMLR segments, aggregates these locations into anonymous speed statistics.
3. Open Traffic Datastore receives and merges together the anonymous speed statistics from all of the Reporter instances. Datastore creates a variety of public data extracts from its historical records of traffic statistics, including space/time tile files to power the Analyst User Interface and routing graph tile files to power a traffic-influenced routing engine.
4. An instance of Valhalla, an open-source routing engine by Mapzen, serves as a routing engine.
5. Open Traffic Analyst User Interface serves as an easy-to-use view into Open Traffic's historical speed and observation count data, allowing basic querying by area, time, and route.

More detailed component diagram

The same as above, with more detail on the specific pieces within each component and the data flows between components:

System components

Basemap Generator

The Basemap Generator is a set of centralized processes that regularly create the road network data common to all other OTv2 components.

OSMLR

On a regular basis (likely quarter), the Basemap Generator runs the OSMLR application to create traffic segments according to the OSMLR tile spec. Outputs are in protocol buffer and GeoJSON formats. The resulting worldwide OSMLR tile set is available for free use by all through the Amazon Public Datasets program.

→ See this blog post and this blog post for an introduction to OSMLR.

→ To download OSMLR segment tiles, go to these S3 bucket listings.

Valhalla routing graph tiles with OSMLR segment associations

On a regularly basis (weekly), the Basemap Generator combines OpenStreetMap data with OSMLR segments to create a set of Valhalla routing graph tiles. These routing graph tiles are available to power Valhalla for map-matching purposes, as is done inside the Open Traffic Reporter.

Reporter

(In OTv1, this component was called the Traffic Engine.)

Reporter is run by each organization that is providing probe data to the Open Traffic platform. Reporter is distributed as a set of Docker containers, or can be run anywhere Java, Python, and Apache Kafka are available. Reporter can also be run as a script against historical speed archives, without Kafka.

Reporter performs the following steps:

1. Reformat incoming GPS-derived points to match the expected key names and structure
2. Collect points into a moving window. That is, a time-ordered slice of a single vehicle's positions.
3. Match the trajectory of points in the window to OSMLR segments, using the map-matching utilities provided by Valhalla Meili and the Valhalla routing graph tiles created by the Basemap Producer
4. Pools matches to ensure they pass privacy thresholds
5. When privacy thresholds are met, send resulting binned times and observation counts to the centralized Open Traffic Datastore

→ See more documentation in the Reporter repository.

→ The nature of map-matching and generating traffic statistics is probabilistic; noisy GPS input can produce variable results . See the Reporter Quality Testing Rig for more information on how to evaluate and tune a Reporter deployment. Also described in a series of blog posts: one, two, three, four.

Datastore

(In OTv1, this component was called the Data Pool. Its API was called the Traffic Engine App.)

The Datastore ingests input from distributed Reporter instances. It stores the aggregated statistics as histogram tile files.

From the histogram tile files, Datastore produces a variety of public data extracts:

• historical average speed tiles (.spd tiles) → documentation
• intersection delay and queue length tiles (.nex tiles) → documentation
• reference speed tiles (.ref tiles) → documentation
• coverage maps → documentation

→ See more documentation in the Datastore repository.

Data Privacy and Competitive Concerns

Note that the Datastore does not store any personally identifiable information. The contents of the histogram tile files have been aggregated such that it should not be possible to reconstruct the paths of any individuals.

The histogram tile files include the number of vehicles/drivers observed along each OSMLR segment. This volume information is important for weighting averages and other statistics. However, releasing to the public the exact number of vehicles/drivers along specific roads may reveal too much information about the business operations of Open Traffic's data providers. Therefore, public data extracts produced by the Datastore only include vehicle/driver counts as a binned "prevalence" index.

Analyst User Interface

(In OTv1, this component was the Traffic Engine App.)

The Analyst UI provides a simple way of performing basic exploration and querying of OTv2 data. It is powered by the Datastore's public data extracts, just by downloading tile files from AWS S3.

→ See more documentation in the Analyst UI repository.

Routing engine

Analyst UI uses the Valhalla routing engine to plan journeys between origins and one or more destinations. Analyst UI then uses the Datastore's public data extracts to calculate the ETA along these routes.

In the future, Valhalla may also offer journey-planning that is weighted by historical and/or real-time speeds.

→ See this documentation for more information on Valhalla's routing API.

→ See this documentation for more information on Valhalla's trace_attributes action, which is also used by Analyst UI.

→ See this blog post for a proof-of-concept using traffic speeds to influence Valhalla routing.

Stand-alone traffic maps

Datastore's public data extracts can be turned into traffic maps for display. While not formally part of the OTv2 platform, we are demonstrating future possibilities during the development process:

→ See this demo using the Tangram map rendering library to display an entire day's worth of traffic in Manila (from the OTv1 platform). Here is an animated screenshot of some of the map: