All Projects → SolaceLabs → covid19-stream-processors

SolaceLabs / covid19-stream-processors

Licence: Apache-2.0 license
Stream Information & Example Applications for Processing JHU and CovidTracking.com COVID-19 data available as streams over Solace

Programming Languages

java
68154 projects - #9 most used programming language

Projects that are alternatives of or similar to covid19-stream-processors

covid-19-image-repository
Anonymized dataset of COVID-19 cases with a focus on radiological imaging. This includes images (x-ray / ct) with extensive metadata, such as admission-, ICU-, laboratory-, and patient master-data.
Stars: ✭ 42 (+20%)
Mutual labels:  covid-19, covid19, covid19-data
covid19africa
Africa open COVID-19 data working group
Stars: ✭ 47 (+34.29%)
Mutual labels:  covid-19, covid19, covid19-data
open-data-covid-19
Open Data Repository for the Covid-19 dataset.
Stars: ✭ 19 (-45.71%)
Mutual labels:  covid-19, covid19, covid19-data
cl-covid19
Explore COVID-19 data with Common Lisp, gnuplot, SQL and Grafana
Stars: ✭ 51 (+45.71%)
Mutual labels:  covid-19, covid19, covid19-data
coronavirus-data
This repository contains data on Coronavirus Disease 2019 (COVID-19) in New York City (NYC), from the NYC Department of Health and Mental Hygiene.
Stars: ✭ 926 (+2545.71%)
Mutual labels:  covid-19, covid19, covid19-data
covid-dashboard
Help welcomed if you have expertise in public health web technology, data modeling and munging, or visualization.
Stars: ✭ 106 (+202.86%)
Mutual labels:  covid-19, covid19, covid19-data
data
Collecting and organising COVID-19 data for Slovenia as they come in from various sources
Stars: ✭ 20 (-42.86%)
Mutual labels:  covid-19, covid19, covid19-data
covid19gr
Open Data Aggregation & Knowledge Base Repository for the evolution of the SARS-COV-2 pandemic in Greece.
Stars: ✭ 21 (-40%)
Mutual labels:  covid-19, covid19, covid19-data
covid19-pr-api
COVID-19 Open API for Datasets in Puerto Rico
Stars: ✭ 21 (-40%)
Mutual labels:  covid-19, covid19, covid19-data
PhoNER COVID19
COVID-19 Named Entity Recognition for Vietnamese (NAACL 2021)
Stars: ✭ 55 (+57.14%)
Mutual labels:  covid-19, covid19, covid19-data
covid19-florida
Florida COVID19 Data parsed from Florida DOH Dashboard and PDF reports
Stars: ✭ 32 (-8.57%)
Mutual labels:  covid-19, covid19, covid19-data
covid-19
An app made with Flutter to track COVID-19 case counts.
Stars: ✭ 47 (+34.29%)
Mutual labels:  covid-19, covid19, covid19-data
COVID-19-Datasets
Novel Coronavirus (COVID-19) Cases for India, provided by University of Kalyani.
Stars: ✭ 19 (-45.71%)
Mutual labels:  covid-19, covid19, covid19-data
COVID-19-DETECTION
Detect Covid-19 with Chest X-Ray Data
Stars: ✭ 43 (+22.86%)
Mutual labels:  covid-19, covid19, covid19-data
COVID-19-AI
Collection of AI resources to fight against Coronavirus (COVID-19)
Stars: ✭ 25 (-28.57%)
Mutual labels:  covid-19, covid19, covid19-data
COVID19
A web app to display the live graphical state-wise reported corona cases in India so far. It also shows the latest news for COVID-19. Stay Home, Stay Safe!
Stars: ✭ 122 (+248.57%)
Mutual labels:  covid-19, covid19, covid19-data
us-covid19
Data repository of State's Health Department stats for COVID19 in the United States
Stars: ✭ 37 (+5.71%)
Mutual labels:  covid-19, covid19, covid19-data
COVID19Py
A tiny Python package for easy access to up-to-date Coronavirus (COVID-19, SARS-CoV-2) cases data.
Stars: ✭ 86 (+145.71%)
Mutual labels:  covid-19, covid19, covid19-data
covid19-timeseries
Covid19 timeseries data store
Stars: ✭ 38 (+8.57%)
Mutual labels:  covid-19, covid19, covid19-data
coronainfobd
Real-time corona-virus tracker of Bangladesh 🇧🇩 which includes latest updates, data visualization, public awareness from WHO and some advice to aware people. 🥰❤
Stars: ✭ 46 (+31.43%)
Mutual labels:  covid-19, covid19, covid19-data
NOTICE The streams below are no longer continuously running on a publicly available broker. If you would like to use them you can spin up a Solace Broker for free in Solace Cloud and run the Publisher apps in the Stream Processors section to publish the event streams.

Table of Contents

Overview & Data Sources

This repo provides streams information and example applications on how to consume COVID19 related topics aggregated from different data sources. Below is a list of the different data sources used to feed the published streams.

The Covid Tracking Project

A team from the The Atlantic launched The Covid Tracking Project to collect and publish testing data available for US states and territories.

John Hopkins University

A team at Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE) has developed this interactive web-based dashboard to provide researchers, public health authorities, and the general public with a user-friendly tool to track the COVID-19 pandemic as it unfolds. Their team has also been nice enough to periodically (usually once or twice a day) upload the data to this Github Repo which has been very popular in the developer community; but has led to developers wanting to receive updates in a more efficient manner. To further their teams’ efforts, help the community, and to “do our part”, we at Solace have created an application that polls the feature service, looks for differences in the data, and publishes updates into an Event Broker we are making available for public use. This allows anyone to consume data updates in near real-time. The data is being published into the event brokers using dynamic topics which allows subscribers to pick, choose and filter on the specific data that they want to consume; e.g., a developer could choose to only get updated when an update it available for a specific Country & Province/State that they are interested in.

How to consume

Any client application could consume those streams by subscribing to the topic of choice as per the diagram below. Note that the streams are provided over Solace PubSub+ Event Broker that has built-in support for a variety of open standard protocols and APIs.

PubSub+ E2E

1. Connection information

Connect to the PubSub+ broker using one of the connection end points below. For more details, check out the Solace Developer Portal and the API docs samples for the different messaging APIs and protocols
Supported APIs and Protocols

Connections credentials

Credentials

MessageVPN : covid-19-streams
Client Username : covid-public-client
Client Password: covid19

Connection Points

API/Protocol Host
SMF Host tcp://covid.messaging.solace.cloud:55555
Web Messaging Host ws://covid.messaging.solace.cloud:80
Secured Web Messaging Host wss://covid.messaging.solace.cloud:443
MQTT Host tcp://covid.messaging.solace.cloud:1883
Secured MQTT Host ssl://covid.messaging.solace.cloud:8883
WebSocket MQTT Host ws://covid.messaging.solace.cloud:8000
AMQP 1.0 amqp://covid.messaging.solace.cloud:5672

Note: if you want to receive notifications via a REST end point that supports POST, contact us at [email protected]

2. Choose your Topics

Streams are available on these topics

As of right now, the following is the list of streams available for consumption

John Hopkins University

Description Schema Topic Notes
Raw data for all countries published every ~45 seconds Raw Data Schema jhu/csse/covid19/raw
Updated Active/Deaths/Confirmed/Recovered for a region/state in one event Regional Update Schema jhu/csse/covid19/cases/region/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Deaths for a region/state Update Type Schema jhu/csse/covid19/cases/deaths/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Active cases for a region/state Update Type Schema jhu/csse/covid19/cases/active/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Confirmed cases for a region/state Update Type Schema jhu/csse/covid19/cases/confirmed/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Recovered cases for a region/state Update Type Schema jhu/csse/covid19/cases/recovered/update/{attributes.countryRegion}/{attributes.provinceState}
Shows the updated percentage of the population affected Update Population Schema jhu/csse/covid19/cases/active/population/update/US/{attributes.provinceState} Only US for now

The Covid Tracking Project

Description Schema Topic Notes
Raw data for all the states in the US published every ~45 seconds Raw Data Schema com/covidtracking/states/current/get/raw
Updated US state information Raw Data Schema com/covidtracking/states/current/update/{state}

Subscribe to one or more of the available topics above to receive the required data. Note that the streams defined as update above only send events when updates actually occur which can be infrequently. For development purposes we are providing test topics that get published every few minutes

For more more information on using topic wildcards check out SMF Topic Subscriptions and MQTT Topic Subscriptions

Note: If you have your own COVID-19 event stream that you'd like to share please contact us at [email protected]

Example topic subscriptions

Subscribing to the following streams will give you the corresponding results:

Subscription Result
jhu/csse/covid19/cases/region/update/> All region updates
jhu/csse/covid19/cases/region/update/US/> All updates for country US
jhu/csse/covid19/cases/region/update/US/Florida All updates where state is Florida
jhu/csse/covid19/cases/recovered/update/Canada/Ontario All recovered in Ontario, Canada
jhu/csse/covid19/cases/active/update/Canada/* All active in Canada
jhu/csse/covid19/cases/deaths/update/Brazil/* All deaths in Brazil (Note: country with no province/state)
jhu/csse/covid19/cases/*/update/Diamond Princess/* All case types from the Diamond Princess cruise ship

3. Create your application

After connecting and choosing the topics to listen on, it's time to build your application. Check out the examples below for applications that consume the streams.

Sample Applications to Get Started Quickly

Application Language Description
Spring Cloud Stream Sample Java + Spring This microservice shows how to consume events using the Spring Cloud Stream project, which abstracts the messaging APIs from the developer and allows them to focus on implementing their business logic.
JavaScript Sample JavaScript This html page allows you to test out multiple topics and view the stream body response. You can check it out on the CovidStreamViewer
MQTT JS Sample JavaScript This html page demonstrates connecting to the Solace PubSub+ COVID19 Broker using MQTT JS Library. You can check it out on the MQTT CovidStreamViewer
NodeJS Sample NodeJS This NodeJs application runs in the terminal using mqtt.js npm library
Python Sample Python This NodeJs executable runs in the terminal using the paho-mqtt python library

COVID19 Stream Processors

Below are Spring Boot microservices used to create the event current streams available for consumption. They are also good examples if you would like to consume the streams and republish them into an event broker. Note that they are not intended to be run with the provided credentials as we are running them as a services.

Application Version Integration Description
COVID19RawPublisher 0.1 Spring This application publihses RAW JHU data
COVID19CaseSplitter 0.1 Spring This application consumes RAW JHU COVID19 data, and publishes smaller updates
COVID19RegionalSplitter 0.1 Spring This application consumes RAW JHU COVID19 data, splits it into regional updates and publishes smaller regional updates
COVID19PopulationProvider 0.1 Spring This application publishes the percentage of population affected in the areas
CovidTrackingStatesPublisher 0.1 Spring This application consumes RAW CovidTracking.com data, and publishes smaller updates
CovidTrackingStatesSplitterPublisher 0.1 Spring This application consumes RAW CovidTracking.com data, splits it into states and publishes smaller updates

EventPortal

Note: Contact us at [email protected] to add more event stream into the current broker.

Test Topics

This section includes information about test streams that are available. Since actual updates can occur infrequently, we are providing these test streams strictly for development purposes. Instead of only receiving events when updates occur, these topics, which include test as a level, will regularly receive events whether or not an update has actually occurred. Once you have completed development you should remove the test level from your topic and you’ll only receive events when actual updates occur.

Description Topic Notes
Updated Active/Deaths/Confirmed/Recovered for a region/state in one event jhu/csse/covid19/test/cases/region/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Deaths for a region/state jhu/csse/covid19/test/cases/deaths/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Active cases for a region/state jhu/csse/covid19/test/cases/active/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Confirmed cases for a region/state jhu/csse/covid19/test/cases/confirmed/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Recovered cases for a region/state jhu/csse/covid19/test/cases/recovered/update/{attributes.countryRegion}/{attributes.provinceState}
Shows the updated percentage of the population affected jhu/csse/covid19/test/cases/active/population/update/US/{attributes.provinceState} Only US for now
Updated US state information com/covidtracking/test/states/current/update/{state}

Note that the raw streams is regularly updated every ~45 seconds

Contribution

See Contribution guidelines form more details

Credit

Disclaimer

  1. As per Johns Hopkins University Terms of Use this data is public strictly for educational and academic research purposes.
  2. Data streams are provided and maintained by Solace under the same terms of use.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].