Wetterdienst - Open weather data for humans
“You must be the change you wish to see in the world.” — Gandhi
"Three things are (almost) infinite: the universe, human stupidity and the temperature time series of Hohenpeissenberg I got with the help of wetterdienst; and I'm not sure about the universe." - Albert Einstein
"We are the first generation to feel the effect of climate change and the last generation who can do something about it." - Barack Obama
Overview
Introduction
Welcome to Wetterdienst, your friendly weather service library for Python.
We are a group of like-minded people trying to make access to weather data in
Python feel like a warm summer breeze, similar to other projects like
rdwd for the R language, which originally drew our interest in this project.
Our long-term goal is to provide access to multiple weather services as well as other
related agencies such as river measurements. With wetterdienst we try to use modern
Python technologies all over the place. The library is based on pandas across the board,
uses Poetry for package administration and GitHub Actions for all things CI.
Our users are an important part of the development as we are not currently using the
data we are providing and only implement what we think would be the best. Therefore
contributions and feedback whether it be data related or library related are very
welcome! Just hand in a PR or Issue if you think we should include a new feature or data
source.
Acknowledgements
We want to acknowledge all environmental agencies which provide their data open and free of charge first and foremost for the sake of endless research possibilities.
We want to acknowledge Jetbrains and their open source team for providing us with licenses for Pycharm Pro, which we are using for the development.
We want to acknowledge all contributors for being part of the improvements to this library that make it better and better every day.
Coverage
- DWD (Deutscher Wetterdienst / German Weather Service / Germany)
- Historical Weather Observations
- Historical (last ~300 years), recent (500 days to yesterday), now (yesterday up to last hour)
- Every minute to yearly resolution
- Time series of stations in Germany
- Mosmix - statistical optimized scalar forecasts extracted from weather models
- Point forecast
- 5400 stations worldwide
- Both MOSMIX-L and MOSMIX-S is supported
- Up to 115 parameters
- Radar
- 16 locations in Germany
- All of Composite, Radolan, Radvor, Sites and Radolan_CDC
- Radolan: calibrated radar precipitation
- Radvor: radar precipitation forecast
- ECCC (Environnement et Changement Climatique Canada / Environment and Climate Change Canada / Canada)
- Historical Weather Observations
- Historical (last ~180 years)
- Hourly, daily, monthly, (annual) resolution
- Time series of stations in Canada
- NOAA (National Oceanic And Atmospheric Administration / National Oceanic And Atmospheric Administration / United States Of America)
- Global Historical Climatology Network
- Historical, daily weather observations from around the globe
- more then 100k stations
- data for weather services which don't publish data themselves
- WSV (Wasserstraßen- und Schifffahrtsverwaltung des Bundes / Federal Waterways and Shipping Administration)
- Pegelonline
- data of river network of Germany
- coverage of last 30 days
- parameters like stage, runoff and more related to rivers
- EA (Environment Agency)
- Hydrology
- data of river network of UK
- parameters flow and ground water stage
To get better insight on which data we have currently made available and under which license those are published take a look at the data section.
Features
- API(s) for stations (metadata) and values
- Get station(s) nearby a selected location
- Define your request by arguments such as parameter, period, resolution, start date, end date
- Command line interface
- Web-API via FastAPI
- Run SQL queries on the results
- Export results to databases and other data sinks
- Public Docker image
Setup
wetterdienst can be used by either installing it on your workstation or within a Docker
container.
Native
Via PyPi (standard):
pip install wetterdienstVia Github (most recent):
pip install git+https://github.com/earthobservations/wetterdienstThere are some extras available for wetterdienst. Use them like:
pip install wetterdienst[http,sql]- docs: Install the Sphinx documentation generator.
- ipython: Install iPython stack.
- export: Install openpyxl for Excel export and pyarrow for writing files in Feather- and Parquet-format.
- http: Install HTTP API prerequisites.
- sql: Install DuckDB for querying data using SQL.
- duckdb: Install support for DuckDB.
- influxdb: Install support for InfluxDB.
- cratedb: Install support for CrateDB.
- mysql: Install support for MySQL.
- postgresql: Install support for PostgreSQL.
In order to check the installation, invoke:
wetterdienst --helpDocker
Docker images for each stable release will get pushed to GitHub Container Registry.
There are images in two variants, wetterdienst-standard and wetterdienst-full.
wetterdienst-standard will contain a minimum set of 3rd-party packages,
while wetterdienst-full will try to serve a full environment by also
including packages like GDAL and wradlib.
Pull the Docker image:
docker pull ghcr.io/earthobservations/wetterdienst-standardLibrary
Use the latest stable version of wetterdienst:
$ docker run -ti ghcr.io/earthobservations/wetterdienst-standard
Python 3.8.5 (default, Sep 10 2020, 16:58:22)
[GCC 8.3.0] on linuximport wetterdienst
wetterdienst.__version__Command line script
The wetterdienst command is also available:
# Make an alias to use it conveniently from your shell.
alias wetterdienst='docker run -ti ghcr.io/earthobservations/wetterdienst-standard wetterdienst'
wetterdienst --help
wetterdienst version
wetterdienst infoExample
Acquisition of historical data for specific stations using wetterdienst as library:
Load required request class:
>>> import pandas as pd
>>> pd.options.display.max_columns = 8
>>> from wetterdienst.provider.dwd.observation import DwdObservationRequest
>>> from wetterdienst import SettingsAlternatively, though without argument/type hinting:
>>> from wetterdienst import Wetterdienst
>>> API = Wetterdienst("dwd", "observation")Get data:
>>> Settings.tidy = True # default, tidy data
>>> Settings.humanize = True # default, humanized parameters
>>> Settings.si_units = True # default, convert values to SI units
>>> request = DwdObservationRequest(
... parameter=["climate_summary"],
... resolution="daily",
... start_date="1990-01-01", # if not given timezone defaulted to UTC
... end_date="2020-01-01", # if not given timezone defaulted to UTC
... ).filter_by_station_id(station_id=(1048, 4411))
>>> request.df.head() # station list
station_id from_date to_date height \
... 01048 1934-01-01 00:00:00+00:00 ... 00:00:00+00:00 228.0
... 04411 1979-12-01 00:00:00+00:00 ... 00:00:00+00:00 155.0
<BLANKLINE>
latitude longitude name state
... 51.1278 13.7543 Dresden-Klotzsche Sachsen
... 49.9195 8.9671 Schaafheim-Schlierbach Hessen
>>> request.values.all().df.head() # values
station_id dataset parameter date value \
0 01048 climate_summary wind_gust_max 1990-01-01 00:00:00+00:00 NaN
1 01048 climate_summary wind_gust_max 1990-01-02 00:00:00+00:00 NaN
2 01048 climate_summary wind_gust_max 1990-01-03 00:00:00+00:00 5.0
3 01048 climate_summary wind_gust_max 1990-01-04 00:00:00+00:00 9.0
4 01048 climate_summary wind_gust_max 1990-01-05 00:00:00+00:00 7.0
<BLANKLINE>
quality
0 NaN
1 NaN
2 10.0
3 10.0
4 10.0Receiving of stations for defined parameters using the wetterdienst client:
# Get list of all stations for daily climate summary data in JSON format
wetterdienst dwd observations stations --parameter=kl --resolution=daily --period=recent
# Get daily climate summary data for specific stations
wetterdienst dwd observations values --station=1048,4411 --parameter=kl --resolution=daily --period=recentFurther examples (code samples) can be found in the examples folder.
Documentation
We strongly recommend reading the full documentation, which will be updated continuously as we make progress with this library:
https://wetterdienst.readthedocs.io/
For the whole functionality, check out the Usage documentation and examples section of our documentation, which will be constantly updated. To stay up to date with the development, take a look at the changelog. Also, don't miss out our examples.
Data license
Licenses of the available data can be found in our documentation at the data license section. Licenses and usage requirements may differ so check this out before including the data in your project to be sure to fulfill copyright issues beforehand.
Contribution
There are different ways in which you can contribute to this library:
- by handing in a PR which describes the feature/issue that was solved including tests for newly added features
- by using our library and reporting bugs to us either by mail or by creating a new Issue
- by letting us know either via issue or discussion what function or data source we may include into this library describing possible solutions or acquisition methods/endpoints/APIs
Development
Clone the library and install the environment.
This setup procedure will outline how to install the library and the minimum dependencies required to run the whole test suite. If, for some reason, you are not available to install all the packages, just leave out some of the "extras" dependency tags.
git clone https://github.com/earthobservations/wetterdienst
cd wetterdienst
# Prerequisites
brew install --cask firefox
brew install git python geckodriver
# Option 1: Basic
git clone https://github.com/earthobservations/wetterdienst
cd wetterdienst
python3 -m venv .venv
source .venv/bin/activate
pip install --requirement=requirements.txt
python setup.py develop
# (Option 2: Install package with extras)
pip install ".[sql,export,restapi,explorer]"
# Option 3: Install package with extras using poetry.
poetry install --extras=sql --extras=export --extras=restapi --extras=explorer
poetry shellFor running the whole test suite, you will need to have Firefox and geckodriver installed on your machine. Install them like:
# macOS brew install --cask firefox brew install geckodriver # Other OS # You can also get installers and/or release archives for Linux, macOS # and Windows at # # - https://www.mozilla.org/en-US/firefox/new/ # - https://github.com/mozilla/geckodriver/releases
If this does not work for some reason and you would like to skip ui-related tests on your machine, please invoke the test suite with:
poe test -m "not ui"
Edit the source code, add corresponding tests and documentation for your changes. While editing, you might want to continuously run the test suite by invoking:
poe test
In order to run only specific tests, invoke:
# Run tests by module name or function name. poe test -k test_cli # Run tests by tags. poe test -m "not (remote or slow)"
Before committing your changes, please als run those steps in order to make the patch adhere to the coding standards used here.
poe format # black code formatting
poe lint # lint checking
poe export # export of requirements (for Github Dependency Graph)Push your changes and submit them as pull request
Thank you in advance!
Note
If you need to extend the list of package dependencies, invoke:
# Add package to runtime dependencies.
poetry add new-package
# Add package to development dependencies.
poetry add --dev new-packageKnown Issues
MAC ARM64 (M1)
You need to install pandas, numpy and scipy as follows before continuing with the regular setup:
pip install pandas --no-use-pep517
pip install numpy --no-use-pep517
pip install --no-binary :all: --no-use-pep517 scipyFurther additional libraries are affected and have to be installed in a similar manner:
# SQL related
brew install postgresql
brew link openssl (and export ENVS as given)
pip install psycopg2-binary --no-use-pep517LINUX ARM (Raspberry Pi)
Running wetterdienst on Raspberry Pi, you need to install numpy and lxml prior to installing wetterdienst running the following lines:
sudo apt-get install libatlas-base-dev
sudo apt-get install python3-lxml