Py-QGIS-WPS

New in 1.8: OGC api processes support

Py-QGIS-WPS is an implementation of the Web Processing Service standard from the Open Geospatial Consortium based on the QGIS Processing API.

Since 1.8 Py-QGIS-WPS supports OGC API REST processes api

This implementation allows you to expose and run on a server:

• QGIS Processing algorithms available on Desktop
• QGIS Processing models and scripts
• QGIS plugins having a Processing provider according to their metadata.txtfile

It is written in Python and was originally a fork of PyWPS.

Requirements and limitations :

• Python 3.7 minimum
• Windows not officially supported
• Redis server

Documentation

Latest documentation is available on docs.3liz.org

Why Py-QGIS-WPS ?

Py-QGIS-WPS differs from PyWPS in the following:

• QGIS centric
• Handle all request in asynchronous way: all jobs run in a non-blocking way, even when synchronous operation is requested.
• Use multiprocessing Pool to handle task queue instead instantiating a new process each time.
• Uniform Logging with the 'logging' module
• Implements OGC processes api.
• Use Redis for asynchronous status storage.
• Support streamed/chunked requests for stored data
• Add extensions to WPS: TIMEOUT and EXPIRE
• No Windows support

All these changes where not easy to implement without some drastic changes of the original code and we think that it deviates too much from the PyWPS original intentions.

That is, we have decided to fork the original project and go along with it.

So, we are really grateful to the original authors of PyWPS for the nice piece of software that helped us very much to start quickly this project.

Why moving to Tornado instead WSGI

• We need to support asyncio: asyncio requires a blocking running loop. This cannot be achieved simply in a WSGI architecture.
• Tornado is fully integrated with native python asyncio library and provide a great framework for developing a http server.

Extensions to WPS

TIMEOUT extension

Specify the timeout for a process: if the process takes more than TIMEOUT seconds to run, the worker is then killed and an error status is returned.

Set the TIMEOUT=<seconds> in GET requests.

In POST requests, set the timeout=<seconds> attribut in the <ResponseDocument> tag.

The server may configure maximum timeout value.

EXPIRE extension

Specify the expiration time for stored results: after EXPIRE seconds after the end of the wps process, all results will be flushed from disks and local cache. Trying to request the results again will return a 404 HTTP error.

Set the EXPIRE=<seconds> in GET requests.

In POST requests, set the expire=<seconds> attribut int the <ResponseDocument> tag.

The server may configure maximum expiration value.

Status API

Now implemented with the processes api: The status REST api will return the list of the stored status for all running and terminated wps processes.

Example for returning all stored status:

http://localhost:8080/jobs

Example for returning status for one given job from its id:

http://localhost:8080/jobs/<job_id>

Extensions to processes api:

Files

http://localhost:8080/jobs/<job_id>/files

Running QGIS processing

WPS input/output layer mapping

With QGIS desktop, QGIS processing algorithms usually apply on a QGIS source project and computed layers are displayed in the same context as the source project.

Py-qgis-wps works the same way: a qgis project will be used as a source of input layers. The difference is that, when an algorithm runs, it creates a qgis project file associated to the current task and register computed layers to it.

The created project may be used as OWS source with Qgis Server. Output layers are returned as complex objects holding a reference to a WMS/WFS uri that can be used directly with Qgis server. The uri template is configurable using the server/wms_response_uri configuration setting.

Contextualized input parameters

Tasks parameters are contextualized using the MAP query param. If a MAP parameters is given when doing a DescripProcess requests, allowed values for input layers will be taken from the qgis source project according the type of the input layers.

QGIS project (.qgs) files and project stored in Postgres databases are both supported.

The best practice is to always provide a MAP parameters and include the possible input layer in a qgs project. This way you may connect whatever data source supported by qgis and use them as input data in a safe way.

If you need to pass data to your algorithm from client-side, prefer inputs file parameter and small payloads.

Dependencies

See requirements.txt file.

Installation from python package

ADVICE: You should always install in a python virtualenv. If you want to use system packages, setup your environment with the --system-site-packages option.

See the official documentation for how to setup a python virtualenv: https://virtualenv.pypa.io/en/stable/.

From source

Install in development mode

make build
pip install -e .

From python package server

pip install py-qgis-wps

Running the server

The server from a command line interface:

The server does not run as a daemon by itself, there are several ways to run a command as a daemon.

For example:

• Use Supervisor http://supervisord.org/ will give you full control over logs and server status notifications.
• Use the daemon command.
• Use Docker

Running the server

Usage

usage: wpsserver [-h] [-d] [-c [PATH]]
                 [--version] [-p PORT] [-b IP] [-u SETUID]

WPS server

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Set debug mode  
  -c [PATH], --config [PATH]
                        Configuration file
  --version             Return version number and exit
  -p PORT, --port PORT  http port
  -b IP, --bind IP      Interface to bind to
  -u SETUID, --setuid SETUID
                        uid to switch to

Requests to OWS services

The OWS requests use the following format: /ows/?<ows_query_params>

Example:

http://myserver:8080/ows/?SERVICE=WPS&VERSION=1.0.0&REQUEST=GetCapabilities

Configuration

From config ini file

By default, the wps server is not using any config file, but one can be used with the --config option. A config file is a simple ini file, a sample config file is given with the sources.

From environment variables

The server can be configured with environnement variables:

Configuration is done with environment variables:

• QGSWPS_SERVER_WORKDIR: set the current dir processes, all processes will be running in that directory.
• QGSWPS_SERVER_HOST_PROXY: When the service is behind a reverse proxy, set this to the proxy entrypoint.
• QGSWPS_SERVER_PARALLELPROCESSES: Number of parallel process workers
• QGSWPS_SERVER_RESPONSE_TIMEOUT: The max response time before killing a process.
• QGSWPS_SERVER_RESPONSE_EXPIRATION: The max time (in seconds) the response from a WPS process will be available.
• QGSWPS_SERVER_WMS_SERVICE_URL: The base url for WMS service. Default to /wms. Responses from processing will be retourned as WMS urls. This configuration variable set the base url for accessing results.
• QGSWPS_SERVER_RESULTS_MAP_URI

Logging

• QGSWPS_LOGLEVEL: the log level, should be INFO in production mode, DEBUG for debug output.

REDIS storage configuration

• QGSWPS_REDIS_HOST: The redis host
• QGSWPS_REDIS_PORT: The redis port. Default to 6379
• QGSWPS_REDIS_DBNUM: The redis database number used. Default to 0

Qgis project Cache configuration

• QGSWPS_CACHE_ROOTDIR: Absolute path to the qgis projects root directory, projects referenced with the MAP parameter will be searched at this location

Processing configuration

• QGSWPS_PROCESSSING_PROVIDERS_MODULE_PATH: Path to look for processing algorithms provider to publish, algorithms from providers specified here will be runnable as WPS processes.

Exposing algorithms as WPS services

Note that since 1.1 , the __algorithms__.py method for declaring providers is no longer supported.

Processing providers following the same rules as Qgis regular plugin with a special factory entrypoint: WPSClassFactory(iface) in the __init__.py file.

The metadata.txt file

As regular QGIS plugin, a metadata.txt file must be present with a special entry wps=True indicating that the plugin is available as a WPS service provider.

Registering providers

The iface parameter is a instance of WPSServerInterface which provide a registerProvider( provider: QgsAlgorithmProvider, expose: bool = True) -> Any method.

Exposed providers as WPS services must be registered using the registerProvider method

Example:

def WPSClassFactory(iface: WPSServerInterface) -> Any:

    from TestAlgorithmProvider1 import  AlgorithmProvider1
    from TestAlgorithmProvider2 import  AlgorithmProvider2

    iface.registerProvider( AlgorithmProvider1() )
    iface.registerProvider( AlgorithmProvider2() )

Controlling what is exposed:

Processing algorithm with the flag FlagHideFromToolbox set will not be exposed as WPS process.

Parameters with the flag FlagHidden set won't be exposed in a DescribeProcess request

References

• OGC standards
• OGC Api processes
• Introduction to WPS
• Py-qgis-server at FOSS4G 2019.