edge-sql

A Cloudflare Worker embedding SQLite with WASM and a simple Forex dataset.

You can preview it here: https://sql.lspgn.workers.dev.

Why?

I had an idea. Here's a little backstory.

Many services provide a REST API on top of a SQL database. For certain datasources, flexibility in terms of queries is key. Often those use-cases are dashboard prototyping, for example with Grafana.

This data is usually read-only statistical datasets where the user needs to run many queries in order to troubleshoot an issue or just find the best visualization. Often this requires complex queries and extensive accesses or quotas. Using regular REST APIs would require a lot of back-and-forth between the developers and the analysts. Recent initiatives like GraphQL are aimed at these issues and hope to provide more flexibility.

The idea behind this fun project is that serverless concepts can be applied for this use-case and have SQL as an execution model. Foreign Data Wrappers are similar as they provide a framework that works with a Query Processor but need to rely on more traditional user control and quotas configurations.

Currently, tools like BigQuery are used in multi-tenancy environments and allow user to execute their SQL program without worrying about data infrastructure.

Through this fun prototype on the Cloudflare Workers platform, we can put a sandbox around a SQL program and play with data stored in Worker KVs.

The current setup is meant to use minimal resources (Free limits). This is a proof of concept that was made possible by SQLite code that can fit inside the plateform (<1MB). Obviously, a rework of the code would be required to make it usable in a production environment. Features like sharding the data and the querying are necessary for large datasets, along with user control. The current quotas are controlled by the platform which can stop the execution of the entire Worker if it goes above the allowed limits.

Data

The data used in production is a reduced version (only EUR, JPY, GBP and CHF currencies) of the European Central Bank Forex Rates on Humdata. And is distributed under the license CC-BY.

Try

The following query will return the days when the British pound was at its lowest and highest against the dollar.

$ curl -XPOST --data \
"
SELECT *,1/EUR,1/JPY,1/GBP,1/CHF
FROM forex
WHERE
  GBP = (
    SELECT
      max(GBP)
    FROM forex)
  OR GBP = (
    SELECT
      min(GBP)
    FROM forex)
" \
-H 'content-type: application/text' \
https://sql.lspgn.workers.dev/
Date,EUR,JPY,GBP,CHF,1/EUR,1/JPY,1/GBP,1/CHF
2007-11-08,1.4666,0.008840265220012,2.10642728904847,0.883440756580929,0.681849174962498,113.118778126279,0.47473748806764,1.13193781535524

Builtin SQLite functions along with extra features are avaialble.

$ curl -XPOST --data \
"
SELECT
  getdata('country') AS country,
  random() AS rnd,
  date('now') AS now
" \
-H 'content-type: application/text' \
https://sql.lspgn.workers.dev/
country,rnd,now
US,-7348224717915799868,2021-01-18