stoqs / stoqs

Spatial Temporal Oceanographic Query System

STOQS is a geospatial database and web application designed to give oceanographers efficient integrated access to in situ measurement and ex situ sample data. See http://www.stoqs.org.

Getting started with a STOQS development system

First, install Vagrant and and VirtualBox — there are standard installers for Mac, Windows, and Linux. (You will also need X Windows System sofware on your computer.) Then create an empty folder off your home directory such as Vagrants/stoqsvm, open a command prompt window, cd to that folder, and enter these commands:

curl "https://raw.githubusercontent.com/stoqs/stoqs/master/Vagrantfile" -o Vagrantfile
curl "https://raw.githubusercontent.com/stoqs/stoqs/master/provision.sh" -o provision.sh
vagrant plugin install vagrant-vbguest
vagrant up --provider virtualbox

The Vagrantfile and provision.sh will provision a development system with an NFS mounted directory from your host operating system. If your host doesn't support serving files via NFS (most Windows hosts don't support NFS file serving) then you'll need to edit these files before executing vagrant up. Look for the support NFS file serving comments in these files for the lines you need to change.

The vagrant up command takes an hour or so to provision and setup a complete CentOS 7 STOQS Virtual Machine that also includes MB-System, InstantReality, and all the Python data science tools bundled in packages such as Anaconda. You will be prompted for your admin password for configuring a shared folder from the VM (unless you've disabled the NFS mount). All connections to this VM are performed from the the directory you installed it in; you must cd to it (e.g. cd ~/Vagrants/stoqsvm) before logging in with the vagrant ssh -- -X command. After installation finishes log into your new VM and test it:

vagrant ssh -- -X                        # Wait for [vagrant@localhost ~]$ prompt
export STOQS_HOME=/vagrant/dev/stoqsgit  # Use STOQS_HOME=/home/vagrant/dev/stoqsgit if not using NFS mount
cd $STOQS_HOME && source venv-stoqs/bin/activate
export DATABASE_URL=postgis://stoqsadm:[email protected]:5438/stoqs
./test.sh CHANGEME load noextraload

In another terminal window start the development server (after a cd ~/Vagrants/stoqsvm):

vagrant ssh -- -X                        # Wait for [vagrant@localhost ~]$ prompt
export STOQS_HOME=/vagrant/dev/stoqsgit  # Use STOQS_HOME=/home/vagrant/dev/stoqsgit if not using NFS mount
cd $STOQS_HOME && source venv-stoqs/bin/activate
export DATABASE_URL=postgis://stoqsadm:[email protected]:5438/stoqs
stoqs/manage.py runserver 0.0.0.0:8000 --settings=config.settings.local

Visit your server's STOQS User Interface using your host computer's browser:

http://localhost:8008

More instructions are in the doc/instructions directory — see LOADING for specifics on loading your own data. For example, you may create your own database of an archived MBARI campaign:

cd stoqs
ln -s mbari_campaigns.py campaigns.py
loaders/load.py --db stoqs_cce2015

You are encouraged to contribute to the STOQS project! Please see CONTRIBUTING for how to share your work. Also, see example Jupyter Notebooks that demonstrate specific analyses and visualizations that go beyond the capabilities of the STOQS User Interface. Visit the STOQS Wiki pages for updates and links to presentations. The stoqs-discuss list in Google Groups is also a good place to ask questions and engage in discussion with the STOQS user and developer communities.

Supported by the David and Lucile Packard Foundation, STOQS undergoes continual development to help support the mission of the Monterey Bay Aquarium Research Institue. If you have your own server you will occasionally want to get new features with:

git pull

Production Deployment with Docker

First, install Docker and docker-compose on your system. Then clone the repository; in the docker directory copy the template.env file to .env and edit it for your specific installation, then execute docker-compose up:

git clone https://github.com/stoqs/stoqs.git stoqsgit
cd stoqsgit/docker
cp template.env .env
chmod 600 .env      # You must then edit .env and change settings for your environment
docker-compose up

If the directory set to the STOQS_VOLS_DIR variable in your .env file doesn't exist then the execution of docker-compose up will create the postgresql database cluster, load a default stoqs database, and execute the unit and functional tests of the stoqs application. If you don't see these tests being executed (they will take several minutes) then check for error messages.

Once you see ... [emperor] vassal /etc/uwsgi/django-uwsgi.ini is ready to accept requests you can visit the site at https://localhost — it uses a self-signed certificate, so your browser will complain and you will need to add an exception. (The nginx service also delivers the same app at http://localhost:8000 without the certificate issue.)

The default settings in template.env will run a production nginx/uwsgi/stoqs server configured for https://localhost in a Vagrant virtual machine. To configure a server for intranet or public serving of your data follow the instructions provided in the comments for the settings in your .env file. After editing your .env file you will need to rebuild the images and restart the Docker services, this time with the -d option to run the containers in the background:

docker-compose build
docker-compose up -d

The above commands should also be done following a git pull in order to deploy updated software on your server.

One thing that's good to do is monitor logs and check for error messages, this can be done with:

docker-compose logs -f

Using STOQS in Docker

You can execute Python code in the stoqs server from your host by prefacing it with docker-compose exec stoqs (Use docker-compose run stoqs to launch another container for long-running processes), for example to load some existing MBARI campaign data:

docker-compose run stoqs stoqs/loaders/load.py --db stoqs_simz_aug2013

(To load MBARI Campaigns you will need to have uncommented the CAMPAIGNS_MODULE=stoqs/mbari_campaigns.py line in your .env file. Make sure that you do not have a symbolic link named campaigns.py in the stoqs directory. This is needed only for a Vagrant development machine — it's best to keep the directory used for a Docker deployment separate from one used for Vagrant.)

In another window monitor its output:

docker-compose run stoqs tail -f /srv/stoqs/loaders/MolecularEcology/loadSIMZ_aug2013.out
# Or (The stoqs code is bound as a volume in the container from the GitHub cloned location)
tail -f stoqsgit/stoqs/loaders/MolecularEcology/loadSIMZ_aug2013.out

You may also use pg_restore to more quickly load an existing Campaign database on your system. For instructions click on the Campaign name in the top bar of a Campaign on another STOQS server, for example on MBARI's Public STOQS Server.

If you use STOQS for your research please cite this publication:

McCann, M.; Schramm, R.; Cline, D.; Michisaki, R.; Harvey, J.; Ryan, J., "Using STOQS (The spatial temporal oceanographic query system) to manage, visualize, and understand AUV, glider, and mooring data," in Autonomous Underwater Vehicles (AUV), 2014 IEEE/OES, pp.1-10, 6-9 Oct. 2014 doi: 10.1109/AUV.2014.7054414