Origami Image Service
Optimises and resize images. See the production service for API information.
Table Of Contents
- Requirements
- Running Locally
- Configuration
- Adding Images
- Testing
- Deployment
- Monitoring
- Trouble-Shooting
- License
Requirements
Running Origami Image Service requires Node.js, npm and git-lfs.
Running Locally
Before we can run the application, we'll need to install dependencies:
npm install
Run the application in development mode with
make run-dev
Now you can access the app over HTTP on port 8080
: http://localhost:8080/
Configuration
We configure Origami Image Service using environment variables. In development, configurations are set in a .env
file. In production, these are set through Heroku config. Further documentation on the available options can be found in the Origami Service documentation.
Required everywhere
CONTENT_API_KEY
: The API key for the FT UPP Content API.CLOUDINARY_ACCOUNT_NAME
: The name of the Cloudinary account to use in image transforms.CLOUDINARY_API_KEY
: The Cloudinary API key corresponding toCLOUDINARY_ACCOUNT_NAME
.CLOUDINARY_API_SECRET
: The Cloudinary API secret corresponding toCLOUDINARY_ACCOUNT_NAME
.CUSTOM_SCHEME_STORE
: The location of the images used in custom schemes. This should be set to the base path under which images live.CUSTOM_SCHEME_CACHE_BUST
: A key used to manually cache-bust custom scheme images.HOSTNAME
: The hostname to use for tinting SVGs. This defaults to the hostname given in the request. See the trouble-shooting guide for more information.NODE_ENV
: The environment to run the application in. One ofproduction
,development
(default), ortest
(for use in automated tests).PORT
: The port to run the application on.
Required in Heroku
FASTLY_PURGE_API_KEY
: A Fastly API key which is used to purge URLs (when somebody POSTs to the/purge
endpoint)GRAPHITE_API_KEY
: The FT's internal Graphite API keyPURGE_API_KEY
: The API key to require when somebody POSTs to the/purge
endpoint. This should be a non-memorable string, for example a UUIDREGION
: The region the application is running in. One ofQA
,EU
, orUS
RELEASE_LOG_API_KEY
: The change request API key to use when creating and closing release logsRELEASE_LOG_ENVIRONMENT
: The Salesforce environment to include in release logs. One ofTest
orProduction
SENTRY_DSN
: The Sentry URL to send error information to
TODO: The options below are required at the moment, but are duplicates of other options above. This will be addressed once all services are using Origami Makefile.
FASTLY_API_KEY
: The Fastly API key to use when purging assets. If not set, purge endpoints are not registered. This should be the same value asFASTLY_PURGE_API_KEY
FASTLY_SERVICE_ID
: The Fastly service to purge assets fromAPI_KEY
: The API key to use when purging assets. If not set, endpoints which require an API key are not registered. This should be the same value asPURGE_API_KEY
Required locally
GRAFANA_API_KEY
: The API key to use when using Grafana push/pull
Headers
The service can also be configured by sending HTTP headers, these would normally be set in your CDN config:
FT-Origami-Service-Base-Path
: The base path for the service, this gets prepended to all paths in the HTML and ensures that redirects work when the CDN rewrites URLs.FT-Origami-Api-Key
: The API key for the service, this is used when calling API endpoints which are restricted to FT Origami developers.
Adding Images
The Origami Image Service fetches and transforms images from external hosts, such as FT APIs or any given URL. It also hosts a number of image sets directly.
To add, edit, or remove an image in one of these image sets see the image-sets
directory. Some image sets have their own readme.md
or contribution.md
with further guidance specific to the image set. For example see the fticon contributing guide.
Removing an image from an image set is considered a major change. To remove an image a new major version of the Origami Image Service API must be released. Therefore it's typical to deprecate images first, and remove multiple deprecated images later as a batch. To manage this each image set directory has a deprecated.json
file containing a list of images in the set which are deprecated and should be removed in the next major version of the Origami Image Service. Deprecated images are hidden on the Origami Image Service image sets page.
Testing
The tests are split into unit tests and integration tests. To run tests on your machine you'll need to install Node.js and run npm install
. Then you can run the following commands:
make test # run all the tests
make test-unit # run the unit tests
make test-integration # run the integration tests
You can run the unit tests with coverage reporting, which expects 90% coverage or more:
make test-unit-coverage verify-coverage
The code will also need to pass linting on CI, you can run the linter locally with:
make verify
We run the tests and linter on CI, you can view [results on CI][ci]. make test
and make lint
must pass before we merge a pull request.
You can run the integration tests against a URL by setting a HOST
environment variable to the URL you want to test. This is useful for testing a Heroku application after it is deployed, which we do on CI.
HOST="https://www.example.com" make test-integration
Deployment
The production (EU/US) and QA applications run on Heroku. We deploy continuously to QA via [CI][ci], you should never need to deploy to QA manually. We use a Heroku pipeline to promote QA deployments to production.
You can promote either through the Heroku interface, or by running the following command locally:
make promote
Monitoring
- Grafana dashboard: graph memory, load, and number of requests
- Pingdom check (Production EU): checks that the EU production app is responding
- Pingdom check (Production US): checks that the US production app is responding
- Sentry dashboard (Production): records application errors in the production app
- Sentry dashboard (QA): records application errors in the QA app
- Splunk dashboard (Production): query application logs
Trouble-Shooting
We've outlined some common issues that can occur in the running of the Image Service:
Requesting a PNG but being returned a JPG, why is that?
When a png image is requested, and the requested image has no alpha channel (no transparency in the image), a jpg is instead returned because it will have a smaller filesize.
I need to purge an image
Please read the purging documentation on the website.
I need to purge all images, is this possible?
Please contact [email protected] - There is a way to purge all images, but this will incur a large cost.
What do I do if memory usage is high?
For now, restart the Heroku dynos:
heroku restart --app origami-image-service-eu
heroku restart --app origami-image-service-us
If this doesn't help, then a temporary measure could be to add more dynos to the production applications, or switch the existing ones to higher performance dynos.
What if I need to deploy manually?
If you really need to deploy manually, you should only do so to QA. Production deploys should always be a promotion from QA.
You'll need to provide an API key for change request logging. You can get this from the Origami LastPass folder in the note named Change Request API Keys
. Now deploy to QA using the following:
make deploy
SVGs don't work when running locally
When an SVG image is requested we rewrite the URL to go route back through the Image Service, this is to sanatize the SVG of any cross-site-scripting attack vectors and to tint the SVG if tinting has been requested. It looks something like this:
- User requests:
http://imageservice/v2/images/raw/http://mysite/example.svg?tint=red
- Image service rewrites to:
http://imageservice/v2/images/raw/http://imageservice/images/svgtint/http://mysite/example.svg%3Fcolor=red
- Cloudinary receives the image URL:
http://imageservice/images/svgtint/http://mysite/example.svg?color=red
When you're running locally this won't work because Cloudinary cannot access your localhost
. The flow would look like this:
- User requests:
http://localhost/v2/images/raw/http://mysite/example.svg?tint=red
- Image service rewrites to:
http://localhost/v2/images/raw/http://localhost/images/svgtint/http://mysite/example.svg%3Fcolor=red
- Cloudinary receives the image URL:
http://localhost/images/svgtint/http://mysite/example.svg?color=red
So Cloudinary responds with a 404
, and you may see an error like connect ECONNREFUSED 127.0.0.1:443
. You can get around this by manually specifying a hostname in your configuration. You'll need to tell the service to rely on the QA instance for SVG tinting. Add the following to your .env
file:
HOSTNAME=origami-image-service-qa.herokuapp.com
License
The Financial Times has published this software under the MIT license.