ASIS Server
ASIS (Advanced Social Image Search) indexes Flickr and MRSS images and provides a search API across both indexes.
Current version
You are reading documentation for ASIS API v1.
Contribute to the code
The server code that runs the image search component of Search.gov is here on Github. Fork this repo to add features like additional datasets, or to fix bugs.
Dependencies
Ruby
Use rvm to install the version of Ruby specified in .ruby-version
.
Configuration
- Copy
config/flickr.yml.example
toconfig/flickr.yml
and update the fields with your Flickr credentials.
Gems
We use bundler to manage gems. You can install bundler and other required gems like this:
gem install bundler
bundle install
Services
The required services (MySQL, Elasticsearch, & Redis) can be run using Docker. Refer to search-services for detailed instructions.
The Elasticsearch services provided by searchgov-services
is configured to run on the default port, 9200. To use a different host (with or without port) or set of hosts, set the ES_HOSTS
environment variable. For example, use following command to run the specs using Elasticsearch running on localhost:9207
:
ES_HOSTS=localhost:9207 bundle exec rspec spec
Development/Usage
Seed some image data
You can bootstrap the system with some government Flickr profiles and MRSS feeds to see the system working.
Sample lists are in config/flickr_profiles.csvand
config/mrss_profiles.csv`.
bundle exec rake oasis:seed_profiles
You can keep the indexes up to date by periodically refreshing the last day's images. To do this manually via the Rails console:
MrssPhotosImporter.refresh
FlickrPhotosImporter.refresh
Running it
Fire up a server and try it all out.
bundle exec rails s
Here are the profiles you have just bootstrapped. Note: Chrome does a nice job of pretty-printing the JSON response.
http://localhost:3000/api/v1/flickr_profiles.json
http://localhost:3000/api/v1/mrss_profiles.json
You can add a new profile manually via the REST API:
curl -XPOST "http://localhost:3000/api/v1/flickr_profiles.json?name=commercegov&id=61913304@N07&profile_type=user"
curl -XPOST "http://localhost:3000/api/v1/mrss_profiles.json?url=https://share-ng.sandia.gov/news/resources/news_releases/feed/"
MRSS profiles work a little differently than Flickr profiles. When you create the MRSS profile, Oasis assigns a short name to it that you will use when performing searches. The JSON result from the POST request will look something like this:
{
"created_at": "2014-10-26T18:25:21.167+00:00",
"updated_at": "2014-10-26T18:25:21.173+00:00",
"name": "72",
"id": "https://share-ng.sandia.gov/news/resources/news_releases/feed/"
}
Asynchronous job processing
We use Sidekiq for job processing. You can see all your jobs queued up here:
Kick off the indexing process:
bundle exec sidekiq
Searching
In the Rails console, you can query each index manually using the Elasticsearch Query DSL:
bin/rails c
FlickrPhoto.count
MrssPhoto.count
FlickrPhoto.all(query:{term:{owner:'28634332@N05'}}).results
MrssPhoto.all(query:{match:{description:'air'}}).results
Parameters
These parameters are accepted for the blended search API:
- query
- flickr_groups (comma separated)
- flickr_users (comma separated)
- mrss_names (comma separated list of Oasis-assigned names)
- size
- from
Results
The top level JSON contains these fields:
total
offset
suggestion
: The overridden spelling suggestionresults
Each result contains these fields:
type
: FlickrPhototitle
url
thumbnail_url
taken_at
API versioning
We support API versioning with the JSON format. The current version is v1. You can specify a specific JSON API version like this:
curl "http://localhost:3000/api/v1/image.json?flickr_groups=1058319@N21&flickr_users=35067687@n04,24662369@n07&mrss_names=72,73&query=earth"
Tests
These require an Elasticsearch server and Redis server to be running.
bundle exec rspec
Code coverage
We track test coverage of the codebase over time, to help identify areas where we could write better tests and to see when poorly tested code got introduced.
After running your tests, view the report by opening coverage/index.html
.
Click around on the files that have < 100% coverage to see what lines weren't exercised.
Code Quality
We use Rubocop for static code analysis. Settings specific to ASIS are configured via .rubocop.yml. Settings that can be shared among all Search.gov repos should be configured via the searchgov_style gem.
Code samples
We "eat our own dog food" and use this ASIS API to display image results on the government websites that use Search.gov.
See the results for a search for bird on FWS.gov.
Feedback
You can send feedback via Github Issues.