All Projects → WordPress → openverse-api

WordPress / openverse-api

Licence: MIT License
The Openverse API allows programmatic access to search for CC-licensed and public domain digital media.

Programming Languages

python
139335 projects - #7 most used programming language
HTML
75241 projects
shell
77523 projects
Dockerfile
14818 projects

Projects that are alternatives of or similar to openverse-api

openverse-catalog
Identifies and collects data on cc-licensed content across web crawl data and public apis.
Stars: ✭ 27 (-34.15%)
Mutual labels:  search-engine, creative-commons, openverse
Vue Django Rest Auth
An example project featuring Vue.js and Django Rest Framework using django-rest-auth
Stars: ✭ 153 (+273.17%)
Mutual labels:  architecture, django-rest-framework
Tutorialdb
A search 🔎 engine for programming/dev tutorials, See it in action 👉
Stars: ✭ 93 (+126.83%)
Mutual labels:  search-engine, django-rest-framework
Movies-PagingLibrary-Arch-Components
Sample to practice PagingLibrary & RX
Stars: ✭ 92 (+124.39%)
Mutual labels:  architecture
falcon-policy
Policy Middleware for Falcon APIs
Stars: ✭ 30 (-26.83%)
Mutual labels:  falcon
stable-systems-checklist
An opinionated list of attributes and policies that need to be met in order to establish a stable software system.
Stars: ✭ 43 (+4.88%)
Mutual labels:  architecture
ReMVVM
ReMVVM is an application architecture concept, marriage of Unidirectional Data Flow (Redux) with MVVM.
Stars: ✭ 180 (+339.02%)
Mutual labels:  architecture
arch-pattern-modularization
🌵Implementation of Modularization on Architecture Pattern
Stars: ✭ 27 (-34.15%)
Mutual labels:  architecture
django-rest-framework-aggregates
Exposes aggregation features of the Django model queryset to the DRF API.
Stars: ✭ 23 (-43.9%)
Mutual labels:  django-rest-framework
react-microfrontends-demo
Demo of microfrontends using React
Stars: ✭ 36 (-12.2%)
Mutual labels:  architecture
bing-ip2hosts
bingip2hosts is a Bing.com web scraper that discovers websites by IP address
Stars: ✭ 99 (+141.46%)
Mutual labels:  search-engine
iresearch
IResearch is a cross-platform, high-performance document oriented search engine library written entirely in C++ with the focus on a pluggability of different ranking/similarity models
Stars: ✭ 121 (+195.12%)
Mutual labels:  search-engine
archunit-junit5-kotlin
Generic Architecture Tests written in Kotlin using ArchUnit and Junit5
Stars: ✭ 22 (-46.34%)
Mutual labels:  architecture
collector-filesystem
Norconex Filesystem Collector is a flexible crawler for collecting, parsing, and manipulating data ranging from local hard drives to network locations into various data repositories such as search engines.
Stars: ✭ 17 (-58.54%)
Mutual labels:  search-engine
fc4-framework
A Docs as Code tool that helps software creators and documentarians author software architecture diagrams using the C4 model for visualising software architecture.
Stars: ✭ 121 (+195.12%)
Mutual labels:  architecture
RabbitMQ-with-Django
Example for using Microservices with RabbitMQ in a Django Web-Application
Stars: ✭ 26 (-36.59%)
Mutual labels:  django-rest-framework
plantuml-libs
A set of PlantUML libraries and a NPM cli tool to design diagrams which focus on several technologies/approaches: Amazon Web Services (AWS), Azure, Google Cloud Platform (GCP), C4 Model or even EventStorming and more.
Stars: ✭ 75 (+82.93%)
Mutual labels:  architecture
architecture
Beauty and the Mess - “Performance of the system depends on how the parts fit, not how they act taken separately.”
Stars: ✭ 32 (-21.95%)
Mutual labels:  architecture
pub-apis
Find public APIs, powered by api(dot)publicapis(dot)org
Stars: ✭ 24 (-41.46%)
Mutual labels:  public-apis
django-tutorial
Django 4 tutorial projects
Stars: ✭ 11 (-73.17%)
Mutual labels:  django-rest-framework

Openverse API

Purpose

This repository is primarily concerned with back end infrastructure like datastores, servers, and APIs. The pipeline that feeds data into this system can be found in the Openverse Catalog repository. A front end web application that interfaces with the API can be found at the Openverse frontend repository.

Getting started

Our quickstart guide and other documentation can be found in our developer docs (or within the repo at ./api/docs/guides/quickstart.md). Our API documentation can also be found at https://api.openverse.engineering.

System architecture

System architecture

Basic flow of data

Search data is ingested from upstream sources provided by the data pipeline. As of the time of writing, this includes data from Common Crawl and multiple 3rd party APIs. Once the data has been scraped and cleaned, it is transferred to the upstream database, indicating that it is ready for production use.

Every week, the latest version of the data is automatically bulk copied ("ingested") from the upstream database to the production database by the Ingestion Server. Once the data has been downloaded and indexed inside of the database, the data is indexed in Elasticsearch, at which point the new data can be served up from the Openverse API servers.

Description of subprojects

  • api/: a Django Rest Framework API server For a full description of its capabilities, please see the browsable documentation.
  • ingestion_server/: a service for downloading and indexing search data once it has been prepared by the Openverse Catalog

Troubleshooting

Diagnosing local Elasticsearch issues

If the API server container failed to start, there's a good chance that Elasticsearch failed to start on your machine. Ensure that you have allocated enough memory to Docker applications, otherwise the container will instantly exit with an error. Also, if the logs mention "insufficient max map count", increase the number of open files allowed on your system. For most Linux machines, you can fix this by adding the following line to /etc/sysctl.conf:

vm.max_map_count=262144

To make this setting take effect, update kernel state.

sudo sysctl -p

Contributing

Pull requests are welcome! Feel free to join us on Slack and discuss the project with the engineers and community members on #openverse.

You are welcome to take any open issue in the tracker labeled help wanted or good first issue; there's no need to ask for permission in advance. Other issues are open for contribution as well, but may be less accessible or well-defined in comparison to those that are explicitly labeled.

See the CONTRIBUTING file for details.

Acknowledgments

Openverse, previously known as CC Search, was conceived and built at Creative Commons. We thank them for their commitment to open source and openly licensed content, with particular thanks to previous team members @ryanmerkley, @janetpkr, @lizadaly, @sebworks, @pa-w, @kgodey, @annatuma, @mathemancer, @aldenstpage, @brenoferreira, and @sclachar, along with their community of volunteers.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].