Masakhane WEB - A Machine Translation Web Platform for African Languages
Masakhane meaning ‘we build together’, is a research effort for machine translation for African languages which is open source and online. So far, the community has built translation models based on Joey NMT for over 38 African languages. As such, Masakhane Web is a platform that aims to host the already trained models from the community and allow contributions from users to create new data for retraining. The objective of this web application is to provide access to an open-source platform that makes available relatively accurate translations for languages across Africa. If you can't find your language and/or would like to train your own machine translation model in your language, see https://github.com/masakhane-io/masakhane-mt on how you can contribute.
Disclaimer: This system is for research purposes only and should be taken as work in progress. None of the trained models are suitable for production usage.
Table of Contents
How to run
As a stand alone app
Backend
- run the app :
- move to the server directory :
cd src/server/ - Install required packages
pip install -r requirements.txt
export FLASK_APP=core/__init__.pyexport FLASK_ENV=developmentpython manage.py run
- move to the server directory :
Note: The stand alone app uses sqlite as db instead of postgreSQL like our live app, you then need to run the command below to create and initialize the datbase.
- Create table relations
python manage.py create_db
- Add language
python manage.py add_language en-sw-JW300
- Remove language
python manage.py remove_language en-sw-JW300
- Check available languages
python manage.py all_languages
- Update known languages
curl --request GET 'http://127.0.0.1:5000/update'
- Run tests
python manage.py tests
You can check content saved in the dabase using the code below (move to src/server/core/ and run python):
import sqlite3, os
conn = sqlite3.connect("masakhane.db")
c = conn.cursor()
for row in c.execute('SELECT * FROM feedback'):
print(row)
for row in c.execute('SELECT * FROM language'):
print(row)Frontend
-
install the following: \
-
To run:
- move to the client directory :
cd src/client/ - run
npm install - run
npm i webpack webpack-cli --legacy-peer-deps - run
npm i @babel/core @babel/preset-env @babel/preset-react babel-loader - run
npm run develop
- move to the client directory :
This runs the app in development mode.
Open http://localhost:3000 to view it in the browser.
Using Docker (Prefered)
The better way to run the app is to use Docker, which will take care of running both the frontend and backend easily.
Install Docker and Docker-compose
Run the following command to check if you have docker and docker-compose installed in your computer :
- docker --version
- docker-compose --version
If you get the information about the version and not an error you can skeep the details below and just to the Using Docker section.
- Docker :
https://docs.docker.com/engine/install/ - Docker-compose :
https://docs.docker.com/compose/install/
To make sure that it is well installed you can run the code above to check the version of the installed docker and docker-compose
Run the App
-
run the app
docker-compose -f docker-compose.prod.yml up -d --buildfrom the root project.
-
shutdown the app
docker-compose -f docker-compose.prod.yml down
-
check the database
docker-compose -f docker-compose.prod.yml exec db psql --username=masakhane --dbname=masakhane- list databases
\l - connect to the masakhane database
\c masakhane - list relations
\dt - to quit
\q - to see saved information in a relation
select * from language;or - to see feedbacks in a relation
select * from feedback;
- list databases
Add, Delete and Update supported languages
-
check the available models in memory
docker-compose -f docker-compose.prod.yml exec api python manage.py all_languages -
add a new language,
- e.g English-Swahili (note: we are using JW300 shortform)
docker-compose -f docker-compose.prod.yml exec api python manage.py add_language en-sw-JW300 - (English-Yoruba)
docker-compose -f docker-compose.prod.yml exec api python manage.py add_language en-yo-
- e.g English-Swahili (note: we are using JW300 shortform)
-
delete a language
docker-compose -f docker-compose.prod.yml exec api python manage.py remove_language en-sw-JW300 -
run this on the production server to update the models
curl --request GET 'http://127.0.0.1:5000/update' -
Run tests
docker-compose -f docker-compose.prod.yml exec api python manage.py tests
View https://github.com/dsfsi/masakhane-web/tree/master/src/client on how you can contribute to improve the look of the website.
Contributing
Options
- Can't see your language as one of the supported languages: Visit Masakhane:Building your first machine translation model to learn more about how you can train a model for your language.
- I have an idea or a new feature: Create a new issue first, assign it to yourself and then fork the repo
- I want to help in improving the accuracy of the models: Check out below on how you can reach out to us
Submitting Changes[Pull Request]
Contributors
Made with contributors-img.
Contact Us
- Vukosi Marivate - [email protected]
- Abiodun Modupe - [email protected]
- Salomon Kabongo - [email protected]
- Catherine Gitau - [email protected]
License
Citing the project
On a visualisation/notebook/webapp:
Data Science for Social Impact Research Group @ University of Pretoria, Masakhane NLP, Masakhane WEB - A Machine Translation Web Platform for African Languages Available on: https://github.com/dsfsi/masakhane-web.
In a publication
Software
@software{marivate_vukosi_2021_4745501, author = {Marivate, Vukosi and Gitau, Catherine and Kabenamualu, Salomon and Modupe, Abiodun and Masakhane NLP}, title = {{Masakhane WEB - A Machine Translation Web Platform for African Languages}}, month = may, year = 2021, publisher = {Zenodo}, version = {0.9}, doi = {10.5281/zenodo.4745501}, url = {https://doi.org/10.5281/zenodo.4745501} }
Acknowledgements
We want to acknowledge support from the following organisations