edx / Devstack
Programming Languages
Labels
Projects that are alternatives of or similar to Devstack
Open edX Devstack |Build Status| |docs|
Get up and running quickly with Open edX services.
Documentation is on Read the Docs
. Code repository is on GitHub
.
.. _Read the Docs: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/ .. _GitHub: https://github.com/edx/devstack
This project replaces the older Vagrant-based devstack with a
multi-container approach driven by Docker Compose
_.
A Devstack installation includes the following Open edX components by default:
- The Learning Management System (LMS)
- Open Response Assessments (ORA2), among other LMS plug-ins.
- Open edX Studio
- Discussion Forums
- E-Commerce
- Credentials
- Notes
- Course Discovery
- Open edX Search
- A demonstration Open edX course
- The Publisher and Gradebook micro-frontends
It also includes the following extra components:
- XQueue
- The Learning micro-frontend (A.K.A the new Courseware experience)
- The Program Console micro-frontend
- The Library Authoring micro-frontend
- edX Registrar service.
- The course-authoring micro-frontend
.. Because GitHub doesn't support toctree
, the Table of Contents is hand-written.
.. Please keep it up-to-date with all the top-level headings.
.. Regenerate: grep '^----' README.rst -B 1 | grep -v -e '--' | sed 's/(.)/ \1
_/' | tail -n+2
Table of Contents
-
Where to Find Help
_ -
Notices
_ -
Prerequisites
_ -
Roadmap
_ -
Getting Started
_ -
Usernames and Passwords
_ -
Service List
_ -
Known Issues
_ -
Advanced Configuration Options
_
Where to Find Help
There are a number of places to get help, including mailing lists and real-time chat. Please choose an appropriate venue for your question. This helps ensure that you get good prompt advice, and keeps discussion focused. For details of your options, see the Community
_ pages.
- See the
most common development workflow
_ (after you've finishedGetting Started
_). - See the
Devstack Interface
_ - See some
helpful troubleshooting tips
_. - See the
Frequently Asked Questions
_. - Or learn about
testing and debugging your code in devstack
_.
You can also browse all the documentation in Read the Docs
_.
.. _most common development workflow: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/workflow.html .. _Devstack Interface: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/devstack_interface.html .. _helpful troubleshooting tips: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/troubleshoot_general_tips.html .. _Frequently Asked Questions: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/devstack_faq.html .. _testing and debugging your code in devstack: .. _testing_and_debugging.rst: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/testing_and_debugging.html
.. _Read the Docs: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/
Notices
NOTE: LMS is now using MySql 5.7 by default. You have to run make dev.pull.lms
and make dev.provision.lms
(more details in Getting Started
_) to fetch latest images and reprovision local copies of databases in order for an existing devstack setup to keep working.
Prerequisites
You will need to have the following installed:
- make
- Python 3
- Docker
This project requires Docker 17.06+ CE. We recommend Docker Stable, but Docker Edge should work as well.
NOTE: Switching between Docker Stable and Docker Edge will remove all images and settings. Don't forget to restore your memory setting and be prepared to provision.
For macOS users, please use Docker for Mac
_. Previous Mac-based tools (e.g.
boot2docker) are not supported.
Since a Docker-based devstack runs many containers,
you should configure Docker with a sufficient
amount of resources. We find that configuring Docker for Mac
_ with
a minimum of 2 CPUs, 8GB of memory, and a disk image size of 96GB does work.
Docker for Windows
_ may work but has not been tested and is not supported.
If you are using Linux, use the overlay2
storage driver, kernel version
4.0+ and not overlay
. To check which storage driver your
docker-daemon
uses, run the following command.
.. code:: sh
docker info | grep -i 'storage driver'
Please note
You should run all ``make`` commands described below on your local machinge, *not*
from within a Virtual Machine, as these commands are meant to stand up a VM-like environment using
Docker containers.
However, you may want to run the ``make`` commands from within a Python 3 virtual
environment, as described in `Getting Started`_. This will keep the Python packages required for Devstack separate from
the ones installed globally on your system.
Roadmap
-------
This repository is in sustained status. The goal is to deprecate this codebase and move the development environment setup into the repos with the application code.
Documentation for future of devstack can be found at: `decentralized devstack`_
Documentation for first prototype of decentralized devstack can be found at: `decentralized devstack workflows`_
.. _decentralized devstack: https://github.com/edx/open-edx-proposals/blob/master/oeps/oep-0005/decisions/0002-why-decentralized-devstack.rst
.. _decentralized devstack workflows: https://github.com/edx/enterprise-catalog/blob/master/docs/decentralized_devstack_workflows.rst
Getting Started
---------------
The default devstack services can be run by following the steps below.
1. Install the requirements inside of a `Python virtualenv`_.
.. code:: sh
make requirements
This will install docker-compose and other utilities into your virtualenv.
2. The Docker Compose file mounts a host volume for each service's executing
code. The host directory defaults to be a sibling of this directory. For
example, if this repo is cloned to ``~/workspace/devstack``, host volumes
will be expected in ``~/workspace/course-discovery``,
``~/workspace/ecommerce``, etc. These repos can be cloned with the command
below.
.. code:: sh
make dev.clone # or, `make dev.clone.https` if you don't have SSH keys set up.
You may customize where the local repositories are found by setting the
``DEVSTACK_WORKSPACE`` environment variable.
(macOS only) Share the cloned service directories in Docker, using
**Docker -> Preferences -> File Sharing** in the Docker menu.
.. _step 3:
3. Pull any changes made to the various images on which the devstack depends.
.. code:: sh
make dev.pull
.. Update rst to point to readthedocs once published.
Note -
If you are setting up devstack to develop on Open edx named releases, see this `document on developing on named releases`_ before following this step 3.
.. _document on developing on named releases: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/developing_on_named_release_branches.html
3. Optional: You have an option to use NFS on MacOS which may improve the performance significantly. To set it up ONLY ON MAC, do
.. code:: sh
make dev.nfs.setup
4. Run the provision command, if you haven't already, to configure the various
services with superusers (for development without the auth service) and
tenants (for multi-tenancy).
**NOTE:** When running the provision command, databases for ecommerce and edxapp
will be dropped and recreated.
The username and password for the superusers are both ``edx``. You can access
the services directly via Django admin at the ``/admin/`` path, or login via
single sign-on at ``/login/``.
Default:
.. code:: sh
make dev.provision
Provision using `docker-sync`_:
.. code:: sh
make dev.sync.provision
Provision using NFS:
.. code:: sh
make dev.nfs.provision
This is expected to take a while, produce a lot of output from a bunch of steps, and finally end with ``Provisioning complete!``
**NOTE:** This command will bring up both MySQL 5.6 and 5.7 databases until all services are upgraded to 5.7.
5. Start the services. This command will mount the repositories under the
``DEVSTACK_WORKSPACE`` directory.
**NOTE:** it may take up to 60 seconds for the LMS to start, even after the ``make dev.up`` command outputs ``done``.
Default:
.. code:: sh
make dev.up
Start using `docker-sync`_:
.. code:: sh
make dev.sync.up
Start using NFS:
.. code:: sh
make dev.nfs.up
To stop a service, use ``make dev.stop.<service>``, and to both stop it
and remove the container (along with any changes you have made
to the filesystem in the container) use ``make dev.down.<service>``.
After the services have started, if you need shell access to one of the
services, run ``make dev.shell.<service>``. For example to access the
Catalog/Course Discovery Service, you can run:
.. code:: sh
make dev.shell.discovery
To see logs from containers running in detached mode, you can either use
"Kitematic" (available from the "Docker for Mac" menu), or by running the
following:
.. code:: sh
make dev.logs
To view the logs of a specific service container run ``make dev.logs.<service>``.
For example, to access the logs for Ecommerce, you can run:
.. code:: sh
make dev.logs.ecommerce
For information on the supported ``make`` commands, you can run:
.. code:: sh
make help
Now that you're up and running, read about the `most common development workflow`_.
Usernames and Passwords
-----------------------
The provisioning script creates a Django superuser for every service.
::
Email: [email protected]
Username: edx
Password: edx
The LMS also includes demo accounts. The passwords for each of these accounts
is ``edx``.
.. list-table::
:widths: 20 60
:header-rows: 1
* - Account
- Description
* - ``[email protected]``
- An LMS and Studio user with course creation and editing permissions.
This user is a course team member with the Admin role, which gives
rights to work with the demonstration course in Studio, the LMS, and
Insights.
* - ``[email protected]``
- A student account that you can use to access the LMS for testing
verified certificates.
* - ``[email protected]``
- A student account that you can use to access the LMS for testing course
auditing.
* - ``[email protected]``
- A student account that you can use to access the LMS for testing honor
code certificates.
Service List
------------
These are the edX services that Devstack can provision, pull, run, attach to, etc.
Each service is accessible at ``localhost`` on a specific port.
The table below provides links to the homepage, API root, or API docs of each service,
as well as links to the repository where each service's code lives.
The services marked as ``Default`` are provisioned/pulled/run whenever you run
``make dev.provision`` / ``make dev.pull`` / ``make dev.up``, respectively.
The extra services are provisioned/pulled/run when specifically requested (e.g.,
``make dev.provision.xqueue`` / ``make dev.pull.xqueue`` / ``make dev.up.xqueue``).
Alternatively, you can run these by modifying the ``DEFAULT_SERVICES`` option as described in the `Advanced Configuration Options`_ section.
+------------------------------------+-------------------------------------+----------------+--------------+
| Service | URL | Type | Role |
+====================================+=====================================+================+==============+
| `lms`_ | http://localhost:18000/ | Python/Django | Default |
+------------------------------------+-------------------------------------+----------------+--------------+
| `studio`_ | http://localhost:18010/ | Python/Django | Default |
+------------------------------------+-------------------------------------+----------------+--------------+
| `forum`_ | http://localhost:44567/api/v1/ | Ruby/Sinatra | Default |
+------------------------------------+-------------------------------------+----------------+--------------+
| `discovery`_ | http://localhost:18381/api-docs/ | Python/Django | Default |
+------------------------------------+-------------------------------------+----------------+--------------+
| `ecommerce`_ | http://localhost:18130/dashboard/ | Python/Django | Default |
+------------------------------------+-------------------------------------+----------------+--------------+
| `credentials`_ | http://localhost:18150/api/v2/ | Python/Django | Default |
+------------------------------------+-------------------------------------+----------------+--------------+
| `edx_notes_api`_ | http://localhost:18120/api/v1/ | Python/Django | Default |
+------------------------------------+-------------------------------------+----------------+--------------+
| `frontend-app-payment`_ | http://localhost:1998/ | MFE (React.js) | Default |
+------------------------------------+-------------------------------------+----------------+--------------+
| `frontend-app-publisher`_ | http://localhost:18400/ | MFE (React.js) | Default |
+------------------------------------+-------------------------------------+----------------+--------------+
| `gradebook`_ | http://localhost:1994/ | MFE (React.js) | Default |
+------------------------------------+-------------------------------------+----------------+--------------+
| `registrar`_ | http://localhost:18734/api-docs/ | Python/Django | Extra |
+------------------------------------+-------------------------------------+----------------+--------------+
| `program-console`_ | http://localhost:1976/ | MFE (React.js) | Extra |
+------------------------------------+-------------------------------------+----------------+--------------+
| `frontend-app-learning`_ | http://localhost:2000/ | MFE (React.js) | Extra |
+------------------------------------+-------------------------------------+----------------+--------------+
| `frontend-app-library-authoring`_ | http://localhost:3001/ | MFE (React.js) | Extra |
+------------------------------------+-------------------------------------+----------------+--------------+
| `course-authoring`_ | http://localhost:2001/ | MFE (React.js) | Extra |
+------------------------------------+-------------------------------------+----------------+--------------+
| `xqueue`_ | http://localhost:18040/api/v1/ | Python/Django | Extra |
+------------------------------------+-------------------------------------+----------------+--------------+
.. _credentials: https://github.com/edx/credentials
.. _discovery: https://github.com/edx/course-discovery
.. _ecommerce: https://github.com/edx/ecommerce
.. _edx_notes_api: https://github.com/edx/edx-notes-api
.. _forum: https://github.com/edx/cs_comments_service
.. _frontend-app-payment: https://github.com/edx/frontend-app-payment
.. _frontend-app-publisher: https://github.com/edx/frontend-app-publisher
.. _gradebook: https://github.com/edx/frontend-app-gradebook
.. _lms: https://github.com/edx/edx-platform
.. _program-console: https://github.com/edx/frontend-app-program-console
.. _registrar: https://github.com/edx/registrar
.. _studio: https://github.com/edx/edx-platform
.. _lms: https://github.com/edx/edx-platform
.. _frontend-app-learning: https://github.com/edx/frontend-app-learning
.. _frontend-app-library-authoring: https://github.com/edx/frontend-app-library-authoring
.. _course-authoring: https://github.com/edx/frontend-app-course-authoring
.. _xqueue: https://github.com/edx/xqueue
Known Issues
------------
Currently, some containers rely on Elasticsearch 7 and some rely on Elasticsearch 1.5. This is
because services are in the process of being upgraded to Elasticsearch 7, but not all of them
support Elasticsearch 7 yet. As we complete these migrations, we will update the dependencies
of these containers.
Advanced Configuration Options
------------------------------
The file ``options.mk`` sets several configuration options to default values.
For example ``DEVSTACK_WORKSPACE`` (the folder where your Git repos are expected to be)
is set to this directory's parent directory by default,
and ``DEFAULT_SERVICES`` (the list of services that are provisioned and run by default)
is set to a fairly long list of services out of the box.
For more detail, refer to the comments in the file itself.
If you're feeling brave, you can create an git-ignored overrides file called
``options.local.mk`` in the same directory and set your own values. In general,
it's good to bring down containers before changing any settings.
Changing the Docker Compose Project Name
The COMPOSE_PROJECT_NAME
variable is used to define Docker namespaced volumes
and network based on this value, so changing it will give you a separate set of databases.
This is handled for you automatically by setting the OPENEDX_RELEASE
environment variable in options.mk
(e.g. COMPOSE_PROJECT_NAME=devstack-juniper.master
. Should you want to manually override this, edit the options.local.mk
in the root of this repo and create the file if it does not exist. Change the devstack project name by adding the following line::
Example: COMPOSE_PROJECT_NAME=secondarydevstack
COMPOSE_PROJECT_NAME=
As a specific example, if OPENEDX_RELEASE
is set in your environment as juniper.master
, then COMPOSE_PROJECT_NAME
will default to devstack-juniper.master
instead of devstack
.
.. _Docker Compose: https://docs.docker.com/compose/ .. _Docker for Mac: https://docs.docker.com/docker-for-mac/ .. _Docker for Windows: https://docs.docker.com/docker-for-windows/ .. _configuring Docker for Mac: https://docs.docker.com/docker-for-mac/#/advanced .. _feature added in Docker 17.05: https://github.com/edx/configuration/pull/3864 .. _edx-e2e-tests README: https://github.com/edx/edx-e2e-tests/#how-to-run-lms-and-studio-tests .. _edxops Docker image: https://hub.docker.com/r/edxops/ .. _Docker Hub: https://hub.docker.com/ .. _Pycharm Integration documentation: docs/pycharm_integration.rst .. _devpi documentation: docs/devpi.rst .. _edx-platform testing documentation: https://github.com/edx/edx-platform/blob/master/docs/guides/testing/testing.rst#running-python-unit-tests .. _docker-sync: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/troubleshoot_general_tips.html#improve-mac-osx-performance-with-docker-sync .. |Build Status| image:: https://travis-ci.com/edx/devstack.svg?branch=master :target: https://travis-ci.com/edx/devstack :alt: Travis .. |docs| image:: https://readthedocs.org/projects/docs/badge/?version=latest :alt: Documentation Status :scale: 100% :target: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/ .. _Python virtualenv: https://docs.python-guide.org/en/latest/dev/virtualenvs/#lower-level-virtualenv .. _Community: https://open.edx.org/community/connect/