Debian Build System

About Molior 1.4.0

Molior is based on aptly for managing Debian package repositories and sbuild for building Debian packages for multiple distributions and architectures.

Molior allows the following via WebUI, REST API or commandline tools:

• Manage Debian repository mirrors
• Manage Debian repositories grouped in projects and versions
• Manage project dependencies between base mirrors and other projects
• Provide build nodes (amd64, arm64) on VMs or bare metal for running sbuild
• Build debianized git repositories for multiple projects and architectures (i386, amd64, armhf, arm64)
• Create project deployments (ISO Installers, VM images, containers, ...)

Table of content

• Screenshots
• Installation
• Components
• Usage
  • Login to the Web UI
  • Creating Mirrors
    • Example Debian base mirror
  • Example a non-base mirrors
  • Create a project
  • Add a source repo
  • Integration
    • Trigger builds from gitlab
    • Build notification hooks
• Contributing
  • Clone the source repositories
  • Build
• Authors

Screenshots

Build List

Build Log

Build Context Menu

Build Errors

Molior Project List

Molior Project / Context Menu

Git Repositories

APT Repository Mirrors

Build Nodes

Installation

See INSTALL.md.

Components

The molior Debian Build System consists of the following components:

• molior-server
  • based on aiohttp
  • manages git repositories
  • manages project and versions
  • manages Debian repositories (aptly)
  • creates Debian source packages
  • provides REST API
  • provides build and deployment environments (schroot, debootstrap)
  • uses aptly REST API
  • uses PostgreSQL database
• molior-web
  • based on AngularJS, nodejs
  • uses molior REST API
• aptly
  • see http://aptly.info
  • manages Debian repository mirrors
  • manages Debian project repositories
  • provides REST API
  • contains molior specific API improvements
• molior-client
  • based on aiohttp
  • runs on build nodes
  • uses molior REST API
  • uses build environment
• molior-tools
  • create releases
  • create deployments
  • automation scripts

Usage

Login to the Web UI

Point your browser to the URL of the molior server, login with admin and the password defined in /etc/molior/molior/yml.

Creating Mirrors

Example Debian base mirror

The following will mirror Debian/stretch for amd64 and arm64. It will take approx. 73GB of disk space.

• Click on Mirror, New Mirror
• Name: stretch
• Version: 9.9
• Continue
• Check: Basemirror
• UnCheck: Mirror Source
• Uncheck: Mirror Installer
• Distribution: stretch
• Select Architectures: amd64, arm64
• Components: main
• Source: http://httpredir.debian.org/debian (or use apt-cacher-ng URL)
• Select: Use Mirror Key
• Continue
• Add the keys (separately): EF0F382A1A7B6500 8B48AD6246925553 7638D0442B90D010 04EE7237B7D453EC
• Key Server: hkp://keyserver.ubuntu.com:80
• Confirm

Depending on the network and disk performance, this might take a 2-3 hours.

Note: if you are mirroring Debian/buster use these keys: 04EE7237B7D453EC 648ACFD622F3D138 DCC9EFBF77E11517)

Example a non-base mirrors

Molior can create mirror of APT repositories, for example mono:

Name: mono
Version: 5.2
Dist: stretch
Archs: amd64
Source: http://download.mono-project.com/repo/debian
Keys: 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
Key Server: hkp://keyserver.ubuntu.com:80

or Docker:

Name: docker
Version: 17.09
Dist: stretch
Archs: amd64
Mirror source packages: no
Base mirror: no
Components: stable
Source: https://download.docker.com/linux/debian
Key URL: https://download.docker.com/linux/debian/gpg

Create a project

• Name: test
• Click new
• Project Version: 1.0
• Choose base mirror
• Choose amd64, arm64 arhcitecutres

Add a source repo

• Click project: test
• Click project version: 1.0
• Click Repositories
• Click NEW REPOSITORY
• URL: https://github.com/neolynx/sold.git
• Click Continue
• Click Continue

Integration

Trigger builds from gitlab

In GitLab:

• Go to Settings/Integrations (or Administration/System-Hooks)
• Enter URL: http://moliorserver/api/build/gitlab (replace with your molior instance)
• Choose secret token id authenticated triggers are desired
  • Configure secret token in /etc/molior/molior/yml (gitlab/auth_token)
• Select "Push events" if CI builds are desired
• Select "Tag push events"

Build notification hooks

Molior can trigger a REST API when build states change.

    POST https://remoteserver/api/{{build.commit|urlencode}}

    {
        "key":"molior-{{platform.distrelease}}-{{platform.version}}-{{platform.architecture}}-{{project.name}}-{{project.version}}",

        "name":"Molior {{platform.architecture}} / {{platform.version}} / {{platform.distrelease}} Build for {{build.commit}}",
        {% if build.status == "building" %}
        "state":"INPROGRESS",
        {% elif build.status == "successful" %}
        "state":"SUCCESSFUL",
        {% else %}
        "state":"FAILED",
        {% endif %}
        "description":"{{build.status}}",
        "url":"{{build.url}}"
    }

Contributing

You are welcome to contribute to the project !

• Feel free to open issues with questions, suggestions and improvements
• Pull requests are welcome, please consider the following:
  • Follow the rules of PEP8
  • Use Google docstrings. (for automated docs generation)
  • Make changes backward compatible and upgradeable (especially the database changes)
  • Unit test the changes, whenever possible

The following chapters describe how to setup a development environment. For building Debian packages in molior, the creation of a Debian mirror is required. This needs approximately 80GB of free disk space.

Clone the source repositories

The following source respositories are needed:

# molior
git clone https://github.com/molior-dbs/molior.git
git clone https://github.com/molior-dbs/molior-web.git

# aptly
git clone https://github.com/molior-dbs/aptlydeb.git
cd aptlydeb
git submodule init
git submodule update
cd ..

# optionally:
git clone https://github.com/neolynx/launchy.git
git clone https://github.com/neolynx/cirrina.git
git clone https://github.com/neolynx/aiohttp_jrpc.git
git clone https://github.com/neolynx/aiohttp-session.git
git clone https://github.com/neolynx/aiohttp-swagger.git
git clone https://github.com/molior-dbs/aiofile.git
git clone https://github.com/molior-dbs/async-cron.git
git clone https://github.com/molior-dbs/git-url-parse.git

Build

All git repositories are debianized. Build dependencies can be installed by running the following in a git repo:

apt-get build-dep .

Debian packages can be build with:

debuild -us -uc -b

Authors

• André Roth
• Karol Swiderski
• Benjamin Fassbind