OBS Source Service obs-service-go_modules

Contents

• Overview
• Usage for packagers
• Compression format support
• OBS Source Service Build Mode support
• Building Go applications with vendored dependency modules
• Example
• Example _service configuration
• Transition note
• openSUSE RPM packages built using obs-service-go_modules
• Dependencies
• FAQ
• Support
• Contributing
• License

Overview

This is the git repository for devel:languages:go/obs-service-go_modules, an Open Build Service (OBS) Source Service to download, verify, and vendor Go module dependency sources. The authoritative source is https://github.com/openSUSE/obs-service-go_modules.

Using go.mod and go.sum present in a Go application, obs-service-go_modules will call Go tools in sequence:

go mod download
go mod verify
go mod vendor

obs-service-go_modules will create a vendor.tar.gz archive or other supported compression type containing the vendor/ directory populated by go mod vendor. The archive is generated in the rpm package directory, and can be committed to OBS to facilitate offline Go application package builds for openSUSE, SUSE, and numerous other distributions.

Usage for packagers

Presently it is assumed the Go application source is distributed as a compressed tarball named app-0.1.0.tar.gz or other supported compression type, unpacking to app-0.1.0/. The compression type can be specified using the compression parameter, and defaults to gz (gzip). obs-service-go_modules will autodetect tarball archives of the form app-0.1.0.tar.gz, where the RPM packaging uses spec file app.spec sharing the base name app.

Create a _service file containing:

<services>
  <service name="go_modules" mode="disabled">
  </service>
</services>

The archive name can alternatively be specified using service parameter archive.

Run osc command locally:

osc service disabledrun

See Example below for typical output with a complete _service file.

Compression format support

obs-service-go_modules reads and writes compressed tar archives using libarchive via the Python3 ctypes wrapper python3-libarchive-c. While libarchive supports numerous compression formats, obs-service-go_modules usage recommends limiting selections to gz (default), xz and zstd. Tables of representative compression method relative sizes and timings with Go vendored dependency sources (vendor/) are shown below.

Compression sizes with Go dependency sources

Compression timing with Go dependency sources

The above are an average of five runs on Intel i7-6820HQ CPU. The zstd format has a clear advantage in speed with reasonable compression ratio, and is likely to become the default compression method in a future release. Decompression timings are closely matched among gz, xz, and zstd compression methods.

OBS Source Service Build Mode support

OBS Source Services can run in one of several modes as shown in OBS Documentation: Using Source Services: Modes of Services.

Currently the recommended mode is disabled (aliased as manual) which implies:

• The service is run locally via explicit CLI call osc service disabledrun

• obs-service-go_modules and its dependencies python3, libarchive and python3-libarchive-c are installed on the local machine. In particular, python3-libarchive-c packages are not available in all SUSE repositories at this time.

• The resulting vendor.tar.gz or other supported compression type should be committed and referenced as an RPM Source:.

• Source: file names are as given in the RPM .spec, no _service: prefix is applied.

If and when obs-service-go_modules is available on OBS, additional modes including Default will be supported. Default runs server side after each commit and locally before every local build. This will enable tighter integration with the tar_scm / obs_scm source service, including uncommitted server-managed .obscpio source and vendor archives.

Building Go applications with vendored dependency modules

Go commands support building with vendored dependencies, but it is no longer on by default. Upstream has stated vendoring is not going away. To ensure the top-level vendor/ directory is used by go build, either:

• pass the argument go build -mod=vendor to each invocation

• set environment variable GOFLAGS=-mod=vendor to apply the setting to all invocations

More information about additional controls is available at: Go Module Knobs, Go Wiki: Modules: How do I use vendoring and Go Wiki: Modules: Old vs. new behavior

Example

Using the hugo static site generator as an example of a Go application with a large number of Go module dependencies, obs-service-go_modules produces vendor.tar.gz:

$ ls -1
hugo-0.57.2.tar.gz
hugo.changes
hugo.spec
_service
_servicedata

$ osc service disabledrun
INFO:obs-service-go_modules:Autodetecting archive since no archive param provided in _service
INFO:obs-service-go_modules:Archive autodetected at /path/to/prj/pkg/hugo-0.57.2.tar.gz
INFO:obs-service-go_modules:Using archive hugo-0.57.2.tar.gz
INFO:obs-service-go_modules:Extracting hugo-0.57.2.tar.gz to /path/to/tmpdir
INFO:obs-service-go_modules:Using go.mod found at /path/to/tmpdir/hugo-0.57.2/go.mod
INFO:obs-service-go_modules:go mod download
go: finding cloud.google.com/go v0.39.0
go: finding contrib.go.opencensus.io/exporter/aws v0.0.0-20181029163544-2befc13012d0
go: finding contrib.go.opencensus.io/exporter/ocagent v0.4.12
go: finding contrib.go.opencensus.io/exporter/stackdriver v0.11.0
go: finding contrib.go.opencensus.io/integrations/ocsql v0.1.4
go: finding contrib.go.opencensus.io/resource v0.0.0-20190131005048-21591786a5e0
go: finding github.com/Azure/azure-amqp-common-go v1.1.4
go: finding github.com/Azure/azure-pipeline-go v0.1.9
go: finding github.com/Azure/azure-sdk-for-go v27.3.0+incompatible
(elided: 193 additional entries)
go: finding gopkg.in/fsnotify.v1 v1.4.7
go: finding gopkg.in/resty.v1 v1.12.0
go: finding gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: finding gopkg.in/yaml.v2 v2.2.2
go: finding honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a
go: finding pack.ag/amqp v0.11.0
INFO:obs-service-go_modules:go mod verify
INFO:obs-service-go_modules:all modules verified
INFO:obs-service-go_modules:go mod vendor
INFO:obs-service-go_modules:Vendor go.mod dependencies to vendor.tar.gz

$ ls -1
hugo-0.57.2.tar.gz
hugo.changes
hugo.spec
_service
_servicedata
vendor.tar.gz

Example _service configuration

OBS Source Services obs-service-tar_scm, obs-service-set_version and obs-service-recompress can be used together to automate Go application source archive handling and support local development workflows:

<services>
  <service name="tar_scm" mode="disabled">
    <param name="url">git://github.com/gohugoio/hugo.git</param>
    <param name="scm">git</param>
    <param name="exclude">.git</param>
    <param name="revision">v0.57.2</param>
    <param name="versionformat">@PARENT_TAG@</param>
    <param name="changesgenerate">enable</param>
    <param name="versionrewrite-pattern">v(.*)</param>
  </service>
  <service name="set_version" mode="disabled">
    <param name="basename">hugo</param>
  </service>
  <service name="recompress" mode="disabled">
    <param name="file">*.tar</param>
    <param name="compression">gz</param>
  </service>
  <service name="go_modules" mode="disabled">
  </service>
</services>

Persistent state for changelog generation is stored in _servicedata.

Transition note

Until such time as obs-service-go_modules is available on OBS, vendor.tar.gz should be committed along with the Go application release tarball.

openSUSE RPM packages built using obs-service-go_modules

• hugo
• go-modiff
• gohack
• mod
• mgit

Dependencies

obs-service-go_modules requires:

• python3
• python-libarchive-c ctypes wrapper for the libarchive C library (added in v0.5.0)

The Python standard library supports only gzipped tar archives. The libarchive dependency was chosen to support additional compression types including xz and the cpio_newc used by .obscpio archives. Supported compression types are intentionally limited to gz, xz, zstd and cpio_newc to preserve future flexibility in the event eliminating the python-libarchive-c dependency is desirable.

FAQ

Q: Does vendor.tar.gz need to be committed to OBS package?

A: Currently yes. As long as obs-service-go_modules is run locally via osc service disabledrun, then vendor.tar.gz should be committed and referenced as an additional Source:. If and when obs-service-go_modules is available on OBS, additional strategies should be possible such as a vendor.cpio where the vendored dependencies are managed on the fly and will not need to be committed to OBS package revisions. The single source of truth go.mod and go.sum always remain with the application source code.

Q: Does obs-service-go_modules update dependencies to newer versions?

A: No. Go modules use Minimum Version Selection, selecting the minimum (oldest) version of a Go module that satisfies all go.mod entries in the transitive dependency set. Go modules are relatively new and real-world use remains to be seen, but the expectation is that dependency versions will increment at a measured pace driven by upstream projects making releases with a well-tested dependency set. It is a design goal that there should be no surprise updates pulled in, and the dependency set selected remains repeatable over time. These characteristics should be quite favorable for distribution packagers.

Q: Does obs-service-go_modules cache Go module downloads to save time and bandwidth?

A: Yes. For local use with osc service disabledrun, obs-service-go_modules uses the standard module cache ~/go/pkg/mod. Subsequent runs of obs-service-go_modules with a populated cache will finish in less time.

Q: Would obs-service-go_modules installed on OBS cache Go module downloads to conserve server resources?

A: Not directly. The Go module cache ~/go/pkg/mod would not persist between OBS build runs. Running a private Go proxy inside OBS could accomplish this, as well as provide protections against third-party service outages and upstream Go modules being removed by the author.

Support

obs-service-go_modules intends to be compatible with most upstream Go projects that use best practice source layouts and module conventions. If you are packaging a Go application with an uncommon project layout or nonstandard go.mod usage pattern that presents compatibility problems, please file an issue with description and reference the specific upstream tag or commit. While it may not be possible to support every unique upstream layout, a maintainer will evaluate feasibility of adding support for that special case or improve output messages to clearly indicate the error.

Contributing

In keeping with support objectives, feature ideas are welcome, particularly those that improve idiomatic OBS and RPM usage, packaging automation and commonality among Go application package sources. It is also a goal to keep manual configuration parameters in _service to a minimum for maintainability. Please file proposals as an issue to discuss feasibility and feature design leading to a subsequent implementation via pull request.

When creating a pull request, please enable the permission: "Maintainers are allowed to edit this pull request" if it is not on by default.

License

GNU General Public License v2.0 or later