All Projects → google → apt-golang-s3

google / apt-golang-s3

Licence: Apache-2.0 License
An s3 transport method for the apt package management system

Programming Languages

go
31211 projects - #10 most used programming language
shell
77523 projects
Dockerfile
14818 projects

Labels

Projects that are alternatives of or similar to apt-golang-s3

ansible-apt
Ansible role to manage packages and up(date|grade)s in Debian-like systems
Stars: ✭ 21 (-36.36%)
Mutual labels:  apt
punic
Punic is a remote cache CLI built for Carthage and Apple .xcframework
Stars: ✭ 25 (-24.24%)
Mutual labels:  s3
laravel-uppy-s3-multipart-upload
Multipart Uploads using Laravel, AWS S3, and Uppy
Stars: ✭ 30 (-9.09%)
Mutual labels:  s3
astro
Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
Stars: ✭ 79 (+139.39%)
Mutual labels:  s3
LOLBAS222
APT || Execution || Launch || APTs || ( Authors harr0ey, bohops )
Stars: ✭ 100 (+203.03%)
Mutual labels:  apt
apt-mirror-docker
Up to date apt-mirror script, containerized for mirroring + serving.
Stars: ✭ 15 (-54.55%)
Mutual labels:  apt
acid-store
A library for secure, deduplicated, transactional, and verifiable data storage
Stars: ✭ 48 (+45.45%)
Mutual labels:  s3
gitlab-mattermost-backup
A simple backup script for mattermost in gitlab omnibus package
Stars: ✭ 23 (-30.3%)
Mutual labels:  s3
Training
Defensive Origins Training Schedule
Stars: ✭ 33 (+0%)
Mutual labels:  apt
docker-aws-s3-sync
Docker container to sync a folder to Amazon S3
Stars: ✭ 21 (-36.36%)
Mutual labels:  s3
nginx-s3-gateway
NGINX S3 Caching Gateway
Stars: ✭ 124 (+275.76%)
Mutual labels:  s3
decimation.github.io
Cydia repo
Stars: ✭ 18 (-45.45%)
Mutual labels:  apt
websync
Like `aws s3 sync` with automatic CloudFront invalidations and more.
Stars: ✭ 45 (+36.36%)
Mutual labels:  s3
fastaop
🚀 lightweight, high-performance AOP framework based on Java Annotation Processing, similar to Lombok
Stars: ✭ 87 (+163.64%)
Mutual labels:  apt
aptdec
NOAA APT satellite imagery decoder (not-so-WIP)
Stars: ✭ 43 (+30.3%)
Mutual labels:  apt
GooglePlay-Web-Crawler
Mapreduce project by Hadoop, Nutch, AWS EMR, Pig, Tez, Hive
Stars: ✭ 18 (-45.45%)
Mutual labels:  s3
S4
🔄 Fast and cheap synchronisation of files using Amazon S3
Stars: ✭ 69 (+109.09%)
Mutual labels:  s3
Less3
Less3 is an S3-compatible object storage server that runs on your laptop, servers, just about anywhere!
Stars: ✭ 16 (-51.52%)
Mutual labels:  s3
herman
Herman is a tool to simplify deployment of AWS Services using ECS and Lambda, and the provisioning of various AWS services.
Stars: ✭ 33 (+0%)
Mutual labels:  s3
RapidORM
Quick solutions for Android ORM
Stars: ✭ 24 (-27.27%)
Mutual labels:  apt

apt-golang-s3

An s3 transport method for the apt package management system

Build Status Go Report Card GoDoc

The apt-golang-s3 project provides support for hosting private apt repositories in Amazon S3. This is useful if you have private packages, vendored public packages, or forks of public packages that your software or business depend on. There are several opensource projects that solve this problem, but they come with some limitations.

  1. They are unmaintained.
  2. They don't support the S3v4 request signature method.
  3. They are written in a language that requires a runtime or other dependencies.

This project is an attempt to address those limitations.

TL;DR

  1. Build the binary $ go build -o apt-golang-s3 main.go
  2. Install the binary $ sudo cp apt-golang-s3 /usr/lib/apt/methods/s3
  3. Add your s3 based source to a package list $ echo "deb s3://access-key:[email protected]/private-repo-bucket stable main" > /etc/apt/sources.list.d/private-repo.list
  4. Update and install packages $ sudo apt-get update && sudo apt-get install your-private-package

Building the go program

There is an included Dockerfile to setup an environment for building the binary in a sandboxed environment:

$ ls
Dockerfile  main.go  method  README.md

$ docker build -t apt-golang-s3 .
...

$ docker run -it --rm -v $(pwd):/app apt-golang-s3 bash

root@83823fffd369:/app# ls
Dockerfile  README.md  build-deb.sh  go.mod  go.sum  main.go  method

root@83823fffd369:/app# go build -o apt-golang-s3 main.go
...

root@83823fffd369:/app# ls
Dockerfile  README.md  apt-golang-s3  build-deb.sh  go.mod  go.sum  main.go  method

root@83823fffd369:/app# exit
exit

$ ls
apt-golang-s3  build-deb.sh  Dockerfile  go.mod  go.sum  main.go  method  README.md

Building a debian package

For convenience, there is a small bash script in the repository that can build the binary and package it as a .deb.

$ ls
build-deb.sh  Dockerfile  go.mod  go.sum  main.go  method  README.md

$ docker build -t apt-golang-s3 .

$ docker run -it --rm -v $(pwd):/app apt-golang-s3 /app/build-deb.sh
...
Created package {:path=>"apt-golang-s3_1_amd64.deb"}

$ ls
apt-golang-s3  apt-golang-s3_1_amd64.deb  build-deb.sh  Dockerfile  go.mod  go.sum  main.go  method  README.md

Installing in production

The apt-golang-s3 binary is an executable. To install it copy it to /usr/lib/apt/methods/s3 on your computer. The .deb file produced by build-deb.sh will install the method in the correct place.

Configuration

APT Repository Source Configuration

We recommend issuing a separate set of API keys, with read-only access, to the S3 bucket that hosts your repository. The keys are specified in the apt sources list configuration as follows:

$ cat /etc/apt/sources.list.d/my-private-repo.list
deb s3://aws-access-key-id:[email protected]/my-private-repo-bucket stable main

APT Method Configuration

The current default AWS region is set to us-east-1, but can be overridden by adding an option in your apt configuration, e.g.

echo "Acquire::s3::region us-east-1;" > /etc/apt/apt.conf.d/s3

Alternatively, you may specify an IAM role to assume before connecting to S3. The role will be assumed using the default credential chain; this option is mutually exclusive with static credentials in the S3 URL.

echo "Acquire::s3::role arn:aws:iam::123456789012:role/s3-apt-reader;" > /etc/apt/apt.conf.d/s3

Additional configuration options may be added in the future.

How it works

Apt creates a child process using the /usr/lib/apt/methods/s3 binary and writes to that processes standard input using a specific protocol. The method interprets the input, downloads the requested files, and communicates back to apt by writing to its standard output. The protocol spec is available here http://www.fifi.org/doc/libapt-pkg-doc/method.html/ch2.html.

Similar Projects

Disclaimer

This is not an officially supported Google product.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].