All Projects → EmbarkStudios → buildkite-jobify

EmbarkStudios / buildkite-jobify

Licence: other
👷 Kubekite, but in Rust, using configuration from your repos 🦀

Programming Languages

rust
11053 projects

Labels

Projects that are alternatives of or similar to buildkite-jobify

buildkite-agent-scaler
📈A lambda for scaling an AutoScalingGroup based on Buildkite metrics
Stars: ✭ 25 (+56.25%)
Mutual labels:  buildkite
junit-annotate-buildkite-plugin
📈 Summarise your test failures as a build annotation
Stars: ✭ 18 (+12.5%)
Mutual labels:  buildkite
bksr
Run and test your Buildkite pipeline steps locally, just as they'd run in CI (discontinued)
Stars: ✭ 22 (+37.5%)
Mutual labels:  buildkite
knapsack pro-ruby
Knapsack Pro gem splits tests across parallel CI nodes and makes sure that tests will run in optimal time on each node.
Stars: ✭ 101 (+531.25%)
Mutual labels:  buildkite
k8s-buildkite-plugin
Run any buildkite build step as a Kubernetes Job
Stars: ✭ 37 (+131.25%)
Mutual labels:  buildkite
golang-docker-example
An example of how to run a Golang project in Docker in a Buildkite pipeline
Stars: ✭ 18 (+12.5%)
Mutual labels:  buildkite
rails-docker-parallel-example
An example of how to run Rails CI and test steps in parallel with Docker and Buildkite
Stars: ✭ 19 (+18.75%)
Mutual labels:  buildkite
nodejs-docker-example
An example of how to run a Node.js project in Docker in a Buildkite pipeline
Stars: ✭ 39 (+143.75%)
Mutual labels:  buildkite
artifacts-buildkite-plugin
🆙 Automatically upload and download artifacts
Stars: ✭ 26 (+62.5%)
Mutual labels:  buildkite
ecr-buildkite-plugin
🔐 Login to an AWS ECR registry
Stars: ✭ 24 (+50%)
Mutual labels:  buildkite
pybuildkite
A Python library for the Buildkite API
Stars: ✭ 29 (+81.25%)
Mutual labels:  buildkite
on-demand
CloudFormation resources for scheduling On-Demand Buildkite Agents with AWS ECS and AWS Fargate
Stars: ✭ 19 (+18.75%)
Mutual labels:  buildkite
docker-compose-buildkite-plugin
🐳⚡️ Run build scripts, and build + push images, w/ Docker Compose
Stars: ✭ 137 (+756.25%)
Mutual labels:  buildkite
docker-buildkite-plugin
🐳📦 Run any build step in a Docker container
Stars: ✭ 90 (+462.5%)
Mutual labels:  buildkite
gitea-buildkite-connector
Connect Gitea & Buildkite
Stars: ✭ 16 (+0%)
Mutual labels:  buildkite

👷 buildkite-jobify

Embark Embark dependency status Build status

Watches 1 or more Buildkite pipelines via the GraphQL API to kick off temporary Kubernetes jobs that (presumably) spin up Buildkite agents to pick up the Buildkite jobs that Buildkite has scheduled but not yet found an agent for.

Repository Setup

The configuration of the Kubernetes jobs specs and the which agent tags correspond to them are expected to be stored in the repositories that the pipelines are attached to, thus keeping the CI configuration within that repository.

The pipeline upload job for your repository should create a gzipped tarball of your configuration and upload it before adding the rest of the pipeline's steps so that the program can download the exact configuration for the commit being built.

Here is an example bash script, intended to work in an Alpine Linux container.

#!/bin/bash
set -eu

echo "--- :aws-artifact: Uploading Jobify templates"
archive="jobify.tar.gz"

# Create a tarball of our jobify files
tar c -zf "$archive" -C .ci/jobify .

# Checksum the tarball and use that as our key
chksum=$(sha1sum $archive | awk '{ print $1 }')

# TODO: Just support CAS artifacts, and only upload unique ones (but maybe buildkite already does?)

# Upload the tarball as a Buildkite artifact
buildkite-agent artifact upload "$archive" &> upload_out
rm "$archive"

upload_info=$(cat upload_out)

# Buildkite doesn't seem to have a convenient way to know the ID assigned to the artifact
# so just cheat and parse its output for now
rx='Uploading\sartifact\s([0-9a-f-]{36})\sjobify\.tar\.gz'
if [[ "$upload_info" =~ $rx ]]; then
    # Set a metadata key to point to the archive we uploaded so that jobify can pick it up
    buildkite-agent meta-data set "jobify-artifact-id" "${BASH_REMATCH[1]}"
    # Set the chksum so we can avoid an additional lookup if jobify already has the data
    buildkite-agent meta-data set "jobify-artifact-chksum" "$chksum"
else
    echo "Failed to upload jobify configuration"
    exit 1
fi

echo "--- Uploading :pipeline:"
# Upload our main pipeline to kick off the rest of the build
buildkite-agent pipeline upload "$(dirname "$0")/pipeline.yml"

Contributing

Contributor Covenant

We welcome community contributions to this project.

Please read our Contributor Guide for more information on how to get started.

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].