jenkins-infra / Pipeline Library

= Pipeline Global Library for ci.jenkins.io

image:https://ci.jenkins.io/job/Infra/job/pipeline-library/job/master/badge/icon[link="https://ci.jenkins.io/job/Infra/job/pipeline-library/job/master/"] image:https://img.shields.io/github/v/release/jenkins-infra/pipeline-library[link="https://github.com/jenkins-infra/pipeline-library/releases"]

This repository contains a series of steps and variables for use inside of the Jenkins project's own link:https://ci.jenkins.io[Jenkins] instance(s).

Check link:https://github.com/jenkins-infra/documentation/blob/master/ci.adoc[this description] of available services.

== Useful steps:

=== buildPlugin

WARNING: Gradle support in buildPlugin() is deprecated and will be eventually removed. Please use buildPluginWithGradle()

Applies the appropriate defaults for building a Maven-based plugin project on Linux and Windows.

You are advised to be using a link:https://github.com/jenkinsci/plugin-pom/blob/master/README.md[2.x parent POM].

.Jenkinsfile [source,groovy]

buildPlugin()

==== Optional arguments

• repo (default: null inherit from Multibranch) - custom Git repository to check out
• useAci (default: false) - uses link:https://azure.microsoft.com/en-us/services/container-instances/[Azure Container Instances] for build instead of link:https://azure.microsoft.com/en-us/services/virtual-machines/[Azure Virtual Machines].
• failFast (default: true) - instruct Maven tests to fail fast
• platforms (default: ['linux', 'windows']) - Labels matching platforms to execute the steps against in parallel
• jdkVersions (default: [8]) - JDK version numbers, must match a version number jdk tool installed
• jenkinsVersions: (default: [null]) - a matrix of Jenkins baseline versions to build/test against in parallel (null means default, only available for Maven projects)
• configurations: An alternative way to specify platforms, jdkVersions and jenkinsVersions (that can not be combined with any of them). ** Those options will run the build for all combinations of their values. While that is desirable in many cases, configurations permit to provide a specific combinations of label and java/jenkins versions to use

[source,groovy]

buildPlugin(/.../, configurations: [ [ platform: "linux", jdk: "8", jenkins: null ], [ platform: "windows", jdk: "8", jenkins: null ], [ platform: "linux", jdk: "11", jenkins: "2.150", javaLevel: 8 ] ])

** It is also possible to use a buildPlugin.recommendedConfigurations() method to get recommended configurations for testing. Note that the recommended configuration may change over time, and hence your CI may break if the new recommended configuration is not compatible + [source,groovy]

buildPlugin(/.../, configurations: buildPlugin.recommendedConfigurations())

• tests: (default: null) - a map of parameters to run tests during the build. The test results and the JaCoCo code coverage results are recorded after the build with the corresponding Jenkins plugins. ** skip - If true, skip all the tests by setting the -skipTests profile. It will also skip FindBugs in modern Plugin POMs.
• spotbugs, checkstyle, pmd, cpd: (default: null) - a map of parameters to archive SpotBugs, CheckStyle, PMD, or CPD warnings, respectively (only available for Maven projects). These values can replace or amend the default configuration for the recordIssues step of the https://github.com/jenkinsci/warnings-ng-plugin[Warnings NG Plugin]. See https://github.com/jenkinsci/warnings-ng-plugin/blob/master/doc/Documentation.md#configuration[Warnings NG Plugin documentation] for a list of available configuration parameters.
• timeout: (default: 60) - the number of minutes for build timeout, cannot be bigger than 180, i.e. 3 hours.

NOTE: The recordIssues steps of the warnings plugin run on the first platform/jdkVersion,jenkinsVersion combination only. So in the example below it will run for linux/jdk7 but not on jdk8.

Usage:

.Jenkinsfile [source,groovy]

buildPlugin(platforms: ['linux'], jdkVersions: [7, 8], checkstyle: [qualityGates: [[threshold: 1, type: 'NEW', unstable: true]]], pmd: [trendChartType: 'TOOLS_ONLY', qualityGates: [[threshold: 1, type: 'NEW', unstable: true]]])

=== buildPluginWithGradle()

Builds a Jenkins plugin using Gradle. The implementation follows the standard build/test/archive pattern. The method targets compatibility with link:https://github.com/jenkinsci/gradle-jpi-plugin[Gradle JPI Plugin], and it may not work for other use-cases.

==== Optional arguments

• repo (default: null inherit from Multibranch) - custom Git repository to check out
• failFast (default: true) - instruct the build to fail fast when one of the configurations fail
• platforms (default: ['linux', 'windows']) - Labels matching platforms to execute the steps against in parallel
• jdkVersions (default: [8]) - JDK version numbers, must match a version number jdk tool installed
• configurations: An alternative way to specify platforms, jdkVersions (that can not be combined with any of them) ** Those options will run the build for all combinations of their values. While that is desirable in many cases, configurations permit to provide a specific combinations of label and java/jenkins versions to use

[source,groovy]

buildPluginWithGradle(/.../, configurations: [ [ platform: "linux", jdk: "8" ], [ platform: "windows", jdk: "8"], ])

** It is also possible to use a buildPlugin.recommendedConfigurations() method to get recommended configurations for testing. Note that the recommended configuration may change over time, and hence your CI may break if the new recommended configuration is not compatible + [source,groovy]

buildPluginWithGradle(/.../, configurations: buildPlugin.recommendedConfigurations())

• tests: (default: null) - a map of parameters to run tests during the build ** skip - If true, skip all the tests.
• timeout: (default: 60) - the number of minutes for build timeout, cannot be bigger than 180, i.e. 3 hours.

==== Limitations

Not all features of buildPlugin() for Maven are supported in the gradle flow. Examples of not supported features:

• Deployment of incremental versions (link:https://github.com/jenkinsci/jep/tree/master/jep/305[JEP-305])
• Publishing of static analysis and coverage reports (Checkstyle, SpotBugs, JaCoCo)
• Configuring jenkinsVersion and javaLevel for the build flow (as standalone arguments or as configurations)
• Usage of link:https://azure.microsoft.com/en-us/services/container-instances/[Azure Container Instances] as agents (only Maven agents are configured)

=== infra.isTrusted()

Determine whether the Pipeline is executing in an internal "trusted" Jenkins environment

.Jenkinsfile [source,groovy]

if (infra.isTrusted()) { /* perform some trusted action like a deployment */ }

=== infra.ensureInNode(env, nodeLabels, body)

Ensures that the given code block is runs in a node with the specified labels

.Jenkinsfile [source,groovy]

infra.ensureInNode(env, "docker,highmem", { sh "docker -v" })

=== infra.stashJenkinsWar(jenkins, stashName)

Given a version of jenkins downloads it if neccesary and stashes it under the given name (which defaults to "jenkinsWar", see the step doc for more documentation about he allowed versions

.Jenkinsfile [source,groovy]

infra.stashJenkinsWar("2.110")

=== runATH

Runs the link:https://github.com/jenkinsci/acceptance-test-harness[Acceptance Test Harness] in a configurable way.

The configuration is divided into two parts, one related to the step itself and another related to how the ATH is run. To configure the step just use the step's parameters described below, to configure the ATH runs a metadata file (in YAML format) is used. Further sections describe the metadata file in detail. Note that if the metadata file does not exist this step will do nothing at all.

The list of step's params and the related default values are:

athUrl:: The URL to get the ATH sources. It can point to a local path (by using the file:// protocol) or a github destination. Defaults to https://github.com/jenkinsci/acceptance-test-harness.git. Can be overridden from the metadata file athRevision:: The ATH revision to use, can be a branch or tag name or a commit id. Defaults to branch master. Can be overridden from the metadata file athImage :: The docker image used for the environment where to run the ATH. Defaults to "jenkins/ath". Use "local" to build the image directly from the ATH sources. Can be overridden from the metadata file metadataFile:: A String indicating the file path (relative to where this step is executed) to use as metadata file for the build, more details about the metadata file are provided belows. Defaults to essentials.yml at the location where this step is invoked jenkins:: URI to the jenkins.war, Jenkins version or one of "latest", "latest-rc", "lts" and "lts-rc". Defaults to "latest". For local war files use the file:// protocol in the URI. Can be overriden from the metadata file jdks:: Java versions to use when running ATH. Defaults to 8. Only 8 and 11 are supported. Can be overridden from the metadata file configFile:: (Optional) Relative (to the workspace) path of a groovy script to customize the ATH behaviour .Step call example [source,groovy]

runATH(metadataFile:"metadata.yml", athRevision: "master", athUrl:"https://github.com/jenkinsci/acceptance-test-harness.git", jenkins: "2.110")

To make it usable in PR builders this step allows users to run the ATH using custom (typically previously built in the same Jenkinsfile) versions of any plugin, for that you need to set the metadata file's useLocalSnapshots property to true and stash the plugins you want to use in the ATH run. By default you need to stash them with the name localPlugins the step will unstash them when appropriate and use the ATHsLOCAL_JARSproperty to run the ATH. You can stash any number of plugins, all of them will be used. You can also stash under other name by setting the env variableRUN_ATH_LOCAL_PLUGINS_STASH_NAME`

.Using development versions of plugins [source,groovy]

node("linux") { dir("sources") { checkout scm List mavenEnv = [ "JAVA_HOME=${tool 'jdk8'}", 'PATH+JAVA=${JAVA_HOME}/bin', "PATH+MAVEN=${tool 'mvn'}/bin"] withEnv(mavenEnv) { sh "mvn clean install -DskipTests" } dir("target") { stash includes: '*.hpi', name: 'localPlugins' }

      runATH(metadataFile:"metadata.yml", athRevision: "master")
    }
}

.Using custom stash name [source,groovy]

node("linux") {
    dir("sources") {
      checkout scm
      List<String> mavenEnv = [
                "JAVA_HOME=${tool 'jdk8'}",
                'PATH+JAVA=${JAVA_HOME}/bin',
                "PATH+MAVEN=${tool 'mvn'}/bin"]
      withEnv(mavenEnv) {
        sh "mvn clean install -DskipTests"
      }
      dir("target") {
       stash includes: '*.hpi', name: 'snapshots'
      }

      env.RUN_ATH_LOCAL_PLUGINS_STASH_NAME="snapshots"
      runATH(metadataFile:"metadata.yml", athRevision: "master")
    }
}

The metadata file is a YAML file with the following structure:

.metadata [source,yaml]

ath: athUrl: https://github.com/jenkinsci/acceptance-test-harness.git athRevision: acceptance-test-harness-1.59 athImage: "jenkins/ath" jenkins: 2.89.4 failFast: false rerunFailingTestsCount: 0 useLocalSnapshots: true browsers: - firefox - chrome tests: - Test1 - Test2 - Test3 categories: - Category1 - Category2 jdks: - 8 - 11

Where:

athUrl:: (Optional) The URL to get the ATH sources. It can point to a local path or a github destination. If specified it will override the parameter in the runATH step athRevision:: (Optional) The ATH revision to use can be a branch or tag name or a commit id. If specified it will override the parameter in the runATH step athImage :: (Optional) The docker image used for the environment where to run the ATH. Defaults to "jenkins/ath". Use "local" to build the image directly from the ATH sources. jenkins:: (Optional) URI to the jenkins.war file, Jenkins version or one of "latest", "latest-rc", "lts" and "lts-rc". If specified it will override the parameter in the runATH step failFast:: If the run has to fail fast or not. Defaults to false if not specified rerunFailingTestsCount:: The number of runs per failing test (a la maven). Defaults to zero useLocalSnapshots:: If the ATH should use local versions of the plugins. Defaults to true. Note that if true the runATH expects the local plugins to be stashed, setting this to true without the stash will make the step fail browsers:: The list of browsers to use when running ATH Defaults to firefox. Note that currently only firefox browser is supported, any other will be ignored tests:: The list of tests to run for the component that calls the step. If no particular set of tests or categories is defined the SmokeTest Category of the ATH will be run categories:: The list of Categories to run. Defaults to nothing jdks:: The list of jdks to use when running ATH. Defaults to 8. Note that currently only 8 and 11 are supported, any other will be ignored

In case you want to use the defaults for all properties you can use

.metadata [source,yaml]

ath: "default"

Please note that a blank metadata file will result in an error

=== runPCT

Runs the link:https://github.com/jenkinsci/plugin-compat-tester[Plugin Compat Tester] in a configurable way.

The configuration is divided into two parts, one related to the step itself and another related to how the PCT is run. To configure the step just use the step's parameters described below, to configure the PCT runs a metadata file (in YAML format) is used. Further sections describe the metadata file in detail. Note that if the metadata file does not exist this step will do nothing at all.

The list of step's params and the related default values are:

pctUrl:: The URL to get the PCT Dockerfile or the pct docker image to use. It can point to a local path of PCT sources (by using the file:// protocol) or a github destination. You can also use this to directly specify a prebuilt PCT docker image by using the docker:// protocol, for example "docker://jenkins/pct". Can be overridden from the metadata file pctRevision:: The PCT revision to use in case that pctUrl points to a github destination, can be a branch or tag name or a commit id. Defaults to branch master. Can be overridden from the metadata file metadataFile:: A String indicating the file path (relative to where this step is executed) to use as metadata file for the build, more details about the metadata file are provided belows. Defaults to essentials.yml at the location where this step is invoked jenkins:: URI to the jenkins.war, Jenkins version or one of "latest", "latest-rc", "lts" and "lts-rc". Defaults to "latest". For local war files use the file:// protocol in the URI. Can be overriden from the metadata file pctExtraOptions:: List of extra PCT options to be passed to the PCT executable. Defaults to empty list. javaOptions:: List of extra Java options to be passed to the PCT executable. Defaults to empty list. dockerOptions:: List of extra options to be passed to PCT containers ( e.g. maven-repo:/root/.m2) jdkVersion:: The version of the JDK to use to run the tests. Should be 8 or 11. Defaults to 8.

.Step call example [source,groovy]

runPCT(metadataFile:"metadata.yml", pctUrl:"docker://mynamspace/pct", jenkins: "2.110")

To make it usable in PR builders this step allows users to run the PCT using custom (typically previously built in the same Jenkinsfile) versions of any plugin, for that you need to set the metadata file's useLocalSnapshots property to true and stash the plugins you want to use in the PCT run. By default you need to stash them with the namelocalPlugins the step will unstash them when appropriate and use the PCT`s docker image /pct/plugin-src volume to access the sources. You can stash any number of plugins, all of them will be tested as long as they are specified in the metadata file. You can also stash under other name by setting the env variable RUN_PCT_LOCAL_PLUGIN_SOURCES_STASH_NAME

.Using development versions of plugins [source,groovy]

node("docker&&highmem") { deleteDir() dir("localPlugins") { sh "git clone https://github.com/jenkinsci/ssh-slaves-plugin.git ssh-slaves -b ssh-slaves-1.25" sh "git clone https://github.com/jenkinsci/credentials-plugin.git Credentials" stash 'localPlugins' } runPCT() }

The metadata file is a YAML file with the following structure:

.metadata [source,yaml]

pct: pctUrl: "https://github.com/jenkinsci/plugin-compat-tester.git" pctRevision: "master" jenkins: 2.89.4 useLocalSnapshots: true plugins: - Credentials

Where:

pctUrl:: (Optional) The URL to get the PCT Dockerfile or the pct docker image to use. It can point to a local path of PCT sources (by using the file:// protocol) or a github destination. You can also use this to directly specify a prebuilt PCT docker image by using the docker:// protocol, for example "docker://jenkins/pct". pctRevision:: (Optional) The PCT revision to use in case that pctUrl points to a github destination, can be a branch or tag name or a commit id. Defaults to branch master. jenkins:: (Optional) URI to the jenkins.war file, Jenkins version or one of "latest", "latest-rc", "lts" and "lts-rc". If specified it will override the parameter in the runATH step useLocalSnapshots:: If the ATH should use local versions of the plugins. Defaults to true. Note that if true the runPCT expects the local plugins to be stashed, setting this to true without the stash will make the step fail plugins:: The list of plugins to run, you must specify the artifactID of the plugin. Defaults to nothing

In case you want to use the defaults for all properties you can use

.metadata [source,yaml]

pct: "default"

Please note that a blank metadata file will result in an error

=== runBenchmarks

Runs JMH benchmarks and archives benchmark reports on highmem nodes.

Supported parameters:

artifacts:: (Optional) If artifacts is not null, invokes archiveArtifacts with the given string value.

==== Example

[source, groovy]

runBenchmarks('jmh-report.json')

=== buildDockerAndPublishImage(imageName, config)

Lints, Builds, then publishes a docker image.

Adds a bunch of build args you can use in your docker image:

• GIT_COMMIT_REV - The commit that triggered this build
• GIT_SCM_URL - Url to repo
• BUILD_DATE - Date that the image was built (now)

Supported parameters:

imageName:: Name of the docker image to build

configs:: (Optional) extra flags

registry: override the smart default of jenkinsciinfra/ or jenkins4eval/ dockerfile: override the default dockerfile of Dockerfile

==== Example [source, groovy]

buildDockerImage_k8s('plugins-site-api')

=== Design documents for runATH and runPCT

The design and some more details about the runATH and runPCT steps can be found link:https://wiki.jenkins.io/display/JENKINS/runATH+and+runPCT+step+design[here]

== Contribute

=== Requirements

• (Open)JDK v8
• Maven 3.6.x

===