Testing gitlab-ci pipelines locally

Introduction

Debugging gitlab-ci pipelines can be a tedious task, especially as the pipeline does not run in the inner loop.

Fortunately the gitlab-runner can be installed locally, allowing you to test many aspects of the CI/CD pipeline prior to commit.

Understanding how environment variables are handled and which ones will be available in the runner is critical for testing.

General recommendations

Generally I recommend putting some effort in defining what has to run in the pipeline, vs. what can be refactored into a stand-alone script (e.g. bash) so that it can be run locally as well. I have the habbit of doing my deployments from a script that can be called from the pipeline as well as for local deployment. For this to work in both environments certain environment variables (of my own and as per the gitlab $CI_ variables) need to be set before running the script locally.

Setting the environment variables

For local deployments I create a setenv script that can be sourced:

export NAMESPACE=blog-asksven-io
export DEPLOY_ENV=production # is either "production"or "development"
export CI_COMMIT_REF_SLUG=master
export CI_BUILD_REF=123456

• NAMESPACE is the namespace to deploy to, and is also set in .gitlab-ci.yml
• CI_COMMIT_REF_SLUGis the name of the branch
• CI_BUILD_REF is the SHA of the commit

Inner loop

When deploying locally I will be using the same bash-script deploy.sh as the one used in the pipeline, but prior to running the script I make sure the environment variables are set properly: source setenv && ./deploy.sh

Running gitlab-ci jobs locally

Install the runner

The runner can be installed for different platform from here

Running jobs

The syntax for running jobs locally is:

gitlab-runner exec docker "<job-name>"

Environment variables

When run locally the runner will make a good guess about the $CI_ environment variables, and will create the variables defined in your job or globally for the pipeline. It is important to understand what variables will be available by default and which ones need to be injected.

Let’s look into this repo’s .gilab-ci.yml:

variables:
  # Gitlab namespace with branch name: "namespace-branchname"
  NAMESPACE: ${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG}
  DOCKER_IMAGE_URL: asksven/blog.asksven.io
...
show env:
  image: docker:19.03.11
  variables:
    PLATFORMS: "linux/amd64,linux/arm64,linux/arm/v7"  
    DOCKER_CLI_EXPERIMENTAL: "enabled"  
  script:
    - env

NAMESPACE and DOCKER_IMAGE_URL are globally defined, PLATFORMS and DOCKER_CLI_EXPERIMENTAL at the job level.

Running gitlab-runner exec docker "show env" results in:

...
Executing "step_script" stage of the job script
$ env
CI_SERVER_REVISION=
FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=true
CI=true
CI_RUNNER_REVISION=ee2d2772
HOSTNAME=runner--project-0-concurrent-0
CI_JOB_STAGE=test
CI_SERVER_VERSION=
SHLVL=3
HOME=/root
OLDPWD=/
FF_NETWORK_PER_BUILD=false
CI_JOB_ID=1
CI_COMMIT_REF_NAME=master
CI_RUNNER_VERSION=13.2.0~beta.225.gee2d2772
FF_SKIP_NOOP_BUILD_STAGES=true
CI_BUILDS_DIR=/builds
CI_PROJECT_ID=0
GITLAB_CI=true
CI_COMMIT_SHA=c1e3074d6c933f55b94a77ee29bc3658f9d3d545
CI_CONCURRENT_ID=0
CI_PROJECT_DIR=/builds/project-0
DOCKER_IMAGE_URL=asksven/blog.asksven.io
FF_USE_DIRECT_DOWNLOAD=true
FF_SHELL_EXECUTOR_USE_LEGACY_PROCESS_KILL=false
CI_SERVER_NAME=GitLab CI
CI_JOB_TOKEN=
CI_CONCURRENT_PROJECT_ID=0
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DOCKER_CHANNEL=stable
CI_SERVER=yes
FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false
CI_JOB_IMAGE=docker:19.03.11
CI_RUNNER_SHORT_TOKEN=
PLATFORMS=linux/amd64,linux/arm64,linux/arm/v7
CI_REPOSITORY_URL=/home/sven/git/blog-asksven-io
DOCKER_VERSION=19.03.11
NAMESPACE=-
DOCKER_TLS_CERTDIR=/certs
DOCKER_HOST=tcp://docker:2375
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_DISPOSABLE_ENVIRONMENT=true
PWD=/builds/project-0
CI_COMMIT_BEFORE_SHA=c920d82c3f86246563a764f7ffb1395fa8fde0ff
DOCKER_CLI_EXPERIMENTAL=enabled
CI_JOB_NAME=show env

Let’s look at what happened to your environment variables:

• NAMESPACE is set to “-”
• DOCKER_IMAGE_URL is set properly to asksven/blog.asksven.io
• PLATFORMS is set properly to linux/amd64,linux/arm64,linux/arm/v7
• DOCKER_CLI_EXPERIMENTAL is set properly to true

In .gitlab-ci.yml NAMESPACE is defined as ${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG} and you should note that none of these variables as set: they need to be injected into the runner!

Injecting enviroment variables

Using --env on the command-line you can inject critical values to the runner:

gitlab-runner exec docker --env CI_COMMIT_REF_SLUG="master" --env CI_PROJECT_NAME="blog-asksven-io" "show env"

results in:

Skipping Git submodules setup
Executing "step_script" stage of the job script
$ env
...
CI_PROJECT_NAME=blog-asksven-io
...
NAMESPACE=blog-asksven-io-master
...
CI_COMMIT_REF_SLUG=master

Job succeeded

Local runner script

Since I already have a script setenv.sh to set-up the environment variables I need for local deployments I have decided to stay with the same pattern, and create a second script set-runner-env.sh (not checked-in, in .gitignore, since it contains critical information like the docker regitry credentials).

Finally I have created a wrapper to the runner: ops/local-runner.sh:

# !bin/bash

if [ "$1" == "" ]; then
  echo "local-runner.sh requires the job-name as argument"
  exit 1
fi  

source setenv.sh
source set-runner-env.sh

cd .. && gitlab-runner exec docker \
    --env CI_COMMIT_REF_SLUG=${CI_COMMIT_REF_SLUG} \
    --env CI_PROJECT_NAME=${CI_PROJECT_NAME} \
    --env DOCKER_REGISTRY_USER=${DOCKER_REGISTRY_USER} \
    --env DOCKER_REGISTRY=${DOCKER_REGISTRY} \
    -env DOCKER_REGISTRY_PASSWORD=${DOCKER_REGISTRY_PASSWORD} \
    "$1"

The script takes one argument: the name of the job.

Limitations

I have experienced following limitations:

• if in your target environment tht runner was deployed on kuberntes you will experience issues with the DOCKER_HOST variable