kubernetes backup to Azure with velero

Introduction

I run my workloads on a Kubernetes cluster in my home-lab and wanted to create an offsite (cloud) backup.

Velero (formerly ark) is a neat project that supports a lot of options and cloud providers so I decided to take it for a spin. My specific scenario is currently only aiming at backing up the Kubernetes objects from a selected list of namespaces; backing up state (e.g. databases) will come later, either with Velero or with another tool like stash: I have not decided yet.

Setup

This is how my setup looks like:

    ----------------------
   | (3)  Azure Storage   |
    ----------------------
             ^
             |                 Azure Cloud
             |  ----------------------------------------------
             |              On-prem Datacenter
    ---------------------
   | (2)     Velero      |
    ---------------------
   | (1) on-prem k8s     |
    ---------------------

1. My workloads run on Pubernetes (Rancher) on my local Proxmox cluster
2. Velero runs on the cluster
3. The backups go to Azure storage

Preparing the Azure side

Make sure to run these commands on one shell as we need all the environment variables in the last step!

On the azure side we will create a storage account and a storage container:

export AZURE_BACKUP_RESOURCE_GROUP=Velero_Backups_PRD
export AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"

az login

az group create -n $AZURE_BACKUP_RESOURCE_GROUP --location WestEurope


az storage account create \
    --name $AZURE_STORAGE_ACCOUNT_ID \
    --resource-group $AZURE_BACKUP_RESOURCE_GROUP \
    --sku Standard_LRS \
    --encryption-services blob \
    --https-only true \
    --kind BlobStorage \
    --access-tier Hot

az storage container create -n velero-prd --public-access off --account-name $AZURE_STORAGE_ACCOUNT_ID

and a service principal that velero will use to access the storage:

export AZURE_RESOURCE_GROUP=Velero_Backups_PRD
export AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
export AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`

export AZURE_CLIENT_SECRET=$(openssl rand -base64 32)
echo $AZURE_CLIENT_SECRET

az ad sp create-for-rbac --name "velero-prd" --role "Contributor" --password $AZURE_CLIENT_SECRET

export AZURE_CLIENT_ID=`az ad sp list --display-name "velero" --query '[0].appId' -o tsv`

Finally we stash the credentials:

cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
EOF

Note: please make sure that you preserve these credentials in a safe place as they will be needed to access the backups from another cluster, e.g. in the case of a disaster or when moving workloads between clusters.

Install velero

Still in the same shell as we need some values from above.

All details can be found here

1. Download and install the binaries of the latest version. I am using 1.0.0
2. Install

velero install \
    --provider azure \
    --bucket velero-prd \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID

Schedule backups

I run backups for chosen namespaces and schedule independent scheduled backups for each:

1. Obtain the list of namespaces: kubectl get ns | sed 1d > namespaces
2. Edit the file and keep the namespaces you are interested in
3. Create the schedules:

#!/bin/bash

for NS in $(cat namespaces)
do
  echo Create schedule for namespace $NS
  #velero schedule delete $NS
  velero schedule create  $NS \
  --schedule="@daily" \
  --include-namespaces $NS
done