Kubernetes RBAC explained

Introduction

Whether it is from CI/CD or from the command-line, I often see the default kube-config with cluster-admin rights being used. This is like permanently working with root privileges and there certainly are more secure ways.

In this post we will look into demystifying Kubernetes RBAC, and setting-up more suitable permissions for two use-cases:

• a CI/CD pipeline that needs full permissions on anything located in a given Namespace
• a reader who needs to access resources for troubleshooting purposes

Concepts

Roles and ClusterRoles define sets of permissions to objects at the namespace and cluster scope. From my point of view this is the part that is the most difficult to get right: permissions are granted by declaring what verbs should be allowed for objects, by API group. To navigate this complexity kubectl api-resources -o wide comes to the rescue for finding-out the verbs a Kubernetes object support, and what API group it belongs to.

RoleBindings and ClusterRoleBindings assign Roles and ClusterRoles to either service accounts, users or groups.

In this post we will focus on service accounts. First we will define a Role for our first use case (CI/CD) and a ClusterRole for our reader. Then we will create two service accounts, and assign them the Role and ClusterRole, using a RoleBinding and a ClusterRoleBinding. And finally we will create kube-configs for these two use cases, so that kubernetes can be accessed using the API server in a least-privileged manner.

Note that you could use the same approach to create a service account to be used by a pod to get certain restricted permissions.

Use-case 1

In this use case we want to create a kube-config for a service account having full permissions in a namespace.

Create the namespace

kubectl create ns demo

Create the service account

kubectl -n demo create sa sa-cicd

Create the Role

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: demo
  name: cicd-full-permissions
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
EOF  

Create the role binding

cat <<EOF | kubectl apply -f -
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cicd-full-binding
  namespace: demo
subjects:
- kind: ServiceAccount
  name: sa-cicd
  namespace: demo
roleRef:
  kind: Role
  name: cicd-full-permissions
  apiGroup: rbac.authorization.k8s.io
EOF  

Create the kube-config

This is how a kube-config looks like:

apiVersion: v1
kind: Config
clusters:
- name: "my-cluster"
  cluster:
    server: "<api-server-url-goes-here>"
    certificate-authority-data: "<certificate-goes-here>"

users:
- name: "<username-goes-here>"
  user:
    token: "<token-goes-here>"

contexts:
- name: "my-context"
  context:
    user: "<username-goes-here>"
    cluster: "my-cluster"

current-context: "my-context"

We will save this to kube-config-cicd.yaml

Now we need to resolve the following placeholders:

• <api-server-url-goes-here>: run kubectl cluster-info and replace with the URL of the Kubernetes master, e.g. https://192.168.2.10:6443
• <username-goes-here>: this is the name of our service account: sa-cicd
• <token-goes-here>: see below
• <certificate-goes-here>: see below

First we will get the name of the secret containing the token and certificate: it is references by our service-account:

$ kubectl -n demo get sa sa-cicd -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-cicd
  namespace: demo
secrets:
- name: sa-cicd-token-5x4m5

Then we will retrieve the secret belonging to the service-account:

$ kubectl -n demo get secret sa-cicd-token-5x4m5 -o yaml
apiVersion: v1
data:
  ca.crt: [snip]
  namespace: ZGVtbw==
  token: [snip]
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: sa-cicd
  name: sa-cicd-token-5x4m5
  namespace: demo
type: kubernetes.io/service-account-token

Note: the tokenfield is base64-encoded, so they need to be decoded using echo -n '<value>' | base64 -d

Test the kube-config

Once you have finished replacing the placeholders in your kube-config-cicd.yaml it is time to test it.

Negative test

Note that we pass the KUBECONFIG to kubectl. This will not change the kube-config permanently, as opposed to export KUBECONFIG=<path-to-kube-config> would do.

$ KUBECONFIG=./kube-config-cicd.yaml kubectl -n kube-system get all
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:demo:sa-cicd" cannot list resource "pods" in API group "" in the namespace "kube-system"
[...]
Error from server (Forbidden): cronjobs.batch is forbidden: User "system:serviceaccount:demo:sa-cicd" cannot list resource "cronjobs" in API group "batch" in the namespace "kube-system"

As sa-cicd does not have any permissions in the namespace kube-system the API call fails.

Positive test

$ KUBECONFIG=./kube-config-cicd.yaml kubectl -n demo create sa another-sa
serviceaccount/another-sa created

Use-case 2

In this use case we want to create a kube-config for a service account having cluster-wide limited permissions: it should be able to enumerate objects, describe pods, deployments and replica sets (but not secrets), and get logs from pods.

Prepare the API groups and verbs

Here kubectl api-resources -o wide comes to the rescue:

1. Get the API group and verbs for pods: kubectl api-resources -o wide | grep pods
2. Get the API group and verbs for pods: kubectl api-resources -o wide | grep deployments
3. Get the API group and verbs for pods: kubectl api-resources -o wide | grep replicasets

Create the service account

kubectl -n default create sa sa-reader

Create the Role

cat <<EOF | kubectl apply -f -
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: reader-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]  
- apiGroups: ["apps"]
  resources: ["replicasets", "deployments"]
  verbs: ["get", "watch", "list"]
EOF  

Create the role binding

cat <<EOF | kubectl apply -f -
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: reader-clusterrolebinding
subjects:
- kind: ServiceAccount
  name: sa-reader
  namespace: default
roleRef:
  kind: ClusterRole
  name: reader-clusterrole
  apiGroup: rbac.authorization.k8s.io
EOF  

Create the kube-config

Same exercise as earlier, except the certificate and token come from sa-reader’s secret, and the user should be sa-reader. We will save the kube-config kube-config-reader.yaml

Test the kube-config

Positive test

$ KUBECONFIG=./kube-config-reader.yaml kubectl -n kube-system get pods
NAME                                      READY   STATUS    RESTARTS   AGE
metrics-server-6d684c7b5-jnwfg            1/1     Running   6          24d
coredns-6c6bb68b64-wr8jt                  1/1     Running   6          24d
local-path-provisioner-58fb86bdfd-jhlk2   1/1     Running   11         24d

$ KUBECONFIG=./kube-config-reader.yaml kubectl -n kube-system delete pod metrics-server-6d684c7b5-jnwfg
I0516 21:17:49.702917       1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0516 21:17:58.021430       1 secure_serving.go:116] Serving securely on [::]:443
E0516 21:19:04.279329       1 manager.go:111] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:armk8sgpunode1: unable to get CPU for container "traefik-internal" in pod traefik/traefik-internal-596d7765b7-b5ftf on node "armk8sgpunode1", discarding data: missing cpu usage metric

Negative test

$ KUBECONFIG=./kube-config-reader.yaml kubectl -n kube-system delete pod coredns-6c6bb68b64-wr8jt
Error from server (Forbidden): pods "metrics-server-6d684c7b5-jnwfg" is forbidden: User "system:serviceaccount:default:sa-reader" cannot delete resource "pods" in API group "" in the namespace "kube-system"

$ KUBECONFIG=./kube-config-reader.yaml kubectl -n kube-system get secret
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:sa-reader" cannot list resource "secrets" in API group "" in the namespace "kube-system"