Deploying a docker registry on Kubernetes

This document will detail the procedure to deploy a private docker registry as a pod on Kubernetes. Having your own private repository for docker images gives absolute control over storage options, tightening access control and implementing authentication practices, all customized per your need. A Kubernetes docker registry is a docker registry running as a Kubernetes pod. Like any storage service, there will be a volume attached to the pod that stores your private docker images, and you can set up access controls for the registry via access controls of the Kubernetes pod. Good examples of a docker registry is the public Dockerhub, there is also Amazon Elastic Container Registry, Azure Container Registry and Artifact Registry from Google. There is also tools like Sonatype’s Nexus, JFrog’s Artifactory and VMware Harbor are popularly used tools for systems that prefer staying cloud agnostic.

Namespace

We will deploy everything in a namespace to make management easier and clean.

kubectl create namespace docker-registry

Security

We will start with authentication to have basic security.

htpasswd

We will generate the user / password for authentication. For this we will use htpasswd to generate a file that will be used to access the registry. For this demonstration we will use myuser as the username and mypasswd as the password.

$ mkdir auth

$  docker run --rm --entrypoint htpasswd registry:2.7.0 -Bbn myuser mypasswd > auth/htpasswd
Unable to find image 'registry:2.7.0' locally
2.7.0: Pulling from library/registry
cd784148e348: Pull complete
0ecb9b11388e: Pull complete
918b3ddb9613: Pull complete
5aa847785533: Pull complete
adee6f546269: Pull complete
Digest: sha256:1cd9409a311350c3072fe510b52046f104416376c126a479cef9a4dfe692cf57
Status: Downloaded newer image for registry:2.7.0

Generic secret

Here is the generic secret created in the cluster

$ kubectl create secret generic auth-secret --from-file=auth/htpasswd --namespace docker-registry

secret/auth-secret created

Persistent Storage

Now that we have basic security done, we need to create storage for our registry. For this we will use Persistent Volume and Claim. At Leaseweb, we will use the default storage class cloudstack-custom. We will not need to specify it during deployment as this is the default and only storage class on new cluster. This might differ if you are using S3 or other storage class and would need to be specified.

From Kubernetes documentation:

A Persistent Volume (PV) is a piece of volume and a cluster resource that you provision just like the nodes of the cluster. 

A Persistent Volume Claim (PVC) is a request for storage by a user in the cluster. Just like a Pod consumes node resources, PVCs request and consume PV resources. 

Click here to read more about PV and PVC. 

For this we will need to first create a manifest and then deploy it on the cluster.

Manifest

In a file called registry-pvc.yml we will define a PersistentVolumeClaim. You can change the storage size to your need. We do not need to specify the StorageClassName as the cloudstack csi is define as default on all deployed cluster at Leaseweb.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: docker-registry-pvc
  namespace: docker-registry
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  

Deployment

We then apply this to our cluster

$ kubectl apply -f registry-pvc.yml --namespace docker-registry

persistentvolumeclaim/docker-registry-pvc created

Now, we have a PVC to use the volume resources from. Let us start using this volume.

Registry

We will use Helm to deploy the registry. Helm is the package manager for Kubernetes, focused on automating the installation of all kinds of Kubernetes applications. We will add the repo, then create a values.yml to configure it and then deploy the registry.

Adding 

First let's add the repo twuni/docker-registry which is the successor of the original stable docker registry helm chart and then update the local cache to make sure we have the latest charts.

$ helm repo add twuni https://helm.twun.io

"twuni" has been added to your repositories

$ helm repo update

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "twuni" chart repository
Update Complete. ⎈Happy Helming!⎈

We can take a look and see the version we will install.

$ helm search repo docker-registry

NAME                 	CHART VERSION	APP VERSION	DESCRIPTION
twuni/docker-registry	2.2.2        	2.8.1      	A Helm chart for Docker Registry

Configuring

We will now create the values.yml that will be use to deploy the registry. Again here we do not need to specify the storageClass as it is defaulted to the cloudstack csi

---
replicaCount: 1
persistence:
  enabled: true
  size: 10Gi # <- adapt this value with the size you want to use
  deleteEnabled: true
  existingClaim: docker-registry-pvc # <- here we specify the pvc we created earlier
secrets:
  htpasswd: myuser:$2y$05$mDso3c3.tzcPUAE5NQsqmuYX0JoP8ihttoNBa59VaFPn8Dz6RQbkS # <- This is the username / password we created in the security section. You can replace this with your own username and password
ingress:
  enabled: true
  className: traefik
  path: /
  hosts:
    - registry.example.org   tls:
    - secretName: docker-registry-tls
      hosts:
        - registry.example.org 

Deployment

We then deploy to our cluster

$  helm install -f values.yml docker-registry --namespace docker-registry twuni/docker-registry

NAME: docker-registry
LAST DEPLOYED: Fri Nov 10 14:09:53 2023
NAMESPACE: docker-registry
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  https://registry.example.org/

Usage

Now that everything as been deployed we will go into how to use this.

For this we will use a simple nginx image to pull and push in our new private docker registry.

login

We will log on our new registry

$ docker login https://registry.example.org/

Username: myuser
Password:
Login Succeeded

We pull an image for example nginx:latest

$ docker pull nginx

Using default tag: latest
latest: Pulling from library/nginx
578acb154839: Pull complete
e398db710407: Pull complete
85c41ebe6d66: Pull complete
7170a263b582: Pull complete
8f28d06e2e2e: Pull complete
6f837de2f887: Pull complete
c1dfc7e1671e: Pull complete
Digest: sha256:86e53c4c16a6a276b204b0fd3a8143d86547c967dc8258b3d47c3a21bb68d3c6
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview nginx

We will now tag this with the private docker registry domain name and a custom tag

$ docker tag nginx registry.example.org/my-nginx

And finally we push on the private docker registry our newly tag image. 

$ docker push registry.example.org/my-nginx

Using default tag: latest
The push refers to repository [registry.example.org/my-nginx]
505f49f13fbe: Pushed
9920f1ebf52b: Pushed
768e28a222fd: Pushed
715b32fa0f12: Pushed
e503754c9a26: Pushed
609f2a18d224: Pushed
ec983b166360: Pushed
latest: digest: sha256:d2e65182b5fd330470eca9b8e23e8a1a0d87cc9b820eb1fb3f034bf8248d37ee size: 1778

Now we have an image in our private docker registry hosted on our kubernetes cluster. Let's use this in a deployment.

Using Docker Registry To Pull Images In Your Kubernetes Cluster

As you have a Docker registry deployed in your Kubernetes cluster, you can start using it by pulling previously pushed images for your Kubernetes Pods.

To learn how a private Docker registry can be used for pulling images, you will create a simple Kubernetes pod in a newly created  test namespace. This Kubernetes Pod will use the previously pushed image registry.example.org/my-nginx.

Namespace

First, you have to create a test Kubernetes namespace

$ kubectl create namespace test

namespace/test created

Secrets

Kubernetes Secret 

Excerpt from the official Kubernetes documentation, A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret means that you don't need to include confidential data in your application code. Because Secrets can be created independently of the Pods that use them, there is less risk of the Secret (and its data) being exposed during the workflow of creating, viewing, and editing Pods. Kubernetes, and applications that run in your cluster, can also take additional precautions with Secrets, such as avoiding writing sensitive data to nonvolatile storage. So we will now create a TLS type of secret and a Generic type of secret to mount our certificate and password, respectively.

We will reuse the username and password that was created earlier in this demonstration to create a Kubernetes secret that will be used by the deployment to access the registry.

We create the secret in the test namespace that we create a minute ago.

$ kubectl create secret docker-registry regcred --docker-server=registry.example.org --docker-username=myuser --docker-password=mypasswd -n test

secret/regcred created

Deployment

We now will deploy a pod to test all of this.

Manifest

We will create a pod manifest name test-nginx.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: test
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: registry.example.org/my-nginx
          ports:
            - containerPort: 80
      imagePullSecrets:
        - name: regcred # <- This point to the secret we created earlier so we can authenticate against the registry.

Deploy

We then apply our manifest

$ kubectl apply -f test-nginx.yml

deployment.apps/nginx-deployment created

We look at the deployment

$ kubectl get pods -n test

NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-5fc878d7b5-gjvsh   1/1     Running   0          2m8s


$ kubectl describe pods nginx-deployment-5fc878d7b5-gjvsh -n test

Name:             nginx-deployment-5fc878d7b5-gjvsh
Namespace:        test
Priority:         0
Service Account:  default

[...]

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  75s   default-scheduler  Successfully assigned test/nginx-deployment-5fc878d7b5-gjvsh to k8s-0000001-worker-0-2z6dj
  Normal  Pulling    75s   kubelet            Pulling image "registry.example.org/my-nginx"
  Normal  Pulled     71s   kubelet            Successfully pulled image "registry.example.org/my-nginx" in 3.885317258s (3.885349506s including waiting)
  Normal  Created    71s   kubelet            Created container nginx
  Normal  Started    71s   kubelet            Started container nginx

Hopefully, this article gave you a good overview of how to set up a private Docker registry in your Kubernetes cluster. Keep in mind that having a Private Docker registry is essential if deploying Docker services that are not Open Source or need more security.