How To Setup A Private Image Registry On K3s

Image for post
Image for post
Photo by Maksym Kaharlytskyi on Unsplash

It is not always appropriate to push ones own container images to a public registry. This post shows a quick way to create a private image registry inside a K3s Kubernetes cluster.

Please note, that with the following manifest, when the registry resources are being removed from the cluster, all images will be removed as well. There is a TODO in the very last line of the manifest that addresses this.

Also important to note: The registry is unsecured. Further steps have to be taken to secure it with methods like username/password or certificates which is not scope of this tutorial.

Before this manifest gets applied, the domain name under spec:rules:hostneeds to be changed accordingly.

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: docker-registry-ingress
annotations:
kubernetes.io/ingress.class: "traefik"
spec:
rules:
- host: registry.domain.de
http:
paths:
- path: /
backend:
serviceName: docker-registry-service
servicePort: 5000

---
apiVersion: v1
kind: Service
metadata:
name: docker-registry-service
labels:
run: docker-registry
spec:
selector:
app: docker-registry
ports:
- protocol: TCP
port: 5000

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: docker-registry
labels:
app: docker-registry
spec:
replicas: 1
selector:
matchLabels:
app: docker-registry
template:
metadata:
labels:
app: docker-registry
spec:
containers:
- name: docker-registry
image: registry
ports:
- containerPort: 5000
protocol: TCP
volumeMounts:
- name: storage
mountPath: /var/lib/registry
env:
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
volumes:
- name: storage
emptyDir: {} # TODO -make this more permanent later

The manifest gets applied with:

kubectl apply -f registry.yaml

On the K3s nodes, the file /etc/rancher/k3s/registries.yaml needs to be created with the following content. The domain name needs to match the one from above manifest.

mirrors:
"registry.domain.de":
endpoint:
- "http://registry.domain.de"

I did this on server and agent nodes. I am not sure if it really has to be done on the agents, I just did.

After that, server and agents need to be restarted.

On the server execute:

systemctl restart k3s

And on the agent node:

systemctl restart k3s-agent

If the changes applied can be checked with:

crictl info

There is a section called registry that should list the newly created private registry.

The local workstation also needs to know about the new registry. I am working on macOS with Docker Desktop. Under “Preferences -> Docker Engine”, the settings had to be extended with the following entry:

{
...

"insecure-registries": [
"registry.domain.de"
]
}

In order to test if an image can be pushed to the new registry, I have built a small image just containing Nginx with a custom HTML-file:

<html>
<head><title>Hello World!</title>
<style>
html {
font-size: 500.0%;
}
div {
text-align: center;
}
</style>
</head>
<body>
<div>Hello World!</div>
</body>
</html>

The Dockerfile for the new image:

FROM nginx:alpine
COPY index.html /usr/share/nginx/html

Build and tag the new image according to your registry domain:

docker build -t registry.domain.de/hello:latest .

Finally the image can be pushed.

docker push registry.domain.de/hello:latest

Hopefully that worked without any issues.

For an additional test, the image can also be removed from the local workstation and then be pulled again from the private registry.

docker rmi registry.domain.de/hello:latest
docker pull registry.domain.de/hello:latest

Documenting my Tech-Stack

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store