This document will be the basis for deploying ingress on a Kubernetes cluster. We will demonstrate both Ingress NGINX Controller and traefik with cert-manager and Let’s Encrypt. The demonstration will use normal deployment as well as Helm Chart deployment. We will also use the whoami application as an example of an application which also give information about headers and client IP.
We will also indicate how to bind one of the /29 IPs given to each client when they order a managed Kubernetes cluster.
Version at time of writing are : cert-manager: v1.13.2, NGINX Controller: controller-v1.9.4, traefik: v2.10.5
cert-manager
We will start with cert-manager. This will be a basic installation with no customization needed and is agnostic of the ingress controller used.
Helm
We add the repo to be able to deploy from it:
$ helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
$ helm repo update
...Successfully got an update from the "jetstack" chart repositorycert-manager requires a number of Custom resource definitions (CRDs) to be installed to work so we need to add parameter to the helm deployment.
To install with the CRDs we need to execute this command:
$ helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.13.2 \
--set installCRDs=true # <- this will install the custom resources (CRDs)
NAME: cert-manager
LAST DEPLOYED: Mon Oct 30 14:34:09 2023
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager v1.13.2 has been deployed successfully!
In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).
More information on the different types of issuers and how to configure them
can be found in our documentation:
https://cert-manager.io/docs/configuration/
For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:
https://cert-manager.io/docs/usage/ingress/kubectl deployment
This is the default static way which also include the CRDs:
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
configmap/cert-manager created
configmap/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook createdMore details and other information can be found on the official site : https://cert-manager.io/docs/.
NGINX Ingress
We will now deploy the NGINX ingress:
Helm
For the ingress to work properly we need to customize it to play nice with our Cloud Controller Manager (CCM). we will need to create a values.yml to add to the parameter needed. We are using a Loadbalancer name to make it easier for both the ingress and the certificate manager to function as expected. You will need to create the DNS entry for this domain.
---
controller:
config:
use-proxy-protocol: 'true'
hostNetwork: false
ingressClass: nginx
service:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true'
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: 'workaround.example.org'To bind the ingress controller to a specific IP we can add the setting loadBalancerIP: ‘<IP_FROM_SUBNET_HERE>’
For example:
---
controller:
config:
use-proxy-protocol: 'true'
ingressClass: nginx
service:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true'
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: 'workaround.example.org'
loadBalancerIP: '111.222.333.444' # <- To specify the IP used for all ingress created under the controller.We then install the controller:
$ helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --create-namespace -f values.yml
NAME: ingress-nginx
LAST DEPLOYED: Mon Oct 30 15:08:57 2023
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the Loadbalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'
An example Ingress that makes use of the controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: foo
spec:
ingressClassName: nginx
rules:
- host: www.example.org
http:
paths:
- pathType: Prefix
backend:
service:
name: exampleService
port:
number: 80
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.org
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt:
tls.key:
type: kubernetes.io/tlsWe then can validate that the annotation is correct, and the Load balancer was configured:
$ kubectl describe svc ingress-nginx-controller -n ingress-nginx
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.9.4
helm.sh/chart=ingress-nginx-4.8.3
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: workaround.example.org # <-- This need to show up
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: true # <-- This need to show up
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: Loadbalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP:
IPs:
IP:
Loadbalancer Ingress: workaround.example.org
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 30542/TCP
Endpoints: :80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 30832/TCP
Endpoints: :443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadbalancer 4m36s service-controller Ensuring Loadbalancer # <-- This indicate that the cluster was able to communicate with the Cloudstack Loadbalancer
Normal EnsuredLoadbalancer 4m19s service-controller Ensured Loadbalancer # <-- This indicate that the cluster was able to configure and obtain an External IP with the Cloudstack Loadbalancerkubectl deployment
We will need to download the controller yaml file from the official github and then modify it so that it will work with our CloudStack load balancer.
$ curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.4/deploy/static/provider/cloud/deploy.yaml -o ingress-nginx-controller.ymlThen edit ingress-nginx-controller.yml to add these parameters to the yaml file:
[...]
apiVersion: v1
data:
allow-snippet-annotations: 'false'
use-proxy-protocol: 'true' # <-- Add this parameter as this will enable proxy mode between the ingress and the Loadbalancer kind: ConfigMap
metadata:
[...]
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.9.4
name: ingress-nginx-controller
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true' # <-- Add this so the Loadbalancer is configure with TCP PROXY
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: 'workaround.example.org' # <-- Add this so that kube-proxy does not hijack traffic
namespace: ingress-nginxExample with a specified IP:
[...]
apiVersion: v1
data:
allow-snippet-annotations: 'false'
use-proxy-protocol: 'true' # <-- Add this parameter as this will enable proxy mode between the ingress and the Loadbalancer
kind: ConfigMap
metadata:
[...]
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.9.4
name: ingress-nginx-controller
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true' # <-- Add this so the Loadbalancer is configure with TCP PROXY
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: 'workaround.example.org' # <-- Add this so that kube-proxy does not hijack traffic
namespace: ingress-nginx
spec:
externalTrafficPolicy: Local
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: LoadBalancer
loadBalancerIP: '111.222.333.444' # <- To specify the IP used for all ingress created under the controller. Then we deploy the modified yaml:
$ kubectl apply -f ingress/nginx/ingress-nginx-controller.yml
namespace/ingress-nginx configured
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
networkpolicy.networking.k8s.io/ingress-nginx-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission createdWe can validate that everything is deploy correctly:
$ kubectl describe svc ingress-nginx-controller -n ingress-nginx
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.9.4
helm.sh/chart=ingress-nginx-4.8.3
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: workaround.example.org # <-- This need to show up
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: true # <-- This need to show up
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: Loadbalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP:
IPs:
IP:
Loadbalancer Ingress: workaround.example.org
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 30542/TCP
Endpoints: :80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 30832/TCP
Endpoints: :443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadbalancer 4m36s service-controller Ensuring Loadbalancer # <-- This indicate that the cluster was able to communicate with the Cloudstack Load balancer
Normal EnsuredLoadbalancer 4m19s service-controller Ensured Loadbalancer # <-- This indicate that the cluster was able to configure and obtain an External IP with the Cloudstack Load balancerCluster Issuer
We will now install let’s encrypt as our certificate issuer. We will use the http01 challenge to validate the control of the domain in question.
First let’s create the yaml file. In this example we will do both the staging and the production issuer. For the rest of the demonstration, we will use the staging Issuer.
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: issuer@example.org
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: issuer@example.org
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginxAnd then we deploy them:
$ kubectl create -f ingress/nginx/cluster-issuer.yml
clusterissuer.cert-manager.io/letsencrypt-staging created
clusterissuer.cert-manager.io/letsencrypt-prod createdWe can validate:
$ kubectl get clusterissuers -A -owide
NAME READY STATUS AGE
letsencrypt-prod True The ACME account was registered with the ACME server 3m5s
letsencrypt-staging True The ACME account was registered with the ACME server 3m5sWe can see that we are now register with the ACME server and ready to issue certificate.
whoami
We will then deploy an application and create an ingress which in turn will create and apply a certificate to it.
First deployment of the application:
helm
We add the repo for the application:
$ helm repo add cowboysysop https://cowboysysop.github.io/charts/
"cowboysysop" has been added to your repositories
$ helm repo update
...Successfully got an update from the "cowboysysop" chart repositoryWe will create a values.yml file with the ingress details needed.
ingress:
## @param ingress.enabled Enable ingress controller resource
enabled: true
## @param ingress.ingressClassName IngressClass that will be be used to implement the Ingress
ingressClassName: "nginx"
## @param ingress.pathType Ingress path type
pathType: ImplementationSpecific
## @param ingress.annotations Ingress annotations
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-staging"
## @param ingress.hosts[0].host Hostname to your Whoami installation
## @param ingress.hosts[0].paths Paths within the url structure
hosts:
- host: whoami.example.org
paths:
- /
## @param ingress.tls TLS configuration
tls:
- secretName: whoami-tls
hosts:
- whoami.example.orgWe then deploy the app:
$ helm install whoami cowboysysop/whoami -f values.yml
NAME: whoami
LAST DEPLOYED: Tue Oct 31 09:25:47 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
https://whoami.example.org/Then we can look at the result with a browser or we can use curl:
Hostname: whoami-648cb6db54-kg9tz
IP: 127.0.0.1
IP: ::1
IP:
IP: fe80::90b6:1ff:fec0:a147
RemoteAddr: :50650
GET / HTTP/1.1
Host: whoami.example.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br
Accept-Language: en-CA,en;q=0.9,fr-CA;q=0.8,fr;q=0.7
Cache-Control: max-age=0
Sec-Ch-Ua: "Chromium";v="118", "Google Chrome";v="118", "Not=A?Brand";v="99"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "macOS"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-Forwarded-For:
X-Forwarded-Host: whoami.example.org
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Scheme: https
X-Real-Ip:
X-Request-Id: a995503bf15c003d2d4e26f3e674983c
X-Scheme: httpsWe can also look at the complete chain:
# We validate the certificate
$ kubectl get certificates.cert-manager.io -A -owide
NAMESPACE NAME READY SECRET ISSUER STATUS AGE
default whoami-tls True whoami-tls letsencrypt-staging Certificate is up to date and has not expired 6m13s
# We can examine the certifcate
$ kubectl describe certificates.cert-manager.io whoami-tls
[...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 11m cert-manager-certificates-trigger Issuing certificate as Secret does not exist
Normal Generated 11m cert-manager-certificates-key-manager Stored new private key in temporary Secret resource "whoami-tls-ks569"
Normal Requested 11m cert-manager-certificates-request-manager Created new CertificateRequest resource "whoami-tls-1"
Normal Issuing 10m cert-manager-certificates-issuing The certificate has been successfully issued
# If there is any error we can troubleshoot with the follwing commands:
# We can check the certificate request
$ kubectl get certificaterequests.cert-manager.io -A -owide
NAMESPACE NAME APPROVED DENIED READY ISSUER REQUESTOR STATUS AGE
default whoami-tls-1 True True letsencrypt-staging system:serviceaccount:cert-manager:cert-manager Certificate fetched from issuer successfully 3m58s
# We can also check the order
$ kubectl get orders.acme.cert-manager.io -A -owide
NAMESPACE NAME STATE ISSUER REASON AGE
default whoami-tls-1-540199672 valid letsencrypt-staging 8m50s
# Finally we can also see the challenge used and the result of it
$ kubectl get challenges
[...]
NAME STATE DOMAIN REASON AGE
example-com-2745722290-4391602865-0 pending example.org Waiting for http-01 challenge propagation 22s
$ kubectl describe challenge example-com-2745722290-4391602865-0
[...]
Status:
Presented: true
Processing: true
Reason: Waiting for http-01 challenge propagation
State: pending
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 19s cert-manager Challenge scheduled for processing
Normal Presented 16s cert-manager Presented challenge using http-01 challenge mechanism
We can also get the certificate from the website/application:
$ openssl s_client -showcerts -servername whoami.example.org -connect whoami.example.org:443 </dev/null
CONNECTED(00000005)
depth=2 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
verify return:1
depth=0 CN = whoami.example.org
verify return:1
---
Certificate chain
0 s:CN = whoami.example.org
i:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
1 s:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
2 s:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Doctored Durian Root CA X3
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
---
Server certificate
subject=CN = whoami.example.org
issuer=C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4642 bytes and written 407 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
DONEThis conclude the demonstration with ingress NGINX controller.
Traefik
We will now look at the traefik ingress controller. We need to create a values.yml file to add the relevant setting to enable proxy mode.
Helm
ports:
web: # This would be for the port 80
# Enable forwarded headers information (X-Forwarded-*).
forwardedHeaders:
insecure: true
proxyProtocol:
insecure: true
websecure: # This would be for the port 443
# Enable forwarded headers information (X-Forwarded-*).
forwardedHeaders:
insecure: true
proxyProtocol:
insecure: true
service:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true'
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: 'workaround.example.org'
spec:
loadBalancerIP: '111.222.333.444' # <- To specify the IP used for all ingress created under the controller.We then deploy the helm chart:
$ helm install traefik traefik/traefik --namespace=traefik --create-namespace -f values.yml
NAME: traefik
LAST DEPLOYED: Tue Oct 31 12:55:08 2023
NAMESPACE: traefik
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Traefik Proxy v2.10.5 has been deployed successfully on traefik namespace !kubectl deployment
For the static deployment we use this yaml. It contains all the necessary ClusterRole, ServiceAccount, ClusterRoleBinding, Deployment and Service needed to work with our cluster.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: traefik-traefik
rules:
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingressclasses
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.io
- traefik.containo.us
resources:
- ingressroutes
- ingressroutetcps
- ingressrouteudps
- middlewares
- middlewaretcps
- tlsoptions
- tlsstores
- traefikservices
- serverstransports
verbs:
- get
- list
- watch
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik
namespace: traefik
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: traefik-traefik
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-traefik
subjects:
- kind: ServiceAccount
name: traefik
namespace: traefik
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
name: traefik
namespace: traefik
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "9100"
prometheus.io/scrape: "true"
labels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
spec:
containers:
- args:
- --global.checknewversion
- --global.sendanonymoususage
- --entrypoints.metrics.address=:9100/tcp
- --entrypoints.traefik.address=:9000/tcp
- --entrypoints.web.address=:8000/tcp
- --entrypoints.websecure.address=:8443/tcp
- --api.dashboard=true
- --ping=true
- --metrics.prometheus=true
- --metrics.prometheus.entrypoint=metrics
- --providers.kubernetescrd
- --providers.kubernetesingress
- --entrypoints.web.forwardedHeaders.insecure
- --entrypoints.web.proxyProtocol.insecure
- --entrypoints.websecure.http.tls=true
- --entrypoints.websecure.forwardedHeaders.insecure
- --entrypoints.websecure.proxyProtocol.insecure
- --log.level=DEBUG
- --accesslog=true
- --accesslog.fields.defaultmode=keep
- --accesslog.fields.headers.defaultmode=keep
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: docker.io/traefik:v2.10.5
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: 9000
scheme: HTTP
initialDelaySeconds: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
name: traefik
ports:
- containerPort: 9100
name: metrics
protocol: TCP
- containerPort: 9000
name: traefik
protocol: TCP
- containerPort: 8000
name: web
protocol: TCP
- containerPort: 8443
name: websecure
protocol: TCP
readinessProbe:
failureThreshold: 1
httpGet:
path: /ping
port: 9000
scheme: HTTP
initialDelaySeconds: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
serviceAccount: traefik
serviceAccountName: traefik
terminationGracePeriodSeconds: 60
volumes:
- emptyDir: {}
name: data
- emptyDir: {}
name: tmp
---
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: workaround.example.org
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true'
labels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
name: traefik
namespace: traefik
spec:
allocateLoadBalancerNodePorts: true
ports:
- name: web
port: 80
protocol: TCP
targetPort: web
- name: websecure
port: 443
protocol: TCP
targetPort: websecure
selector:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
sessionAffinity: None
type: LoadBalancer
If we want to specify the IP we modify the Service, the rest is the same.
---
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: workaround.example.org
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true'
labels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
name: traefik
namespace: traefik
spec:
allocateLoadBalancerNodePorts: true
loadBalancerIP: '111.222.333.444' # <- To specify the IP used for all ingress created under the controller.
ports:
- name: web
port: 80
protocol: TCP
targetPort: web
- name: websecure
port: 443
protocol: TCP
targetPort: websecure
selector:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
sessionAffinity: None
type: LoadBalancerWe then apply the manifest:
$ kubectl apply -f ingress/traefik/traefik-deploy.yml
clusterrole.rbac.authorization.k8s.io/traefik-traefik created
serviceaccount/traefik created
clusterrolebinding.rbac.authorization.k8s.io/traefik-traefik created
deployment.apps/traefik created
service/traefik createdWe can validate with the following command:
$ kubectl describe svc traefik -n traefik
Name: traefik
Namespace: traefik
Labels: app.kubernetes.io/instance=traefik-traefik
app.kubernetes.io/name=traefik
Annotations: service.beta.kubernetes.io/cloudstack-load-balancer-hostname: workaround.example.org
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: true
Selector: app.kubernetes.io/instance=traefik-traefik,app.kubernetes.io/name=traefik
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP:
IPs:
IP:
LoadBalancer Ingress: workaround.example.org
Port: web 80/TCP
TargetPort: web/TCP
NodePort: web 30243/TCP
Endpoints: :8000
Port: websecure 443/TCP
TargetPort: websecure/TCP
NodePort: websecure 30707/TCP
Endpoints: :8443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 3m19s service-controller Ensuring load balancer # <-- This indicate that the cluster was able to communicate with the Cloudstack Load balancer
Normal EnsuredLoadBalancer 3m2s service-controller Ensured load balancer # <-- This indicate that the cluster was able to configure and obtain an External IP with the Cloudstack Load balancerWith this result we know everything is now installed.
ClusterIssuer
As we did with the NGINX ingress we will define and deploy our clusterIssuer pointing to Let’s Encrypt and specifying traefik as the ingress class and using HTTP-01 challenge for verification.
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: issuer@example.org
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
ingressClassName: traefik
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: issuer@example.org
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
ingressClassName: traefikWe then deploy the manifest:
$ kubectl create -f cluster-issuer.yml
clusterissuer.cert-manager.io/letsencrypt-staging created
clusterissuer.cert-manager.io/letsencrypt-prod createdWe validate:
$ kubectl get clusterissuers -A -owide
NAME READY STATUS AGE
letsencrypt-prod True The ACME account was registered with the ACME server 3m5s
letsencrypt-staging True The ACME account was registered with the ACME server 3m5swhoami
We will deploy a sample app to test and validate that everything is working as expected.
helm
We add the repo for the application:
$ helm repo add cowboysysop https://cowboysysop.github.io/charts/
"cowboysysop" has been added to your repositories
$ helm repo update
...Successfully got an update from the "cowboysysop" chart repositoryWe will create a values.yml file with the ingress details needed:
ingress:
## @param ingress.enabled Enable ingress controller resource
enabled: true
## @param ingress.ingressClassName IngressClass that will be be used to implement the Ingress
ingressClassName: "traefik"
## @param ingress.pathType Ingress path type
pathType: ImplementationSpecific
## @param ingress.annotations Ingress annotations
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-staging"
## @param ingress.hosts[0].host Hostname to your Whoami installation
## @param ingress.hosts[0].paths Paths within the url structure
hosts:
- host: whoami.example.org
paths:
- /
## @param ingress.tls TLS configuration
tls:
- secretName: whoami-traefik-tls
hosts:
- whoami.example.orgWe then deploy the app:
$ helm install whoami cowboysysop/whoami -f values.yml
NAME: whoami
LAST DEPLOYED: Tue Oct 31 09:25:47 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
https://whoami.example.org/Then we can look at the result with a browser:
Hostname: whoami-traefik-7ffbbd6c56-gzx2b
IP: 127.0.0.1
IP: ::1
IP:
IP: fe80::78b8:2eff:fe71:e455
RemoteAddr: :34628
GET / HTTP/1.1
Host: whoami.example.org
User-Agent: curl/7.85.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For:
X-Forwarded-Host: whoami.example.org
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-74f4c67788-ntm9r
X-Real-Ip: We can examine the complete chain:
# We validate the certificate
$ kubectl get certificates.cert-manager.io -A -owide
NAMESPACE NAME READY SECRET ISSUER STATUS AGE
default whoami-traefik-tls True whoami-traefik-tls letsencrypt-staging Certificate is up to date and has not expired 21h
# We can examine the certifcate
$ kubectl describe certificates.cert-manager.io whoami-tls
[...] Status:
Conditions:
Last Transition Time: 2023-10-31T15:40:44Z
Message: Certificate is up to date and has not expired
Observed Generation: 2
Reason: Ready
Status: True
Type: Ready
Not After: 2024-01-29T14:40:39Z
Not Before: 2023-10-31T14:40:40Z
Renewal Time: 2023-12-30T14:40:39Z
Revision: 1
# If there is any error we can troubleshoot with the following commands:
# We can also check the certificate request
$ kubectl get certificaterequests.cert-manager.io -A -owide
NAMESPACE NAME APPROVED DENIED READY ISSUER REQUESTOR STATUS AGE
default whoami-traefik-tls-1 True True letsencrypt-staging system:serviceaccount:cert-manager:cert-manager Certificate fetched from issuer successfully 21h
# We can also check the order
$ kubectl get orders.acme.cert-manager.io -A -owide
NAMESPACE NAME STATE ISSUER REASON AGE
default whoami-traefik-tls-1-686561438 valid letsencrypt-staging 21h # Finally we can also see the challenge used and the result of it
$ kubectl get challenges
[...]
NAME STATE DOMAIN REASON AGE
example-com-2745722290-4391602865-0 pending example.org Waiting for http-01 challenge propagation 22s
$ kubectl describe challenge example-com-2745722290-4391602865-0
[...]
Status:
Presented: true
Processing: true
Reason: Waiting for http-01 challenge propagation
State: pending
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 19s cert-manager Challenge scheduled for processing
Normal Presented 16s cert-manager Presented challenge using http-01 challenge mechanism
