This document will be the basis for deploying ingress on a Kubernetes cluster. We will demonstrate both Ingress NGINX Controller and traefik with cert-manager and Let's Encrypt. The demonstration will use normal deployment as well as Helm Chart deployment. We will also use the whoami application as an example of an application which also give information about headers and client IP.
We will also indicate how to bind one of the /29 IPs given to each client when they order a managed Kubernetes cluster.
Version at time of writing are : cert-manager: v1.13.2, NGINX Controller: controller-v1.9.4, traefik: v2.10.5
cert-manager
We will start with cert-manager. This will be a basic installation with no customization needed and is agnostic of the ingress controller used.
Helm
We add the repo to be able to deploy from it
$ helm repo add jetstack https://charts.jetstack.io "jetstack" has been added to your repositories $ helm repo update ...Successfully got an update from the "jetstack" chart repository
cert-manager requires a number of Custom resource definitions (CRDs) to be installed to work so we need to add parameter to the helm deployment.
To install with the CRDs we need to execute this command
$ helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --version v1.13.2 \ --set installCRDs=true # <- this will install the custom resources (CRDs) NAME: cert-manager LAST DEPLOYED: Mon Oct 30 14:34:09 2023 NAMESPACE: cert-manager STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: cert-manager v1.13.2 has been deployed successfully! In order to begin issuing certificates, you will need to set up a ClusterIssuer or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer). More information on the different types of issuers and how to configure them can be found in our documentation: https://cert-manager.io/docs/configuration/ For information on how to configure cert-manager to automatically provision Certificates for Ingress resources, take a look at the `ingress-shim` documentation: https://cert-manager.io/docs/usage/ingress/
kubectl deployment
This is the default static way which also include the CRDs
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml namespace/cert-manager created customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created serviceaccount/cert-manager-cainjector created serviceaccount/cert-manager created serviceaccount/cert-manager-webhook created configmap/cert-manager created configmap/cert-manager-webhook created clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created clusterrole.rbac.authorization.k8s.io/cert-manager-view created clusterrole.rbac.authorization.k8s.io/cert-manager-edit created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created role.rbac.authorization.k8s.io/cert-manager:leaderelection created role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created service/cert-manager created service/cert-manager-webhook created deployment.apps/cert-manager-cainjector created deployment.apps/cert-manager created deployment.apps/cert-manager-webhook created mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
More details and other information can be found on the official site : https://cert-manager.io/docs/
NGINX Ingress
We will now deploy the NGINX ingress
Helm
For the ingress to work properly we need to customize it to play nice with our Cloud Controller Manager (CCM). we will need to create a values.yml to add to the parameter needed. We are using a Loadbalancer name to make it easier for both the ingress and the certificate manager to function as expected. You will need to create the DNS entry for this domain.
---
controller:
config:
use-proxy-protocol: 'true'
hostNetwork: false
ingressClass: nginx
service:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true'
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: 'workaround.example.org'
To bind the ingress controller to a specific IP we can add the setting loadBalancerIP: '<IP_FROM_SUBNET_HERE>'
for example
---
controller:
config:
use-proxy-protocol: 'true'
ingressClass: nginx
service:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true'
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: 'workaround.example.org'
loadBalancerIP: '111.222.333.444' # <- To specify the IP used for all ingress created under the controller.
We then install the controller
$ helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --create-namespace -f values.yml
NAME: ingress-nginx
LAST DEPLOYED: Mon Oct 30 15:08:57 2023
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the Loadbalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'
An example Ingress that makes use of the controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: foo
spec:
ingressClassName: nginx
rules:
- host: www.example.org
http:
paths:
- pathType: Prefix
backend:
service:
name: exampleService
port:
number: 80
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.org
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls
We then can validate that the annotation is correct and the Load balancer was configure
$ kubectl describe svc ingress-nginx-controller -n ingress-nginx
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.9.4
helm.sh/chart=ingress-nginx-4.8.3
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: workaround.example.org # <-- This need to show up
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: true # <-- This need to show up
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: Loadbalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: <IP_ADDRESS>
IPs: <IP_ADDRESS>
IP: <PUBLIC_IP_ADDRESS>
Loadbalancer Ingress: workaround.example.org
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 30542/TCP
Endpoints: <IP_ADDRESS>:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 30832/TCP
Endpoints: <IP_ADDRESS>:443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadbalancer 4m36s service-controller Ensuring Loadbalancer # <-- This indicate that the cluster was able to communicate with the Cloudstack Loadbalancer
Normal EnsuredLoadbalancer 4m19s service-controller Ensured Loadbalancer # <-- This indicate that the cluster was able to configure and obtain an External IP with the Cloudstack Loadbalancer
Everything is deployed as expected.
kubectl deployment
We will need to download the controller yaml file from the official github and then modify it so that it will work with our CloudStack load balancer.
$ curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.4/deploy/static/provider/cloud/deploy.yaml -o ingress-nginx-controller.yml
Then edit ingress-nginx-controller.yml to add these parameter to the yaml file
[...]
apiVersion: v1
data:
allow-snippet-annotations: 'false'
use-proxy-protocol: 'true' # <-- Add this parameter as this will enable proxy mode between the ingress and the Loadbalancer kind: ConfigMap
metadata:
[...]
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.9.4
name: ingress-nginx-controller
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true' # <-- Add this so the Loadbalancer is configure with TCP PROXY
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: 'workaround.example.org' # <-- Add this so that kube-proxy does not hijack traffic
namespace: ingress-nginx
Example with a specified IP
[...]
apiVersion: v1
data:
allow-snippet-annotations: 'false'
use-proxy-protocol: 'true' # <-- Add this parameter as this will enable proxy mode between the ingress and the Loadbalancer
kind: ConfigMap
metadata:
[...]
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.9.4
name: ingress-nginx-controller
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true' # <-- Add this so the Loadbalancer is configure with TCP PROXY
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: 'workaround.example.org' # <-- Add this so that kube-proxy does not hijack traffic
namespace: ingress-nginx
spec:
externalTrafficPolicy: Local
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: LoadBalancer
loadBalancerIP: '111.222.333.444' # <- To specify the IP used for all ingress created under the controller.
Then we deploy the modified yaml
$ kubectl apply -f ingress/nginx/ingress-nginx-controller.yml namespace/ingress-nginx configured serviceaccount/ingress-nginx created serviceaccount/ingress-nginx-admission created role.rbac.authorization.k8s.io/ingress-nginx created role.rbac.authorization.k8s.io/ingress-nginx-admission created clusterrole.rbac.authorization.k8s.io/ingress-nginx created clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created rolebinding.rbac.authorization.k8s.io/ingress-nginx created rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created configmap/ingress-nginx-controller created service/ingress-nginx-controller created service/ingress-nginx-controller-admission created deployment.apps/ingress-nginx-controller created job.batch/ingress-nginx-admission-create created job.batch/ingress-nginx-admission-patch created ingressclass.networking.k8s.io/nginx created networkpolicy.networking.k8s.io/ingress-nginx-admission created validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
We can validate that everything is deploy correctly
$ kubectl describe svc ingress-nginx-controller -n ingress-nginx
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.9.4
helm.sh/chart=ingress-nginx-4.8.3
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: workaround.example.org # <-- This need to show up
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: true # <-- This need to show up
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: Loadbalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: <IP_ADDRESS>
IPs: <IP_ADDRESS>
IP: <PUBLIC_IP_ADDRESS>
Loadbalancer Ingress: workaround.example.org
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 30542/TCP
Endpoints: <IP_ADDRESS>:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 30832/TCP
Endpoints: <IP_ADDRESS>:443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadbalancer 4m36s service-controller Ensuring Loadbalancer # <-- This indicate that the cluster was able to communicate with the Cloudstack Load balancer
Normal EnsuredLoadbalancer 4m19s service-controller Ensured Loadbalancer # <-- This indicate that the cluster was able to configure and obtain an External IP with the Cloudstack Load balancer
Cluster Issuer
We will now install let's encrypt as our certificate issuer. We will use the http01 challenge to validate the control of the domain in question.
First let's create the yaml file. In this example we will do both the staging and the production issuer. For the rest of the demonstration we will use the staging Issuer.
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: issuer@example.org
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: issuer@example.org
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
And then we deploy them
$ kubectl create -f ingress/nginx/cluster-issuer.yml clusterissuer.cert-manager.io/letsencrypt-staging created clusterissuer.cert-manager.io/letsencrypt-prod created
We can validate
$ kubectl get clusterissuers -A -owide NAME READY STATUS AGE letsencrypt-prod True The ACME account was registered with the ACME server 3m5s letsencrypt-staging True The ACME account was registered with the ACME server 3m5s
We can see that we are now register with the ACME server and ready to issue certificate.
whoami
We will then deploy an application and create an ingress which in turn will create and apply a certificate to it.
First deployment of the application
helm
We add the repo for the application
$ helm repo add cowboysysop https://cowboysysop.github.io/charts/ "cowboysysop" has been added to your repositories $ helm repo update ...Successfully got an update from the "cowboysysop" chart repository
We will create a values.yml file with the ingress details needed.
ingress:
## @param ingress.enabled Enable ingress controller resource
enabled: true
## @param ingress.ingressClassName IngressClass that will be be used to implement the Ingress
ingressClassName: "nginx"
## @param ingress.pathType Ingress path type
pathType: ImplementationSpecific
## @param ingress.annotations Ingress annotations
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-staging"
## @param ingress.hosts[0].host Hostname to your Whoami installation
## @param ingress.hosts[0].paths Paths within the url structure
hosts:
- host: whoami.example.org
paths:
- /
## @param ingress.tls TLS configuration
tls:
- secretName: whoami-tls
hosts:
- whoami.example.org
We then deploy the app
$ helm install whoami cowboysysop/whoami -f values.yml NAME: whoami LAST DEPLOYED: Tue Oct 31 09:25:47 2023 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: 1. Get the application URL by running these commands: https://whoami.example.org/
Then we can look at the result with a browser or we can use curl
Hostname: whoami-648cb6db54-kg9tz IP: 127.0.0.1 IP: ::1 IP: <POD_IP_WHOAMI> IP: fe80::90b6:1ff:fec0:a147 RemoteAddr: <NGINX_INGRESS_CONTROLLER_IP>:50650 GET / HTTP/1.1 Host: whoami.example.org User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 Accept-Encoding: gzip, deflate, br Accept-Language: en-CA,en;q=0.9,fr-CA;q=0.8,fr;q=0.7 Cache-Control: max-age=0 Sec-Ch-Ua: "Chromium";v="118", "Google Chrome";v="118", "Not=A?Brand";v="99" Sec-Ch-Ua-Mobile: ?0 Sec-Ch-Ua-Platform: "macOS" Sec-Fetch-Dest: document Sec-Fetch-Mode: navigate Sec-Fetch-Site: none Sec-Fetch-User: ?1 Upgrade-Insecure-Requests: 1 X-Forwarded-For: <MY_PUBLIC_CLIENT_IP_HERE> X-Forwarded-Host: whoami.example.org X-Forwarded-Port: 443 X-Forwarded-Proto: https X-Forwarded-Scheme: https X-Real-Ip: <MY_PUBLIC_CLIENT_IP_HERE> X-Request-Id: a995503bf15c003d2d4e26f3e674983c X-Scheme: https
We can also look at the complete chain
# We validate the certificate $ kubectl get certificates.cert-manager.io -A -owide NAMESPACE NAME READY SECRET ISSUER STATUS AGE default whoami-tls True whoami-tls letsencrypt-staging Certificate is up to date and has not expired 6m13s # We can examine the certifcate $ kubectl describe certificates.cert-manager.io whoami-tls [...] Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Issuing 11m cert-manager-certificates-trigger Issuing certificate as Secret does not exist Normal Generated 11m cert-manager-certificates-key-manager Stored new private key in temporary Secret resource "whoami-tls-ks569" Normal Requested 11m cert-manager-certificates-request-manager Created new CertificateRequest resource "whoami-tls-1" Normal Issuing 10m cert-manager-certificates-issuing The certificate has been successfully issued # If there is any error we can troubleshoot with the follwing commands: # We can check the certificate request $ kubectl get certificaterequests.cert-manager.io -A -owide NAMESPACE NAME APPROVED DENIED READY ISSUER REQUESTOR STATUS AGE default whoami-tls-1 True True letsencrypt-staging system:serviceaccount:cert-manager:cert-manager Certificate fetched from issuer successfully 3m58s # We can also check the order $ kubectl get orders.acme.cert-manager.io -A -owide NAMESPACE NAME STATE ISSUER REASON AGE default whoami-tls-1-540199672 valid letsencrypt-staging 8m50s # Finally we can also see the challenge used and the result of it $ kubectl get challenges [...] NAME STATE DOMAIN REASON AGE example-com-2745722290-4391602865-0 pending example.org Waiting for http-01 challenge propagation 22s $ kubectl describe challenge example-com-2745722290-4391602865-0 [...] Status: Presented: true Processing: true Reason: Waiting for http-01 challenge propagation State: pending Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Started 19s cert-manager Challenge scheduled for processing Normal Presented 16s cert-manager Presented challenge using http-01 challenge mechanism
We can also get the certificate from the website / application
$ openssl s_client -showcerts -servername whoami.example.org -connect whoami.example.org:443 </dev/null CONNECTED(00000005) depth=2 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1 verify error:num=20:unable to get local issuer certificate verify return:1 depth=1 C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3 verify return:1 depth=0 CN = whoami.example.org verify return:1 --- Certificate chain 0 s:CN = whoami.example.org i:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- 1 s:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3 i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- 2 s:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1 i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Doctored Durian Root CA X3 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- --- Server certificate subject=CN = whoami.example.org issuer=C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3 --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 4642 bytes and written 407 bytes Verification error: unable to get local issuer certificate --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported No ALPN negotiated Early data was not sent Verify return code: 20 (unable to get local issuer certificate) --- DONE
This conclude the demonstration with ingress NGINX controller.
Traefik
We will now look at the traefik ingress controller.
we need to create a values.yml file to add the relevant setting to enable proxy mode.
Helm
ports:
web: # This would be for the port 80
# Enable forwarded headers information (X-Forwarded-*).
forwardedHeaders:
insecure: true
proxyProtocol:
insecure: true
websecure: # This would be for the port 443
# Enable forwarded headers information (X-Forwarded-*).
forwardedHeaders:
insecure: true
proxyProtocol:
insecure: true
service:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true'
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: 'workaround.example.org'
spec:
loadBalancerIP: '111.222.333.444' # <- To specify the IP used for all ingress created under the controller.
We then deploy the helm chart
$ helm install traefik traefik/traefik --namespace=traefik --create-namespace -f values.yml NAME: traefik LAST DEPLOYED: Tue Oct 31 12:55:08 2023 NAMESPACE: traefik STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Traefik Proxy v2.10.5 has been deployed successfully on traefik namespace !
kubectl deployment
For the static deployment we use this yaml. It contain all the necessary ClusterRole, ServiceAccount, ClusterRoleBinding, Deployment and Service needed to work with our cluster.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: traefik-traefik
rules:
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingressclasses
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.io
- traefik.containo.us
resources:
- ingressroutes
- ingressroutetcps
- ingressrouteudps
- middlewares
- middlewaretcps
- tlsoptions
- tlsstores
- traefikservices
- serverstransports
verbs:
- get
- list
- watch
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik
namespace: traefik
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: traefik-traefik
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-traefik
subjects:
- kind: ServiceAccount
name: traefik
namespace: traefik
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
name: traefik
namespace: traefik
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "9100"
prometheus.io/scrape: "true"
labels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
spec:
containers:
- args:
- --global.checknewversion
- --global.sendanonymoususage
- --entrypoints.metrics.address=:9100/tcp
- --entrypoints.traefik.address=:9000/tcp
- --entrypoints.web.address=:8000/tcp
- --entrypoints.websecure.address=:8443/tcp
- --api.dashboard=true
- --ping=true
- --metrics.prometheus=true
- --metrics.prometheus.entrypoint=metrics
- --providers.kubernetescrd
- --providers.kubernetesingress
- --entrypoints.web.forwardedHeaders.insecure
- --entrypoints.web.proxyProtocol.insecure
- --entrypoints.websecure.http.tls=true
- --entrypoints.websecure.forwardedHeaders.insecure
- --entrypoints.websecure.proxyProtocol.insecure
- --log.level=DEBUG
- --accesslog=true
- --accesslog.fields.defaultmode=keep
- --accesslog.fields.headers.defaultmode=keep
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: docker.io/traefik:v2.10.5
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: 9000
scheme: HTTP
initialDelaySeconds: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
name: traefik
ports:
- containerPort: 9100
name: metrics
protocol: TCP
- containerPort: 9000
name: traefik
protocol: TCP
- containerPort: 8000
name: web
protocol: TCP
- containerPort: 8443
name: websecure
protocol: TCP
readinessProbe:
failureThreshold: 1
httpGet:
path: /ping
port: 9000
scheme: HTTP
initialDelaySeconds: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
serviceAccount: traefik
serviceAccountName: traefik
terminationGracePeriodSeconds: 60
volumes:
- emptyDir: {}
name: data
- emptyDir: {}
name: tmp
---
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: workaround.example.org
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true'
labels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
name: traefik
namespace: traefik
spec:
allocateLoadBalancerNodePorts: true
ports:
- name: web
port: 80
protocol: TCP
targetPort: web
- name: websecure
port: 443
protocol: TCP
targetPort: websecure
selector:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
sessionAffinity: None
type: LoadBalancer
If we want to specify the IP we modify the Service, the rest is the same.
---
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-hostname: workaround.example.org
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: 'true'
labels:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
name: traefik
namespace: traefik
spec:
allocateLoadBalancerNodePorts: true
loadBalancerIP: '111.222.333.444' # <- To specify the IP used for all ingress created under the controller.
ports:
- name: web
port: 80
protocol: TCP
targetPort: web
- name: websecure
port: 443
protocol: TCP
targetPort: websecure
selector:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
sessionAffinity: None
type: LoadBalancer
We then apply the manifest.
$ kubectl apply -f ingress/traefik/traefik-deploy.yml clusterrole.rbac.authorization.k8s.io/traefik-traefik created serviceaccount/traefik created clusterrolebinding.rbac.authorization.k8s.io/traefik-traefik created deployment.apps/traefik created service/traefik created
We can validate with the following command
$ kubectl describe svc traefik -n traefik
Name: traefik
Namespace: traefik
Labels: app.kubernetes.io/instance=traefik-traefik
app.kubernetes.io/name=traefik
Annotations: service.beta.kubernetes.io/cloudstack-load-balancer-hostname: workaround.example.org
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: true
Selector: app.kubernetes.io/instance=traefik-traefik,app.kubernetes.io/name=traefik
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: <IP_ADDRESS>
IPs: <IP_ADDRESS>
IP: <PUBLIC_IP_ADDRESS>
LoadBalancer Ingress: workaround.example.org
Port: web 80/TCP
TargetPort: web/TCP
NodePort: web 30243/TCP
Endpoints: <IP_ADDRESS>:8000
Port: websecure 443/TCP
TargetPort: websecure/TCP
NodePort: websecure 30707/TCP
Endpoints: <IP_ADDRESS>:8443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 3m19s service-controller Ensuring load balancer # <-- This indicate that the cluster was able to communicate with the Cloudstack Load balancer
Normal EnsuredLoadBalancer 3m2s service-controller Ensured load balancer # <-- This indicate that the cluster was able to configure and obtain an External IP with the Cloudstack Load balancer
With this result we know everything is now installed.
ClusterIssuer
As we did with the NGINX ingress we will define and deploy our clusterIssuer pointing to Let's Encrypt and specifying traefik as the ingress class and using HTTP-01 challenge for verification.
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: issuer@example.org
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
ingressClassName: traefik
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: issuer@example.org
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
ingressClassName: traefik
We then deploy the manifest
$ kubectl create -f cluster-issuer.yml clusterissuer.cert-manager.io/letsencrypt-staging created clusterissuer.cert-manager.io/letsencrypt-prod created
We validate
$ kubectl get clusterissuers -A -owide NAME READY STATUS AGE letsencrypt-prod True The ACME account was registered with the ACME server 3m5s letsencrypt-staging True The ACME account was registered with the ACME server 3m5s
whoami
We will deploy a sample app to test and validate that everything is working as expected.
helm
We add the repo for the application
$ helm repo add cowboysysop https://cowboysysop.github.io/charts/ "cowboysysop" has been added to your repositories $ helm repo update ...Successfully got an update from the "cowboysysop" chart repository
We will create a values.yml file with the ingress details needed.
ingress:
## @param ingress.enabled Enable ingress controller resource
enabled: true
## @param ingress.ingressClassName IngressClass that will be be used to implement the Ingress
ingressClassName: "traefik"
## @param ingress.pathType Ingress path type
pathType: ImplementationSpecific
## @param ingress.annotations Ingress annotations
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-staging"
## @param ingress.hosts[0].host Hostname to your Whoami installation
## @param ingress.hosts[0].paths Paths within the url structure
hosts:
- host: whoami.example.org
paths:
- /
## @param ingress.tls TLS configuration
tls:
- secretName: whoami-traefik-tls
hosts:
- whoami.example.org
We then deploy the app
$ helm install whoami cowboysysop/whoami -f values.yml NAME: whoami LAST DEPLOYED: Tue Oct 31 09:25:47 2023 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: 1. Get the application URL by running these commands: https://whoami.example.org/
Then we can look at the result with a browser
Hostname: whoami-traefik-7ffbbd6c56-gzx2b IP: 127.0.0.1 IP: ::1 IP: <IP_WHOAMI_APP> IP: fe80::78b8:2eff:fe71:e455 RemoteAddr: <TRAEFIK_INGRESS_CONTROLLER_IP>:34628 GET / HTTP/1.1 Host: whoami.example.org User-Agent: curl/7.85.0 Accept: */* Accept-Encoding: gzip X-Forwarded-For: <MY_PUBLIC_CLIENT_IP_HERE> X-Forwarded-Host: whoami.example.org X-Forwarded-Port: 443 X-Forwarded-Proto: https X-Forwarded-Server: traefik-74f4c67788-ntm9r X-Real-Ip: <MY_PUBLIC_CLIENT_IP_HERE>
We can examine the complete chain.
# We validate the certificate
$ kubectl get certificates.cert-manager.io -A -owide
NAMESPACE NAME READY SECRET ISSUER STATUS AGE
default whoami-traefik-tls True whoami-traefik-tls letsencrypt-staging Certificate is up to date and has not expired 21h
# We can examine the certifcate
$ kubectl describe certificates.cert-manager.io whoami-tls
[...] Status:
Conditions:
Last Transition Time: 2023-10-31T15:40:44Z
Message: Certificate is up to date and has not expired
Observed Generation: 2
Reason: Ready
Status: True
Type: Ready
Not After: 2024-01-29T14:40:39Z
Not Before: 2023-10-31T14:40:40Z
Renewal Time: 2023-12-30T14:40:39Z
Revision: 1
# If there is any error we can troubleshoot with the following commands:
# We can also check the certificate request
$ kubectl get certificaterequests.cert-manager.io -A -owide
NAMESPACE NAME APPROVED DENIED READY ISSUER REQUESTOR STATUS AGE
default whoami-traefik-tls-1 True True letsencrypt-staging system:serviceaccount:cert-manager:cert-manager Certificate fetched from issuer successfully 21h
# We can also check the order
$ kubectl get orders.acme.cert-manager.io -A -owide
NAMESPACE NAME STATE ISSUER REASON AGE
default whoami-traefik-tls-1-686561438 valid letsencrypt-staging 21h # Finally we can also see the challenge used and the result of it
$ kubectl get challenges
[...]
NAME STATE DOMAIN REASON AGE
example-com-2745722290-4391602865-0 pending example.org Waiting for http-01 challenge propagation 22s
$ kubectl describe challenge example-com-2745722290-4391602865-0
[...]
Status:
Presented: true
Processing: true
Reason: Waiting for http-01 challenge propagation
State: pending
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 19s cert-manager Challenge scheduled for processing
Normal Presented 16s cert-manager Presented challenge using http-01 challenge mechanism