Kustomize

Since we have migrated all our application deployment to Kubernetes, we needed to automatize the deployment of each application for different environments (integration and production).

By default kubectl files are not really customizable. That’s why we choose Kustomize to deploy our apps. And now Kustomize is part of kubectl binary. You can just put your kustomiszation file in a folder and apply it using the :

kubectl apply -k my-app

The Idea of Kustomize is to use a base to define your Kubernetes application deployment, service, ingress, etc.. and after you add overlay to override / patch your base for each environment.

Let’s start with and example. I want to deploy a simple docker image in my k8s cluster. So I have some kubectl files : namespace.yml, service.yml deployment.yml and ingress.yml all stored in a base directory.

Then I add a kustomization.yml in this directory which reference all this files :

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: demo
resources:
  - namespace.yml
  - service.yml
  - deployment.yml
  - ingress.yml
configMapGenerator:
  - name: demo-config
    behavior: create
    files:
      - config.json

Once I’ve create my base Kustomization, I can patch Kubernetes resources for different environment. For example for each environment, i want to create a Secret containing the db password.

I can use the Kustomize sops decoder plugin which is open sourced by Malt ^_^’

Here is the kustomization.yml stored in the staging folder :

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: demo
resources:
  - ../base
generators:
  - sops-decoder.yml
vars:
  - name: INGRESS_PREFIX
    objref:
      apiVersion: v1
      kind: ConfigMap
      name: demo-config
    fieldref:
      fieldpath: data.INGRESS_PREFIX
configMapGenerator:
  - name: demo-config
    behavior: merge
    literals:
      - environment=staging
      - INGRESS_PREFIX="staging."

You can see that I’ve included the base Kustomization.yml, then I’ve overrided some variables with a sops decoder generator (see bellow) and a configMap which contain specific values for this environment.

Another trick here is to fetch the value of the INGRESS_PREFIX from the configMapGenerator into a Kustomize variable. This variable can be used after in my ingress to have a different ingress host for each environment :

host: demo.$(INGRESS_PREFIX)aperogeek.fr

Now, Let’s have a look to the sops-decoder.yml generator :

apiVersion: gitlab.com/maltcommunity
kind: SopsDecoder
metadata:
  name: password
name: password
files:
  - path: password.enc
    key_name: password
    type: raw

The Kustomize sops decoder plugin will automatically decrypt the Mozzilla Sops encrypted file and generate a Kubernetes Secret with the content in the secrets data.

Here is a layout example of my Kustomize workspace :

base/
  config.json
  deployment.yml
  ingress.yml
  kustomization.yml
  namespace.yml
  service.yml

production/
  kustomization.yml
  password.enc
  sops-decoder.yml

staging/
  kustomization.yml
  password.enc
  sops-decoder.yml

Now I can validate and apply my Kustomize files running the command

kustomize build staging --enable_alpha_plugins

The Kustomize build will return all the desired kubectl resources to deploy my kubernetes app :

apiVersion: v1
kind: Namespace
metadata:
  name: demo
---
apiVersion: v1
data:
  INGRESS_PREFIX: integration.
  environment: integration
  environment-type: staging
  config.json: |-
    {"port":8080}
kind: ConfigMap
metadata:
  annotations: {}
  labels: {}
  name: demo-config-bdbg5f6dd5
  namespace: demo
---
apiVersion: v1
data:
  password: Qydlc3QgYm9uIGxhIGJpw6hyZSAhCg==
kind: Secret
metadata:
  name: password-mcgc26kf8m
  namespace: demo
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  name: demo
  namespace: demo
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: demo
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: demo
  name: demo
  namespace: demo
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - env:
        - name: ENVIRONMENT
          valueFrom:
            configMapKeyRef:
              key: environment
              name: demo-config-bdbg5f6dd5
        - name: PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: password-mcgc26kf8m
              optional: false
        image: mon-image:latest
        imagePullPolicy: IfNotPresent
        name: demo
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/my-app
          name: config
          readOnly: true
      volumes:
      - configMap:
          items:
          - key: config.json
            path: config.json
          name: demo-config-bdbg5f6dd5
        name: config
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip
  name: demo
  namespace: demo
spec:
  rules:
  - host: demo.staging.aperogeek.fr
    http:
      paths:
      - backend:
          serviceName: demo
          servicePort: 8080
        path: /

Finally I can apply this with a pipe to kubectl

kustomize build staging --enable_alpha_plugins | kubectl apply -f -