The complete Kustomize tutorial

What is Kustomize in the first place?

Kustomize is a native Kubernetes configuration management tool designed to customize and manage resource configurations in a declarative and reusable way. Instead of relying on templating or parameterization, it works directly with Kubernetes manifests without altering their underlying structure. A key feature is its ability to dynamically create variations of Kubernetes resources while maintaining human-readable manifests throughout the process. As a fully declarative tool, Kustomize is highly GitOps-friendly. By using patches, transformers, and generators, it allows resource modifications without changing the base files, ensuring a clean separation of concerns and simplifying long-term maintenance.

Do you still think that offering on-premises installations is hard?

Glasskube Cloud is a battle tested software distribution platform that helps you scale from your first on-premises customers to dozens and even thousands.

Try for freeBook free demo

Getting Started with Kustomize

Before diving into Kustomize, let's first understand how application deployment and management works through the perspective of the Kubernetes API. The Kubernetes API is given instructions to create objects within clusters, it can only understand those instructions if they are delivered in a particular language: verbose YAML. Each Kubernetes object is highly customized with tags, labels, annotations, and various parameters that define its behavior and placement.

So, how can we generate manifests that tailor each object to contain all the necessary lines of YAML code which specify discrete environments and requirements? Helm approaches this by using templates that inject values from external data files to dynamically generate the manifests. Kustomize, on the other hand, works directly with the plain YAML files and uses a series of methods to create file variants without templating.

What is a Variant?

Kustomize has a straightforward goal: to create environment-specific variants of base Kubernetes manifests. Built into the tool are different methods of doing so. Generators, transformers, and patches are the levers that Kubernetes admins have at their disposal to modify the base YAML manifests which will later be handed off for the Kubernetes API to apply.

If these methods are the agents of variation, the kustomization.yaml file is the vehicle that delivers them. Each directory level has its own kustomization.yaml, serving as the glue that merges base files with environment-specific overlays.

The default Kustomize directory structure

A typical Kustomize setup has two main directories: base and overlays.

• base: Contains the base Kubernetes resource definitions, with files for each object that makes up the application as well as the base level kustomization.yaml file.

• overlays: Holds environment-specific folders (e.g., prod, dev), each with its own patch files and a kustomization.yaml which contains environment specific transformers and generator. These overlays modify the base resources for each environment. For example, ingress-patch.yaml in prod will update the base ingress.yaml with production-specific settings.

|-- base
    ingress.yaml
    deployment.yaml
    kustomization.yaml
|-- overlays
    |-- prod
        ingress-patch.yaml
        kustomization.yaml
    |-- dev
        ingress-patch.yaml
        kustomization.yaml

The role of the kustomize.yaml file

In Kustomize, overlays layer modifications on top of base manifests to create customized versions. The kustomization.yaml file guides this process, listing the base resources and applying the specified overlays. This can be seen as stacking changes on a temporary copy of the original manifest, resulting in a customized version that can be applied to Kubernetes with kubectl apply. Kustomize uses "transformer and generator" functions to modify manifests according to simple declarative rules through the Kustomization.yaml file.

Kustomize Transformers

Kustomize transformers are embedded functions that allow for modification of Kubernetes manifests in the following ways:

Setting Labels and Annotations

To add labels and/or annotations to all resources. If the label or annotation keys are already present on the resource, the value will be overridden.

Example:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

#setting labels
commonLabels:
  someName: someValue
  owner: jake
  managed-by: gitops

#setting annotations
commonAnnotations:
  version: "v1.2.3"

resources:
- deployment.yaml

Setting Namespaces and Names

Enforcing consistency across a project's resources can be useful by ensuring all resources are in the correct namespace and follow a common naming convention and case be easily done.

Example:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

#setting namepace
namespace: staging

#setting a name suffix to be appended to the deployment.yaml name
nameSuffix: -staging

resources:
- deployment.yaml

Customizing Container images

This transformer can modify the following container images specifications:

• Name
• Tag
• Digest
• Tag from the latest commit SHA
• Tag from an Environment Variable

Example:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

images:
- name: my-demo-app
  newName: my-app
  newTag: 1.8.0

resources:
- deployment.yaml

Result of applying all of the transformers

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: staging
  name: jake-app-staging
  labels:
    someName: someValue
    owner: jake
    managed-by: gitops
  annotations:
    version: "v1.2.3"
spec:
  containers:
    - name: my-demo-app-container
      image: my-app:1.8.0

Kustomize Generators

A generator creates or uses pre-existing resources that can be layered onto base manifests or modified further by transformers. Kustomize generators can be used to create or use both secret and configMap objects.

configMapsGenerator

Configmaps can be behave in three different ways is can create, replace or merge. And the generation of the actual config map can be also done in different ways:

Creating a new configMap

• A configMap can be created using .env or .properties files:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# using a .properties file
configMapGenerator:
- name: my-application-properties
  files:
  - application.properties

#using a .env file
configMapGenerator:
- name: tracing-options
  envs:
  - tracing.env

• configMaps can also be created from literals directly:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# using literals
configMapGenerator:
- name: my-server-env-vars
  literals:
  - SPRING_PROFILES_ACTIVE=production
  - LOG_LEVEL=INFO

Updating and existing configMap

The configMapGenerator can also be used to modify existing configMap manifests that exist in /base

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

configMapGenerator:
  - name: existing-name
    namespace: existing-ns # needs to match target ConfigMap from base
    behavior: replace
    literals:
      - ENV=staging

resources:
- ../base

secretsGenerator

Use the secretGenerator to generate Kubernetes secrets dynamically from files, environment variables, or key-value pairs. Here is an example of different types of secret generations.

secretGenerator:
- name: test-tls
  files:
  - secret/tls.crt
  - secret/tls.key
  type: "kubernetes.io/tls"

- name: test-tls-namespaced
  namespace: test-apps
  files:
  - tls.crt=catsecret/tls.crt
  - tls.key=secret/tls.key
  type: "kubernetes.io/tls"

- name: env_file_secret
  envs:
  - env.txt
  type: Opaque

- name: secret-with-annotation
  files:
  - app-config.yaml
  type: Opaque
  options:
    annotations:
      app_config: "true"
    labels:
      app.kubernetes.io/name: "test-app"

1. test-tls: This generates a TLS secret using the tls.crt and tls.key files, setting the type to kubernetes.io/tls.
2. test-tls-namespaced: Similar to test-tls, but this secret is created in the test-apps namespace. It uses the specified files and assigns custom names to the keys (e.g., tls.crt=secret/tls.crt).
3. env_file_secret: This secret is created from an environment file (env.txt), setting the type to Opaque.
4. secret-with-annotation: This creates a secret from the file app-config.yaml, with additional metadata like annotations and labels for easier management.

It is important to note that the secrets are base64 encoded

generatorOptions

Additionally, generatorOptions can be set on a per resource level within each generator. Which will create labels and annotation to add to all thr Kustomized generated resources.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generatorOptions:
  # labels to add to all generated resources
  labels:
    kustomize.generated.resources: somevalue
  # annotations to add to all generated resources
  annotations:
    kustomize.generated.resource: somevalue

Kustomize Patches

There are two main types of patches Kustomize can apply: strategic merge patches and JSON patches. Each is suited for different use cases and each plays a specific role in customizing Kubernetes configurations.

In the following example, both types of patches with reference the same base deployment which is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app-container
          image: old-image:1.0.0
          env:
            - name: EXISTING_VAR
              value: "existing value"

Strategic merge patching

Strategic merge patching uses JSON/YAML merge semantics. This approach merges the patch with the existing resource, preserving the original configuration while introducing changes from the patch.

Patch Logic

When Kustomize applies a strategic merge patch, it starts with the base resource configuration and combines it with the patch. Any existing fields in the base that are not addressed by the patch remain untouched. If a field in the patch is new, it gets added to the base configuration. If a field already exists, the patch's value overwrites the base value.

Use Case and Benefits

Strategic merging is ideal when you need to preserve most of the base configuration while only adjusting specific parts. This approach is particularly useful for handling complex Kubernetes resources like Deployments, where you may want to update only the container image or replica count without affecting other settings.

Example deployment-patch.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app  # Matches the name in the base manifest
spec:
  namespace: staging
  template:
    spec:
      containers:
        - name: my-app-container
          image: new-image:2.0.0  # Updates the container image
          env:
            - name: NEW_VAR       # Adds a new environment variable
              value: "new value"

The strategic merge patch will update the container image, add a new environment variable, and label the deployment as environment: production while leaving any other fields in the base unchanged.

JSON Patch

Kustomize JSON patching, based on the JSON Patch specification, allows for precise modifications to Kubernetes resources by applying explicit changes in JSON format. Unlike strategic patching, which merges configurations, JSON patching directly adds, removes, or replaces fields.

Patch syntax:

A patch consists of an array of patch operations, each defined by the following fields:

• op: The operation to perform ( add, remove, replace, move, copy, or test).
• path: The JSON path to the field that needs to be modified.
• value: (for add, replace, and move operations) The new value to be set or moved.

Use Case and Benefits

JSON patching is ideal for fine-grained control, allowing you to update individual fields, remove unnecessary entries, or add new ones without altering the rest of the configuration. These patches can also be applied directly in the kustomization.yaml file for convenience.

Here is a combined kustomization.yaml file that includes both strategic merge and JSON patching:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

patches:
  # Strategic merge patch
  - path: deployment-patch.yaml
    target:
      group: apps
      version: v1
      kind: Deployment
      name: my-app

  # JSON patch
  - target:
      group: apps
      version: v1
      kind: Deployment
      name: my-app
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 5
      - op: add
        path: /metadata/annotations/app-version
        value: "v2.0"

resources:
  - ../base

And this would be the output of the modified base deployment.yaml file:

# result after appying both patches to the base deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
  annotations:
    app-version: "v2.0"  # Added by the JSON patch
spec:
  replicas: 5  # Updated by the JSON patch
  namespace: staging  # Added by the strategic merge patch
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app-container
          image: new-image:2.0.0  # Updated by the strategic merge patch
          env:
            - name: NEW_VAR  # Added by the strategic merge patch
              value: "new value"

A practical GitOps example: Using Kustomize with ArgoCD

Because Kustomize works with un-templated Kubernetes base manifests and applies YAML overlays on top, these files can be easily stored in a version control system like GitHub. A GitOps tool like ArgoCD can then apply the contents of the environment-specific overlay folders directly. Argo CD reads the kustomization.yaml file, applies the necessary patches and transformations, and knows where to find the base Kubernetes manifest files to deploy.

You can create a file structure like this, add the base manifests, patch files, and transformations discussed in the previous sections, and then push them to a platform like GitHub.

my-app/
├── base/
│   ├── deployment.yaml
│   ├── kustomization.yaml
├── overlays/
    ├── staging/
    │   ├── kustomization.yaml
    │   └── env-config.yaml
    ├── production/
        ├── kustomization.yaml
        └── env-config.yaml

With the file structure above, Argo CD has everything needed to create two applications, each pointing to the environment-specific overlay folders for deploying the app to both the staging and production environments.

Here’s an example of the ArgoCD application file called my-app-staging-yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-staging
spec:
  project: default
  source:
    repoURL: 'https://github.com/my-repo/my-app.git'
    targetRevision: HEAD
    path: overlays/staging # use overlays/production for the other app
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: staging
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

If a kustomization.yaml file exists at the location pointed to by repoURL and path, Argo CD will render the manifests using Kustomize.

To apply the file and create the apps, make sure you are logged in to an instance of ArgoCD and run:

# create the staging app
argocd app create --file path/to/my-app-staging.yaml
# create the production app
argocd app create --file path/to/my-app-production.yaml

Once deployed, any changes to the source manifests in GitHub will be automatically picked up and deployed by ArgoCD. While Kustomize on its own lacks state management, rollback capabilities, and release history, combining it with ArgoCD provides these features, offering a more robust deployment process.

Kustomize best practices and testing

Testing Kustomize transformations

Always be sure to test Kustomize patches and transformations before applying them. Two useful commands for this are using kubectl apply -k with the --dry-run=client -o yaml flag as well as and kubectl diff plugin.

Example usage:

kubectl apply -k <path-to-your-kustomization-directory> --dry-run=client -o yaml

This command outputs the YAML rendered by combining the base resources and applying the patches defined in the <path-to-your-kustomization-directory> directory, without actually applying any changes. You can inspect the output to ensure the patches were applied correctly.

Use kubectl diff to compare the changes between your live cluster resources and the new configurations generated by Kustomize.

Example usage:

kubectl diff -k <path-to-your-kustomization-directory>

This will compare the current state of your cluster with the resources that Kustomize would apply and show a diff of the differences.

Kustomize Best practices

When working with Kustomize, following best practices can help maintain clarity, efficiency, and scalability across your Kubernetes configurations. Here are some key tips to keep in mind:

1. Fell free to use either patchStrategicMerge or patchJson6902 path, keep in ming that patchJson6902 can do almost everything a patchStrategicMerge can do, but with a briefer syntax.
2. Name kustomization directories after their corresponding namespaces.
3. Name patch files descriptively Use clear and descriptive names for your patch files that indicate what resource the patch targets.
4. If something is the same between multiple overlays, consider putting it in your base.
5. Test patches thoroughly.
6. Securely manage secrets Use secretGenerator in Kustomize to handle sensitive information like passwords or API keys. Avoid hardcoding sensitive data in plain YAML files.

Kustomize FAQs

How to use the Kustomize standalone CLI tool?

Kustomize is integrated directly into kubectl starting from version 1.14, allowing you to run commands like kubectl apply -k to apply configurations without needing to install Kustomize separately. However, if you prefer a standalone installation, there are multiple methods available.

For Homebrew users, simply run:

brew install kustomize

That said, the most common and convenient way to use Kustomize is through kubectl.

Can Kustomize and Helm be used together?

Yes, Kustomize and Helm can complement each other in certain workflows, even though they serve different purposes. Kustomize is widely regarded as the second most popular Kubernetes configuration tool after Helm, largely because it enhances Helm’s ability to manage environment-specific configurations.

For example, you can use Helm to generate the base Kubernetes manifests and then use Kustomize to apply additional customizations as patches. This allows you to dynamically modify the generated resources without altering the Helm chart itself.

However, if you want to modify an existing Helm deployment using Kustomize, additional tooling like ArgoCD is required, as Kustomize doesn’t manage or track Helm releases natively.

In an upcoming blog post, we’ll explore the key differences between Kustomize and Helm, and provide a working example that demonstrates how these tools can be effectively combined.

What are Kustomize components? 

As of v3.7.0, Kustomize supports a special new type of kustomization called components, which allows reusable configuration logic to be included across multiple overlays.

Components are useful when dealing with applications that support multiple optional features, enabling you to activate different features in specific overlays for different environments or audiences.

Read more about Kustomize components here.

Conclusion

Kustomize was born out of a need to evolve Kubernetes resource configuration tools, offering a fresh, template-free approach to managing configurations across different environments. It was designed to build upon existing Kubernetes API practices, avoiding complex configuration generators, while introducing a powerful yet intuitive way to handle parameters.

By embracing declarative resource specifications, Kustomize enables composable and reusable configurations and has emerged as a robust Kubernetes-native solution that conveniently extends the capabilities of kubectl yet leaving the complexities of packaging to other tools.

Do you still think that offering on-premises installations is hard?

Glasskube Cloud is a battle tested software distribution platform that helps you scale from your first on-premises customers to dozens and even thousands.

Try for freeBook free demo