Skip to main content

Install the Kube Prometheus Stack

Prometheus and Grafana are open-source, community-backed solutions with stellar reputations. They bring immense value by fetching and storing metrics while enabling the creation of dashboards that are not only useful but also easy on the eyes.

The uncomfortable truth is that anyone who has ever set up Prometheus alongside Grafana as their environment's monitoring stack from scratch has probably felt the frustration of not getting value especially quickly. Metric exporter configuration, dashboard widget customisation and deciding what to monitor and alert on in the first place takes time.

That's why kube-prometheus-stack was created. It installs a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules, providing an easy-to-operate, end-to-end Kubernetes cluster monitoring solution with Prometheus using the Prometheus Operator.

This sounds like good news, and it is, but the stack is bundled in a Helm chart, and just the values.yaml file has over 4000 lines. Configuring and maintaining the Helm chart isn’t necessarily straightforward or “fun.”

With so many configuration options, we must be getting something good right? Well yeah, we are, by deploying kube-prometheus-stack we get all of this right out of the box:

stack-diagram

Top Layer:

  • User: Interacts with Grafana and Kubernetes API.

Visualization and Alerting Layer:

  • Grafana: Connects to Prometheus for data visualization.
  • Alertmanager: Connected to Prometheus for alert management.
  • Prometheus Server: Central component collecting and storing metrics.

Exporters Layer:

  • Node Exporter: Collects node-level metrics.
  • Kube State Metrics: Collects metrics from Kubernetes API objects.
  • Other Exporters: Additional exporters for various applications and services.

Kubernetes Cluster:

  • Kubernetes Nodes: Running applications and system components.
  • Applications: Monitored by the Kube Prometheus Stack.

Luckily, Glasskube now supports the kube-prometheus-stack. Package configuration, lifecycle management, and installation can be done in record time.

In this blog post, we will explore the steps to configure and install the kube-prometheus-stack using Glasskube, wasting no unnecessary time wrestling with never-ending values files and getting you working dashboards and alerts quicker than ever before.

Requirements:​

  • Access to a Kubernetes cluster (Minikube will be fine)
  • Glasskube installed
  • An extra screen for all the cool dashboards you are going to want to look at all the time. 🤣

Create a cluster​

Install Minikube then run:

minikube start

Check your installation by running:

minukube status

Desired output:

➜  ~ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Install Glasskube​

If you already installed glasskube you can skip this step. If not, glasskube can easily be installed the way you usually install packages for your operating system.

On macOS, you can use Homebrew to install and update Glasskube.

brew install glasskube/tap/glasskube

After installing Glasskube on your local machine, make sure to install the necessary components in your Kubernetes cluster by running glasskube bootstrap. For more information, check out our bootstrap guide.

Install kube-prometheus-stack​

Start the UI via the command line:

glasskube serve

Install kube-prometheus-stack via the Glasskube UI by following the package customization steps below.

Package Customization​

Glasskube offers a series of customisations that we can be tweaked and adjusted from the CLI or GUI, saving you from having to render and configure the values.yaml file directly.

parameter config

Let’s take them one by one.

Enable Alertmanager 🚨​

We want Alertmanager to be enabled so we can leverage the metrics prometheus exposes to create helpful alerts.

Grafana Domain 📊​

We will leave this empty for this demo since we would need to deploy an ingress controller to our cluster to handle the ingress object associated with the grafana service. We could use kube-prometheus-stack or caddy-ingress which are also supported by Glasskube for this.

Glasskube will automatically port-forward the Grafana pod so we can access the dashboard via the Open button.

Node Exporter host network 💽​

Let’s also enable this to export node level metrics like memory and node level CPU usage.

Prometheus retention 📅​

This is a duration in days for how long we want to persist the collected metrics.

Prometheus storage size​

The amount of storage requests we consider the package will need.

Parameter input methods 🗄️​

Glasskube allows for various methods of parameter input:

  • From a Kubernetes Secret
  • From ConfigMap
  • Value from Package Configuration
  • Via the UI

data input methods

By choosing to inject data via Kubernetes Secrets, ConfigMaps and Package configuration we can maintain simplicity without compromising security.

Here is the example of how we would reference a specific configMap we have already created and deployed to our cluster.

Value from ConfigMap

💡 If you're using kube-prometheus-stack and considering Glasskube for package lifecycle management but need support for specific key parameter customizations, please open an issue on GitHub with your use case. We'll do our best to expand the parameter list accordingly.

Install via Glasskkube​

Once the configuration section is complete, install kube-prometheus-stack.

Glasskube UI

Upon installation you can see that the kube-prometheus-stack namespace has been created. And a series of pods have been deployed, including the grafana dashboard, the prometheus operator and the kube state metrics pods too.

cli output

Access the dashboards​

In next weeks blog post we will access the dashboard via a custom dedicated Grafana URL

open command

Hit the Open button or if you want to access Grafana on a different port you can simply port-forward the pod, which will map the exposed Grafana port to a port on your localhost. I've arbitrarily chosen to port-forward to localhost 52222 since it's available.

kubectl port-forward POD_NAME 52222:3000

Head over to http://localhost:52222/ and you will then be greated by the Granfana login page. To find your credentials which are stored in a Kubernetes secret that was generated as part of the deployed stack, run:

kubectl get secret kube-prometheus-stack-kube-prometheus-stack-grafana  -o go-template='
{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

Which will output something like:

admin-password: <password>
admin-user: admin
ldap-toml:

Upon access you we be greeted by a long list of powerful pre-configured Grafana dashboards which are already showing local cluster metrics:

dashboards

Easily access CPU usage information​

CPU dashboard

Here is a segment of the nifty CoreDNS dashboard that also comes preconfigured​

CoreDNS

Alerting​

We already get many useful alerts created for us right out of the box.

alerting rules

In this snippet you can see that some of the preconfigured alerts are already firing: ↘️

firing alerts

If you want to be notified in via email, PagerDuty or any number of third party supported you will just need to add your contact points of preference and then add them as destination inside custom notification policies.

The kube-prometheus-stack offers tremendous "out-of-the-box" value for Kubernetes cluster monitoring, eliminating the need to start from scratch. It bundles essential components for metrics exposure, extraction, alerting, and visualization, helping you establish a robust monitoring posture from the get-go. With official support from Glasskube, managing and updating a comprehensive, best practice-compliant monitoring stack has never been easier.

If you like this sort of content and would like to see more of it, please consider supporting us by giving us a Star on GitHub 🙏 cats-like--github-stars