Diagram of product resources

Module deploys the monitoring stack to the Kubernetes (K8s) cluster. Key features:

  • flexible configuration where components can be toggled or customized
  • default setup deploys a set of key components with a minimal configuration for logs and metrics overview
  • main exporters and collectors are enabled
  • persistent storage for all the components configured
  • Grafana is exposed via Ingress
  • Loki and Prometheus are available only inside the cluster
  • if Prometheus or Loki are marked as enabled, corresponding local datasources for Grafana will be created

Components overview

Nginx is used as ingress_class in Ingress Annotations by default for all main monitoring stack components - all main components expect Nginx as Ingress class as dependency for this module. If custom ingress_class and ingress_auth_enabled are used, specific auth Annotations must be provided through loki.custom_values and prometheus.custom_values.

Dashboard interface

Grafana - data visualization web dashboard. Managed as a Grafana Custom Resource with the Grafana Operator pre-installed. Can be managed with the grafana input block.

Metrics

Prometheus - metrics aggregator and storage. Managed as a single Helm release with Node exporter, Pushgateway, and Alertmanager. Can be managed with the prometheus input block. Prometheus uses Recreate update strategy type that causes short downtime between deleting old Pod and creating a new one to properly re-attach volumes.

Node exporter - collects metrics from the K8s hosts and workloads.

Pushgateway - allows ephemeral and batch jobs to expose their metrics to Prometheus. Disabled by default. Can be managed with the pushgateway input block.

Alertmanager - handles alerts sent by client applications such as the Prometheus server. Disabled by default.

Logs

Loki - logs aggregator and storage. Managed as a Helm release. Can be managed with the loki input block. Supported deployment modes are: monolithic (deprecated: single binary) and simple scalable.

NOTE: Loki in simple scalable deployment (SSD) or distributed mode requires object storage.
In SingleBinary mode, filesystem object store is commonly used. Existing logs are not retained unless they are migrated.

NOTE: In Loki simple scalable deployment (SSD) mode, log write operations require a quorum of ingesters (floor(replication_factor / 2) + 1), which run in the write replicas. For example, 3 write replicas require 2/3 successful writes, while 2 write replicas require 2/2. If quorum is not reached, the write operation is retried.

NOTE: In Loki distributed mode, log write operations require a quorum of ingesters (floor(replication_factor / 2) + 1). For example, 3 ingester replicas require 2/3 running pods, while 2 ingester replicas require 2/2. If quorum is not reached, the ingest operation is retried.

NOTE: Enable loki.bind_memberlist_endpoint if your K8s node with an address in the non-standard IP range (outside RFC 1918 or RFC 6598: 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12). This prevents Loki from failing to identify a private advertise address during deployment:

 bashfailed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided

Fluent Bit - logs and optionally K8s event processor and forwarder. Managed as a Helm release. Can be managed with the fluent_bit input block.

Traces

Tempo - tracing backend. Disabled by default. Managed as a Helm release. Can be managed with the tempo input block. Tempo works in distributed (microservice) mode. Tempo core components are: compactor, distributor, ingester, querier, query-frontend, and memcached. The gateway component can be optionally enabled to receive traces outside of the cluster in which Tempo is installed.

Log in to Corewide IaC registry

Once you have a Corewide Solutions Portal account, this one-time action will use your browser session to retrieve credentials:

 shellterraform login solutions.corewide.com
Provision instructions

Initialize mandatory providers:

Copy and paste into your Terraform configuration and insert the variables:

 hclmodule "tf_k8s_monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 6.1.0"

  # specify module inputs here or try one of the examples below
  ...
}

Initialize the setup:

 shellterraform init
Define update strategy

Corewide DevOps team strictly follows Semantic Versioning Specification to provide our clients with products that have predictable upgrades between versions. We recommend pinning patch versions of our modules using pessimistic constraint operator (~>) to prevent breaking changes during upgrades.

To get new features during the upgrades (without breaking compatibility), use ~> 6.1 and run terraform init -upgrade

For the safest setup, use strict pinning with version = "6.1.0"

Diagram of product resources

Module deploys the monitoring stack to the Kubernetes (K8s) cluster. Key features:

  • flexible configuration where components can be toggled or customized
  • default setup deploys a set of key components with a minimal configuration for logs and metrics overview
  • main exporters and collectors are enabled
  • persistent storage for all the components configured
  • Grafana is exposed via Ingress
  • Loki and Prometheus are available only inside the cluster
  • if Prometheus or Loki are marked as enabled, corresponding local datasources for Grafana will be created

Components overview

Nginx is used as ingress_class in Ingress Annotations by default for all main monitoring stack components - all main components expect Nginx as Ingress class as dependency for this module. If custom ingress_class and ingress_auth_enabled are used, specific auth Annotations must be provided through loki.custom_values and prometheus.custom_values.

Dashboard interface

Grafana - data visualization web dashboard. Managed as a Grafana Custom Resource with the Grafana Operator pre-installed. Can be managed with the grafana input block.

Metrics

Prometheus - metrics aggregator and storage. Managed as a single Helm release with Node exporter, Pushgateway, and Alertmanager. Can be managed with the prometheus input block. Prometheus uses Recreate update strategy type that causes short downtime between deleting old Pod and creating a new one to properly re-attach volumes.

Node exporter - collects metrics from the K8s hosts and workloads.

Pushgateway - allows ephemeral and batch jobs to expose their metrics to Prometheus. Disabled by default. Can be managed with the pushgateway input block.

Alertmanager - handles alerts sent by client applications such as the Prometheus server. Disabled by default.

Logs

Loki - logs aggregator and storage. Managed as a Helm release. Can be managed with the loki input block. Supported deployment modes are: monolithic (deprecated: single binary) and simple scalable.

NOTE: Loki in simple scalable deployment (SSD) or distributed mode requires object storage.
In SingleBinary mode, filesystem object store is commonly used. Existing logs are not retained unless they are migrated.

NOTE: In Loki simple scalable deployment (SSD) mode, log write operations require a quorum of ingesters (floor(replication_factor / 2) + 1), which run in the write replicas. For example, 3 write replicas require 2/3 successful writes, while 2 write replicas require 2/2. If quorum is not reached, the write operation is retried.

NOTE: In Loki distributed mode, log write operations require a quorum of ingesters (floor(replication_factor / 2) + 1). For example, 3 ingester replicas require 2/3 running pods, while 2 ingester replicas require 2/2. If quorum is not reached, the ingest operation is retried.

NOTE: Enable loki.bind_memberlist_endpoint if your K8s node with an address in the non-standard IP range (outside RFC 1918 or RFC 6598: 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12). This prevents Loki from failing to identify a private advertise address during deployment:

 bashfailed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided

Fluent Bit - logs and optionally K8s event processor and forwarder. Managed as a Helm release. Can be managed with the fluent_bit input block.

Traces

Tempo - tracing backend. Disabled by default. Managed as a Helm release. Can be managed with the tempo input block. Tempo works in distributed (microservice) mode. Tempo core components are: compactor, distributor, ingester, querier, query-frontend, and memcached. The gateway component can be optionally enabled to receive traces outside of the cluster in which Tempo is installed.

Deploy complete stack with only mandatory values:

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 6.1"

  grafana = {
    ingress_host = "testmon.example.com"
  }
}

Deploy partial stack with some customization, Prometheus and Node Exporter are disabled, Tempo with gateway enabled:

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 6.1"

  grafana = {
    ingress_host  = "testmon.example.com"
    admin_pass    = "YYYY-YYYY-YYYY"
    storage_class = "standard"
  }

  prometheus = {
    enabled               = false
    node_exporter_enabled = false
  }

  tempo = {
    enabled = true

    gateway = {
      enabled              = true
      ingress_auth_enabled = true
      host                 = "tempo.example.com"
    }

    node_selector = {
      "kubernetes.azure.com/agentpool" = "maintenance"
    }
  }
}

Deploy full stack and an additional datasource.

Set node selectors for already existing Prometheus, add Pushgateway from Prometheus chart, set Loki deployment mode to simple scalable, add custom value for Grafana, enable basic authentication and its credentials:

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 6.1"

  name_prefix = "dev"

  auth_credentials = {
    password = "XXXX-XXXX-XXXX"
  }

  grafana = {
    ingress_host = "testmon.example.com"
    admin_pass   = "YYYY-YYYY-YYYY"

    node_selector = {
      "cloud.google.com/gke-nodepool" = "maintenance"
    }

    env_vars = {
      GF_PLUGIN_GRAFANA_IMAGE_RENDERER_RENDERING_IGNORE_HTTPS_ERRORS = true
    }
  }

  prometheus = {
    ingress_enabled      = true
    ingress_auth_enabled = true
    ingress_host         = "testmetrics.example.com"
    remote_write_enabled = true

    node_selector = {
      "cloud.google.com/gke-nodepool" = "maintenance"
    }
  }

  pushgateway = {
    enabled         = true
    ingress_enabled = true
    ingress_host    = "pushgw.example.com"
    volume_size     = "5Gi"
  }

  loki = {
    deployment_mode    = "SimpleScalable"
    bucket_region      = "ams3"
    chunks_bucket_name = "loki-chunks"

    object_storage_do_spaces = {
      access_key_id     = "XXXX"
      secret_access_key = "XXXX"
    }

    node_selector = {
      "cloud.google.com/gke-nodepool" = "maintenance"
    }
  }

  grafana_datasources = [
    {
      name               = "Prometheus Dev"
      type               = "prometheus"
      url                = "https://devprom.example.com"
      basic_auth_enabled = true
      basic_auth_pass    = "XXXX-XXXX-XXXX"
      basic_auth_user    = "monitoring"
    },
  ]
}

Loki deployment mode set to simple scalable with Azure object storage configured:

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 6.1"

  name_prefix = "dev"

  loki = {
    deployment_mode    = "SimpleScalable"
    chunks_bucket_name = "loki-chunks"

    object_storage_azure = {
      client_id            = "abcdefgh-1234-5678-9012-a1b2c3d4e5f6g7"
      storage_account_name = "loki-account"
    }
  }

  auth_credentials = {
    # ...
  }

  grafana = {
    # ...
  }

  prometheus = {
    # ...
  }

  pushgateway = {
    # ...
  }

  grafana_datasources = [
    {
      # ...
    },
  ]
}

Loki deployment mode set to simple scalable with GCP object storage configured:

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 6.1"

  name_prefix = "dev"

  loki = {
    deployment_mode    = "SimpleScalable"
    chunks_bucket_name = "loki-chunks"

    object_storage_gcs = {
      service_account_email = "[email protected]"
    }
  }

  auth_credentials = {
    # ...
  }

  grafana = {
    # ...
  }

  prometheus = {
    # ...
  }

  pushgateway = {
    # ...
  }

  grafana_datasources = [
    {
      # ...
    },
  ]
}

Loki deployment mode set to distributed with DO object storage configured

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 6.1"

  name_prefix = "dev"

  loki = {
    deployment_mode    = "Distributed"
    chunks_bucket_name = "loki-chunks"
    bucket_region      = "fra1"

    object_storage_do_spaces = {
      access_key_id     = "DO00AA11BB22CC33DD"
      secret_access_key = "+1a2B3cDE456fGH78iJkL0mNO1234pQrST567UvW+89X"
    }
  }

  auth_credentials = {
    # ...
  }

  grafana = {
    # ...
  }

  prometheus = {
    # ...
  }

  pushgateway = {
    # ...
  }

  grafana_datasources = [
    {
      # ...
    },
  ]
}
Variable Description Type Default Required Sensitive
grafana Grafana parameters object yes no
auth_credentials Ingress Nginx basic auth login credentials object {} no yes
auth_credentials.password Ingress Nginx basic auth login password (will be randomly generated if it's not set) string no yes
auth_credentials.username Ingress Nginx basic auth login username string monitoring no yes
create_namespace Indicates creation of dedicated namespace for monitoring components bool true no no
fluent_bit Fluent Bit parameters object {} no no
fluent_bit.app_version Fluent Bit image version string 5.0.7 no no
fluent_bit.chart_version Helm chart version string 0.57.7 no no
fluent_bit.collect_kubernetes_events Toggle Kubernetes events collection bool true no no
fluent_bit.enabled Toggle Fluent Bit installation bool true no no
fluent_bit.node_selector Node selector to place Fluent Bit pods in map(any) {} no no
grafana.admin_pass Grafana admin password (will be randomly generated if it's not set) string no no
grafana.admin_user Grafana admin username string admin no no
grafana.enabled Toggle Grafana installation bool true no no
grafana.env_vars Environment variables for Grafana container in key-value format map(any) {} no no
grafana.grafana_version Grafana server version string 13.0.2 no no
grafana.ingress_annotations Map of ingress annotations (applied additionally to cert-manager.io/cluster-issuer) map(any) {} no no
grafana.ingress_host Hostname to use with Ingress (required if enabled is true) string no no
grafana.log_level Grafana log level (Supported levels: trace, debug, info, warn, error or critical) string warn no no
grafana.node_selector Node selector to place Grafana pods in map(any) {} no no
grafana.operator_app_version Grafana operator image version string v5.24.0 no no
grafana.operator_chart_version Grafana operator Helm chart version string 5.24.0 no no
grafana.recreate_on_changes Whether the Grafana CRD should be recreated and not updated during apply phase bool false no no
grafana.storage_class Storage class name (will be used a cluster's default if not specified) string no no
grafana.volume_size Volume data size string 5Gi no no
grafana_datasources Grafana datasources for datasource provisioning list(object) [] no yes
grafana_datasources[*].basic_auth_enabled Toggle Ingress basic auth bool no yes
grafana_datasources[*].basic_auth_pass Ingress basic auth password string no yes
grafana_datasources[*].basic_auth_user Ingress basic auth user string no yes
grafana_datasources[*].name Name of the datasource string no yes
grafana_datasources[*].type Type of the datasource string no yes
grafana_datasources[*].url URL of the datasource string no yes
ingress_cert_issuer Ingress TLS certificate issuer string letsencrypt no no
ingress_class Ingress Class definition string nginx no no
loki Loki parameters object {} no no
loki.app_version Loki server version string 3.7.2 no no
loki.backend_replicas Number of Loki backend replicas (simple scalable mode only) number 1 no no
loki.bind_memberlist_endpoint Toggle explicit bind of POD IP to Loki-Memberlist Kubernetes service. Required only for deployment on a K8s node with an address in the non-standard IP range (outside RFC 1918 or RFC 6598) bool false no no
loki.bucket_region Region of the object storage provider where buckets for Loki are located (simple scalable or distributed mode only) string no no
loki.chart_version Helm chart version (compatible versions are 12.0.0 and newer) string 17.4.9 no no
loki.chunks_bucket_name Bucket name for Loki chunk storage (simple scalable or distributed mode only) string no no
loki.compactor_replicas Number of Loki compactor replicas (distributed mode only) number 1 no no
loki.custom_values Custom values to pass to the Loki Helm chart as a block any {} no no
loki.deployment_mode Specify how to deploy Loki. Possible values: Monolithic (deprecated: SingleBinary), SimpleScalable, and Distributed string Monolithic no no
loki.distributor_replicas Number of Loki distributor replicas (distributed mode only) number 2 no no
loki.enabled Toggle Loki installation bool true no no
loki.ingester_replicas Number of Loki ingester replicas (distributed mode only) number 2 no no
loki.ingester_zone_replication Whether to enable zone-aware replication for the ingester pods (distributed mode only, requires 3 availability zones by default) bool false no no
loki.ingress_auth_enabled Toggle Ingress basic auth (effective if ingress_enabled is true) bool false no no
loki.ingress_enabled Whether to expose with Ingress bool false no no
loki.ingress_host Hostname to use with Ingress (effective if ingress_enabled is true) string no no
loki.node_selector Node selector to place Loki pods in map(any) {} no no
loki.object_storage_aws_s3 AWS S3 object storage configuration enabling Loki bucket access (simple scalable or distributed mode only) object no no
loki.object_storage_aws_s3.aws_iam_role_arn ARN of the AWS IAM role assumed by Loki for accessing S3 (IRSA) string no no
loki.object_storage_azure Azure Blob Storage configuration enabling Loki bucket access (simple scalable or distributed mode only) object no no
loki.object_storage_azure.client_id Azure user-assigned managed identity application ID string no no
loki.object_storage_azure.storage_account_name Azure storage account name string no no
loki.object_storage_do_spaces DigitalOcean Spaces object storage configuration enabling Loki bucket access (simple scalable or distributed mode only) object no no
loki.object_storage_do_spaces.access_key_id DigitalOcean Spaces access key ID string no no
loki.object_storage_do_spaces.k8s_secret_name Name of the Kubernetes Secret that will store DigitalOcean Spaces credentials string loki-do-spaces-creds no no
loki.object_storage_do_spaces.secret_access_key DigitalOcean Spaces secret access key string no no
loki.object_storage_gcs Google Cloud Storage configuration enabling Loki bucket access (simple scalable or distributed mode only) object no no
loki.object_storage_gcs.service_account_email Google Cloud service account email string no no
loki.querier_replicas Number of Loki querier replicas (distributed mode only) number 2 no no
loki.query_frontend_replicas Number of Loki query frontend replicas (distributed mode only) number 1 no no
loki.query_scheduler_replicas Number of Loki query scheduler replicas (distributed mode only) number 1 no no
loki.read_replicas Number of Loki read replicas (simple scalable mode only) number 1 no no
loki.retention_period Data retention period string 93d no no
loki.sa_name Kubernetes ServiceAccount name used by all Loki components string loki no no
loki.storage_class Storage class name (monolithic mode only) string no no
loki.volume_size Volume data size (monolithic mode only) string 100Gi no no
loki.write_replicas Number of Loki write replicas (simple scalable mode only) number 1 no no
name_prefix Name prefix for resources creation string no no
namespace Monitoring stack namespace string monitoring no no
prometheus Prometheus parameters object {} no no
prometheus.alertmanager_enabled Toggle Prometheus alertmanager installation bool false no no
prometheus.app_version Prometheus server version string v3.12.0 no no
prometheus.chart_version Helm chart version (compatible versions are 22.0.0 and newer) string 29.10.0 no no
prometheus.custom_values Custom values to pass to the Prometheus Helm chart as a block any {} no no
prometheus.enabled Toggle Prometheus installation bool true no no
prometheus.ingress_auth_enabled Toggle Nginx basic auth (effective if ingress_enabled is true) bool false no no
prometheus.ingress_enabled Whether to expose with Ingress bool false no no
prometheus.ingress_host Hostname to use with Ingress (effective if ingress_enabled is true) string no no
prometheus.node_exporter_enabled Toggle Node exporter installation bool true no no
prometheus.node_selector Node selector to place Prometheus pods in map(any) {} no no
prometheus.remote_write_enabled Toggle the remote write receiver feature bool false no no
prometheus.retention_period Data retention period string 93d no no
prometheus.storage_class Storage class name string no no
prometheus.volume_size Volume data size string 100Gi no no
pushgateway Pushgateway parameters object {} no no
pushgateway.app_version Pushgateway server version string v1.11.3 no no
pushgateway.enabled Toggle Pushgateway installation bool false no no
pushgateway.ingress_auth_enabled Toggle Nginx basic auth (effective if ingress_enabled is true) bool false no no
pushgateway.ingress_enabled Whether to expose with Ingress bool false no no
pushgateway.ingress_host Hostname to use with Ingress (effective if ingress_enabled is true) string no no
pushgateway.volume_size Volume data size string 2Gi no no
tempo Tempo parameters object {} no no
tempo.app_version Tempo components version (compatible versions are 2.9.0 and newer) string 2.10.7 no no
tempo.azure_remote_storage Azure blob storage for Tempo data object no no
tempo.azure_remote_storage.container_name Storage account container name string no no
tempo.azure_remote_storage.storage_account_key Storage account key string no no
tempo.azure_remote_storage.storage_account_name Storage account name string no no
tempo.chart_version Tempo distributed Helm chart version (compatible versions are 2.18.0 and newer) string 2.25.5 no no
tempo.custom_values Custom values to pass to the Tempo Helm chart as a block any {} no no
tempo.enabled Toggle Tempo installation bool false no no
tempo.gateway Gateway configuration block object no no
tempo.gateway.enabled Toggle gateway for Tempo bool false no no
tempo.gateway.host Hostname to expose the gateway on string no no
tempo.gateway.ingress_auth_enabled Toggle Nginx basic auth for the gateway service bool false no no
tempo.ingress_auth_enabled Toggle Nginx basic auth (effective if ingress_enabled is true) bool false no no
tempo.ingress_enabled Whether to expose with Ingress bool false no no
tempo.ingress_host Hostname to use with Ingress (effective if ingress_enabled is true) string no no
tempo.node_selector Node selector to place Tempo components in map(any) {} no no
tempo.span_end_time_shift Shifts the end time for the logs query, based on the span's end time string -1h no no
tempo.span_start_time_shift Shifts the start time for the logs query, based on the span's start time string 1h no no
tempo.traces_tags Define additional tags when provisioning traces to logs feature map(any) {} no no
tempo.traces_to_logs Toggle traces to logs feature bool false no no
Output Description Type Sensitive
basic_auth_credentials Ingress basic auth credentials computed yes
basic_auth_credentials.password Password for basic HTTP authentication computed yes
basic_auth_credentials.username Username for basic HTTP authentication computed yes
grafana_admin_credentials Contains admin user credentials for Grafana web UI map yes
grafana_admin_credentials.admin_pass Password of the Grafana web UI admin user map yes
grafana_admin_credentials.admin_user Name of the Grafana web UI admin user map yes
grafana_instance_selector_label Selector label to match dashboard and data source CRs to Grafana CR computed no
ingress_hosts Ingress exposed hosts map no
ingress_hosts.grafana Grafana web UI hostname map no
ingress_hosts.loki Loki endpoint hostname map no
ingress_hosts.prometheus Prometheus endpoint hostname map no
ingress_hosts.tempo Tempo query endpoint hostname map no
ingress_hosts.tempo_gateway Tempo gateway endpoint hostname map no
Dependency Version Kind
terraform >= 1.3 CLI
hashicorp/helm ~> 3.1 provider
hashicorp/kubernetes ~> 3.0 provider
hashicorp/random ~> 3.3 provider
tf-k8s-crd ~> 3.0 module
tf-k8s-grafana ~> 2.1 module

Not sure where to start?
Let's find your perfect match.