All products Terraform Kubernetes helm
tf-k8s-monitoring-stack
Diagram of product resources

Module installs the monitoring stack where components can be disabled or customized.
The default setup deploys a complete set with a minimal configuration - Grafana is exposed via Ingress, Loki and Prometheus are available only inside the cluster, Tempo disabled, persistent storage for all the components configured, main exporters are enabled.
Tempo works in distributed (microservice) mode.

Tempo core components are: compactor, distributor, ingester, querier, query-frontend, memcached.
Grafana and Prometheus use Recreate update strategy type that causes short downtime between deleting old pod and creating a new one to properly re-attach volumes.
All main components expect Nginx as Ingress class as dependency for this module.
If Prometheus or Loki are marked as enabled, corresponding local datasources for Grafana will be created.
Pushgateway installed within Prometheus helm chart and disabled by default. Use pushgateway values from variables table to install and configure it. The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus.
Nginx is used as ingress_class in Ingress Annotations by default for all main monitoring-stack components. If custom ingress_class and ingress_auth_enabled are used, specific auth Annotations must be provided through loki.custom_values and prometheus.custom_values.

NOTE: Loki in simple scalable deployment (SSD) mode requires object storage.
In SingleBinary mode, filesystem object store is commonly used. Existing logs are not retained unless they are migrated.

NOTE: In Loki simple scalable deployment (SSD) mode, log write operations require a quorum of ingesters (floor(replication_factor / 2) + 1), which run in the write replicas. For example, 3 write replicas require 2/3 successful writes, while 2 write replicas require 2/2. If quorum is not reached, the write operation is retried.

NOTE: Enable loki.bind_memberlist_endpoint if your K8s node with an address in the non-standard IP range (outside RFC 1918 or RFC 6598: 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12). This prevents Loki from failing to identify a private advertise address during deployment:

 bashfailed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided
Log in to Corewide IaC registry

Once you have a Corewide Solutions Portal account, this one-time action will use your browser session to retrieve credentials:

 shellterraform login solutions.corewide.com
Provision instructions

Initialize mandatory providers:

Copy and paste into your Terraform configuration and insert the variables:

 hclmodule "tf_k8s_monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 5.1.3"

  # specify module inputs here or try one of the examples below
  ...
}

Initialize the setup:

 shellterraform init
Define update strategy

Corewide DevOps team strictly follows Semantic Versioning Specification to provide our clients with products that have predictable upgrades between versions. We recommend pinning patch versions of our modules using pessimistic constraint operator (~>) to prevent breaking changes during upgrades.

To get new features during the upgrades (without breaking compatibility), use ~> 5.1 and run terraform init -upgrade

For the safest setup, use strict pinning with version = "5.1.3"

Diagram of product resources

Module installs the monitoring stack where components can be disabled or customized.
The default setup deploys a complete set with a minimal configuration - Grafana is exposed via Ingress, Loki and Prometheus are available only inside the cluster, Tempo disabled, persistent storage for all the components configured, main exporters are enabled.
Tempo works in distributed (microservice) mode.

Tempo core components are: compactor, distributor, ingester, querier, query-frontend, memcached.
Grafana and Prometheus use Recreate update strategy type that causes short downtime between deleting old pod and creating a new one to properly re-attach volumes.
All main components expect Nginx as Ingress class as dependency for this module.
If Prometheus or Loki are marked as enabled, corresponding local datasources for Grafana will be created.
Pushgateway installed within Prometheus helm chart and disabled by default. Use pushgateway values from variables table to install and configure it. The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus.
Nginx is used as ingress_class in Ingress Annotations by default for all main monitoring-stack components. If custom ingress_class and ingress_auth_enabled are used, specific auth Annotations must be provided through loki.custom_values and prometheus.custom_values.

NOTE: Loki in simple scalable deployment (SSD) mode requires object storage.
In SingleBinary mode, filesystem object store is commonly used. Existing logs are not retained unless they are migrated.

NOTE: In Loki simple scalable deployment (SSD) mode, log write operations require a quorum of ingesters (floor(replication_factor / 2) + 1), which run in the write replicas. For example, 3 write replicas require 2/3 successful writes, while 2 write replicas require 2/2. If quorum is not reached, the write operation is retried.

NOTE: Enable loki.bind_memberlist_endpoint if your K8s node with an address in the non-standard IP range (outside RFC 1918 or RFC 6598: 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12). This prevents Loki from failing to identify a private advertise address during deployment:

 bashfailed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided

Deploy complete stack with only mandatory values:

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 5.1"

  grafana = {
    ingress_host = "testmon.example.com"
  }
}

Deploy partial stack with some customization, Prometheus and Node Exporter are disabled, Tempo with gateway enabled:

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 5.1"

  grafana = {
    ingress_host  = "testmon.example.com"
    admin_pass    = "YYYY-YYYY-YYYY"
    storage_class = "standard"
  }

  prometheus = {
    enabled               = false
    node_exporter_enabled = false
  }

  tempo = {
    enabled = true

    gateway = {
      enabled              = true
      ingress_auth_enabled = true
      host                 = "tempo.example.com"
    }

    node_selector = {
      "kubernetes.azure.com/agentpool" = "maintenance"
    }
  }
}

Deploy full stack and an additional datasource.

Set node selectors for already existing Prometheus, add Pushgateway from Prometheus chart, set Loki deployment mode to simple scalable, add custom value for Grafana, enable basic authentication and its credentials:

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 5.1"

  name_prefix = "dev"

  auth_credentials = {
    password = "XXXX-XXXX-XXXX"
  }

  grafana = {
    ingress_host = "testmon.example.com"
    admin_pass   = "YYYY-YYYY-YYYY"

    node_selector = {
      "cloud.google.com/gke-nodepool" = "maintenance"
    }

    env_vars = {
      GF_PLUGIN_GRAFANA_IMAGE_RENDERER_RENDERING_IGNORE_HTTPS_ERRORS = true
    }
  }

  prometheus = {
    ingress_enabled      = true
    ingress_auth_enabled = true
    ingress_host         = "testmetrics.example.com"
    remote_write_enabled = true

    node_selector = {
      "cloud.google.com/gke-nodepool" = "maintenance"
    }
  }

  pushgateway = {
    enabled         = true
    ingress_enabled = true
    ingress_host    = "pushgw.example.com"
    volume_size     = "5Gi"
  }

  loki = {
    enable_simple_scalable_mode = true
    bucket_region               = "ams3"
    chunks_bucket_name          = "loki-chunks"
    ruler_bucket_name           = "loki-ruler"

    object_storage_do_spaces = {
      access_key_id     = "XXXX"
      secret_access_key = "XXXX"
    }

    node_selector = {
      "cloud.google.com/gke-nodepool" = "maintenance"
    }
  }

  grafana_datasources = [
    {
      name               = "Prometheus Dev"
      type               = "prometheus"
      url                = "https://devprom.example.com"
      basic_auth_enabled = true
      basic_auth_pass    = "XXXX-XXXX-XXXX"
      basic_auth_user    = "monitoring"
    },
  ]
}

Loki deployment mode set to simple scalable with Azure object storage configured:

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 5.1"

  name_prefix = "dev"

  loki = {
    enable_simple_scalable_mode = true
    chunks_bucket_name          = "loki-chunks"
    ruler_bucket_name           = "loki-ruler"

    object_storage_azure = {
      client_id            = "abcdefgh-1234-5678-9012-a1b2c3d4e5f6g7"
      storage_account_name = "loki-account"
    }
  }

  auth_credentials = {
    # ...
  }

  grafana = {
    # ...
  }

  prometheus = {
    # ...
  }

  pushgateway = {
    # ...
  }

  grafana_datasources = [
    {
      # ...
    },
  ]
}

Loki deployment mode set to simple scalable with GCP object storage configured:

 hclmodule "monitoring_stack" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-monitoring-stack/helm"
  version = "~> 5.1"

  name_prefix = "dev"

  loki = {
    enable_simple_scalable_mode = true
    chunks_bucket_name          = "loki-chunks"
    ruler_bucket_name           = "loki-ruler"

    object_storage_gcs = {
      service_account_email = "[email protected]"
    }
  }

  auth_credentials = {
    # ...
  }

  grafana = {
    # ...
  }

  prometheus = {
    # ...
  }

  pushgateway = {
    # ...
  }

  grafana_datasources = [
    {
      # ...
    },
  ]
}
Variable Description Type Default Required Sensitive
grafana Grafana parameters object yes no
auth_credentials Ingress Nginx basic auth login credentials object {} no yes
auth_credentials.password Ingress Nginx basic auth login password (will be randomly generated if it's not set) string no yes
auth_credentials.username Ingress Nginx basic auth login username string monitoring no yes
create_namespace Indicates creation of dedicated namespace for monitoring components bool true no no
grafana.admin_pass Grafana admin password (will be randomly generated if it's not set) string no no
grafana.admin_user Grafana admin username string admin no no
grafana.enabled Toggle Grafana installation bool true no no
grafana.env_vars Environment variables for Grafana container in key-value format map(any) {} no no
grafana.grafana_version Grafana server version string 11.2.2 no no
grafana.ingress_annotations Map of ingress annotations (applied additionally to cert-manager.io/cluster-issuer) map(any) {} no no
grafana.ingress_host Hostname to use with Ingress (required if enabled is true) string no no
grafana.log_level Grafana log level (Supported levels: trace, debug, info, warn, error or critical) string warn no no
grafana.node_selector Node selector to place Grafana pods in map(any) {} no no
grafana.operator_app_version Grafana operator image version string v5.9.2 no no
grafana.operator_chart_version Grafana operator Helm chart version string v5.9.2 no no
grafana.recreate_on_changes Whether the Grafana CRD should be recreated and not updated during apply phase bool false no no
grafana.storage_class Storage class name string no no
grafana.volume_size Volume data size string 5Gi no no
grafana_datasources Grafana datasources for datasource provisioning list(object) [] no yes
grafana_datasources[*].basic_auth_enabled Toggle Ingress basic auth bool no yes
grafana_datasources[*].basic_auth_pass Ingress basic auth password string no yes
grafana_datasources[*].basic_auth_user Ingress basic auth user string no yes
grafana_datasources[*].name Name of the datasource string no yes
grafana_datasources[*].type Type of the datasource string no yes
grafana_datasources[*].url URL of the datasource string no yes
ingress_cert_issuer Ingress TLS certificate issuer string letsencrypt no no
ingress_class Ingress Class definition string nginx no no
loki Loki parameters object {} no no
loki.app_version Loki server version string 3.2.0 no no
loki.backend_replicas Number of Loki backend replicas (simple scalable mode only) number 1 no no
loki.bind_memberlist_endpoint Toggle explicit bind of POD IP to Loki-Memberlist Kubernetes service. Required only for deployment on a K8s node with an address in the non-standard IP range (outside RFC 1918 or RFC 6598) bool false no no
loki.bucket_region Region of the object storage provider where buckets for Loki are located (simple scalable mode only) string no no
loki.chart_version Helm chart version (compatible chart version is 6.0.0 and newer) string 6.18.0 no no
loki.chunks_bucket_name Bucket name for Loki chunk storage (simple scalable mode only) string no no
loki.custom_values Custom values to pass to the Loki Helm chart as a block any {} no no
loki.enable_simple_scalable_mode Specify whether Loki is in simple scalable deployment mode bool false no no
loki.enabled Toggle Loki installation bool true no no
loki.eventrouter_app_version Eventrouter app version string v1.7 no no
loki.eventrouter_chart_version Eventrouter Helm chart version (compatible chart version is 3.0.0 and newer) string 3.2.14 no no
loki.eventrouter_enabled Toggle Eventrouter installation bool true no no
loki.eventrouter_node_selector Node selector to place Eventrouter pods in map(any) {} no no
loki.ingress_auth_enabled Toggle Ingress basic auth (effective if ingress_enabled is true) bool false no no
loki.ingress_enabled Toggle Ingress bool false no no
loki.ingress_host Hostname to use with Ingress (effective if ingress_enabled is true) string no no
loki.node_selector Node selector to place Loki pods in map(any) {} no no
loki.object_storage_aws_s3 AWS S3 object storage configuration enabling Loki bucket access (simple scalable mode only) object no no
loki.object_storage_aws_s3.aws_iam_role_arn ARN of the AWS IAM role assumed by Loki for accessing S3 (IRSA) string no no
loki.object_storage_azure Azure Blob Storage configuration enabling Loki bucket access (simple scalable mode only) object no no
loki.object_storage_azure.client_id Azure user-assigned managed identity application ID string no no
loki.object_storage_azure.storage_account_name Azure storage account name string no no
loki.object_storage_do_spaces DigitalOcean Spaces object storage configuration enabling Loki bucket access (simple scalable mode only) object no no
loki.object_storage_do_spaces.access_key_id DigitalOcean Spaces access key ID string no no
loki.object_storage_do_spaces.k8s_secret_name Name of the Kubernetes Secret that will store DigitalOcean Spaces credentials string loki-do-spaces-creds no no
loki.object_storage_do_spaces.secret_access_key DigitalOcean Spaces secret access key string no no
loki.object_storage_gcs Google Cloud Storage configuration enabling Loki bucket access (simple scalable mode only) object no no
loki.object_storage_gcs.service_account_email Google Cloud service account email string no no
loki.promtail_app_version Promtail app version string 3.0.0 no no
loki.promtail_chart_version Promtail Helm chart version (compatible chart version is 6.0.0 and newer) string 6.16.6 no no
loki.promtail_enabled Toggle Promtail installation bool true no no
loki.promtail_node_selector Node selector to place Promtail pods in map(any) {} no no
loki.read_replicas Number of Loki read replicas (simple scalable mode only) number 1 no no
loki.retention_period Data retention period string 93d no no
loki.ruler_bucket_name Bucket name for Loki ruler storage (simple scalable mode only) string no no
loki.sa_name Kubernetes ServiceAccount name used by all Loki components string loki no no
loki.storage_class Storage class name string no no
loki.volume_size Volume data size string 100Gi no no
loki.write_replicas Number of Loki write replicas (simple scalable mode only) number 1 no no
name_prefix Name prefix for resources creation string no no
namespace Monitoring stack namespace string monitoring no no
prometheus Prometheus parameters object {} no no
prometheus.alertmanager_enabled Toggle Prometheus alertmanager installation bool false no no
prometheus.app_version Prometheus server version string v2.39.1 no no
prometheus.chart_version Helm chart version (compatible chart version must be from 18.0.0 and up to 19.0.0) string 18.4.0 no no
prometheus.custom_values Custom values to pass to the Prometheus Helm chart as a block any {} no no
prometheus.enabled Toggle Prometheus installation bool true no no
prometheus.ingress_auth_enabled Toggle Nginx basic auth (effective if ingress_enabled is true) bool false no no
prometheus.ingress_enabled Toggle Ingress bool false no no
prometheus.ingress_host Hostname to use with Ingress (effective if ingress_enabled is true) string no no
prometheus.node_exporter_enabled Toggle Node exporter installation bool true no no
prometheus.node_selector Node selector to place Prometheus pods in map(any) {} no no
prometheus.remote_write_enabled Toggle the remote write receiver feature bool false no no
prometheus.retention_period Data retention period string 93d no no
prometheus.storage_class Storage class name string no no
prometheus.volume_size Volume data size string 100Gi no no
pushgateway Pushgateway parameters object {} no no
pushgateway.app_version Pushgateway server version string v1.4.3 no no
pushgateway.enabled Toggle Pushgateway installation bool false no no
pushgateway.ingress_auth_enabled Toggle Nginx basic auth (effective if ingress_enabled is true) bool false no no
pushgateway.ingress_enabled Toggle Ingress bool false no no
pushgateway.ingress_host Hostname to use with Ingress (effective if ingress_enabled is true) string no no
pushgateway.volume_size Volume data size string 2Gi no no
tempo Tempo parameters object {} no no
tempo.app_version Tempo components version string 2.6.0 no no
tempo.azure_remote_storage Azure blob storage for Tempo data object no no
tempo.azure_remote_storage.container_name Storage account container name string no no
tempo.azure_remote_storage.storage_account_key Storage account key string no no
tempo.azure_remote_storage.storage_account_name Storage account name string no no
tempo.chart_version Tempo distributed Helm chart version (compatible chart version is 1.0.0 and newer) string 1.20.0 no no
tempo.custom_values Custom values to pass to the Tempo Helm chart as a block any {} no no
tempo.enabled Toggle Tempo installation bool false no no
tempo.gateway Gateway configuration block object no no
tempo.gateway.enabled Toggle gateway for Tempo bool false no no
tempo.gateway.host Hostname to expose the gateway on string no no
tempo.gateway.ingress_auth_enabled Toggle Nginx basic auth for the gateway service bool false no no
tempo.ingress_auth_enabled Toggle Nginx basic auth (effective if ingress_enabled is true) bool false no no
tempo.ingress_enabled Toggle Ingress bool false no no
tempo.ingress_host Hostname to use with Ingress (effective if ingress_enabled is true) string no no
tempo.node_selector Node selector to place Tempo components in map(any) {} no no
tempo.span_end_time_shift Shifts the end time for the logs query, based on the span's end time string -1h no no
tempo.span_start_time_shift Shifts the start time for the logs query, based on the span's start time string 1h no no
tempo.traces_tags Define additional tags when provisioning traces to logs feature map(any) {} no no
tempo.traces_to_logs Toggle traces to logs feature bool false no no
Output Description Type Sensitive
basic_auth_credentials Ingress basic auth credentials computed yes
grafana_admin_credentials Contains admin user credentials for Grafana web UI map yes
grafana_instance_selector_label Selector label to match dashboard and data source CRs to Grafana CR computed no
ingress_hosts Ingress exposed hosts map no
Dependency Version Kind
terraform >= 1.3 CLI
hashicorp/helm ~> 3.1 provider
hashicorp/kubernetes ~> 3.0 provider
hashicorp/random ~> 3.3 provider
tf-k8s-crd ~> 3.0 module
tf-k8s-grafana ~> 2.0 module

These components are included as is under the terms of their corresponding licenses.

Component License
Prometheus Apache-2.0
Grafana Loki AGPL-3.0
Promtail agent Apache-2.0
Grafana Tempo AGPL-3.0
Kubernetes event exporter Apache-2.0
Kubernetes event exporter Helm chart Apache-2.0

References

Not sure where to start?
Let's find your perfect match.