Grafana Loki Distributed in Kubernetes

$750

Dependencies included: $50

$50

License

Helm-based setup of Grafana Loki with Terraform, flexible and highly configurable, using cloud object storage for long term storing logs. The resulting Loki deployment is ready to be used as logs write target for Promtail or Grafana Agent.

For cloud-specific implementations, the module does not support passing credentials to object storage directly - instead, it uses RBAC of a corresponding cloud provider. This somewhat limits the module usage: GCS can only be configured with GKE, Azure Blob Storage with AKS - S3, however, remains an exception to support non-AWS implementations of S3 protocol, like DigitalOcean Spaces.

Supported features:

cloud-specific object storage configuration with RBAC (integrations for AWS, GCP, Azure)
out-of-the-box ready to integrate with Grafana, Grafana Agent or Promtail
per-component scaling
limits control

Loki stores incoming data and WAL in short-term storage (Kubernetes volumes), then transfers them to object storage. Should volume data be destroyed, it leads to losing metrics for only this period.

Memory limit for all Loki components is set as a global parameter, then proportionally distributed between the enabled components. Make sure to increase this value (if needed). The default setup restricts the stack to 4GB of RAM which should be taken into consideration for a minimal deployment.

NOTE: Certain parameters require pre-created storage along with corresponding IAM role and policies (see Variables for reference). Make sure to refer them using Terraform code to set a direct dependency on it in your code or use the depends_on argument to prevent errors at the planning stage.

Note: If ingressClassName for external endpoint is set to nginx Ingress annotations for Basic Auth setup are configured automatically if it is enabled.

Once you have a Corewide Solutions Portal account, this one-time action will use your browser session to retrieve credentials:

 shellterraform login solutions.corewide.com

Provision instructions

Initialize mandatory providers:

Copy and paste into your Terraform configuration and insert the variables:

 hclmodule "tf_k8s_loki" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-loki/helm"
  version = "~> 1.1.0"

  # specify module inputs here or try one of the examples below
  ...
}

Initialize the setup:

 shellterraform init

Define update strategy

Corewide DevOps team strictly follows Semantic Versioning Specification to provide our clients with products that have predictable upgrades between versions. We recommend pinning patch versions of our modules using pessimistic constraint operator (~>) to prevent breaking changes during upgrades.

To get new features during the upgrades (without breaking compatibility), use ~> 1.1 and run terraform init -upgrade

For the safest setup, use strict pinning with version = "1.1.0"

v1.1.0 released 3 days, 5 hours ago

New version approx. every 26 weeks

Changelog

Minimal setup with mandatory parameters:

 hclmodule "loki" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-loki/helm"
  version = "~> 1.1"

  name_prefix = "biz"
  namespace   = kubernetes_namespace_v1.monitoring.metadata[0].name

  object_storage = {
    type             = "s3"
    s3_region        = data.aws_region.current.name
    data_bucket_name = module.s3.bucket.bucket
  }

  rbac = {
    cloud_storage    = "aws"
    aws_iam_role_arn = module.loki_iam_role.role.arn
  }
}

Minimal setup with mandatory parameters, custom Helm values, and external endpoint and multi-tenant mode enabled:

 hclmodule "loki" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-loki/helm"
  version = "~> 1.1"

  name_prefix  = "biz"
  namespace    = kubernetes_namespace_v1.monitoring.metadata[0].name
  auth_enabled = true

  custom_values = {
    loki = {
      image = {
        pullPolicy = "Always"
      }
    }
  }

  object_storage = {
    type             = "s3"
    s3_region        = data.aws_region.current.name
    data_bucket_name = module.s3.bucket.bucket
  }

  external_endpoint = {
    domain_name         = "loki.example.com"
    basic_auth_enabled  = true
    basic_auth_password = "XXXX-XXXX-XXXX"
  }

  rbac = {
    cloud_storage    = "aws"
    aws_iam_role_arn = module.loki_iam_role.role.arn
  }
}

EKS cluster and OIDC provider configured with tf-aws-k8s-eks module, AWS RBAC configured via tf-aws-iam-role module, storage deployed with tf-aws-s3-bucket module and monitoring namespace created:

 hcllocals {
  loki_service_account_name = "loki"
  loki_namespace            = "monitoring"
}

data "aws_region" "current" {}

data "aws_iam_policy_document" "loki" {
  statement {
    actions = [
      "s3:ListBucket",
      "s3:PutObject",
      "s3:GetObject",
      "s3:DeleteObject",
    ]

    resources = [
      module.s3.bucket.arn,
      "${module.s3.bucket.arn}/*",
    ]
  }
}

module "eks" {
  source  = "solutions.corewide.com/aws/tf-aws-k8s-eks/aws"
  version = "~> 5.2"
  # ...
}

module "s3" {
  source  = "solutions.corewide.com/aws/tf-aws-s3-bucket/aws"
  version = "~> 1.0"
  # ...
}

resource "kubernetes_namespace_v1" "monitoring" {
  metadata {
    name = local.loki_namespace
  }
}

module "loki_iam_role" {
  source  = "solutions.corewide.com/aws/tf-aws-iam-role/aws"
  version = "~> 1.1"

  name                     = "foo"
  assume_with_web_identity = true
  custom_assume_policy     = data.aws_iam_policy_document.assume_role_loki.json

  policy_documents = {
    "bar" = data.aws_iam_policy_document.loki.json
  }

  service_accounts = [
    {
      name      = local.loki_service_account_name
      namespace = local.loki_namespace
    },
  ]
}

Minimal setup with mandatory parameters:

 hclmodule "loki" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-loki/helm"
  version = "~> 1.1"

  name_prefix = "foo"
  namespace   = kubernetes_namespace_v1.monitoring.metadata[0].name

  object_storage = {
    type             = "gcs"
    data_bucket_name = module.loki_storage.bucket.name
  }

  rbac = {
    cloud_storage             = "gcs"
    gcp_service_account_email = google_service_account.loki.email
  }
}

Minimal setup with mandatory parameters, custom Helm values, and external endpoint and multi-tenant mode enabled:

 hclmodule "loki" {
  source  = "solutions.corewide.com/kubernetes/tf-k8s-loki/helm"
  version = "~> 1.1"

  name_prefix  = "foo"
  namespace    = kubernetes_namespace_v1.monitoring.metadata[0].name
  auth_enabled = true

  custom_values = {
    loki = {
      image = {
        pullPolicy = "Always"
      }
    }
  }

  object_storage = {
    type             = "gcs"
    data_bucket_name = google_storage_bucket.loki.name
  }

  rbac = {
    cloud_storage             = "gcs"
    gcp_service_account_email = google_service_account.loki.email
  }

  external_endpoint = {
    domain_name         = "loki.example.com"
    basic_auth_enabled  = true
    basic_auth_password = "XXXX-XXXX-XXXX"
  }
}

RBAC configuration, Google Cloud Storage deployed with tf-gcp-storage-bucket module and monitoring namespace created:

 hcldata "google_project" "current" {}

resource "google_service_account" "loki" {
  account_id = "loki"
  # ...
}

resource "google_project_iam_member" "loki" {
  project = data.google_project.current.project_id
  role    = "roles/storage.objectUser"
  member  = "serviceAccount:${google_service_account.loki.email}"
}

resource "kubernetes_namespace_v1" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

module "loki_storage" {
  source  = "solutions.corewide.com/google-cloud/tf-gcp-storage-bucket/google"
  version = "~> 1.0"

  name   = "foo"
  region = "us-central1"
}

Variable	Description	Type	Default	Required	Sensitive
`app_version`	Loki app version to deploy (image tag). If not set, the chart uses its corresponding default	`string`		yes	no
`name_prefix`	Name prefix for module resources	`string`		yes	no
`namespace`	Namespace to deploy Loki into	`string`		yes	no
`object_storage`	Object storage parameters block	`object`		yes	no
`rbac`	Cloud-specific RBAC settings to access object storage	`object`		yes	no
`auth_enabled`	Toggle the multi-tenant mode	`bool`	`false`	no	no
`chart_version`	Loki Helm chart version to deploy	`string`	`0.80.5`	no	no
`components_scaling`	Number of replicas of Loki components. `0` disables the component (if possible)	`object`	`{}`	no	no
`components_scaling.compactor`	Number of `compactor` replicas	`number`	`0`	no	no
`components_scaling.distributor`	Number of `distributor` replicas	`number`	`1`	no	no
`components_scaling.gateway`	Number of `gateway` replicas	`number`	`1`	no	no
`components_scaling.index_gateway`	Number of `index gateway` replicas	`number`	`0`	no	no
`components_scaling.ingester`	Number of `ingester` replicas (per zone)	`number`	`1`	no	no
`components_scaling.querier`	Number of `querier` replicas	`number`	`1`	no	no
`components_scaling.query_frontend`	Number of `query_frontend` replicas	`number`	`1`	no	no
`components_scaling.table_manager`	Number of `table manager` replicas	`number`	`0`	no	no
`custom_values`	Custom Helm chart values in the map format	`map(any)`	`{}`	no	no
`external_endpoint`	Settings to enable external access to Loki	`object`	`{}`	no	yes
`external_endpoint.basic_auth_enabled`	Whether to protect external endpoint with basic HTTP authentication. Only works with Ingress Nginx controller	`bool`	`false`	no	yes
`external_endpoint.basic_auth_password`	Password for basic HTTP authentication. Generated randomly if not specified	`string`		no	yes
`external_endpoint.basic_auth_username`	Username for basic HTTP authentication	`string`	`loki`	no	yes
`external_endpoint.cert_issuer_name`	Name of CertManager `ClusterIssuer` resource to generate certificates with	`string`	`letsencrypt`	no	yes
`external_endpoint.domain_name`	Domain name to make Loki externally accessible at	`string`		no	yes
`external_endpoint.ingress_annotations`	Gateway `Ingress` annotations	`map(any)`	`{}`	no	yes
`external_endpoint.ingress_class`	Gateway `Ingress` class name	`string`	`nginx`	no	yes
`grafana`	Grafana integration settings. CRDs from Grafana Operator must be preinstalled	`object`	`{}`	no	no
`grafana.create_datasource`	Whether to create a Grafana Operator compatible datasource to access Loki. Grafana Operator must be preinstalled for CRDs to exist in the cluster	`bool`	`true`	no	no
`grafana.datasource_name`	Name for Loki datasource visible in Grafana UI	`string`		no	no
`grafana.instance_selector`	Selector to use for Grafana CRs to be detected by Grafana Operator	`map(any)`	`{'instance_name': 'grafana'}`	no	no
`grafana.polling_interval_seconds`	Default time interval for Loki datasource, seconds	`number`	`30`	no	no
`in_cluster_storage`	Parameters of in-cluster persistence (used for storing data before sending it to object storage)	`object`	`{}`	no	no
`in_cluster_storage.compactor_persistence`	Whether to enable PVC creation for `compactor`	`bool`	`false`	no	no
`in_cluster_storage.compactor_volume_size_gb`	Size of Kubernetes volume to store local data of `compactor`, GB	`number`	`10`	no	no
`in_cluster_storage.compactor_wal_volume_size_gb`	Size of Kubernetes volume to store WAL data of `querier`, GB	`number`	`150`	no	no
`in_cluster_storage.index_gateway_persistence`	Whether to enable PVC creation for the `index_gateway`	`bool`	`false`	no	no
`in_cluster_storage.index_gateway_volume_size_gb`	Size of Kubernetes volume to store local data of `index_gateway`, GB	`number`	`10`	no	no
`in_cluster_storage.ingester_volume_size_gb`	Size of Kubernetes volume to store local data of `ingester`, GB	`number`	`10`	no	no
`in_cluster_storage.ingester_wal_volume_size_gb`	Size of Kubernetes volume to store WAL data of `ingester`, GB	`number`	`150`	no	no
`in_cluster_storage.querier_persistence`	Whether to enable PVC creation for the `querier` component	`bool`	`false`	no	no
`in_cluster_storage.querier_volume_size_gb`	Size of Kubernetes volume to store local data of `querier`, GB	`number`	`10`	no	no
`in_cluster_storage.storage_class_name`	Name of the Kubernetes storage class to use for short-term chunks storage. If unset, uses the default storage class of the cluster	`string`		no	no
`limits`	Limits configuration block	`object`	`{}`	no	no
`limits.memory_global_gb`	Global RAM limit for all Loki components together in Gi	`number`	`4`	no	no
`log_level`	Loki components logging level. Possible values: `info`, `error`, `debug`	`string`	`info`	no	no
`node_selector`	Node selector for Loki components	`map(string)`	`{}`	no	no
`object_storage.azure_account_name`	Azure account name to use for object storage connections. Mandatory if `type` is set to `azure`	`string`		no	no
`object_storage.azure_endpoint_suffix`	Endpoint suffix to use for Azure Blob Storage connections. Mandatory if `type` is set to `azure`	`string`		no	no
`object_storage.data_bucket_name`	Name of the bucket to use for Loki data	`string`		yes	no
`object_storage.s3_advanced_settings`	Extra parameters for S3 connection: encryption, explicit credentials, etc. (see official documentation for full list). Only works with `type` set to `s3`	`map(any)`	`{}`	no	no
`object_storage.s3_endpoint`	S3 endpoint to connect to buckets data. Mandatory if `type` is set to `s3`	`string`		no	no
`object_storage.s3_region`	Region to use for S3 requests. Mandatory if `type` is set to `s3`	`string`		no	no
`object_storage.type`	Type of object storage to use. Possible values: `s3`, `gcs`, `azure`	`string`		yes	no
`rbac.aws_iam_role_arn`	ARN of AWS IAM role. This is the role that Service Account will be bound to have privileges to access S3 buckets. Mandatory if `cloud_storage` is set to `aws`	`string`		no	no
`rbac.azure_client_id`	Azure client ID with privileges to access Azure Blob Storage. Mandatory if `cloud_storage` is set to `azure`	`string`		no	no
`rbac.cloud_storage`	Cloud storage to adapt configuration to. Possible values: `aws`, `gcs`, `azure`	`string`		yes	no
`rbac.gcp_service_account_email`	GCP service account email with privileges to access GCS. Mandatory if `cloud_storage` is set to `gcp`	`string`		no	no
`rbac.service_account_name`	Name to use for Kubernetes service account created by the module	`string`	`loki`	no	no
`tolerations`	Kubernetes tolerations for Loki components	`list(object)`	`[]`	no	no
`tolerations[*].effect`	Indicates the taint effect to match	`string`		no	no
`tolerations[*].key`	The taint key that the toleration applies to	`string`		no	no
`tolerations[*].operator`	The operator to check the taint value	`string`		no	no
`tolerations[*].value`	The taint value that the toleration applies to	`string`		no	no

Output	Description	Type	Sensitive
`connection_params_external`	Loki connection parameters (external)	`computed`	yes
`connection_params_internal`	Loki connection parameters (in-cluster)	`map`	no

Dependency	Version	Kind
`terraform`	`>= 1.3`	CLI
`hashicorp/helm`	`~> 2.17`	provider
`hashicorp/kubernetes`	`~> 2.35`	provider
`hashicorp/random`	`~> 3.5`	provider
`tf-k8s-crd`	`~> 2.0`	module