Managed Kafka cluster in GCP

This module manages KMS-encrypted Kafka cluster and its topics in GCP Managed Service for Apache Kafka.

NOTE: This module is meant to be used with an already created VPC.

Once you have a Corewide Solutions Portal account, this one-time action will use your browser session to retrieve credentials:

 shellterraform login solutions.corewide.com

Provision instructions

Initialize mandatory providers:

Copy and paste into your Terraform configuration and insert the variables:

 hclmodule "tf_gcp_kafka" {
  source  = "solutions.corewide.com/google-cloud/tf-gcp-kafka/google"
  version = "~> 2.0.0"

  # specify module inputs here or try one of the examples below
  ...
}

Initialize the setup:

 shellterraform init

Define update strategy

Corewide DevOps team strictly follows Semantic Versioning Specification to provide our clients with products that have predictable upgrades between versions. We recommend pinning patch versions of our modules using pessimistic constraint operator (~>) to prevent breaking changes during upgrades.

To get new features during the upgrades (without breaking compatibility), use ~> 2.0 and run terraform init -upgrade

For the safest setup, use strict pinning with version = "2.0.0"

Module 10% off	~~$600~~ $540
TOTAL	$540

BUY

270

Corewide Proprietary License

v2.0.0 released 8 months, 3 weeks ago

New version approx. every 20 weeks

Upgrade Notes

Changelog

This module manages KMS-encrypted Kafka cluster and its topics in GCP Managed Service for Apache Kafka.

NOTE: This module is meant to be used with an already created VPC.

From `v1.x` to `v2.x`

Module from v2.0 has changed GCP providers version which isn't compatible with an old version. After the module version is upgraded, re-init module to upgrade GCP providers version.
Upgrade GCP provider version on project level to ~> 7.0:

 hclterraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 7.0"
    }

    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 7.0"
    }
  }
}

Upgrade project dependencies:

 bashterraform init --upgrade

Minimal setup with mandatory parameters only:

 hclmodule "vpc" {
  source  = "solutions.corewide.com/google-cloud/tf-gcp-vpc/google"
  version = "~> 2.0"

  name_prefix = "foo"

  subnets = {
    data = {
      cidr_block = "10.0.0.0/8"
      region     = "us-west1"
    }
  }
}

module "kafka" {
  source  = "solutions.corewide.com/google-cloud/tf-gcp-kafka/google"
  version = "~> 2.0"

  name_prefix = "bar"
  region      = "us-west1"
  subnet_id   = module.vpc.subnets["data"].id
}

Deploy Kafka cluster with 4 CPU threads, 4 GiB memory, single topic with custom configuration, and custom labels:

 hclmodule "vpc" {
  source  = "solutions.corewide.com/google-cloud/tf-gcp-vpc/google"
  version = "~> 2.0"

  name_prefix = "foo"

  subnets = {
    data = {
      cidr_block = "10.0.0.0/8"
      region     = "us-west1"
    }
  }
}

module "kafka" {
  source  = "solutions.corewide.com/google-cloud/tf-gcp-kafka/google"
  version = "~> 2.0"

  name_prefix     = "bar"
  region          = "us-west1"
  subnet_id       = module.vpc.subnets["data"].id
  vcpu_count      = 4
  memory_size_gib = 4
  rebalance_mode  = "NO_REBALANCE"

  topics = [
    {
      name               = "biz"
      partition_count    = 3
      replication_factor = 3

      config = {
        "cleanup.policy" = "compact"
      }
    },
  ]

  labels = {
    env = "dev"
  }
}

Variable	Description	Type	Default	Required	Sensitive
`name_prefix`	Name for Kafka cluster and name prefix for related resources	`string`		yes	no
`region`	GCP region to create resources in	`string`		yes	no
`subnet_id`	The name or `self_link` of the Google Compute Engine network subnet to create the cluster endpoint in	`string`		yes	no
`labels`	Map of labels to be assigned to Kafka cluster	`map(string)`	`{}`	no	no
`memory_size_gib`	The memory to provision for the cluster (GiB). The value must be between 1 GiB and 8 GiB per one vCPU	`number`	`3`	no	no
`rebalance_mode`	The rebalance behavior for the cluster. Possible values are: `NO_REBALANCE`, `AUTO_REBALANCE_ON_SCALE_UP`	`string`	`NO_REBALANCE`	no	no
`topics`	List of Kafka topics to create inside the cluster	`list(object)`	`[]`	no	no
`topics[*].config`	Map containing Kafka topic configuration parameters	`map(any)`	`{}`	no	no
`topics[*].name`	Name of Kafka topic	`string`		yes	no
`topics[*].partition_count`	The number of partitions for the topic	`number`	`3`	no	no
`topics[*].replication_factor`	The number of nodes that topics are replicated across. `3` is recommended for high availability	`number`	`3`	no	no
`vcpu_count`	Number of vCPU threads to provision for Kafka. Minimal value is `3`	`number`	`3`	no	no

Output	Description	Type	Sensitive
`bootstrap_host`	Kafka bootstrap host	`computed`	no
`bootstrap_port`	Kafka bootstrap port	`computed`	no
`cluster`	Kafka cluster resource attributes	`resource`	yes
`topics`	Kafka Topic resources' attributes	`computed`	no