#terraform (2024-06)

terraform Discussions related to Terraform or Terraform Modules

Archive: https://archive.sweetops.com/terraform/

2024-06-04

Taylor Turner avatar
Taylor Turner

What are some of the newer age IaC tools that you’ve shown before on the podcast? There was one that was dependency aware and had a visual drag-n-drop UI.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Referring to Cloud Posse’s office hours?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hey everyone wave I’m George the founder of Stakpak, we’re a VC backed startup working on a specialized copilot for Terraform that attributes recommendations back to real people’s work.

I’d love to learn more about your workflows, and get your feedback, please DM me if you’re interested in helping us out.

https://www.youtube.com/watch?v=sNA1wC02pa8

Taylor Turner avatar
Taylor Turner

Hey Erik, good to see you are still going at it!

I’ve been bouncing around DevOps jobs the last 6 months so I haven’t been attending office hours but I plan on changing that starting this week.

That looks like what I was thinking. Thanks for the link!

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@george.m.sedky

Taylor Turner avatar
Taylor Turner

Some people on my team haven’t used Terraform so I was doing a little demo today. We started talking about pain points, one of them being managing really large Terraform deployments for all of prod.

The example was “I want to deploy an EC2”, the tool should ideally know that I need

• VPC

• Subnets

• IGW/NAT

• AMI

• EBS w/ KMS key for encryption

• And so on… Terraform is great but it’s pretty “dumb” compared to where we’ll be in a few years with AI and tools like Stakpak. It may not be a popular idea since managing Terraform, and bringing that knowledge of how to do it well, counts for a big chunk of what many DevOps Engineers are being paid to do.

The tool I’m thinking of had a different name and it would’ve been something you shared on at least a year ago. If I can dig it up I’ll share it. Same idea as Stakpak though but I don’t think it was using AI.

george.m.sedky avatar
george.m.sedky

Hey @Taylor Turner this is exactly what we’re working on at Stakpak I’d love to arrange a call sometime to learn about your use case we’re doing a lot of product discovery interviews this week

sheldonh avatar
sheldonh

remind me… if i’m building a resuable module… does it declare an empty provider or not, the docs got me confused. I see no examples of you doing this outside the examples/ directory in cloudposse modules.

Assuming I should gut any providers.tf in the resuable module and only declare required provider?

So would leave this out?

provider "azurerm" {
  features {}
}

or any other provide mapping to the username/password etc?

RB avatar

It’s not good practices to bake the provider instantiation in the consumable module itself

1
this1
Nate McCurdy avatar
Nate McCurdy

https://developer.hashicorp.com/terraform/language/modules/develop/providers#provider-version-constraints-in-modules

It should declare a required provider with a minimum version. But not a provider block.

5
RB avatar

The provider should be instantiated in the root dir where the reusable module is called

sheldonh avatar
sheldonh

cool, that’s what I thought but couldn’t find confirmation. thank you for this quick help. cheers!

1

2024-06-05

Release notes from terraform avatar
Release notes from terraform
09:43:31 AM

v1.8.5 1.8.5 (June 5, 2024) BUG FIXES:

terraform test: Remove duplicate warning diagnostic when providing values for unknown variables in run blocks. (#35172)

Remove invalid warning during cleanup phase by MicahKimel · Pull Request #35172 · hashicorp/terraformattachment image

Fixes #35061 Target Release

1.8.x Draft CHANGELOG entry

NEW FEATURESUPGRADE NOTESENHANCEMENTSBUG FIXESEXPERIMENTS

This update takes out display warnings during cleanup phase and by…

Zing avatar

what’s the best way to do the following?

• convert from CDKTF (failed experiment) cloudposse eks module to vanilla terraform cloudposse eks module

• convert from terraform-aws-modules/eks to cloudposse eks module? (some of our old clusters) we’re trying to standardize and need to perform module migrations for both of the above. I think the first one might be simpler in terms of state file massaging, but not certain

is terraform state mv the right call? or something fancy / clever with the import blocks?

Destroying / recreating the sensitive resources (eks cluster) is not an option

theherk avatar
theherk

Some combination of import blocks and moved blocks is probably your best bet. The moved block is one of the most handy tools added, in my view.

Refactoring | Terraform | HashiCorp Developerattachment image

How to make backward-compatible changes to modules already in use.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, just to clarify - you cannot destroy/rebuild the cluster?

Zing avatar

yeah cannot destroy rebuild :(

Zing avatar

I haven’t messed with moved blocks yet

Zing avatar

just brainstorming best possible options atm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I suspect it will be very difficult to use any off the shelf terraform module, because the shape of infrastructure will always be different.

2024-06-06

Release notes from terraform avatar
Release notes from terraform
05:43:30 PM

v1.10.0-alpha20240606 1.10.0-alpha20240606 (June 6, 2024) EXPERIMENTS: Experiments are only enabled in alpha releases of Terraform CLI. The following features are not yet available in stable releases.

ephemeral_values: This language experiment introduces a new special kind of value which Terraform allows to change between the plan phase and the apply phase, and between plan/apply rounds….

Release v1.10.0-alpha20240606 · hashicorp/terraformattachment image

1.10.0-alpha20240606 (June 6, 2024) EXPERIMENTS: Experiments are only enabled in alpha releases of Terraform CLI. The following features are not yet available in stable releases.

ephemeral_values:…

Terraform Settings - Configuration Language | Terraform | HashiCorp Developerattachment image

The terraform block allows you to configure Terraform behavior, including the Terraform version, backend, integration with HCP Terraform, and required providers.

2024-06-09

2024-06-10

Zing avatar

have people been using the “new” security group resources?

aws_vpc_security_group_ingress/egress_rule

historically, the group rules have been a PITA with destroy recreates / rule dedupe logic… wondering if folks think it’s worth it to move to the new resources

1
loren avatar

Personally I prefer my security group rules inline for exclusive management by the security group resource, so no

1
Dale avatar

I managed to fix the ‘old’ SG resource we have by switching around the order of the rules so the TF state (which was ordered arbitrarily) matched the AWS console, but it did require me to first delete the individual rules and then reapply them which made me unhappy for the 2 seconds they weren’t applied

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We updated our security group module to use these rules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
The Difficulty of Managing AWS Security Groups with Terraform – Cloud Posse

There are trade-offs when updating AWS Security Groups with Terraform. This article discusses them and the various options for managing them with Cloud Posse’s Terraform security group module.

3
RB avatar

Hmm… It would be nice if they had a new security group v2 resource that enforced inline rules (no clickops rules) and tagging individual rules so you get the best of both worlds.

1
Zing avatar


We updated our security group module to use these rules
am I doing something silly? I’m looking at https://github.com/cloudposse/terraform-aws-security-group/security

and it doesn’t seem to be using the new “ingress/egress” rule resources

Zing avatar


Hmm… It would be nice if they had a new security group v2 resource that enforced inline rules (no clickops rules) and tagging individual rules so you get the best of both worlds.
yeah, that’s been a desirable feature for years haha. it’s one of the many reasons we were considering onboarding a product like spacelift for drift detection

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

On that note, we provide a GitHub action for drift detection if using Atmos. https://atmos.tools/integrations/github-actions/atmos-terraform-drift-detection/

Atmos Terraform Drift Detection | atmos

The Cloud Posse GitHub Action for “Atmos Terraform Drift Detection” and “Atmos Terraform Drift Remediation” define a scalable pattern for detecting and remediating Terraform drift from within GitHub using workflows and Issues. “Atmos Terraform Drift Detection” will determine drifted Terraform state by running Atmos Terraform Plan and creating GitHub Issues for any drifted component and stack. Furthermore, “Atmos Terraform Drift Remediation” will run Atmos Terraform Apply for any open Issue if called and close the given Issue. With these two actions, we can fully support drift detection for Terraform directly within the GitHub UI.

RB avatar

There’s no way to detect drift in security group rules if they are clickopsed which is the issue with using the individual rule resources vs inline resources

1
loren avatar

At least, no way using just terraform. You’d have to query the APIs directly, and compare against the config.

1
Zing avatar

thanks @RB,

security groups have been such a headache for us. I’m debating between creating the security groups outside of the eks module rather than using the in-module one atm

RB avatar

That’s certainly an option. Most of the modules, if not all, provide a way to pass in your own security group

Zing avatar

what are your personal preferences?

Zing avatar

im mainly concerned with scenarios with rule modifications causing downtime

RB avatar

My personal preference is the same as lorens

Zing avatar

yeah I generally like that approach as well, but it seems all the folks out there these days advise against it :p

tbh, I don’t even know why… inline rules just seems… better?

loren avatar

it changes the responsibility a bit. folks sometimes want different processes (or different teams) to manage the creation of the security group, vs the rules in the security group. inline rules can also make some use cases more difficult, like rules that cross-reference other sgs

loren avatar

but if you can change your operating model to one that supports inline rules, then i feel like it works a lot better. and i strongly prefer having a resource that manages the “container” (the security group) and its “items” (the rules) together, so that terraform can alert on drift in the items in the container (e.g. unmanaged rules added to the sg)

loren avatar

i use the container/items terminology for a number of things with similar concerns, like iam roles (container) and policy attachments (items), or route tables (container) and routes (items)

loren avatar

unfortunately terraform providers have been moving the other direction, and instead are aligning a resource purely to a single api action, so we see less and less inline support for “items” and their exclusive management, which ultimately makes drift detection effectively impossible within pure terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

which ultimately makes drift detection effectively impossible within pure terraform I think “drift” is not an entirely fair characterization. You can certainly catch the drift of what is managed in Terraform. You cannot catch things managed outside of that Terraform. That may or may not be drift, if other systems.

I actually am the outlier. I like the current trajectory, enabling a lose coupling between root modules. E.g. one root module provisions a cluster and security group. Another service managed by different root module, that perhaps provisions node pools that work with the cluster, can add then modify security groups as needed that it needs to the cluster’s security group. It’s similar to a hub/spoke. As infrastructure state gets decomposed into it’s various components, this is a nice to have.

loren avatar

I acknowledge that operating model, and it has its use cases, I just don’t prefer it for everything.

1
loren avatar


You can certainly catch the drift of what is managed in Terraform. You cannot catch things managed outside of that Terraform.
You can’t even be aware of drift outside of Terraform, without exclusive management features. That’s the whole point. That “in Terraform” is a massive caveat. It leaves a gaping hole, which gets nicely plugged by exclusive management features at the container level. All I’m arguing for is that it is a valid model with desirable characteristics, and it would be nice if terraform providers put effort into meeting the use case. It is also valid and desirable to support the pure “attachment” model.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I look at that more as IaC coverage than drift.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

E.g. if you use a EKS cluster, and any operators, there will likely be TONS of resources provisioned not managed by Terraform.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That’s not drift. It’s just managed by something else.

loren avatar

like i said, there is a place for both

this1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Security Groups might be a special case though. From a security perspective, I could appreciate wanting tighter controls.

1
loren avatar

once you start thinking about it, there are lots of resources where it’s rather nice to manage everything from the container level, rather than as separate resources. iam roles and users and groups, certainly. s3 buckets used to work this way, which was nice from the perspective of managing the ordering of actions and conflicting api calls. it certainly does put more work on the code for the resource itself, which is of course why the providers don’t want to do it

Juan Pablo Lorier avatar
Juan Pablo Lorier

Hi, I’m trying to understand why the ecs cluster module is trying to recreate the policy attachments every time I add more than one module instance via a for_each. The plan shows the arn will change, but it’s a AWS managed policy, so it won’t change:

update policy_arn : “arnawsiam:awspolicy/AmazonSSMManagedInstanceCore”

change to Known after apply Forces replacement

the resource address is:

module.ecs_clusters[“xxx”].module.ecs_cluster.aws_iam_role_policy_attachment.default[“AmazonSSMManagedInstanceCore”]

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Are you properly setting a unique attribute for each instance

Juan Pablo Lorier avatar
Juan Pablo Lorier

the module takes care of the ids, I only set the cluster id. The id is unique (the name is formed from the tenant,+environment+name)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think I’ll need to see more code

Juan Pablo Lorier avatar
Juan Pablo Lorier

This is the cluster related code

module “ecs_cluster” { source = “cloudposse/ecs-cluster/aws” version = “~>0.6”

enabled = var.enabled context = module.label.context

container_insights_enabled = var.container_insights_enabled logging = var.logging log_configuration = var.log_configuration capacity_providers_fargate = true tags = var.tags depends_on = [module.label] }

Juan Pablo Lorier avatar
Juan Pablo Lorier

most options are defaulted. The context includes tenant and environment.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Where’s the for_each?

Juan Pablo Lorier avatar
Juan Pablo Lorier

I’m managing a local module that uses the cloudposse modules

module "ecs_clusters" {
  for_each = { for v in var.ecs_clusters : "${v.tenant}-${v.environment}-${v.cluster_name}" => v }
  source   = "./modules/ecs"

  vpc_id              = module.vpc.vpc_id
  subnet_ids          = module.vpc.private_subnets
  public_subnet_ids   = module.vpc.public_subnets
  dns_domain          = each.value.dns_domain == null ? var.dns_domain : each.value.dns_domain
  zone_id             = each.value.dns_domain == null ? data.cloudflare_zone.cloudflare_zone[0].id : data.cloudflare_zone.cloudflare_zones[each.value.dns_domain].id
  # If a certificate is provided, pass that. If not, dns_domain determins if we use the root cert for the workspace or create a new one
  cloudwatch_alarm            = each.value.cloudwatch_alarm
  container_definitions       = var.container_definitions
  ecr_repos                   = var.ecr_repos
  enforced_dns_root           = each.value.enforced_dns_root
  task_execution_IAM_role_arn = aws_iam_role.roles["ecs_task_execution_role"].arn

  cluster_name                     = each.value.cluster_name
  enabled                          = each.value.enabled
  alb_enable_logs                  = each.value.alb_enable_logs
  alb_url                          = each.value.alb_url
  https_listener_arn               = each.value.https_listener_arn
  redis_cluster_enabled            = each.value.redis_cluster_enabled
  sdk_redis_enabled                = each.value.sdk_redis_enabled
  tenant                           = each.value.tenant
  namespace                        = each.value.tenant
  environment                      = each.value.environment
  container_insights_enabled       = each.value.container_insights_enabled
  logging                          = each.value.logging
  log_configuration                = each.value.log_configuration
  ecs_metric_topic_subscriptions   = each.value.ecs_metric_topic_subscriptions
  ecs_critical_topic_subscriptions = each.value.ecs_critical_topic_subscriptions
  ecs_services                     = [for v in var.ecs_services : v if v.ecs_cluster_name == each.key]
  tags                             = each.value.tags
  depends_on                       = [aws_ecr_repository.ecs_ecr_repos, aws_ecr_repository.ecr_repos, aws_iam_role_policy.role_policies]
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
(please use code fences)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, so what’s missing from this example is usng var.attributes to a unique instance

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

otherwise the resources will be duplicated

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

At least, that was the way we intended it to be done. We haven’t tried other ways.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, word of caution - you’re building a factory inside of a root module: your terraform state will be massive, slow and brittle.

Juan Pablo Lorier avatar
Juan Pablo Lorier

Thanks. As I said, I’m actually using a module for all ECS related stuff. It’s consumed in the root module by this ecs_cluster module. let me look at the attributes so I can understand what is missing and the impact of modifying any of that. Again, thanks a lot

2024-06-11

2024-06-12

Release notes from terraform avatar
Release notes from terraform
09:33:29 AM

v1.9.0-rc1 No content.

2024-06-13

Soren Jensen avatar
Soren Jensen

I need a bit of help to debug an issue. I have created a RDS Postgres cluster with resource "aws_rds_cluster" "postgres_cluster" and set manage_master_user_password = true I’m now trying to get the master password from Secrets Manager to bootstrap the database with the Postgresql provider.

data "aws_secretsmanager_secret" "postgres_password_secret" {
  arn        = aws_rds_cluster.postgres_cluster.master_user_secret[0].secret_arn
  depends_on = [aws_rds_cluster.postgres_cluster]
}

data "aws_secretsmanager_secret_version" "postgres_password_version" {
  secret_id  = data.aws_secretsmanager_secret.postgres_password_secret.id
  depends_on = [data.aws_secretsmanager_secret.postgres_password_secret]
}

# Parse the secret JSON string to extract the username and password
locals {
  db_credentials = jsondecode(data.aws_secretsmanager_secret_version.postgres_password_version.secret_string)
}

Unfortunately the first data aws_secretsmanager_secret isn’t working on our self-hosted github runners. It works locally on my laptop, as well as in the default GitHub runners. I have spent a significant amount of time trying to narrow down differences in versions, reading the debug outputs to see why it doesn’t work. I see both on the self-hosted runner as well as locally that terraform correctly finds the cluster and resolve aws_rds_cluster.postgres_cluster.master_user_secret[0].secret_arn to the correct arn. Still terraform is stuck:

data.aws_secretsmanager_secret.postgres_password_secret: Still reading... [10s elapsed]
data.aws_secretsmanager_secret.postgres_password_secret: Still reading... [20s elapsed]
data.aws_secretsmanager_secret.postgres_password_secret: Still reading... [30s elapsed] 

Any ideas?

Soren Jensen avatar
Soren Jensen

I have the following debug print in all environments and they are identical aws-cli/2.16.7 Python/3.11.8 Linux/5.15.0-1063-aws exe/x86_64.ubuntu.20 { “UserId”: “AROAYKRB4XIA3M6HSSAPV:GitHubActions”, “Account”: “00000000000”, “Arn”: “arnawssts:assumed-role/prod-github-action/GitHubActions” } Terraform v1.5.7 on linux_amd64

Python 3.11.9

Anchor avatar

maybe a silly question here: did you try to ssh into the self-hosted github runner and retrieve the secret via CLI? I just want to ensure that is not a permission issue

Soren Jensen avatar
Soren Jensen

Anchor, not a silly question at all. I did and I can access the secret that way.

1

2024-06-14

2024-06-17

Jackie Virgo avatar
Jackie Virgo

Has anyone used terrafomr-aws-s3-bucket module for creating bi-directional replication?

    keyboard_arrow_up