#terraform (2022-05)

terraform Discussions related to Terraform or Terraform Modules

Archive: https://archive.sweetops.com/terraform/

2022-05-02

Darrin F avatar
Darrin F

Do you use AWS Control Tower, and do you provision that with IaC or do you just ClickOps the initial setup? We’re starting fresh and I’m not sure what to call “Day 0” operations where it doesn’t make sense to automate…

RB avatar

We use IaC using terraform to create all of our aws accounts

Darrin F avatar
Darrin F

I agree for the account creation. But to create/enable ControlTower and OU’s, do you use terraform or just make a few clicks in the AWS console?

RB avatar

We don’t use control tower but we do configure OUs and accounts through our account terraform component

https://github.com/cloudposse/terraform-aws-components/tree/master/modules/account

Soren Jensen avatar
Soren Jensen

We use control tower and have set it up with a philips module. It’s working like a charm

Soren Jensen avatar
Soren Jensen
New – AWS Control Tower Account Factory for Terraform | Amazon Web Servicesattachment image

AWS Control Tower makes it easier to set up and manage a secure, multi-account AWS environment. AWS Control Tower uses AWS Organizations to create what is called a landing zone, bringing ongoing account management and governance based on our experience working with thousands of customers. If you use AWS CloudFormation to manage your infrastructure as […]

Manage AWS Accounts Using Control Tower Account Factory for Terraform | Terraform - HashiCorp Learnattachment image

Use the AWS Control Tower Account Factory for Terraform to create a pipeline for provisioning and customizing AWS accounts in Control Tower. Create a new account and learn more about AWS Control Tower governance.

aws-ia/terraform-aws-control_tower_account_factory

AWS Control Tower Account Factory

2022-05-03

idan levi avatar
idan levi

Hi all! im trying to create self managed node groups on EKS using Terraform eks module and terragrun. I want to add toleration ,taints and labels to each node group, so i tried to use bootstrap_extra_args = "--node-labels=[node.kubernetes.io/lifecycle=spot,node/role=os-client](http://node.kubernetes.io/lifecycle=spot,node/role=os-client)" and

bootstrap_extra_args = <<-EOT
      [settings.kubernetes.node-labels]
      ingress = "allowed"
      EOT 

but none of them create the node group with the labels/taint . someone know what is the right way to do it ? Thanks !!

Matt H. avatar
Matt H.

Hello, I’m using the cloudposse tgw module with terragrunt to create a cross-account transit gateway with attachments. I created a module-of-modules that basically mimics the multi-account example here https://github.com/cloudposse/terraform-aws-transit-gateway/tree/master/examples/multi-account, only I replaced the config with vars that I pass in from terragrunt. When it runs, I get everyone’s favorite error, “The “count” value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created” message from the local for lookup_transit_gateway local. I am trying to understand why that may be. It occurs on the vpc_attachments, which should be evaulating to true/1 based on https://github.com/cloudposse/terraform-aws-transit-gateway/blob/master/main.tf#L10 since I pass in the existing_transit_gateway_id.

Matt H. avatar
Matt H.

It’s odd because there’s not even a resource to target with -target; it’s just checking for variables.

Niv Weiss avatar
Niv Weiss

Hey guys, can someone please help me? :pray:

I’m creating an EKS cluster using Terraform, and part of it, is to create an ALB. When I run terraform destroy it doesn’t work because the vpc is locked due to the ALB that needs to be deleted first. But, because it has been created from my ingress controller I don’t know how to make terraform recognize it… I want to be able to run terraform destroy without the need to manually delete the alb first.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

your ingress controller must be created in diff module, so it needs to be destroyed first

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it should destroy all the AWS resources it created

Niv Weiss avatar
Niv Weiss

@Andriy Knysh (Cloud Posse) I’m using Helm to create the ingress controller. So, I need to do helm uninstall xxx and then it should work? (or of course to use helm inside terraform)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yes, you need to use whatever you used to create it

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it could be helm, helmfile, terraform helm release, terraform kubernetes resources

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you destroy it first, then run terraform destroy on the EKS cluster

Niv Weiss avatar
Niv Weiss

Thank you!!

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you need to destroy (almost) all helm releases (not only ingress)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

b/c many of them create something in AWS, e.g. Route53 records

Niv Weiss avatar
Niv Weiss

I just did it and it didn’t delete the alb I have only 2 helm charts - one that create an alb controller and the other one create an ingress. and still the ALB on aws is active..

Niv Weiss avatar
Niv Weiss

ok now I see that even after I delete the helm chart the ingress is still remain in k8s

Niv Weiss avatar
Niv Weiss

Another question, when I’m creating my cluster, it’s all the time failing during terraform apply when it’s getting to run things inside my cluster. Is there a way to make terraform to wait a little bit so that the cluster will be online and then it would not fail? I’m getting this error message: Error: Kubernetes cluster unreachable: Get "<http://xxxxxxxxx1682.yl4.us-east-1.eks.amazonaws.com/version>": dail tcp xx.xx.xx.xx:443: i/o timeout

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it’s better to separate the cluster and things inside the cluster into separate components (root modules) and provision them separately

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you separate TF state and live-cycle

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and you will not have the issue with destroying them

Niv Weiss avatar
Niv Weiss

So, just to make I got it correctly, if I will separate them into 2 different modules the will get different states?

1

2022-05-04

Release notes from terraform avatar
Release notes from terraform
05:13:13 PM

v1.2.0-rc1 1.2.0-rc1 (Unreleased) UPGRADE NOTES:

The official Linux packages for the v1.2 series now require Linux kernel version 2.6.32 or later.

When making outgoing HTTPS or other TLS connections as a client, Terraform now requires the server to support TLS v1.2. TLS v1.0 and v1.1 are no longer supported. Any safely up-to-date server should support TLS 1.2, and mainstream web browsers have required it since 2020.

When making outgoing HTTPS or other TLS connections as a client, Terraform will no…

mfridh avatar

Wow… this is incredibly annoying…

I’m attempting to simplify the data I’m passing around from module to module by wrapping data in more of a predefined “context”, so it can be easier passed to other modules…

but as soon as there is an Unknown value inside, it isn’t included in a dynamic block where I’m ultimately utilizing it… thus making the plan output very confusing (because it’s empty…).

mfridh avatar
output lb_context {
  description = "A context object containing most or all needed information about our style of load balancer."
  value = {
    name = aws_lb.this.name
    node_permit = true
    asg_attach = true
    security_group = aws_security_group.this.id
    target_group = {
      name = aws_lb_target_group.this.name
      arn = aws_lb_target_group.this.arn
      port = aws_lb_target_group.this.port
      health_check_port = aws_lb_target_group.this.health_check[0].port
    }  
  }
}

If I change to security_group = "foo-123" it shows up in the plan output…

mfridh avatar

Output:

Changes to Outputs:
  + dynamic_rules = [
      + {
          + cidr_blocks     = []
          + description     = "bf-test-grpc-pub-ingress-wtf LB to bf-test target port (from terraform-aws-eks-dice LB context)"
          + from_port       = 31390
          + protocol        = "TCP"
          + security_groups = [
              + (known after apply),
            ]
          + self            = false
          + to_port         = 31390
        },

If I include the same object in an output, it’s all there except for that little “(known after apply)” inside …

mfridh avatar

This is how it gets dynamically used in an aws_security_group resource:

resource "aws_security_group" "node_extra" {
  name        = "${var.cluster_name}-node-extra"
  description = "${var.cluster_name} Node Extra Security Group"
  vpc_id      = var.aws_account.vpc_id

  dynamic "ingress" {
    for_each = [
      for i in local.node_security_group_rules : {
        cidr_blocks     = i.cidr_blocks
        description     = i.description
        from_port       = i.from_port
        protocol        = i.protocol
        security_groups = i.security_groups
        to_port         = i.to_port
        self            = i.self
      }
    ]

    content {
      cidr_blocks     = ingress.value.cidr_blocks
      description     = ingress.value.description
      from_port       = ingress.value.from_port
      protocol        = ingress.value.protocol
      security_groups = ingress.value.security_groups
      to_port         = ingress.value.to_port
      self            = ingress.value.self
    }
  }

Not sure if there’s a way around it… except for running terraform twice …

mfridh avatar

As soon as the i.security_groups is replaced with a static string, those blocks show up in the plans as expected.

So, this seems like a very specific case with unknown values and dynamic blocks … it won’t work…

If anyone know otherwise, let me know. wave.

mfridh avatar

I guess the way around it is to migrate this to an aws_security_group_rule with for_each instead of including the dynamic block in a aws_security_group resource.

Bit of a gotcha though.. had actually expected Terraform to not silently hide a block. Let me worry about the potential failure, ok? …

mfridh avatar

Same issue there… for_each also requires values to be known.

using count is the only thing that works… which is annoying because if a list of things change - everything is reordered on next apply.

2022-05-05

Grummfy avatar
Grummfy

question, when you have several terraform project, what’s your prefered way to get some ressources ccreated by another? using data to search after a given ressource, reading terraform state? writing afile somewhere and read it?

Soren Jensen avatar
Soren Jensen

We use a combination. Read the state when the resource created got a random name e.g. for S3 buckets as they got to be globally unique. For other resources there can have the same name across all accounts like a VPC, we make a data look up on the resource.

mfridh avatar

Data source is my number 1 go to.

Grummfy avatar
Grummfy

thanks

Dan Herrington avatar
Dan Herrington

hey all, have an arch/folder layout question. We have customer websites deployed in production isolated onto separate instances, but in dev they reside on the same instance. I had started to design the terraform folder structure around application->customer app1, customer app 2, customer app 3, etc. Environment (dev,qa,prod) was defined in tfvars. If I do this though, DEV is going to have different code than QA and Prod. I’m figuring others have run into this similiar issue, and wondering how you setup your terraform code and modules to handle this type of structure?

Tyler Jarjoura avatar
Tyler Jarjoura

Would it be feasible to change dev so that it matches the architecture of prod/qa? Otherwise yeah you would just have separate code for dev. If each app is wrapped in a module then you could do a separate module for the “single-instance” version. Or you could have a variable in the module which changes whether the application is deployed on a new instance or not.

There are a number of options really without knowing the specifics..

Dan Herrington avatar
Dan Herrington

Thanks @Tyler Jarjoura Yeah, I’m looking into coding some branching if we’re in DEV this, in PROD this, but I think that introduces fragility into the code. I agree, modifying so envs match is the best way. I’ll need to cost that option out to make the case though, but by managing active instances that should work.

loren avatar

Popped in my feed today, seems interesting… https://link.medium.com/1YOv5hToNpb

Introducing Terramate — An Orchestrator and Code Generator for Terraformattachment image

Terramate is a tool for managing multiple Stacks containing Terraform code supporting change detection and code generation .

1
Soren Martius avatar
Soren Martius

Thanks for sharing

Introducing Terramate — An Orchestrator and Code Generator for Terraformattachment image

Terramate is a tool for managing multiple Stacks containing Terraform code supporting change detection and code generation .

2022-05-06

Grummfy avatar
Grummfy

a comparison with terraspace could be cool

Soren Martius avatar
Soren Martius

Hey @Grummfy - sure will come up with that

Soren Martius avatar
Soren Martius

Hey Community! We just launched a new open-source orchestrator and code generator Terramate for Terraform.

We’ve built Terramate to solve issues when handling Terraform on scale that we experienced during our day-to-day work with customers.

To mention some of the features that Terramate ships with:

Stacks: Splitting up your state into isolated units. A stack is a runnable Terraform Root Module that operates on a subset of the infrastructure’s resources and has its own state.

Keep you code DRY: Avoid duplication by easily sharing data across your project.

Code Generation: Generate valid Terraform Code to ensure that you can always enter a stack to run plain Terraform commands.

Stack Change detection: Only execute commands in stacks that have been changed in the current branch or since the last merge.

Module Change detection: Enhanced Change Detection allows to identifying stacks that have changes in local modules.

Execute Any Command: Terramate is not a wrapper of Terraform but can execute any commands in (changed) stacks.

Execution Order: Explicitly define an order of execution of stacks.

Forced Stack Execution: Ensure specific stacks are run alongside other stacks.

Pure HCL: All configuration of Terramate can be supplied in the well-known Hashicorp Configuraltion Language (HCL). We’d love to hear your feedback! Thanks!

2
loren avatar

the medium article mentioned terragrunt, but didn’t actually compare anything to terragrunt or mention why you felt terragrunt didn’t meet the need. can you expand on that?

2
jose.amengual avatar
jose.amengual

and how is this compare to atmos?

Marius Tolzmann avatar
Marius Tolzmann

Hi together..

We are currently preparing detailed comparisons to various tools and services and will publish those over the next weeks and will ping you here.

We also plan to e.g. support orchestration for Terragrunt setups by adding detailed change detection (like Atlantis also offers it for Terragrunt via an extension). This would allow to identify changed Terragrunt stacks and only run a subset of stacks that we detected to actually have changed.

In general we designed Terramate to be able to extend existing tooling (which is already great) and existing setups (even plain terraform) by just providing some advanced code generation for configurations and enabling orchestration instead of trying to replace existing tooling.

So our customers or other Terramate users do not need to drop their current tooling in order to make use of the features Terramate offers.

2

2022-05-08

Steve Wade (swade1987) avatar
Steve Wade (swade1987)

Does anyone know of a terraform to cloud formation tool?

2022-05-09

Niv Weiss avatar
Niv Weiss

Hey, I got this error while deploying aws-load-balancer-controller : Error: YAML parse error on aws-load-balancer-controller/templates/deployment.yaml: error converting YAML to JSON: yaml: line 27: could not find expected ':' It’s happening when I’m trying to set podAnnotations with this value: "eks.amazonaws.com/role-arn:${aws_iam_role.alb_role.arn}" Someone knows how can I resolve this problem?

this is the configuration of this resource:

resource "helm_release" "alb" {
  name       = "aws-load-balancer-controller"
  repository = "<https://aws.github.io/eks-charts>"
  chart      = "aws-load-balancer-controller"
  namespace  = "kube-system"

  set {
    name  = "clusterName"
    value = module.eks.cluster_name
  }
  set {
    name  = "podAnnotations"
    value = "eks.amazonaws.com/role-arn:${aws_iam_role.alb_role.arn}"
    type  = "string"
  }
el avatar

have you tried re-running with TF_LOG=trace set so you can see some debug output? it might show you the generated YAML

RB avatar

I think you need backslashes before each special char in the annotation name

RB avatar

something like this

  set {
    name  = "serviceAccount.podAnnotations.eks\\.amazonaws\\.com\\/role-arn"
    value = aws_iam_role.alb_role.arn
    type  = "string"
  }
z0rc3r avatar
Associate an IAM role to a service account - Amazon EKS

In Kubernetes, you define the IAM role to associate with a service account in your cluster by adding the following annotation to the service account.

1
RB avatar

it depends on the helm chart but agreed. i was formatting his set expression correctly where the name should be the full annotation and the value should be the arn

Niv Weiss avatar
Niv Weiss

The problem is that my ALB doesn’t get the right permission. So I tried to manually change them with this annotation and it works. So, now I’m trying to use terraform for implementation.

Niv Weiss avatar
Niv Weiss

@el I tried it without any luck

z0rc3r avatar

Terraform won’t fix misconfigurations. First figure out what is wrong manually, then replicate configuration in terraform.

1
1
Niv Weiss avatar
Niv Weiss

@RB Thank you. I tried what you have suggested and it didn’t work

Niv Weiss avatar
Niv Weiss

@z0rc3r So I followed your advice and it’s working for me with helm chart :smile:(Thank you for that!) but, still in TF I’m getting an error but it’s look easier to solve. This is what I run with helm: helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=staging --set serviceAccount.annotations.'eks\.amazonaws\.com/role-arn'="arn:aws:iam::12345678:role/alb-staging" On TF I’m getting this error: Error: cannot patch "aws-load-balancer-controller" with kind ServiceAccount: ServiceAccount "aws-load-balancer-controller" is invalid: [metadata.annotations: Invalid value: "'[eks.amazonaws.com/role-arn](http://eks.amazonaws.com/role-arn)'": prefix part a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '[example.com](http://example.com)', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), metadata.annotations: Invalid value: "'[eks.amazonaws.com/role-arn](http://eks.amazonaws.com/role-arn)'": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')]

z0rc3r avatar

sorry, but I cannot help with this. I don’t use terraform for helm releases

Niv Weiss avatar
Niv Weiss

@z0rc3r ok, anyway, thank you for your help! by the way, why you don’t use terraform for helm?

z0rc3r avatar

when we were evaluating what to use for helm release management, we found that helmfile works best for us. as there no need to manage terraform state and it uses yaml/templates design similar to helm, it was easier to onboard development teams with this tool

Niv Weiss avatar
Niv Weiss

ok, good to know, thank you!

Andrew Miskell avatar
Andrew Miskell

This is what I use to deploy alb-load-balancer-controller vis helm_terraform

Andrew Miskell avatar
Andrew Miskell
Andrew Miskell avatar
Andrew Miskell

It creates everything needed (IAM Policy, IAM Role, EKS Service Account, etc).

Abhinav Khanna avatar
Abhinav Khanna

Pod annotations should be map and not string. Put a space after :

2022-05-10

Grummfy avatar
Grummfy

hi, I want to create a file (ressource local__file) with terraform that contains the date fof generation, but if the other value doesn’t change I don’t want to regenerate a naw date. But I don’t see how to do it

Adam Roderick avatar
Adam Roderick

Hello, I am setting the ssh_user property of cloudposse/ec2-bastion-server/aws but it does not seem to be taking effect. I can ssh into the server with ec2-user but not with the value I put into ssh_user. Am I misunderstanding something?

batman_93 avatar
batman_93

Hey Folks, I have a use case, where I need to merge a list into a map and iterate it over. I tried several things using for expression but couldn’t get the desired output. Wondering if anyone can help me out here. TIA

list = ["foo", "bar"]

variable "tag" {
  default = {
    "tag1" = {
      product = "xyz"
      env     = "dev"
      lob     = "amazon"
    },
    "tag2" = {
      product = "abc"
      env     = "qa"
      lob     = "google"
    }
  }
}

##Desired  variable output 

variable "tag" {
  default = {
    "tag1" = {
      product = "xyz"
      env     = "dev"
      lob     = "amazon"
      list   = "foo"
    },
    "tag2" = {
      product = "abc"
      env     = "qa"
      lob     = "google"
      list   = "bar"
    }
  }
}
RB avatar

@batman_93

this should work

locals {
  mylist = ["foo", "bar"]
}

variable "tag" {
  default = {
    "tag1" = {
      product = "xyz"
      env     = "dev"
      lob     = "amazon"
    },
    "tag2" = {
      product = "abc"
      env     = "qa"
      lob     = "google"
    }
  }
}

output "tag" {
  value = {
    for key, val in var.tag :
    key => merge(val, { list = local.mylist })
  }
}
loren avatar

How do you decide which item in the list should be present in which tag?

batman_93 avatar
batman_93
> local.desired
{
  "tag1" = {
    "env" = "dev"
    "list" = [
      "foo",
      "bar",
    ]
    "lob" = "amazon"
    "product" = "xyz"
  }
  "tag2" = {
    "env" = "qa"
    "list" = [
      "foo",
      "bar",
    ]
    "lob" = "google"
    "product" = "abc"
  
batman_93 avatar
batman_93

I am expecting the map to loop over the list based on index and only print one value in the output

batman_93 avatar
batman_93

Based on the above I see that its printing all the values in the list

loren avatar

I’m not sure what it means for a map to loop over a list

loren avatar

A map doesn’t have an index, it has a key

batman_93 avatar
batman_93
{
  "tag1" = {
    "env" = "dev"
    "list" = [
      "foo",
    ]
    "lob" = "amazon"
    "product" = "xyz"
  }
  "tag2" = {
    "env" = "qa"
    "list" = [
      "bar"
    ]
    "lob" = "google"
    "product" = "abc"

This is what I am expecting,But the output has all the values in the list

loren avatar

When you loop over a map, you loop over the keys. Terraform sorts map keys lexograhically. So at best you can make this work by convention, but not really by explicit logic and data structure

batman_93 avatar
batman_93

Is it possible for TF to loop over the list and map at the same time and return a map?

loren avatar

try this:

locals {
  mylist = ["foo", "bar"]
}

variable "tag" {
  default = {
    "tag1" = {
      product = "xyz"
      env     = "dev"
      lob     = "amazon"
    },
    "tag2" = {
      product = "abc"
      env     = "qa"
      lob     = "google"
    }
  }
}

output "tag" {
  value = {
    for idx, key in keys(var.tag) :
    key => merge(var.tag[key], { list = [local.mylist[idx]] })
  }
}
loren avatar

but note that we’re relying on the implicit sort order of keys() to match the explicit order of mylist

loren avatar

personally i would reject this code if someone proposed it in our projects, and require them to rethink their inputs and data structures

batman_93 avatar
batman_93

Even I am not happy with the use case, But don’t want to redeploy some of the resources.

1
batman_93 avatar
batman_93

That condition worked. I am gonna try this with my use case to see if there any issues. Thanks a ton for your help.

batman_93 avatar
batman_93

Is it possible to get rid of the "," in the output after foo?

    "list" = [
      "foo",
    ]
loren avatar

Too cursed for me, can’t help there

batman_93 avatar
batman_93

I know it’s too much to ask.. Playing around with trimsuffix. But really thanks for the help. Was stuck on this since yesterday

2022-05-11

Release notes from terraform avatar
Release notes from terraform
07:03:32 PM

v1.2.0-rc2 Version 1.2.0-rc2

Release notes from terraform avatar
Release notes from terraform
07:13:32 PM

v1.2.0-rc2-rc2 1.2.0-rc1 (Unreleased) UPGRADE NOTES:

The official Linux packages for the v1.2 series now require Linux kernel version 2.6.32 or later.

When making outgoing HTTPS or other TLS connections as a client, Terraform now requires the server to support TLS v1.2. TLS v1.0 and v1.1 are no longer supported. Any safely up-to-date server should support TLS 1.2, and mainstream web browsers have required it since 2020.

When making outgoing HTTPS or other TLS connections as a client, Terraform will no…

Craig Wright avatar
Craig Wright

(^^ switching to updated internal release management tooling, still working out kinks in the script)

2
Release notes from terraform avatar
Release notes from terraform
07:43:31 PM

v1.2.0-rc2 1.2.0-rc2 (Unreleased) UPGRADE NOTES:

The official Linux packages for the v1.2 series now require Linux kernel version 2.6.32 or later.

When making outgoing HTTPS or other TLS connections as a client, Terraform now requires the server to support TLS v1.2. TLS v1.0 and v1.1 are no longer supported. Any safely up-to-date server should support TLS 1.2, and mainstream web browsers have required it since 2020.

When making outgoing HTTPS or other TLS connections as a client, Terraform will no…

2022-05-12

Philip Clark avatar
Philip Clark

Hi all, I’m facing the “Error: Invalid count argument” error when trying to use the Cloudposse transit-gateway module for AWS. I’ve seen a number of similar SO and Github issues for other modules. People have mentioned using the -target argument in Terraform as a solution, but I’m curious if there are any other workarounds? That approach is really a non-starter if you’re using an automated platform like Terraform Cloud/Enterprise

loren avatar

hard to say without seeing actual code and error messages, but generally the options are to create the dependent resource in a separate state, or to interpolate arguments from known values instead of passing an attribute of a resource created in the same state

Philip Clark avatar
Philip Clark

I’m more or less following https://github.com/cloudposse/terraform-aws-transit-gateway/tree/master/examples/multi-account exactly. The errors I get are from the sub-account tgw attachments, so yeah creating them in separate state would probably work (but not what I’m going for)

loren avatar

“more or less” and “exactly”?

Adam Panzer avatar
Adam Panzer

Howdy! I’ve been using the eks node group module and noticed something that might be worth making a PR for and wanted to get feedback first.

When you add a node group and don’t specify any ami related things, you end up with a launch template that has no AMI in it, which is fine, that launches the latest/greatest AMI for 1.22. The problem is, dockershim is going away, and if you are in that state of having no ami in the launch template, you might not realize that you still are still using docker as your runtime and not containerd.

More importantly, in order to migrate to using containerd you have to build a new node group. I think it might better, if you aren’t specifying the AMI id, to use the ssm parameters store lookup method instead of the data ami lookup since the parameters store lookup will definitely set an ami and you can modify bootstrap options once that is in place (no ami ID means you can’t modify bootstrap options and that means you can’t enable containerd)

Does this make sense? The PR would essentially ensure you would always have an AMI specified in the LT by using the ssm parameter lookup.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy G (Cloud Posse)

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

@Adam Panzer Thank you for the suggestion. I’ve worked a lot on these modules, but am not afraid to admit I am not entirely following your train of though. Understand that AMI IDs are availabe via SSM as well as through the data.aws_ami Data Source, but that is as far as I follow your argument. I do not know the issues surrounding the switch from Dockerd to Containerd or how getting the AMI ID from SSM is of any help. AFAIK we always specify an AMI ID one way or another in our ASG Launch Template in terraform-aws-eks-node-group and I thought that would be enough. Please educate me.

2022-05-13

2022-05-15

Gitau Wairimu avatar
Gitau Wairimu

On Google Cloud, am lost on why I can’t upload a file to a bucket. Here goes the code.

Gitau Wairimu avatar
Gitau Wairimu

resource “google_storage_bucket” “statics_website_assets” { name = “statics-website-assets” location = “US”

uniform_bucket_level_access = false

website { main_page_suffix = “index.html” #not_found_page = “404.html” } }

resource “google_storage_bucket_object” “website-assets” { name = “website-assets” source = “main.tf” bucket = google_storage_bucket.statics_website_assets.id

}

2022-05-16

Michael Galey avatar
Michael Galey

How do you guys handle ignoring updates on values in a module? I’m using terraform-aws-ec2-instance, using my own ami filter. In my previous personal module I’d use lifecycle{ ignore_changes: [“ami”] }, to ignore updates to that value after the server is built. I guess I could try to set the ami value outside the module, and use the lifecycle settings there? And use that value as the module input

Kelly Correll avatar
Kelly Correll

I have a question regarding the cloudposse/transit-gateway module.

I’m trying to follow the examples on github, however after creating the initial TGW..I attempt to immediately use it to create the attachments. I then get this:

Error: Invalid count argument
│
│   on .terraform/modules/region_tgw.transit_gateway_dev/main.tf line 34, in data "aws_ec2_transit_gateway" "this":
│   34:   count = local.lookup_transit_gateway ? 1 : 0
│
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be
│ created. To work around this, use the -target argument to first apply only the resources that the count depends on.
Kelly Correll avatar
Kelly Correll
module "transit_gateway_prod" {
  source = "cloudposse/transit-gateway/aws"
  version     = "0.9.0"

  ram_resource_share_enabled        = true
  allow_external_principals         = true
  auto_accept_shared_attachments    = "enable"
  ram_principals                    = ["<account number>"]
  config                            = local.transit_gateway_config
  tags                              = "${merge(local.tags, {Name = "prod-tgw-${var.region}" } )}"

  context = module.this.context
  providers = {
    aws = aws.prod
  }
}

module "transit_gateway_dev" {
  source = "cloudposse/transit-gateway/aws"
  version     = "0.9.0"

  existing_transit_gateway_id             = module.transit_gateway_prod.transit_gateway_id
  existing_transit_gateway_route_table_id = module.transit_gateway_prod.transit_gateway_route_table_id

  create_transit_gateway                                         = false
  create_transit_gateway_route_table                             = false
  create_transit_gateway_vpc_attachment                          = true
  create_transit_gateway_route_table_association_and_propagation = false

  config = {
    prod = {
      vpc_id                            = data.aws_vpc.dev_vpc.id
      vpc_cidr                          = data.aws_vpc.dev_vpc.cidr_block
      subnet_ids                        = data.aws_subnets.dev_subnets.ids
      subnet_route_table_ids            = data.aws_route_tables.dev_routes.ids
      route_to                          = null
      route_to_cidr_blocks              = null
      static_routes                     = null
      transit_gateway_vpc_attachment_id = null
    }
  }

  context = module.this.context
  providers = {
    aws = aws.dev
  }

  depends_on = [
    module.transit_gateway_prod
  ]
}
Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

if you were to comment out the transit_gateway_dev module block does it run through as expected with just the transit_gateway_prod resource? If so that might lean towards the depends_on not being honored it would seem to appear.

Kelly Correll avatar
Kelly Correll

Yes, prod runs just fine. I have to run it with -target or comment it out, let it run, re-run/uncomment and re-run

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

yeah I’ve noticed similar quirks when using multiple modules like this and haven’t really found a solution to it other than doing that as well

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

I’ve got a rather complex TGW setup I’ve worked out but it’s multi-VPC but not multi-acct like yours so not sure if that could be related as well.

Kelly Correll avatar
Kelly Correll

Hopefully just bumping this.

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

I believe I found an issue with the transit-gateway module… since v0.7.0 it would appear it doesn’t support v3 of the aws provider or any version <= v4.3.0 as the transit_gateway_cidr_blocks argument was not added to aws_ec2_transit_gateway until v4.4.0. Even with the variable set to the default null it will error about the argument being passed.

RB avatar

Nice catch. Could you create an issue please so we remember to (or perhaps a community member could) pr a fix?

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

@RB sure I’ll get a ticket entered… I finally had time to take a deeper look to see why I couldn’t get any version > 0.6.1 to work because of it. Easy fix would obviously be to set the minimum version as >= 4.4.0 in the versions.tf but that doesn’t fix the released versions.

RB avatar

Ill just add the pr now so we can get it approved sooner

RB avatar

what

• Bump aws provider to v4.4.0

why

• Prevent people from using an older provider version which would cause issues

references

https://github.com/hashicorp/terraform-provider-aws/releases/tag/v4.4.0 • Previous PR #20

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

yeah I just saw the PR… was it really not making use of local or random any longer as well? I hadn’t looked at that

RB avatar

nope

Alex Jurkiewicz avatar
Alex Jurkiewicz

oops, this was my fault

Alex Jurkiewicz avatar
Alex Jurkiewicz

I added this functionality to both the provider and module

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

@Alex Jurkiewicz do you know much about the transit gateway cidr blocks and it’s usage? I’m not familiar with the use cases for it myself and been trying to find some information to see if it’s something I should be including or not in our environment

Alex Jurkiewicz avatar
Alex Jurkiewicz

short version is you don’t need it

Alex Jurkiewicz avatar
Alex Jurkiewicz


You can optionally associate one or more IPv4 or IPv6 CIDR blocks with your transit gateway. You specify an IP address from the CIDR block when you establish a Transit Gateway Connect peer for a transit gateway Connect attachment. You can associate any public or private IP address range, except for addresses in the 169.254.0.0/16 range, and ranges that overlap with addresses for your VPC attachments and on-premises networks. For more information about IPv4 and IPv6 CIDR blocks, see VPCs and subnets in the Amazon VPC User Guide.

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

oh okay so it’s for using Connect attachments not VPC attachments. We’re currently not using Connect attachments so that would be why I haven’t seen it used

Bryan Dady avatar
Bryan Dady

I’m working on setting up a few CloudSearch domains with :terraform: It seems pretty straightforward, and may be worth us creating a module. But it seems the provider / resource is not honoring the expected default_tags, and also complaining if I try to add a tags argument to the resource statement.

Has anyone else encountered this? I appreciate any help confirming if there’s an API limitation, or some other reason we can’t programmatically Manage Tags on CloudSeach Domains

RB avatar

We’ve ran into problems when trying to implement default_tags. There are still lots of open issues on it.

https://github.com/hashicorp/terraform-provider-aws/issues?q=is%3Aissue+is%3Aopen+default_tags

2022-05-17

Eugene avatar

Hi there! What’s the best way to perform/control rolling update during cluster EKS upgrade with https://github.com/cloudposse/terraform-aws-eks-node-group module?

cloudposse/terraform-aws-eks-node-group

Terraform module to provision a fully managed AWS EKS Node Group

Stewart Henderson avatar
Stewart Henderson

That is a problem that I too, am working to solve albeit without that module.

cloudposse/terraform-aws-eks-node-group

Terraform module to provision a fully managed AWS EKS Node Group

Eugene avatar

Thanks! When applying the change after control plane upgrade I assume there should be version update on node group in the code. Amazon EKS creates a new Amazon EC2 launch template version for the Auto Scaling group associated with your node group. The new template uses the target AMI for the update.

I see several vars • ami_release_versionkubernetes_versionlaunch_template_version that goes into https://github.com/cloudposse/terraform-aws-eks-node-group/blob/master/main.tf#L72 Trying to see what is the right var/vars to control launch template version when I need to perform rolling update.

    version         = length(compact(concat([local.launch_template_ami], var.ami_release_version))) == 0 ? try(var.kubernetes_version[0], null) : null
Eugene avatar

The rolling updates system is automatically triggered in most desired situations (AMI change, template change…)

Hila F avatar

hey :slightly_smiling_face: I’m trying using the cloudposse/dynamic-subnets/aws module. It asks me to init, I did, and then I got-

Error: Unsupported block type

  on .terraform/modules/bi-subnets/moved.tf line 3:
   3: moved {

Blocks of type "moved" are not expected here.


Error: Unsupported block type

  on .terraform/modules/bi-subnets/moved.tf line 8:
   8: moved {

Blocks of type "moved" are not expected here.


Error: Unsupported block type

  on .terraform/modules/bi-subnets/moved.tf line 13:
  13: moved {

Blocks of type "moved" are not expected here.

Any idea why?

This is the module call config-

module "subnets" {
  for_each = {
    for subnet in local.private_subnets_per_az_mapping :
    keys(local.private_subnets_per_az_mapping2)[each] => subnet_name
  }
  source              = "cloudposse/dynamic-subnets/aws"
  version             = "2.0.0"
  
  name                = each.subnet_name
  vpc_id              = module.bi-usw2-vpc.vpc_id
  # igw_id              = ["igw-XXXXXXXX"]
  ipv4_cidr_block     = each.subnet_cidr
  availability_zones  = each.availability_zone
}
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

moved is supported in terraform v1

Hila F avatar

oh, so I need to change the version to 1.0.0?

Cause I haven’t used moved at all here

Hila F avatar

I’ve tried init with version 1.0.0 and it works. let’s see all needed functionality is still met on version 1.0.0. Thanks!

1
Tyler Jarjoura avatar
Tyler Jarjoura

Question about the RDS modules…if using them to create a DB from scratch - how are we supposed to manage the DB passwords? Surely we aren’t supposed to just hardcode those as inputs into the modules? Usually I would set them to some default and add an ignore_changes lifecycle rule but it doesn’t look like this is supported?

Alex Jurkiewicz avatar
Alex Jurkiewicz

You can use random_password to generate a password in-stack

mfridh avatar

That’s what I do and the terraform db module I built also puts all details in SSM parameter store in predictable env/service paths so each service can pick everything up from env via either chamber env or kube external secrets operator and injected to files or environment variables.

I put several things in SSM: endpoint, reader endpoint, database schema, username, password.

And then on yet another subpath in ssm I place two items - one in yaml and one in json. These items contain a complete object with everything needed. Endpoints, username, database schema name, passwords. Same as in each individual parameter above.

Credentials are handles the same for both:

  1. The db server or cluster itself… in this case the credentials in ssm are the “root” credentials. Accessible for human operators or other tooling needs.
  2. Db schema(s) on the above server for the actual services themselves. Credentials in SSM accessible from the services themselves (and human operators too if they want of course).

2022-05-18

Release notes from terraform avatar
Release notes from terraform
09:53:33 PM

v1.2.0 1.2.0 (May 18, 2022) UPGRADE NOTES:

If you use the third-party credentials helper plugin terraform-credentials-env, you should disable it as part of upgrading to Terraform v1.2 because similar functionality is now built in to Terraform itself. The new behavior supports the same environment variable naming scheme but has a difference in priority order from the credentials helper: TF_TOKEN_… environment variables will…

GitHub - apparentlymart/terraform-credentials-env: Terraform credentials helper for supplying credentials in environment variablesattachment image

Terraform credentials helper for supplying credentials in environment variables - GitHub - apparentlymart/terraform-credentials-env: Terraform credentials helper for supplying credentials in enviro…

loren avatar

some seriously awesome features in the 1.2 release
precondition and postcondition check blocks for resources, data sources, and module output values: module authors can now document assumptions and assertions about configuration and state values. If these conditions are not met, Terraform will report a custom error message to the user and halt further execution.

replace_triggered_by is a new lifecycle argument for managed resources which triggers replacement of an object based on changes to an upstream dependency.

2022-05-19

brant avatar

Is there a way to define mutually-exclusive input variables in a module? (I could have sworn I’ve seen this implemented somewhere but I can’t remember the details now) I want a user to be able to specify either var.foo or var.bar (or neither), but throw an error if they specify both.

brant avatar

I don’t quite follow, sorry. I thought validation conditions could only refer to the variable itself, not other vars?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

oh, it does support only the variable itself, yes (sorry I misread your question)

loren avatar

in 1.2, yes. see the immediate prior post: https://sweetops.slack.com/archives/CB6GHNLG0/p1652911403789589

some seriously awesome features in the 1.2 release
precondition and postcondition check blocks for resources, data sources, and module output values: module authors can now document assumptions and assertions about configuration and state values. If these conditions are not met, Terraform will report a custom error message to the user and halt further execution.

replace_triggered_by is a new lifecycle argument for managed resources which triggers replacement of an object based on changes to an upstream dependency.

loren avatar

use preconditions…

loren avatar

before 1.2, you have to define it as a single variable object with two attributes. then in your validation you can check both attributes

type = object({
  foo = string
  bar = string
})
brant avatar

@loren thanks, for some reason I skimmed “resources, data sources, and module output values” and dismissed it as an option. I need more coffee. :coffee: The other option I found was locals { check_mutually_exclusive_vars = var.foo != null && var.bar != null ? parseint("Error message here. This is an awful hack.") : null } but uh… I like preconditions much better.

loren avatar

yeah there are some weird hacks you can use to force an error. in the past, we’ve wrapped some of that in a module to make the inputs a little prettier when you use it. but i am also very much looking forward to preconditions!

brant avatar

The only downside is that both methods only surface the error during the plan phase, not validate, but I’ll take it.

loren avatar

yeah, kinda interesting idea. validate has to work without any credentials, so no providers, so what you could validate in a precondition would be pretty limited. but maybe open a feature request, they might be able to find a way to get that to work

Release notes from terraform avatar
Release notes from terraform
06:53:33 PM

v1.3.0-dev Beginning of v1.3.x development.

Release v1.3.0-dev · hashicorp/terraformattachment image

Beginning of v1.3.x development.

2022-05-20

tom637 avatar

This has probably been asked millions of times so far. But what’s the best approach to organising terraform code for multiple aws accounts? Are we thinking multiple repos per account? A mono repo that calls modules?

RB avatar

There’s so many ways to do it

RB avatar

We use atmos here which allows us to structure all of the account infrastructure into yaml files which get deep merged and turned into tfvar inputs for terraform root modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

here are some screenshots of how we organize it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

2022-05-21

Nitin avatar

wave Hello, team!

Nitin avatar

https://github.com/cloudposse/terraform-aws-ecr

While creating permission for lambda then it is allowing other services from that account to call ecr service

Also there should be write_only_permission too needed

cloudposse/terraform-aws-ecr

Terraform Module to manage Docker Container Registries on AWS ECR

Nitin avatar

so that blast redius can be reduce when permission is given to other account principle arns.

2022-05-23

Alex NOVACOVSCHI avatar
Alex NOVACOVSCHI

hi guys, glad to join this space! I have a question regarding aws ecr module: https://registry.terraform.io/modules/cloudposse/ecr/aws/latest How can I change the lifecycle policy params for a particular repo? I found only enable_lifecycle_policy and max_image_count params related to lifecycle policy config.

RB avatar

You can use the escape hatch if you set enable_lifecycle_policy to false and create your own raw resource aws_ecr_lifecycle_policy based on the module output repository_name

Alex NOVACOVSCHI avatar
Alex NOVACOVSCHI

hm, I will try, thank you!

Alex NOVACOVSCHI avatar
Alex NOVACOVSCHI

found a github issue that describes exactly what I need - https://github.com/cloudposse/terraform-aws-ecr/issues/92

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Slack Community

Describe the Feature

Unless I missed it, the current rule available for images expiracy is based on the max number of images for a repo (max_image_count).
Adding the possibility to use the countType “sinceImagePushed” would obviously extend the capabilities of this module.

Describe Ideal Solution

Have the ability to choose between a lifecycle based on either the total number of images or the number of days since their last push.
If both options are set, obviously this should not be allowed.

Alternatives Considered

For the moment I manage the policies outside the module, directly with the resource aws_ecr_lifecycle_policy which will iterate on the list of images on which I want to apply it.

ismail yenigul avatar
ismail yenigul

Hi @Andriy Knysh (Cloud Posse) Can someone from Cloud Posse review this PR? https://github.com/cloudposse/terraform-aws-elasticache-redis/pull/160

what

• Replaces use of deprecated attributes (cluster_mode, replication_group_description, number_cache_clusters) in aws_elasticache_replication_group resurce when using provider registry.terraform.io/hashicorp/aws v4.12.0

why

Eliminate warnings when running terraform plan by moving to latest supported attributes instead.

references Terraform aws provider docs

cluster_mode blocknumber_cache_clustersreplication_group_description • See my comment on #155
closes #155

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we’ll review it, thanks

what

• Replaces use of deprecated attributes (cluster_mode, replication_group_description, number_cache_clusters) in aws_elasticache_replication_group resurce when using provider registry.terraform.io/hashicorp/aws v4.12.0

why

Eliminate warnings when running terraform plan by moving to latest supported attributes instead.

references Terraform aws provider docs

cluster_mode blocknumber_cache_clustersreplication_group_description • See my comment on #155
closes #155

1
ismail yenigul avatar
ismail yenigul

what

• Update minimum version of terraform aws provider to support num_cache_clusters

why

• fix for #160 (comment)

Ross Rochford avatar
Ross Rochford

Against my better judgement, I implemented a Terraform clone in Python/Django: https://github.com/rossrochford/make-it-so

rossrochford/make-it-so

A generic declarative provisioning engine implemented in Python.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

wow, so you read HCL but process it in pure python?

rossrochford/make-it-so

A generic declarative provisioning engine implemented in Python.

Ross Rochford avatar
Ross Rochford

Yeah HCL is the input, luckily there is a library for this so that part wasn’t much work.

These are not Terraform Resources, they are custom classes. It doesn’t have any support for Terraform providers or modules.

Josh B. avatar
Josh B.

Is this something that will approved in the near future? https://github.com/cloudposse/terraform-aws-rds-cluster/pull/138

what

• Add aurora serverlessv2 support

why

• AWS releases aurora serverless v2. • Adopt this new feature in this rds cluster module

references

• closes #137Terraform serverlessv2_scaling_configuration

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy G (Cloud Posse) @Andriy Knysh (Cloud Posse)

what

• Add aurora serverlessv2 support

why

• AWS releases aurora serverless v2. • Adopt this new feature in this rds cluster module

references

• closes #137Terraform serverlessv2_scaling_configuration

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, we should be adding this

1
Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

We need to add testing for it, and @Erik Osterman (Cloud Posse) We need to sign off on this requiring AWS Terraform provider v4.12 (generally our modules still work with the latest 3.x)

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

@Erik Osterman (Cloud Posse) Also need to decide if we are OK with Bridgecrew alerts, such as default storage encrypted and default monitoring interval.

Does serverless support encryption? Our code makes it seem like it does not, but that seems odd. If that differs between v1 and v2, we should account for that, too.

Josh B. avatar
Josh B.

Serverless does support encryption.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m all for moving to 4.x for the module.

1
Release notes from terraform avatar
Release notes from terraform
10:53:31 PM

v1.2.1 Version 1.2.1

Release notes from terraform avatar
Release notes from terraform
11:13:30 PM

v1.2.1 No content.

Release notes from terraform avatar
Release notes from terraform
11:23:30 PM

v1.2.1 1.2.1 (May 23, 2022) BUG FIXES: SSH provisioner connections fail when using signed ed25519 keys (#31092) Crash with invalid module source (#31060) Incorrect “Module is…

Release v1.2.1 · hashicorp/terraformattachment image

1.2.1 (May 23, 2022) BUG FIXES: SSH provisioner connections fail when using signed ed25519 keys (#31092) Crash with invalid module source (#31060) Incorrect “Module is incompatible with count, for…

update golang.org/x/crypto again by nilsding · Pull Request #31092 · hashicorp/terraformattachment image

The version that was in use previously broke my runs using ssh-agent with a signed ed25519 key with a nice ssh: handshake failed: agent unsupported algorithm “ssh-ed25519” error. I took a further …

configs: Fix module loader nil pointer panic by alisdair · Pull Request #31060 · hashicorp/terraformattachment image

In configurations which have already been initialized, updating the source of a non-root module call to an invalid value could cause a nil pointer panic. This commit fixes the bug and adds test cov…

2022-05-25

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

I don’t think I say it nearly enough, that said… I am never at a loss for the availability of so many high quality modules I’m trying to work out the raw resources needed to perform a step that I know how to do manually when I think, “Is there a Cloudposse module to do this?” and low and behold there is

3
4
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks @Jeremy (UnderGrid Network Services)

1
managedkaos avatar
managedkaos

in #office-hours today we were talking about minimizing the output from terraform plan.

Here’s an alias that i use for that:

alias fvp='terraform fmt && terraform validate && terraform plan | grep -v unchanged | grep -E "(#|Plan:)" || true'
RB avatar

But why would you want to minimize the output ? How do you know what’s changing then ?

managedkaos avatar
managedkaos

it still shows the resources that are changing. it only minimizes the verbose parts of the plan output.

so if you expect one resource to change but you see two resources changing, for example, minimize the output and you’ll see it a lot easier.

managedkaos avatar
managedkaos

then if you want to see the specifics (if the minimal plan it not what you expect) just run a normal plan to get all the output.

RB avatar

Hmmm… I don’t know. Even if I could see just the resources changing, I’d still want to see the exact arguments of each resource that differ… otherwise, how would I know if the change would break existing functionality ?

RB avatar

I suppose if I had every pattern codified in an OPA rego rule, then I wouldn’t mind a minimized plan

managedkaos avatar
managedkaos

i guess it depends on how you are viewing the plan. sometimes i just make changes locally, and just do a minimal plan before running in CI. CI does a full plan. i can review it there if needed and then manually apply in CI.

managedkaos avatar
managedkaos

i guess because i am mainly looking for things that i don’t expect to change. when i see resources that i don’t think should be affected by the changes i made. i dig a bit deeper.

managedkaos avatar
managedkaos

output is similar to…

Success! The configuration is valid.

  # module... will be updated in-place
  # module... will be updated in-place
  # module... will be updated in-place
  # module... will be updated in-place
  # module... will be updated in-place
  # module... will be destroyed
  # module... will be updated in-place
Plan: 0 to add, 6 to change, 1 to destroy.
Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

Anyone else using the cloudposse/terraform-aws-eks-cluster module able to confirm what I’m seeing? I’m passing in the map_additional_iam_user argument and I’ve left the aws_auth_yaml_strip_quotes default to true which should be removing the double quotes but I am still seeing them in the aws-auth ConfigMap and Terraform plan output. Interestingly though I note that the map_additional_iam_roles that is being passed in does not have the double quotes, only the mapUsers.

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

okay so I found my issue and it wasn’t related to this afterall and I was just chasing a red herring.

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

That said I did uncover an issue and I could see 2 possible solutions but one is currently the only option.

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

@Jeremy G (Cloud Posse) wanted to run this by you before I made a bug/FR for this… Currently the launch template that is created by the cloudposse/terraform-aws-eks-node-group module does not have the ability to pass in any of the Private DNS Name Options and I’ve found that if you use RBN (resource-name) hostname_type instead of IPBN (ip-name) for your VPC subnets this will cause an issue with EKS worker nodes which expects IPBN hostnames.

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

I don’t see a need for full support for Private DNS Name Options at this time but I could see a case made to go ahead and ensure it is set to IPBN as the worker nodes will need it and this would allow other non-EKS instances to use RBN as the default.

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

Would just need to add a

private_dns_name_options {
  hostname_type = "ip-name"
}

block I believe to the aws_launch_template resource

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

@Jeremy (UnderGrid Network Services) I am working on eks-node-group today to buff up IPv6 support, so please elaborate and I will take care of it. FWIW, EKS does not work in IPv6-only environments and does not require NAT64 when running IPv6, both because it requires IPv4 for certain operations and has built-in NAT64 without using DNS64.

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

The situation I found myself in was that we’re launching VPCs in a new region so I set the hostname_type to resource-name on the VPC subnets to use RBN DNS instead of the old default IPBN when it is set to ip-name. This works fine for all other EC2 instances and gives us a nice hostname that matches the instance ID; however, this causes TLS internal errors with EKS worker nodes as it appears to be expecting the IPBN hostnames to be used. Since the eks-node-group doesn’t have any option to change it in the LC to use IPBN I had to revert all my subnets to use IPBN.

Jeremy (UnderGrid Network Services) avatar
Jeremy (UnderGrid Network Services)

since EKS just doesn’t support RBN I didn’t see any reason why it couldn’t just be included to use IPBN explicitly in the LC that’s created rather than opening up to allow for the various options

2022-05-26

Amit Karpe avatar
Amit Karpe

Hey IaC Guru, Anyone would like to point me good document / guide which share best practice for TG/TF to implement “Multiple AWS Accounts with AWS Organizations”. I am configuring AWS multi account, so want to follow best practises from IaC specifically Terragrunt point of view.

András Sándor avatar
András Sándor

Hi, I’m using the cloudposse/terraform-aws-ecs-web-app module and having some problems with setting the context for the child modules. terraform-aws-ecs-web-app module calls ecs-cloudwatch-sns-alarms child module. The both the root and child modules uses cloudposse/label/null to set context, like so:

main.tf

module "httpcode_target_3xx_alarm_label" {
  source     = "cloudposse/label/null"
  version    = "0.25.0"
  attributes = ["3XX", "count", "high"]

  context = module.this.context
}
...

I set the context in root like so:

context.tf

...

variable "context" {
  type = any
  default = {
    enabled             = true
    namespace           = "company"
    tenant              = null
    environment         = "dev"
    stage               = null
    name                = "backend"
    delimiter           = null
    attributes          = []
    tags                = {}
    additional_tag_map  = {}
    regex_replace_chars = null
    label_order         = ["namespace","environment", "name"]
    id_length_limit     = null
    label_key_case      = "lower"
    label_value_case    = "lower"
    descriptor_formats  = {}
....
  }

...

I’d expect the cloudwatch alarms to come out as company-dev-backend-3XX-count-high, but only company-dev-backend is being used. This results in all Cloudwatch alarms having identical ARNs, which is obviously not desired. Am I missing something with the null label, or how variables are passed down in modules?

András Sándor avatar
András Sándor

To anyone wondering, this was because I set label_order in the context variable to non-default.

RB avatar

What was your input?

RB avatar

Did you resolve the issue?

András Sándor avatar
András Sándor

I resolved it, but not sure if it’s an issue necessarily, might be a user error :(.

It was caused by this line in the context.tf:

label_order         = ["namespace","environment", "name"]

I assume by specifying the labels and their order the attributes will not be appended in the child module. I think either using the default label order or also adding attributes to the list should work:

label_order         = []

or

label_order         = ["namespace","environment", "name", "attributes"]

Both of these solves my problems.

1
Kelly Correll avatar
Kelly Correll

So, I’ve been messing with the “multi-account” example for the transit gateways……and the entire example is completely busted. For example, I’m using terraform 1.1.8, aws provider 4.15.1, and with JUST replacing the providers with something that map to what I have it doesn’t work out of the box. The modules listed in the version are too old. So, I updated them to the latest. Replaced the problem “count” code for determining the tgw owner vs the vpc owner…and now I’ve run into yet another “count” problem.

Kelly Correll avatar
Kelly Correll

i can definitively say that the multi-account example does not work out. too many things require plan time data.

Alex Jurkiewicz avatar
Alex Jurkiewicz

Like a few cloudposse modules, it might be designed deliberately as “eventually consistent”, where you are expected to have several incrementally-more-successful applies interspersed with manual tweaks

Alex Jurkiewicz avatar
Alex Jurkiewicz

it looks like this is a deliberate design decision to keep everything in a single module. The way to use this is:

  1. Run Terraform, it will create the bucket and then fail when creating the MWAA environment
  2. Add your requirements.txt to the bucket
  3. Re-run Terraform, it will work If you don’t like this flow, build the bucket yourself and upload requirements.txt before using this module, and tell the module not to create the bucket itself.
Alex Jurkiewicz avatar
Alex Jurkiewicz

having said that, I tried using the module for a relatively simple 2-account setup, and found the API (eg, variables) extremely confusing, so much so that I don’t use the module

Laurynas avatar
Laurynas

Hi, has anyone done a migration from terraform to terraform cdk? I’m interested in gradual migration where we already have a huge infra in terraform but want to write some stuff in cdk for terraform

Amit Karpe avatar
Amit Karpe

Hi, With ref to above questions, I would to know more from best practice point of view. Is there any document or guide to understand best practice to manage AWS Multi Account using TF/TG? What should be directory structure? Whether we should prefer mono-repo or separate repository for dev/test/qa/prod/cicd/monitoring environment? How to handle other IaC repo like we have like github, azure, Cloudflare?

RB avatar

We use atmos and yaml to configure this.

See https://atmos.tools

1
Amit Karpe avatar
Amit Karpe

thx

2022-05-27

Tarak_Brainboard avatar
Tarak_Brainboard

Hi everyone! We just did a Terraform Training for our community at Brainboard. Would it be helpful / useful to share it with you?

2
Soren Jensen avatar
Soren Jensen

Anyone know how I bring an error from a bash script run in local-exec up to the terraform level so my terraform deploy will be marked as failed? I had an issue running terraform apply in a github action runner. All looked fine, only due to application error we found the bash script had failed.

msharma24 avatar
msharma24

Add set -e

Soren Jensen avatar
Soren Jensen

Thanks, let me check. I’m pretty sure I got that on all my scripts

loren avatar

yeah that should “just work”. basically the script just needs to actually exit with a non-zero return code and terraform should marked it as failed

jose.amengual avatar
jose.amengual
2
1
Sean avatar

Indeed. State is such a pain.

There are a few things that get stored there and not available from the infrastructure, but those are things that I don’t think we should have in state.

Such as when you create an RDS database, the password is stored in the state. It would be better to create the password 1st and store in a better manner which you allow terraform to read it.

jose.amengual avatar
jose.amengual

I completely disagree with the article, I think the state is the most powerful thing terraform have over the other solutions

1
jose.amengual avatar
jose.amengual

treat the state as a secret, that is generally the rule

Ralf Pieper avatar
Ralf Pieper

I like not being able to touch certain resources that I didn’t make, but state is mostly a pain and I sure hated it when I started, now I accept it and am excited about being able to terraformer an env to write code from manually making an env. Azure has this nice save this as JSON everywhere, it is nice to be able to save things that you made by hand.

Ralf Pieper avatar
Ralf Pieper

https://github.com/GoogleCloudPlatform/terraformer is something I am itching to play with.

GoogleCloudPlatform/terraformer

CLI tool to generate terraform files from existing infrastructure (reverse Terraform). Infrastructure to Code

2022-05-29

Sean avatar

What approach have y’all taken to keep secrets out of terraform state?

For example, when you create an AWS RDS database, the master password is stored in state. Even if you pre-create the password in a secure storage (such as AWS Secrets), terraform will still store the password in association with the RDS resource.

Andy Miguel avatar
Andy Miguel

[https://m.youtube.com/watch?v=xw0iPkYGKQM&list=PLhRztDM6Uvnf40vPxs9nP09ZTr7tQQ96&index=13](https://m.youtube.com/watch?v=xw0iPkYGKQM&list=PLhRztDM6Uvnf40vPxs9nP09ZTr7tQQ96&index=13)

Sean avatar

Yeah. We already cover all that, but would prefer secrets not be in the state at all, and don’t see any value in them being there when we have secret storage.

For our S3 buckets:

  1. encrypted
  2. versioned
  3. state locking
  4. strict access policies.
Faizan Raza avatar
Faizan Raza

Not heard of any methods to avoid secrets in state - it is a fundamental flaw. You need to mange your entire state file as a secret - it is the only way

Joe Perez avatar
Joe Perez

for the example of RDS, ignoring changes to master password parameter and updating the values outside of terraform. The same goes for Secrets Manager secrets. Set a temporary value with the random resource to set an initial value for the secret and then updating those secret values outside of terraform so that they don’t end up in the state

Sean avatar

Yep. Have taken that approach. And even deleted from the state so that audits don’t flag it.

jrade avatar

Previous place I worked at built an in-house/custom terraform provider that interacted with GCP Secrets during apply , that solved for this. I wasn’t there for the implementation though and was only a user so my recollection of the interface(s) and design is a bit fuzzy.

OliverS avatar
OliverS

I’m not big on the idea of deliberately drifting resources from their terraform state.

If the state is encrypted at rest and in transport, and has access policies, how is it less safe than pulling from a secret which uses the same mechanisms (aws kms, iam policies, storage somewhere in AWS, etc)?

Would be great if cloud providers allowed sensitive settings to be specified via a path to a secret in their managed secrets service, would provide peace of mind on that question.

jrade avatar

There would be different keys used for encrypting tf state versus the actual secret(s) you do not want in tf state so you can give them different privileges. I’d also rather have a placeholder for a secret accidentally show up somewhere in cleartext than the actual secret value.

Sean avatar

It’s worse for many reasons.

• When you access a secret from a secret store/vault, that specific event is recorded and has specific controls.

• State is 1 big plain-text file which can’t be audited.

• It’s brought to local file-system, which is another risk if you have people running from their laptops. It’s much easier to accidentally leave secrets laying around, than it is when accessing from a secret store/vault.

1
1

2022-05-30

Taz avatar

wave Hello, team! Can anyone advise on a good terraform course that involves high level hands on learning for an experienced AWS infrastructure admin but with limited IaC. Thanks

msharma24 avatar
msharma24

Terraform Up & Running

Taz avatar

Hey, Thanks for getting in touch. I appreciate it. I need to deploy cloud-trail using Terraform and am struggling. This is the level that I find myself. This is the resource I have looked at but cannot work out what i need to change to personalise it https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudtrail

msharma24 avatar
msharma24

Hey @Taz - You would write the same TF configuration that you were to apply if you were creating cloudtrail via AWS Console.

Tarak_Brainboard avatar
Tarak_Brainboard

@Taz we created a Terraform training for our community last week - if it can help you I can share it with you

Taz avatar

Hi @Tarak_Brainboard Sounds great. Please do share.

Tarak_Brainboard avatar
Tarak_Brainboard

Of course! You can find it here. If you’ve any questions, you can ask them directly here or / and in the chat like that everyone can learn together

Tarak :cloud: on LinkedIn: :pushpin: Free Terraform training :school_satchel: | 30 commentsattachment image

Special thanks to our community members from Brainboard.co (YC W22) for assisting to the FREE Terraform Training we had yesterday!   You’ve been… 30 comments on LinkedIn

2022-05-31

DevOpsGuy avatar
DevOpsGuy

Hi All, Question: “Terraform-How to change AWS Instance type without deleting instance if the instances are already running/stopped.”

RB avatar

I don’t think it’s possible to change that of an existing instance

RB avatar

I believe you’d have to recreate it with the correct instance type

DevOpsGuy avatar
DevOpsGuy

Understood, I guess we need to have its owd EBS volume and change the instance type I guess?

loren avatar

you can change the instance type when it’s stopped with the awscli/console, then update your tf config and terraform should figure it out

loren avatar

but i don’t think you can do it entirely in terraform

RB avatar

Wow didnt know this. Thats very cool

loren avatar

i suppose tf could do this automatically… when it refreshes, note the instance state, and condition whether the instance type is a replacing change based on whether the instance is stopped

loren avatar

same with userdata

loren avatar

yeah, it has to be a “compatible” instance type, but you can go from say a m5.large to m5.xlarge. same family is always compatible. i think it might be possible to go from family to family in some cases, like c5 to m5, but that is not as certain

Amit Karpe avatar
Amit Karpe

Hi, Is there any way to install tf/tg with darwin_amd64 support on Mac with M1 chip? My system is Apple M1, and when I upgrade my tf/tg it is now using arm arch. But this is causing issues with all my old and new providers. I want to continue using darwin_amd64 binaries and providers. Any suggestions?

RB avatar

https://www.terraform.io/downloads shows an arm download for 1.21

RB avatar

I use homebrew to install the latest terraform version based on my architecture

Amit Karpe avatar
Amit Karpe

noted I will prefer to darwin_adm64, as not all provider support arm.

thank you

Amit Karpe avatar
Amit Karpe

Is Rosetta going to support darwin_adm64 on M1 chip? i.e. all darwin_arm64 will supported natively

I am just worried old provider, which need darwin_adm64 arch

RB avatar

which providers do not support arm ? i can only think of the deprecated template provider

Joe Perez avatar
Joe Perez

yeah, the template provider gave me issues. I’ve used a docker container and mounting the current dir

RB avatar

@Joe Perez you can add the following to get a forked template provider working on darwin arm

# versions.tf
terraform {
  required_version = ">= 1.0.0"

  required_providers {
    template = {
      # Custom provider forks the original provider.
      # <https://registry.terraform.io/providers/gxben/template/latest>
      # Only difference is the darwin arm binary.
      source  = "gxben/template"
      # version has to be set explicitly instead of using a > sign
      version = "= 2.2.0-m1"
    }
  }
}

ref: https://gist.github.com/nitrocode/cf40a24054a66afe2b19ca54b7be5d68 (for more info)

1
Joe Perez avatar
Joe Perez

@RB thanks for the heads up! I feel weird about using forked/non-official providers though

RB avatar

no worries. you can fork the original, build it in arm, and publish it in the registry too

Joe Perez avatar
Joe Perez

that’s a good point too! it’s mostly for legacy support, hoping to axe that stuff soon too

1
Amit Karpe avatar
Amit Karpe

I have moved my tf/tg back to darwin_adm64. I thought this will work smoothly with all my providers. but I am still getting wired errors, which indicate my (OS arch for ) providers are not correct.

$ tg plan
╷
│ Error: Failed to load plugin schemas
│
│ Error while loading schemas for plugin components: Failed to obtain
│ provider schema: Could not load the schema for provider
│ registry.terraform.io/hashicorp/aws: failed to instantiate provider
│ "registry.terraform.io/hashicorp/aws" to obtain schema: Unrecognized remote
│ plugin message:
│
│ This usually means that the plugin is either invalid or simply
│ needs to be recompiled to support the latest protocol...
╵
ERRO[0017] 1 error occurred:
	* exit status 1

❯ file $(which terraform)
/Users/amit/bin/terraform: Mach-O 64-bit executable x86_64
❯ file $(which terragrunt)
/opt/homebrew/bin/terragrunt: Mach-O 64-bit executable x86_64
RB avatar

try the following

rm -rf .terraform*
terraform init -upgrade
terraform plan
Amit Karpe avatar
Amit Karpe

Thanks but that did not work.

I even tried to use tfenv and TFENV_ARCH=arm64, without any success

RB avatar

what does this return

terraform --version

my output returns architecture

✗ terraform --version
Terraform v1.1.7
on darwin_amd64
+ provider registry.terraform.io/cloudposse/utils v0.17.24
+ provider registry.terraform.io/hashicorp/aws v4.16.0
+ provider registry.terraform.io/hashicorp/external v2.2.2
+ provider registry.terraform.io/hashicorp/local v2.2.3

Your version of Terraform is out of date! The latest version
is 1.2.1. You can update by downloading from <https://www.terraform.io/downloads.html>
Amit Karpe avatar
Amit Karpe
❯ terraform --version
Terraform v1.2.1
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v4.11.0

❯ tg version
WARN[0000] No double-slash (//) found in source URL /terraform-aws-modules/terraform-aws-vpc.git. Relative paths in downloaded Terraform code may not work.
Terraform v1.2.1
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v4.11.0

❯ tf version
Terraform v1.2.1
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v4.11.0
RB avatar

Hm that output looks good. What error do you receive

Amit Karpe avatar
Amit Karpe

any plugin / provider wont work on my Mac System. As it might be getting confussed with amd vs arm binaries.

RB avatar

maybe something is in your .terraform.lock.hcl file that is cached and using the wrong executable ?

RB avatar

hmm but you already tried terraform init -upgrade which should have ignored the lock file

RB avatar

what version of tf are you using ?

RB avatar

does terraform work and only terragrunt fail ?

Amit Karpe avatar
Amit Karpe

I’m using terragrunt version v0.37.1 Terraform v1.2.0

I want to setup my Mac to use darwin_amd64 or darwin_arm64

I spend 4-5 days trying all.

darwin_amd64 will slightly older version of provider, aws 3.61 works.

but team wanted to use latest version and all those are arm based (also amd based) but something is just breaking

Amit Karpe avatar
Amit Karpe

❯ file $(which terraform) /Users/amit/bin/terraform: Mach-O 64-bit executable arm64

Amit Karpe avatar
Amit Karpe

Hi,

I am using “arm64” binaries and providers.

❯ terraform version
Terraform v1.2.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v4.10.0

❯ file $(which terraform)
/Users/amit/bin/terraform: Mach-O 64-bit executable arm64

❯ file $(find ~/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/aws/4.10.0)
/Users/amit/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/aws/4.10.0:                                                directory
/Users/amit/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/aws/4.10.0/darwin_arm64:                                   directory
/Users/amit/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/aws/4.10.0/darwin_arm64/terraform-provider-aws_v4.10.0_x5: Mach-O 64-bit executable arm64

I have tried everything but I’m still getting follow errors.

╷
│ Error: Failed to load plugin schemas
│
│ Error while loading schemas for plugin components: Failed to obtain provider schema: Could not load the schema for provider registry.terraform.io/hashicorp/aws: failed to instantiate provider “registry.terraform.io/hashicorp/aws” to
│ obtain schema: Unrecognized remote plugin message:
│
│ This usually means that the plugin is either invalid or simply
│ needs to be recompiled to support the latest protocol...

In past with aws providers lower version 3.61, everything was working. But when we upgraded it newer aws provider versions it keep failing.

Just to fall back, I tried to setup amd64 arch but now it is also not working for me. ( but that is not a main concern now).

    keyboard_arrow_up