#terraform (2022-05)

We use IaC using terraform to create all of our aws accounts

I agree for the account creation. But to create/enable ControlTower and OU’s, do you use terraform or just make a few clicks in the AWS console?

We don’t use control tower but we do configure OUs and accounts through our account terraform component

https://github.com/cloudposse/terraform-aws-components/tree/master/modules/account

We use control tower and have set it up with a philips module. It’s working like a charm

Some links to get you started.. AWS: https://aws.amazon.com/blogs/aws/new-aws-control-tower-account-factory-for-terraform/ AWS GitHub: https://github.com/aws-ia/terraform-aws-control_tower_account_factory Step by step guide on Youtube, step 1 https://www.youtube.com/watch?v=dIv9dPcuehc and step 2 https://www.youtube.com/watch?v=eYWIn7-waP0 Hashicorp: https://learn.hashicorp.com/tutorials/terraform/aws-control-tower-aft

It’s odd because there’s not even a resource to target with -target; it’s just checking for variables.

your ingress controller must be created in diff module, so it needs to be destroyed first

it should destroy all the AWS resources it created

@Andriy Knysh (Cloud Posse) I’m using Helm to create the ingress controller. So, I need to do helm uninstall xxx and then it should work? (or of course to use helm inside terraform)

yes, you need to use whatever you used to create it

it could be helm, helmfile, terraform helm release, terraform kubernetes resources

you destroy it first, then run terraform destroy on the EKS cluster

Thank you!!

you need to destroy (almost) all helm releases (not only ingress)

b/c many of them create something in AWS, e.g. Route53 records

I just did it and it didn’t delete the alb I have only 2 helm charts - one that create an alb controller and the other one create an ingress. and still the ALB on aws is active..

ok now I see that even after I delete the helm chart the ingress is still remain in k8s

it’s better to separate the cluster and things inside the cluster into separate components (root modules) and provision them separately

you separate TF state and live-cycle

and you will not have the issue with destroying them

So, just to make I got it correctly, if I will separate them into 2 different modules the will get different states?

output lb_context {
  description = "A context object containing most or all needed information about our style of load balancer."
  value = {
    name = aws_lb.this.name
    node_permit = true
    asg_attach = true
    security_group = aws_security_group.this.id
    target_group = {
      name = aws_lb_target_group.this.name
      arn = aws_lb_target_group.this.arn
      port = aws_lb_target_group.this.port
      health_check_port = aws_lb_target_group.this.health_check[0].port
    }  
  }
}

If I change to security_group = "foo-123" it shows up in the plan output…

Output:

Changes to Outputs:
  + dynamic_rules = [
      + {
          + cidr_blocks     = []
          + description     = "bf-test-grpc-pub-ingress-wtf LB to bf-test target port (from terraform-aws-eks-dice LB context)"
          + from_port       = 31390
          + protocol        = "TCP"
          + security_groups = [
              + (known after apply),
            ]
          + self            = false
          + to_port         = 31390
        },

If I include the same object in an output, it’s all there except for that little “(known after apply)” inside …

This is how it gets dynamically used in an aws_security_group resource:

resource "aws_security_group" "node_extra" {
  name        = "${var.cluster_name}-node-extra"
  description = "${var.cluster_name} Node Extra Security Group"
  vpc_id      = var.aws_account.vpc_id

  dynamic "ingress" {
    for_each = [
      for i in local.node_security_group_rules : {
        cidr_blocks     = i.cidr_blocks
        description     = i.description
        from_port       = i.from_port
        protocol        = i.protocol
        security_groups = i.security_groups
        to_port         = i.to_port
        self            = i.self
      }
    ]

    content {
      cidr_blocks     = ingress.value.cidr_blocks
      description     = ingress.value.description
      from_port       = ingress.value.from_port
      protocol        = ingress.value.protocol
      security_groups = ingress.value.security_groups
      to_port         = ingress.value.to_port
      self            = ingress.value.self
    }
  }

Not sure if there’s a way around it… except for running terraform twice …

As soon as the i.security_groups is replaced with a static string, those blocks show up in the plans as expected.

So, this seems like a very specific case with unknown values and dynamic blocks … it won’t work…

If anyone know otherwise, let me know. .

I guess the way around it is to migrate this to an aws_security_group_rule with for_each instead of including the dynamic block in a aws_security_group resource.

Bit of a gotcha though.. had actually expected Terraform to not silently hide a block. Let me worry about the potential failure, ok? …

Same issue there… for_each also requires values to be known.

using count is the only thing that works… which is annoying because if a list of things change - everything is reordered on next apply.

We use a combination. Read the state when the resource created got a random name e.g. for S3 buckets as they got to be globally unique. For other resources there can have the same name across all accounts like a VPC, we make a data look up on the resource.

Data source is my number 1 go to.

thanks

Would it be feasible to change dev so that it matches the architecture of prod/qa? Otherwise yeah you would just have separate code for dev. If each app is wrapped in a module then you could do a separate module for the “single-instance” version. Or you could have a variable in the module which changes whether the application is deployed on a new instance or not.

There are a number of options really without knowing the specifics..

Thanks @Tyler Jarjoura Yeah, I’m looking into coding some branching if we’re in DEV this, in PROD this, but I think that introduces fragility into the code. I agree, modifying so envs match is the best way. I’ll need to cost that option out to make the case though, but by managing active instances that should work.

Thanks for sharing

Hey @Grummfy - sure will come up with that

the medium article mentioned terragrunt, but didn’t actually compare anything to terragrunt or mention why you felt terragrunt didn’t meet the need. can you expand on that?

and how is this compare to atmos?

Hi together..

We are currently preparing detailed comparisons to various tools and services and will publish those over the next weeks and will ping you here.

We also plan to e.g. support orchestration for Terragrunt setups by adding detailed change detection (like Atlantis also offers it for Terragrunt via an extension). This would allow to identify changed Terragrunt stacks and only run a subset of stacks that we detected to actually have changed.

In general we designed Terramate to be able to extend existing tooling (which is already great) and existing setups (even plain terraform) by just providing some advanced code generation for configurations and enabling orchestration instead of trying to replace existing tooling.

So our customers or other Terramate users do not need to drop their current tooling in order to make use of the features Terramate offers.

have you tried re-running with TF_LOG=trace set so you can see some debug output? it might show you the generated YAML

I think you need backslashes before each special char in the annotation name

something like this

  set {
    name  = "serviceAccount.podAnnotations.eks\\.amazonaws\\.com\\/role-arn"
    value = aws_iam_role.alb_role.arn
    type  = "string"
  }

see https://github.com/cloudposse/terraform-aws-helm-release#input_service_account_set_key_path

Annotations should be on ServiceAccount though https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html

it depends on the helm chart but agreed. i was formatting his set expression correctly where the name should be the full annotation and the value should be the arn

I’m talking about this chart: https://github.com/aws/eks-charts/tree/master/stable/aws-load-balancer-controller

The problem is that my ALB doesn’t get the right permission. So I tried to manually change them with this annotation and it works. So, now I’m trying to use terraform for implementation.

@el I tried it without any luck

Terraform won’t fix misconfigurations. First figure out what is wrong manually, then replicate configuration in terraform.

@RB Thank you. I tried what you have suggested and it didn’t work

@z0rc3r So I followed your advice and it’s working for me with helm chart :smile:(Thank you for that!) but, still in TF I’m getting an error but it’s look easier to solve. This is what I run with helm: helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=staging --set serviceAccount.annotations.'eks\.amazonaws\.com/role-arn'="arn:aws:iam::12345678:role/alb-staging" On TF I’m getting this error: Error: cannot patch "aws-load-balancer-controller" with kind ServiceAccount: ServiceAccount "aws-load-balancer-controller" is invalid: [metadata.annotations: Invalid value: "'[eks.amazonaws.com/role-arn](http://eks.amazonaws.com/role-arn)'": prefix part a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '[example.com](http://example.com)', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), metadata.annotations: Invalid value: "'[eks.amazonaws.com/role-arn](http://eks.amazonaws.com/role-arn)'": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')]

sorry, but I cannot help with this. I don’t use terraform for helm releases

@z0rc3r ok, anyway, thank you for your help! by the way, why you don’t use terraform for helm?

when we were evaluating what to use for helm release management, we found that helmfile works best for us. as there no need to manage terraform state and it uses yaml/templates design similar to helm, it was easier to onboard development teams with this tool

ok, good to know, thank you!

This is what I use to deploy alb-load-balancer-controller vis helm_terraform

It creates everything needed (IAM Policy, IAM Role, EKS Service Account, etc).

Pod annotations should be map and not string. Put a space after :

@batman_93

this should work

locals {
  mylist = ["foo", "bar"]
}

variable "tag" {
  default = {
    "tag1" = {
      product = "xyz"
      env     = "dev"
      lob     = "amazon"
    },
    "tag2" = {
      product = "abc"
      env     = "qa"
      lob     = "google"
    }
  }
}

output "tag" {
  value = {
    for key, val in var.tag :
    key => merge(val, { list = local.mylist })
  }
}

How do you decide which item in the list should be present in which tag?

I am expecting the map to loop over the list based on index and only print one value in the output

Based on the above I see that its printing all the values in the list

I’m not sure what it means for a map to loop over a list

A map doesn’t have an index, it has a key

{
  "tag1" = {
    "env" = "dev"
    "list" = [
      "foo",
    ]
    "lob" = "amazon"
    "product" = "xyz"
  }
  "tag2" = {
    "env" = "qa"
    "list" = [
      "bar"
    ]
    "lob" = "google"
    "product" = "abc"

This is what I am expecting,But the output has all the values in the list

When you loop over a map, you loop over the keys. Terraform sorts map keys lexograhically. So at best you can make this work by convention, but not really by explicit logic and data structure

Is it possible for TF to loop over the list and map at the same time and return a map?

try this:

locals {
  mylist = ["foo", "bar"]
}

variable "tag" {
  default = {
    "tag1" = {
      product = "xyz"
      env     = "dev"
      lob     = "amazon"
    },
    "tag2" = {
      product = "abc"
      env     = "qa"
      lob     = "google"
    }
  }
}

output "tag" {
  value = {
    for idx, key in keys(var.tag) :
    key => merge(var.tag[key], { list = [local.mylist[idx]] })
  }
}

but note that we’re relying on the implicit sort order of keys() to match the explicit order of mylist

personally i would reject this code if someone proposed it in our projects, and require them to rethink their inputs and data structures

Even I am not happy with the use case, But don’t want to redeploy some of the resources.

That condition worked. I am gonna try this with my use case to see if there any issues. Thanks a ton for your help.

Is it possible to get rid of the "," in the output after foo?

    "list" = [
      "foo",
    ]

Too cursed for me, can’t help there

I know it’s too much to ask.. Playing around with trimsuffix. But really thanks for the help. Was stuck on this since yesterday

hard to say without seeing actual code and error messages, but generally the options are to create the dependent resource in a separate state, or to interpolate arguments from known values instead of passing an attribute of a resource created in the same state

I’m more or less following https://github.com/cloudposse/terraform-aws-transit-gateway/tree/master/examples/multi-account exactly. The errors I get are from the sub-account tgw attachments, so yeah creating them in separate state would probably work (but not what I’m going for)

“more or less” and “exactly”?

@Jeremy G (Cloud Posse)

@Adam Panzer Thank you for the suggestion. I’ve worked a lot on these modules, but am not afraid to admit I am not entirely following your train of though. Understand that AMI IDs are availabe via SSM as well as through the data.aws_ami Data Source, but that is as far as I follow your argument. I do not know the issues surrounding the switch from Dockerd to Containerd or how getting the AMI ID from SSM is of any help. AFAIK we always specify an AMI ID one way or another in our ASG Launch Template in terraform-aws-eks-node-group and I thought that would be enough. Please educate me.

if you were to comment out the transit_gateway_dev module block does it run through as expected with just the transit_gateway_prod resource? If so that might lean towards the depends_on not being honored it would seem to appear.

Yes, prod runs just fine. I have to run it with -target or comment it out, let it run, re-run/uncomment and re-run

yeah I’ve noticed similar quirks when using multiple modules like this and haven’t really found a solution to it other than doing that as well

I’ve got a rather complex TGW setup I’ve worked out but it’s multi-VPC but not multi-acct like yours so not sure if that could be related as well.

Hopefully just bumping this.

Nice catch. Could you create an issue please so we remember to (or perhaps a community member could) pr a fix?

@RB sure I’ll get a ticket entered… I finally had time to take a deeper look to see why I couldn’t get any version > 0.6.1 to work because of it. Easy fix would obviously be to set the minimum version as >= 4.4.0 in the versions.tf but that doesn’t fix the released versions.

Ill just add the pr now so we can get it approved sooner

https://github.com/cloudposse/terraform-aws-transit-gateway/pull/23

yeah I just saw the PR… was it really not making use of local or random any longer as well? I hadn’t looked at that

nope

oops, this was my fault

I added this functionality to both the provider and module

@Alex Jurkiewicz do you know much about the transit gateway cidr blocks and it’s usage? I’m not familiar with the use cases for it myself and been trying to find some information to see if it’s something I should be including or not in our environment

short version is you don’t need it

You can optionally associate one or more IPv4 or IPv6 CIDR blocks with your transit gateway. You specify an IP address from the CIDR block when you establish a Transit Gateway Connect peer for a transit gateway Connect attachment. You can associate any public or private IP address range, except for addresses in the 169.254.0.0/16 range, and ranges that overlap with addresses for your VPC attachments and on-premises networks. For more information about IPv4 and IPv6 CIDR blocks, see VPCs and subnets in the Amazon VPC User Guide.

oh okay so it’s for using Connect attachments not VPC attachments. We’re currently not using Connect attachments so that would be why I haven’t seen it used

We’ve ran into problems when trying to implement default_tags. There are still lots of open issues on it.

https://github.com/hashicorp/terraform-provider-aws/issues?q=is%3Aissue+is%3Aopen+default_tags

That is a problem that I too, am working to solve albeit without that module.

Looking at the module, I would suspect this needs set: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group#update_config-configuration-block in conjunction with https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group#update and dealing with proper taints

Thanks! When applying the change after control plane upgrade I assume there should be version update on node group in the code. Amazon EKS creates a new Amazon EC2 launch template version for the Auto Scaling group associated with your node group. The new template uses the target AMI for the update.

I see several vars • ami_release_version • kubernetes_version • launch_template_version that goes into https://github.com/cloudposse/terraform-aws-eks-node-group/blob/master/main.tf#L72 Trying to see what is the right var/vars to control launch template version when I need to perform rolling update.

The rolling updates system is automatically triggered in most desired situations (AMI change, template change…)

moved is supported in terraform v1

oh, so I need to change the version to 1.0.0?

Cause I haven’t used moved at all here

I’ve tried init with version 1.0.0 and it works. let’s see all needed functionality is still met on version 1.0.0. Thanks!

You can use random_password to generate a password in-stack

That’s what I do and the terraform db module I built also puts all details in SSM parameter store in predictable env/service paths so each service can pick everything up from env via either chamber env or kube external secrets operator and injected to files or environment variables.

I put several things in SSM: endpoint, reader endpoint, database schema, username, password.

And then on yet another subpath in ssm I place two items - one in yaml and one in json. These items contain a complete object with everything needed. Endpoints, username, database schema name, passwords. Same as in each individual parameter above.

Credentials are handles the same for both:

1. The db server or cluster itself… in this case the credentials in ssm are the “root” credentials. Accessible for human operators or other tooling needs.
2. Db schema(s) on the above server for the actual services themselves. Credentials in SSM accessible from the services themselves (and human operators too if they want of course).

see this as an example https://github.com/cloudposse/terraform-aws-dynamic-subnets/blob/master/variables.tf#L443

I don’t quite follow, sorry. I thought validation conditions could only refer to the variable itself, not other vars?

oh, it does support only the variable itself, yes (sorry I misread your question)

in 1.2, yes. see the immediate prior post: https://sweetops.slack.com/archives/CB6GHNLG0/p1652911403789589

use preconditions…

before 1.2, you have to define it as a single variable object with two attributes. then in your validation you can check both attributes

type = object({
  foo = string
  bar = string
})

@loren thanks, for some reason I skimmed “resources, data sources, and module output values” and dismissed it as an option. I need more coffee. :coffee: The other option I found was locals { check_mutually_exclusive_vars = var.foo != null && var.bar != null ? parseint("Error message here. This is an awful hack.") : null } but uh… I like preconditions much better.

yeah there are some weird hacks you can use to force an error. in the past, we’ve wrapped some of that in a module to make the inputs a little prettier when you use it. but i am also very much looking forward to preconditions!

The only downside is that both methods only surface the error during the plan phase, not validate, but I’ll take it.

yeah, kinda interesting idea. validate has to work without any credentials, so no providers, so what you could validate in a precondition would be pretty limited. but maybe open a feature request, they might be able to find a way to get that to work

There’s so many ways to do it

We use atmos here which allows us to structure all of the account infrastructure into yaml files which get deep merged and turned into tfvar inputs for terraform root modules

https://github.com/cloudposse/atmos

Here’s an example setup of the yaml

https://github.com/cloudposse/atmos/tree/master/examples/complete/stacks

here are some screenshots of how we organize it

You can use the escape hatch if you set enable_lifecycle_policy to false and create your own raw resource aws_ecr_lifecycle_policy based on the module output repository_name

hm, I will try, thank you!

we’ll review it, thanks

thanks and review https://github.com/cloudposse/terraform-aws-elasticache-redis/pull/166 please

wow, so you read HCL but process it in pure python?

Yeah HCL is the input, luckily there is a library for this so that part wasn’t much work.

These are not Terraform Resources, they are custom classes. It doesn’t have any support for Terraform providers or modules.

https://pypi.org/project/python-hcl2/

@Jeremy G (Cloud Posse) @Andriy Knysh (Cloud Posse)

Yes, we should be adding this

We need to add testing for it, and @Erik Osterman (Cloud Posse) We need to sign off on this requiring AWS Terraform provider v4.12 (generally our modules still work with the latest 3.x)

@Erik Osterman (Cloud Posse) Also need to decide if we are OK with Bridgecrew alerts, such as default storage encrypted and default monitoring interval.

Does serverless support encryption? Our code makes it seem like it does not, but that seems odd. If that differs between v1 and v2, we should account for that, too.

Serverless does support encryption.

I’m all for moving to 4.x for the module.

Thanks @Jeremy (UnderGrid Network Services)

But why would you want to minimize the output ? How do you know what’s changing then ?

it still shows the resources that are changing. it only minimizes the verbose parts of the plan output.

so if you expect one resource to change but you see two resources changing, for example, minimize the output and you’ll see it a lot easier.

then if you want to see the specifics (if the minimal plan it not what you expect) just run a normal plan to get all the output.

Hmmm… I don’t know. Even if I could see just the resources changing, I’d still want to see the exact arguments of each resource that differ… otherwise, how would I know if the change would break existing functionality ?

I suppose if I had every pattern codified in an OPA rego rule, then I wouldn’t mind a minimized plan

i guess it depends on how you are viewing the plan. sometimes i just make changes locally, and just do a minimal plan before running in CI. CI does a full plan. i can review it there if needed and then manually apply in CI.

i guess because i am mainly looking for things that i don’t expect to change. when i see resources that i don’t think should be affected by the changes i made. i dig a bit deeper.

okay so I found my issue and it wasn’t related to this afterall and I was just chasing a red herring.

That said I did uncover an issue and I could see 2 possible solutions but one is currently the only option.

@Jeremy G (Cloud Posse) wanted to run this by you before I made a bug/FR for this… Currently the launch template that is created by the cloudposse/terraform-aws-eks-node-group module does not have the ability to pass in any of the Private DNS Name Options and I’ve found that if you use RBN (resource-name) hostname_type instead of IPBN (ip-name) for your VPC subnets this will cause an issue with EKS worker nodes which expects IPBN hostnames.

I don’t see a need for full support for Private DNS Name Options at this time but I could see a case made to go ahead and ensure it is set to IPBN as the worker nodes will need it and this would allow other non-EKS instances to use RBN as the default.

Would just need to add a

private_dns_name_options {
  hostname_type = "ip-name"
}

block I believe to the aws_launch_template resource

@Jeremy (UnderGrid Network Services) I am working on eks-node-group today to buff up IPv6 support, so please elaborate and I will take care of it. FWIW, EKS does not work in IPv6-only environments and does not require NAT64 when running IPv6, both because it requires IPv4 for certain operations and has built-in NAT64 without using DNS64.

The situation I found myself in was that we’re launching VPCs in a new region so I set the hostname_type to resource-name on the VPC subnets to use RBN DNS instead of the old default IPBN when it is set to ip-name. This works fine for all other EC2 instances and gives us a nice hostname that matches the instance ID; however, this causes TLS internal errors with EKS worker nodes as it appears to be expecting the IPBN hostnames to be used. Since the eks-node-group doesn’t have any option to change it in the LC to use IPBN I had to revert all my subnets to use IPBN.

since EKS just doesn’t support RBN I didn’t see any reason why it couldn’t just be included to use IPBN explicitly in the LC that’s created rather than opening up to allow for the various options

To anyone wondering, this was because I set label_order in the context variable to non-default.

What was your input?

Did you resolve the issue?

I resolved it, but not sure if it’s an issue necessarily, might be a user error :(.

It was caused by this line in the context.tf:

label_order         = ["namespace","environment", "name"]

I assume by specifying the labels and their order the attributes will not be appended in the child module. I think either using the default label order or also adding attributes to the list should work:

label_order         = []

or

label_order         = ["namespace","environment", "name", "attributes"]

Both of these solves my problems.

Like a few cloudposse modules, it might be designed deliberately as “eventually consistent”, where you are expected to have several incrementally-more-successful applies interspersed with manual tweaks

(similar to https://sweetops.slack.com/archives/CCT1E7JJY/p1652828857092889?thread_ts=1652793190.549939&cid=CCT1E7JJY)

having said that, I tried using the module for a relatively simple 2-account setup, and found the API (eg, variables) extremely confusing, so much so that I don’t use the module

We use atmos and yaml to configure this.

See https://atmos.tools

thx

Add set -e

Thanks, let me check. I’m pretty sure I got that on all my scripts

yeah that should “just work”. basically the script just needs to actually exit with a non-zero return code and terraform should marked it as failed

Indeed. State is such a pain.

There are a few things that get stored there and not available from the infrastructure, but those are things that I don’t think we should have in state.

Such as when you create an RDS database, the password is stored in the state. It would be better to create the password 1st and store in a better manner which you allow terraform to read it.

I completely disagree with the article, I think the state is the most powerful thing terraform have over the other solutions

treat the state as a secret, that is generally the rule

I like not being able to touch certain resources that I didn’t make, but state is mostly a pain and I sure hated it when I started, now I accept it and am excited about being able to terraformer an env to write code from manually making an env. Azure has this nice save this as JSON everywhere, it is nice to be able to save things that you made by hand.

https://github.com/GoogleCloudPlatform/terraformer is something I am itching to play with.

[https://m.youtube.com/watch?v=xw0iPkYGKQM&list=PLhRztDM6Uvnf40vPxs9nP09ZTr7tQQ96&index=13](https://m.youtube.com/watch?v=xw0iPkYGKQM&list=PLhRztDM6Uvnf40vPxs9nP09ZTr7tQQ96&index=13)

Yeah. We already cover all that, but would prefer secrets not be in the state at all, and don’t see any value in them being there when we have secret storage.

For our S3 buckets:

1. encrypted
2. versioned
3. state locking
4. strict access policies.

Not heard of any methods to avoid secrets in state - it is a fundamental flaw. You need to mange your entire state file as a secret - it is the only way

for the example of RDS, ignoring changes to master password parameter and updating the values outside of terraform. The same goes for Secrets Manager secrets. Set a temporary value with the random resource to set an initial value for the secret and then updating those secret values outside of terraform so that they don’t end up in the state

Yep. Have taken that approach. And even deleted from the state so that audits don’t flag it.

Previous place I worked at built an in-house/custom terraform provider that interacted with GCP Secrets during apply , that solved for this. I wasn’t there for the implementation though and was only a user so my recollection of the interface(s) and design is a bit fuzzy.

I’m not big on the idea of deliberately drifting resources from their terraform state.

If the state is encrypted at rest and in transport, and has access policies, how is it less safe than pulling from a secret which uses the same mechanisms (aws kms, iam policies, storage somewhere in AWS, etc)?

Would be great if cloud providers allowed sensitive settings to be specified via a path to a secret in their managed secrets service, would provide peace of mind on that question.

There would be different keys used for encrypting tf state versus the actual secret(s) you do not want in tf state so you can give them different privileges. I’d also rather have a placeholder for a secret accidentally show up somewhere in cleartext than the actual secret value.

It’s worse for many reasons.

• When you access a secret from a secret store/vault, that specific event is recorded and has specific controls.

• State is 1 big plain-text file which can’t be audited.

• It’s brought to local file-system, which is another risk if you have people running from their laptops. It’s much easier to accidentally leave secrets laying around, than it is when accessing from a secret store/vault.

Terraform Up & Running

Hey, Thanks for getting in touch. I appreciate it. I need to deploy cloud-trail using Terraform and am struggling. This is the level that I find myself. This is the resource I have looked at but cannot work out what i need to change to personalise it https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudtrail

Hey @Taz - You would write the same TF configuration that you were to apply if you were creating cloudtrail via AWS Console.

@Taz we created a Terraform training for our community last week - if it can help you I can share it with you

Hi @Tarak_Brainboard Sounds great. Please do share.

Of course! You can find it here. If you’ve any questions, you can ask them directly here or / and in the chat like that everyone can learn together

I don’t think it’s possible to change that of an existing instance

I believe you’d have to recreate it with the correct instance type

Understood, I guess we need to have its owd EBS volume and change the instance type I guess?

you can change the instance type when it’s stopped with the awscli/console, then update your tf config and terraform should figure it out

but i don’t think you can do it entirely in terraform

Wow didnt know this. Thats very cool

i suppose tf could do this automatically… when it refreshes, note the instance state, and condition whether the instance type is a replacing change based on whether the instance is stopped

same with userdata

yeah, it has to be a “compatible” instance type, but you can go from say a m5.large to m5.xlarge. same family is always compatible. i think it might be possible to go from family to family in some cases, like c5 to m5, but that is not as certain

https://www.terraform.io/downloads shows an arm download for 1.21

I use homebrew to install the latest terraform version based on my architecture

noted I will prefer to darwin_adm64, as not all provider support arm.

thank you

Is Rosetta going to support darwin_adm64 on M1 chip? i.e. all darwin_arm64 will supported natively

I am just worried old provider, which need darwin_adm64 arch

which providers do not support arm ? i can only think of the deprecated template provider

yeah, the template provider gave me issues. I’ve used a docker container and mounting the current dir

@Joe Perez you can add the following to get a forked template provider working on darwin arm

# versions.tf
terraform {
  required_version = ">= 1.0.0"

  required_providers {
    template = {
      # Custom provider forks the original provider.
      # <https://registry.terraform.io/providers/gxben/template/latest>
      # Only difference is the darwin arm binary.
      source  = "gxben/template"
      # version has to be set explicitly instead of using a > sign
      version = "= 2.2.0-m1"
    }
  }
}

ref: https://gist.github.com/nitrocode/cf40a24054a66afe2b19ca54b7be5d68 (for more info)

@RB thanks for the heads up! I feel weird about using forked/non-official providers though

no worries. you can fork the original, build it in arm, and publish it in the registry too

that’s a good point too! it’s mostly for legacy support, hoping to axe that stuff soon too

try the following

rm -rf .terraform*
terraform init -upgrade
terraform plan

Thanks but that did not work.

I even tried to use tfenv and TFENV_ARCH=arm64, without any success

what does this return

terraform --version

my output returns architecture

✗ terraform --version
Terraform v1.1.7
on darwin_amd64
+ provider registry.terraform.io/cloudposse/utils v0.17.24
+ provider registry.terraform.io/hashicorp/aws v4.16.0
+ provider registry.terraform.io/hashicorp/external v2.2.2
+ provider registry.terraform.io/hashicorp/local v2.2.3

Your version of Terraform is out of date! The latest version
is 1.2.1. You can update by downloading from <https://www.terraform.io/downloads.html>

Hm that output looks good. What error do you receive

any plugin / provider wont work on my Mac System. As it might be getting confussed with amd vs arm binaries.

maybe something is in your .terraform.lock.hcl file that is cached and using the wrong executable ?

hmm but you already tried terraform init -upgrade which should have ignored the lock file

what version of tf are you using ?

does terraform work and only terragrunt fail ?

I’m using terragrunt version v0.37.1 Terraform v1.2.0

I want to setup my Mac to use darwin_amd64 or darwin_arm64

I spend 4-5 days trying all.

darwin_amd64 will slightly older version of provider, aws 3.61 works.

but team wanted to use latest version and all those are arm based (also amd based) but something is just breaking

❯ file $(which terraform) /Users/amit/bin/terraform: Mach-O 64-bit executable arm64

Hi,

I am using “arm64” binaries and providers.

❯ terraform version
Terraform v1.2.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v4.10.0

❯ file $(which terraform)
/Users/amit/bin/terraform: Mach-O 64-bit executable arm64

❯ file $(find ~/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/aws/4.10.0)
/Users/amit/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/aws/4.10.0:                                                directory
/Users/amit/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/aws/4.10.0/darwin_arm64:                                   directory
/Users/amit/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/aws/4.10.0/darwin_arm64/terraform-provider-aws_v4.10.0_x5: Mach-O 64-bit executable arm64

I have tried everything but I’m still getting follow errors.

╷
│ Error: Failed to load plugin schemas
│
│ Error while loading schemas for plugin components: Failed to obtain provider schema: Could not load the schema for provider registry.terraform.io/hashicorp/aws: failed to instantiate provider “registry.terraform.io/hashicorp/aws” to
│ obtain schema: Unrecognized remote plugin message:
│
│ This usually means that the plugin is either invalid or simply
│ needs to be recompiled to support the latest protocol...

In past with aws providers lower version 3.61, everything was working. But when we upgraded it newer aws provider versions it keep failing.

Just to fall back, I tried to setup amd64 arch but now it is also not working for me. ( but that is not a main concern now).