What is your software development background, and what type of learner are you? Hands on, or by the book?

@Igor Rodionov

@Andriy Knysh (Cloud Posse)

@Iftach Unfortunately this is chicken-egg problem how are terraform resources for MSK done https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_cluster#configuration_info

configuration has to be created/replaced before the MSK cluster can be created/replaced. And while that’s the right order of initial creation, it causes problems for updates. The configuration is attached to the current MSK cluster (what would be recreated/updated on the next step) so you see the error.

If you fine with downtime and do not care about data - you can do terraform destroy and terraform apply

Can you send me the plan output so I can suggest a solution?

Hi @Igor Rodionov i do care about avoiding downtime as much as possible. generally, if i was using resource blocks to create the cluster/configuration, i could have used this lifecycle option on the configuration resource to ‘fix’ this issue.

  lifecycle {
    create_before_destroy = true
  }

is there a way to provide the cloudposse msk module with external configuration resource? as far as i could tell, no, but maybe im missing something?

this is the relevant part of the plan causing me issues

  # module./../.module.msk.module.msk-apache-kafka-cluster.aws_msk_configuration.config[0] must be replaced
-/+ resource "aws_msk_configuration" "config" {
      ~ arn             = "arn:aws:bla" -> (known after apply)
      ~ id              = "arn:aws:kafka:bla -> (known after apply)
      ~ kafka_versions  = [ # forces replacement
          - "3.5.1",
          + "2.8.0",
        ]
      ~ latest_revision = 1 -> (known after apply)
        name            = "bla"
        # (1 unchanged attribute hidden)
    }

I am capable of working around this issue with some manual intervention (create external config unmanaged by TF, moved existing clusters to use that, destroy the old configs and let tf recreate them)

but i wanted to know if this can be fixed at the TF level

Maybe share how you are invoking the module?

I created the project in my local

terraform init
 terraform  plan -var-file="fixtures.us-east-2.tfvars"

It generates this error for all properties under module “rds_cluster”

Without seeing how you set up that module it is impossible to say. I could for example do:

module "rds_cluster" {
  source = "cloudposse/route53-alias/aws"

  version         = "0.12.0"
  other_attrs = "..."
}

And get errors that look like they’re about terraform-aws-rds-cluster, but they aren’t actually. So need to see how you are setting up the module.

all right, I found an issue with the path and it is resolved. You were right. thanks a lot

To leave an answer here, it does work as expected.

@Ben Smith (Cloud Posse)

Hey @Juan Pablo Lorier to be clear passing the ARNs from one module to another worked for you?

yes, it did. All went fine by using both modules together

Perfect, glad to hear

thanks!

@Ben Smith (Cloud Posse)

We have a component for lambdas that we use

Are you using atmos?

HI @Erik Osterman (Cloud Posse)! Yes I use atmos. Could you please post a sample code snippet on how to create a layer with that module?

Our docs should be publicly available tomorrow

Great! Looking forward!

@Erik Osterman (Cloud Posse), any updates?

@Boris Dyga it will go live today. I will let you know when it is up

https://docs.cloudposse.com/layers/software-delivery/lambda/

Thank you!

So our component here is the starting point. then you’d probably want to enable CICD if this is a constantly updated lambda app. If it’s static then throw the code in an s3 bucket (or local to the component) and be done.

If CICD is wanted then you’ll want a repo with the workflow similar to https://github.com/cloudposse/example-app-on-lambda-with-gha

This will build and publish (push to an s3 bucket) the lambda. Then write SSM Key with the commit sha and s3 key where the code lives (like <my-lambda-s3-bucket>/<my-app>/<sha>.zip). We then have promote steps in main branch commits or releases that promote - this copies the SSM key to other aws accounts so that the lambda can be updated in other environments. Finally in all the workflows we re-deploy the stack that actually deploys the lambda from s3 using terraform.

The link Erik posted handles most of this

https://developer.hashicorp.com/terraform/language/expressions/version-constraints#terraform-core-and-provider-versions

normally, in semver major version can include breaking changes in API

this is correct, it can introduce breaking changes (in fact, it happened a few times in the past with the aws provider)

we do it in TF modules to not update hundreds of them every time a new major version of a provider is released

in case of breaking changes, we fix a particular module

however, in your TF root-modules (components), you can always restrict it

so even if a child module uses a greater-than limit, you can still override it in your root modules to exclude major versions

thanks for quick and comprehensive replies!

I use it every so often when I get stuck figuring the type of the attribute in a resource (especially with complex types). Great tool

@Andriy Knysh (Cloud Posse)

the tfvars.json files and the backend.tf.json are automatically created by Atmos from the config in YAML manifests - we don’t push those to VCS (and you can even exclude them in .gitignore)

if you add any tfvars files, then Terraform will combine them with those that Atmos generates - this is not recommended b/c Terraform processes the files in lexicographical order, and depending on the generated file names, the values in them or in the tfvars files will take precedence (which is not ok)

also, if you have all the vars and backend config (and other settings) in YAML, then the command atmos describe component <component> -s <stack> will show you everything about the component in the stack. If you have some vars defined in the terraform code, then Atmos will not be able to show you those values

thank you for excellent answer, @Andriy Knysh (Cloud Posse)

@Jeremy G (Cloud Posse)

cross posted here: https://sweetops.slack.com/archives/C031KFY9C6M/p1724085769888019

providers don’t provide modules, but if you mean resource, then yes the provider doesn’t seem to have a resource for that API. https://github.com/hashicorp/terraform-provider-aws/issues/30060

because of this, you likely won’t find any modules, as the only option for managing this service in terraform is the awscc provider. Which I have found to be a useful provider, but I haven’t seen it used in any public modules before.

Your options are to write your own module or terraform code using the awscc provider or to upvote the issue and wait.

wtf this image really offended me because i’m a terragrunt pusher

I really enjoy seeing how others tackle terraform organization

I’m not a big fan of terraform cloud’s resource based pricing model, so I lean more into s3 based backends, reuseable terraform modules, data source lookups for loose coupling and tier based separation of resources similar to the author

This article is for Terraform heavy users, who manage complex infrastructures. Any one can say that, and all things are relative. I know one of our customers learned the hard way, this doesn’t work when their terraform plans took 25GB ram to run. Now they use atmos.

That’s crazy, what warrants needing 25GB to run a plan? 1000s of resources in a single state?

I’ve heard of plans taking an hour to finish and using targeting as a bandaid

It can happen easily if creating a resource factory inside of a root module. Imagine a root module, that not only leverages a lot of child modules, but also defines a complete SDLC for dev/staging/prod and N environments.

I can definitely see how someone adding one thing for their project or stand alone ticket creating lots of bloat over time

The worst I’ve experienced of that has just been a single state for each env, which I guess is good to get started, but scales poorly for teams

Really fascinating read, especially since our team just decided to move away from this approach after a little over a year!

Like @Erik Osterman (Cloud Posse) says, expertise is relative. Our trio nearly had a decade of Terraform experience between us when we began the greenfield project, so we went all in on dynamic backends, var-files and even workspaces since we wanted the DRYest setup.

After the year’s experience, some issues became too much to workaround:

• Module version tagging ◦ Simply not possible to have different module tags in dev/stg/prod since all TF config is shared. ◦ Thanks to OpenTofu’s early static evaluation, this is at least possible now.

• More complexity than resource count ◦ I think it’s naive to believe all envs are “identical”, excepting resource counts for cost optimisation. ◦ In order to launch a new feature/implementation, you test and tinker with it in dev. Then move it to stg and probably introduce more changes from feedback, before releasing a polished version in prod. ◦ Some additional features (be that backup or security options) simply don’t exist or are incompatible with how prod requires it.

• Plans per environment for each change ◦ The worst one of all, since the core TF config is shared by all environments, we reguarly had to run plan for all envs for each change, just to make sure we weren’t inadvertently breaking prod by changing something in dev. ◦ This one is unforgiveable, and not even branch-based version strategy can help with this mess.

With that being said, I do not enjoy having to diff between directories nor the repetition brought about by having a dedicated folder for each environment.

But the simplicity of a 1:1 relationship is undeniable. What you see is what you get, and you can rest easy knowing any change in dev will never impact stg, let alone prod.

Great insight, I think you should repost this with “also send to #terraform”

Aw, thanks!

You reckon? Alright then.

I’ve wanted to test moving to that pattern, but I agree that it can be rigid/unforgiving in both good and bad ways

Spot-on, that’s it!

When we started out, we were fully onboard with the strict rigidity. After all, all envs should be (near-)identical for a given setup, right?

Well, we believed in it hard enough to push through for a year, managing 6 different envs with the same DRY setup.

However, the tipping point was regularly waiting for plans to complete for 6 different envs for the most trivial of config changes.

If anything, the experience has re-confirmed this is a perfectly valid (and positive!) approach IF you can guarantee there’ll never be any “major” differences between any of your environments beyond changing counts.

Experimentation is key and the important thing is that you found something that works for your team

I may be missing something but imagine TF config (variables) was separate from “code”. It would be layered in a way Opscode Chef did it with environments and roles, allowing for overrides with total of 11 levels of precedence (some used to find it too complex, I know). Isn’t the reason for this thread to exist and for this problem to be such a “no right way to do it” problem, that TF doesn’t allow for code/config separation and config per env?

Imagine just hypothetically that you would have no env / region directories and just one plain structure in TF repo, and in another repo you would have Chef like variables layout with defaults and env, region, role overrides

@Paweł Rein if you like to keep config separate from code, organized by environments, and leveraging overrides, inheritance and imports, sounds like you may like atmos https://atmos.tools

One other thought. What the author calls “tier 2” - the IaC stack that is the application team concern. If done the way he suggests, with TF in application repo, it can lead to duplicated and orphaned resources. Not sure if that approach is compatible with Atmos or other generators like terramate. Crossplane seems to be suited well for this kind of separation of concerns.

stacks/deploy/dev.yaml

vars:
  env: "dev"
  account: "<AWS ACCOUNT>"

components:
  terraform:
    aws-to-github: {}

import:
  - catalog/aws-to-github

terraform:
  providers:
    aws:
      allowed_account_ids: "{{ .vars.account }}"
      profile: "{{ .vars.env }}-administrator"

the command I’m using is

atmos terraform init aws-to-github -s dev

ok I found the atmos describe stacks command which showed me my stack is named deploy/dev for some reason

ok - for some reason if I use the {stage} naming pattern it works but with {{.vars.env}} it doesn’t

¯_(ツ)_/¯

I think is your stack naming try using something like this: {environment}-{stage} instead of {{.vars.env}}-{{.vars.region}}

it might be that .vars is not available at atmos.yaml file

also this might be a question for atmos channel

awesome that works! I can even remap environment to env in defaults with

vars:
  env: "{{ .vars.environment }}"

super cool, thank you!

Niceee!! Glad I was of help!

We switched 7 months ago cloudformation is so painful lol

https://medium.com/@haque-zubair/aws-lambda-api-gateway-with-terraform-bd143b1c56bb

Shameless plug www.taccoform.com

https://roadmap.sh/terraform

Thanks all, i’ll look into this roadmap too! Nicely done.

Is this the component your refering? https://github.com/cloudposse/terraform-aws-components/blob/main/modules/vpc/README.md if so where do you see that account-map?

@Miguel Zablah, this one exactly. have a look in providers.tf file

I see, you can always modify it to not need that part and let atmos generate the backend for you. I actually use the aws module for this https://github.com/terraform-aws-modules/terraform-aws-vpc so I might not be of much help

that’s exactly what I did. however, I’m still wondering if I could preserve vendor component intact and have that possibility (no account-map component usage) without any removal.

without account and account-map I don’t think it’s possible but you can also ask on atmos maybe someone there knows of a way

Note, the terraform-aws-components repo consists of our opinionated root modules as part of our refarch that is optimized for use with atmos and documented at docs.cloudposse.com

We discuss components here https://atmos.tools/core-concepts/components/

Future versions of our refarch will move away from the account-map, but no ETA on that.

assuming I have a spare moment, would it make sense to prepare PR that would make it more generic? i.e. some flag to enable/disable values sourcing from account-map. or do you plan do change the architecture of this component completely? then, such PR maybe it’s not worth the effort

We were discussing this again last friday on our regular Architecture Review Board call. We have some longer term plans that would be too much, but shorter term there might be something we could do that benefits you.

Out of curiosity, are you also using atmos?

ok, I see. yes, Atmos it is

So I understand your use case better, is your goal to use the components in accounts that were already provisioned, such as with control tower?

yes, therefore I have the part related to accounts creation already in place. sort of brownfield scenario for VPC component usage.

ok, great that is the use-case we were discussing. we would like to make it so all the roles can be imperatively defined in the stack configurations, rather than relying on the outputs of the account-map.

the short term workaround is to use the static account map. I see we are missing the definition of that in the documentation here: https://atmos.tools/core-concepts/components/terraform/brownfield#hacking-remote-state-with-static-backends

@Gabriela Campana (Cloud Posse) can you add a task for @Andriy Knysh (Cloud Posse) to document that? It’s come up a lot lately

@Andriy Knysh (Cloud Posse)

cc @Jeremy G (Cloud Posse)

anything on this ?

the difference between the components which are successfully deleted and not able to delete is this Not able to delete provider.tf provider “aws” { region = var.region

# Profile is deprecated in favor of terraform_role_arn. When profiles are not in use, terraform_profile_name is null. profile = module.iam_roles.terraform_profile_name

dynamic “assume_role” { # module.iam_roles.terraform_role_arn may be null, in which case do not assume a role. for_each = compact([module.iam_roles.terraform_role_arn]) content { role_arn = assume_role.value } } }

and able to delete

provider “aws” { region = var.region

profile = module.iam_roles.profiles_enabled ? coalesce(var.import_profile_name, module.iam_roles.terraform_profile_name) : null

dynamic “assume_role” { for_each = module.iam_roles.profiles_enabled ? [] : [“role”] content { role_arn = coalesce(var.import_role_arn, module.iam_roles.terraform_role_arn) } } }

is profile being null is an issue somehow ?

@Aditya What version of account-map are you using? What version of Terraform or OpenTofu are you using?

Older versions of Terraform have issues with destroy that could be the cause of this. I have not seen this problem with any of our current customers, and I suspect the issue is with Terraform.

The profiles feature was something we used for a short time until we found better solutions, and it is only being supported in a legacy mode. I expect you are not using it, in which case you can simply delete the offending line from the [providers.tf](http://providers.tf), although if I’m right about this being a resolved Terraform bug, the best solution is to use a more current version of Terraform, such as 1.5.7.

cc @Jeremy G (Cloud Posse)

Wondering if this is some other issue on EKS side. The node actuallys shows ready status, even has pods running on it.. but in the AWS console the node group doesn’t show this node associated to it.

If I manually create a node group (clickops) and use a different launch template.. this works. but if I try and use the node group module to point to that launch template, it fails. Doing another run now to get the proper error message and i’ll share the code snippet as well.

Maybe i’m losing my mind.. but this seemed to have worked now.

I’ll call it a layer 8 issue for now

I do still want to understand how I can name the node groups to what I want… It seems the node group module is using the label module internally.. so not sure if it’s possible to pass the name I want myself.

@mlanci91 I’m glad you figured out the issue with the Kubernetes version.

Regarding naming the node group, this module uses the Cloud Posse standard null-label module to construct a name. Give it any of the inputs (e.g. name), and the module will append “-workers” to the constructed ID and use that as the name of the node group. At the moment, there is no way to suppress the “-workers” suffix, but other than that you can set whatever name you want via the null-label inputs.

Thanks @Jeremy G (Cloud Posse)

Do you mean “name” as an input to the node group module?

I didn’t see name as a variable in variables.tf

Yes, name = "foo". The variables associated with naming and the null-label module are in a separate file, [context.tf](http://context.tf), to facilitate automated upgrades and remove all this boilerplate from [variables.tf](http://variables.tf).

Explains why I couldn’t find it….

Need to look further into utilizing the label module elsewhere, thanks so much for the reply.

@Jeremy G (Cloud Posse)

If you are using the recommended API authentication mode, and deploying Windows nodes, then you have to configure the node groups access using EC2_WINDOWS, not EC2_LINUX.

Our eks/cluster component does not currently support Windows node groups, although our terraform-aws-eks-cluster module does.