#terraform (2023-03)

It’s a shame it’s not OSS

I’m pretty sure it’s tightly coupled to the architecture and modules they are selling. It wouldn’t be too useable outside that

Oh good point

yeah I wish the terraform docs would have more complete examples for stuff like this, I found this medium article on efs mount with fargate. I hope this is helpful for cross-referencing with your setup https://medium.com/@bhasvanth/create-aws-efs-and-mount-in-ecs-fargate-container-using-terraform-8ce1d68b7eef

I’m not sure where the breakdown is for you, but I would recommend enabling and using ECS exec to exec into a running fargate container to validate what you’re trying to do. Its been super helpful for me

Thanks for the response! I wasn’t even aware that ECS exec on ECS Fargate was a thing, so I will definitely be checking that out! Appreciate the pointer

So my first q - how did folks ended up going on with this limited functionality ? Can you elaborate on your usecase here? And how TF_VAR_ isn’t suitable?

sure thing, let me expand of it

The context is like this:

I’ve already codebase where i heavily use tfvars & auto.tfvars json files and because of that i want to reuse it and not go down another path just because TFC can’t support it. Having to go the TFC way will turn things messy if folks also have to deal with the local run (see my next points)

sounds like TF_CLI_ARGS might be what you need

thanks @Alex Jurkiewicz will look into it. Any thoughts on the other points ?

sounds like TF_CLI_ARGS might be what you need this won’t work if workspace is configured in VCS mode, no ? or am i missing s’thing here

TF_CLI_ARGS_plan to make the environment variable only visible during the plan phase

only visible during the plan phase oh, right i need to read about that, it won’t work for me. Thx for chiming in @Fizz

Why won’t it work? Tf cloud embeds the variables into the saved plan that is used by the subsequent apply, so in the apply phase, the tfvars isn’t even used or referenced

But if you make the tfvars visible during the apply phase, tf cloud will complain that the vars are already in the saved plan and setting it as an environment variable is not allowed

need to play more on this tbh, there are so many edge cases t think about it.

Id follow something like this

1. Make sure your terrform has 0 changes in it by running a plan
2. Change the module version number
3. Run a plan
4. Analyze the plan results and see if you want the new changes. If not, try to see if there is a way to no-op the change

there’s only one output, so no need to splat on it

data.aws_caller_identity.current.account_id

that is because it has a count iterator

and btw very interesting concepts you, like the pattern there

@DaniC (he/him) we have a few diff concepts here:

1. Atmos components and stacks - defined in YAML stack config files and usually in the stacks folder (it’s configurable)

1. Terraform component, usually in components/terraform folder (it’s configurable)

this is separation of concerns by separating code/logic (terraform components) from configurations (Atmos components and stacks)

regarding https://github.com/cloudposse/terraform-aws-components - this is a collection of open source terraform components which can be used to provision your infrastructure

you def can use them, or create your own components in components/terraform folder

terraform components are terraform root modules https://developer.hashicorp.com/terraform/language/modules#the-root-module

https://docs.cloudposse.com/glossary/root-module/

each component (TF root module) defines some functionality and can be combined from terraform modules and/or terraform resources and data sources

as mentioned above, you can use any open source components and you can verdor them into your repo by using Atmos vendoring https://atmos.tools/core-concepts/components/vendoring

(if you don’t want to copy them manually)

and you can vendor them from the public repo, or from your own public/private repos (as you asked in the question)

you create your infra config by a combination of TF components (root modules) which need to be already in your repo (not in the registry) since you have to have some initial TF code to define what TF modules are being instantiated (you can’t have empty TF code to start with )

that’s what are TF components (root modules) for, they are in your repo anyway. And as mentioned, you can vendor them using Atmos, or copy them, or create your own components (which can use TF modules from public repos or your own private repos)

please look at https://atmos.tools/category/quick-start for more details

hey @Andriy Knysh (Cloud Posse) much thanks for the lengthy info provided.

(also, let’s direct atmos questions to atmos and reference architecture questions in refarch)

what was the context of keeping all those modules locals instead of consuming from your private/ public registry ? As others child modules do point to the the registry I’d like to address this point, because it’s so important (and a great question)

What we found from working with customers is if the root modules (components) were all remote, it make it much more cumbersome for your standard developers to a) search for where something was defined b) forced them to dereference a lot of git URLs to look at it c) required multiple PRs just to test something d) make the upstream repo (e.g. cloud posse) a blocker for introducing new changes and therefore a business liability. By vendoring root modules, you get the best of everything. You can easily search the code and understand what’s going on. You can make changes to it if you need to diverge. You can still pull down the upstream components using the atmos vendor commands.

At the same time, nothing requires you to commit these components to VCS. They can be downloaded on demand using the atmos vendor commands. So you can still do things similar to Terragrunt with remote root modules, if the value proposition of having the local components is not there. To reiterate though: we don’t think remote root modules is a great DX.

(also, let’s direct atmos questions to atmos and reference architecture questions in refarch) (edited) my bad, will do so in the future. Thanks a bunch for pointing out, realised i’ve missed joining that channel

thank you very much Cloud Posse team, very detailed set of information

@kevcube some great context above from Erik that is worth reading.

thanks, we’ll review it

you can use #pr-reviews

oh, thanks!

If you init on one remote configuration, then reconfigure the remote config, terraform will ask you if you want to migrate (copy) the state from bucket location A => bucket location B.

terraform init -migrate-state

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "s3" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: 

Do the buckets need to be able to get/put from each other across the accounts for that?

nm…that was dumb.

1. terraform init
2. … change the backend configuration
3. terraform init -migrate-state The TL;DR because I wasn’t entirely clear ^.

My trouble with this approach is that I have one user in one account and a different one in the target account…and they cant see each other and dont have access to each others buckets. So if I change the back end in accountA its not going to be able to see the bucket in accountB where it all needs to migrate to.

You should be able to copy over all the state files from one bucket to the other then just update your backend config to point to thr new bucket and dynamo db lock table

The state file doesn’t contain references to its hosting bucket

Thats what I was thinking but when i tested it that didnt work. It tried to apply resources that were already there.

Did you update the location of the state file in your backend config? Are you accessing the bucket in the correct account?

You can check a state file exists in the bucket by running terraform state pull

So I have been able to move my state files and that seems to work ok. Now the issue im running into is that if i run a plan I get a message that basically says that the md5 in my Dynamo table doesnt match and I need to set it to whatever the original table has. I can do that but I would think I shouldnt have to. Wouldnt terraform just create a new md5 if im pointing at a new table in a different account from the original? I looked through my test statefile and I dont see a mention of that md5 in it or anything called “Digest value”, so how does it know?? Here is the error i get…

Initializing modules...

Initializing the backend...
Error refreshing state: state data in S3 does not have the expected content.

This may be caused by unusually long delays in S3 processing a previous state
update.  Please wait for a minute or two and try again. If this problem
persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
DynamoDB table to the following value: 12492dff2940c03fafc2058ee7f96bd3

I think after tinkering with it some more what Im going to have to do is export my table and import it to the new account. This is a pain because as we slowly move TF code for services from one account to another other devs are working on the same code. So as we move things we remove the code from the old repo once its setup and working and tested in the new repo.

I would not suggest trying to move all of your code, statefiles, and dynamodb locks from one account to another if you have the option not to do thiat.

You migrated the dynamodb table too?

I would think you would not migrate the dynamo table as all of the stuff in there is derived so can be regenerated if it is missing

My guess is that the md5 contains the bucket arn, which includes the account id

Yea…was the only way to get around the error. Otherwise I’d have to find the md5 in the old table and paste it into the new one. Huge pain

I’ve benefitted from having workspaces associated with branches.

in CI, when the environment is built according to the branch, a corresponding workspace is created. The infra is built and tested.

When feature branches gets merged into the staging branch, the branch, workspace, and infra are destroyed.

there are no code or folder changes as using the terraform workspace name is used in resource naming to avoid conflicts.

If you automate your plans and applies then you build in the workspace to your scripts or whatever…then you dont forget or use the wrong workspace.

So let’s say that the workspaces feature did not exist, would the following be equivalent to what @managedkaos describes:

1. say the repo has a folder called dev that is a root terraform module for the dev stack;
2. ci/cd builder checks out the feature branch called foo
3. it would then copy the tf code from dev folder into a new sibling folder called foo
4. then it would cd to foo and run terraform init
5. it would run terraform apply with --var stack_name=foo
6. it would then deploy code, clone database etc and run tests
7. it woudl eventually run terraform destroy
8. builder would remove foo folder With workspaces, steps 3 and 4 get replaced with terraform workspace new and step 8 with terraform workspace delete. Everything else is the same AFAICT.

Is there anything missing from that list? Any situations where having used a folder instead of a folder copy would be major disadvantage?

not sure I like the idea of representing an environment in a “branch”, we use folders + terragrunt

Im not a fan of the branch model either.

In this case, it works because the environments are short lived and any features are merged into the main branch.

indeed, there are other projects where development is done on branches but the environments are in folders. que sera, sera.

Personally, I prefer using the labels, but it works best for poly repos, where one app per repo.

For monorepos, the version file might work better, since you have control over the version of each component inside the monorepo

We use release-drafter at cloudposse

thank you Erik for insights

aws eks update-kubeconfig

https://docs.aws.amazon.com/cli/latest/reference/eks/update-kubeconfig.html

I meant specifically via terraform. Sorry should’ve stipulated that

maybe like this https://jhowerin.medium.com/easy-way-to-use-terraform-to-update-kubeconfig-951837fb4e8e

Wonder if that’ll work via Atlantis

or, you can templatize it in a file, then get all the values from the cluster, then call templatefile function and get back the kubeconfig, then write it to a file (similar to https://stackoverflow.com/questions/64820975/how-to-retrieve-the-eks-kubeconfig)

I’ll give it a go tomorrow thanks

also, if you work with the cluster in TF, you prob don’t need the whole kubeconfig, there are other ways to access the cluster, for example https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/auth.tf#L77

i believe it is owned by aws… https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/elb_service_account

thanks @loren

you’d have to get creative i think, with fileset and a ** pattern, dirname, and toset

maybe throw regex or replace in there also, depending on how deep your directory structure is

Thanks, yes I was trying out ** and figuring out how to dedup the keys

Terraform gives me an error about duplicate keys and adding an ellipsis, but I don’t want to group, I want to ignore

No fileset does not currently support it, not even double star. You can only glob files not folders.

This is discussed in github issue and I opened a feature request for this not long ago.

As a workaround, I globbed for a file that I know exists in each folder, but this won’t apply to all use cases. I was just lucky.

Yep that’s pretty much what I ended up doing. For the moment, the function I was creating only cared about directories that have a specific file anyway… but this might not always be the case.

No plans at the moment. Please feel free to create a branch, test it, and submit it :)

I have created PR, please check it and approve: https://github.com/cloudposse/terraform-aws-alb-ingress/pull/68 Added only fixed-response action, due to only this kind of action was needed for my case. If smb want to add other types of actions, feel free.

Hi, make sure that azure disk encry is enabled for the data disk. Also, check the Azure Key Vault policies and permissions.

Os disk is getting encrypted without any issues

Anyone have working code for that ? Do we need to run any scripts or anything for encryption My os disk is getting encrypted using disk_encryption_id . Same way I’m passing disk_encryption_id in data disk as well

look this bro https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/disk_encryption_set

My Disk encryption key already exists, I’m using the same to encrypt os disk which works fine

Please post in #pr-reviews

(and thanks for the contribution!)

It is already merged, but thanks, will do next time!

fundamentally, the solution is not to use attributes of resources in the key part of the for_each expression

yeah, but it is based on that resource

well, an output from it

that’s a choice. it doesn’t have to be

create a for_each expression where the key is derived entirely from user input

the value part can continue to be derived from another resource

yeah, was just wanting a more direct name there

i already adjusted it locally. was just curious if there were better solutions

did you see this https://registry.terraform.io/providers/pseudo-dynamic/value/latest/docs/resources/is_fully_known ?

would ya look at it

that’s interesting

i would be shocked if that worked with for_each expressions

someone opened a PR and added this

resource "value_unknown_proposer" "default" {}

resource "value_is_fully_known" "task_exec_policy_arns" {
  value            = var.task_exec_policy_arns
  guid_seed        = "var.task_exec_policy_arns"
  proposed_unknown = value_unknown_proposer.default.value
}

interesting. will have to toy with that at some point

truly a terrifying provider

that’s why we have not approved the PR yet

I think if you’re looking at this provider as a serious production solution, you are missing a more obvious better solution.

What loren said is basically accurate – Terraform simply Doesn’t Support creating a dynamic number of resources with count/for_each. If you statically know the number of resources, you can use this static number instead. If the number of resources is truly dynamic, you’ll need to determine the correct number outside of Terraform and push it in as a variable. A common pattern here is two Terraform configurations which run in sequence.

If you have some sample code, we can make more concrete suggestions.

already solved it. i was just wondering if there was some new hotness to resolve it without the old school ways.

this is an old problem they talked about “fixing” years ago

Yeah, it can make some common patterns more complex than we’d like

they talked about fixing it… but it’s not possible to fix it for all cases

if something depends on not yet provisioned resources, no way to know it

that’s why they have not fixed it, and never will

i don’t think it’s actually possible to fix it. at least, not as long as the key part of the for_each expression ends up as part of the state address

i always thought they’d do a “(known after apply)” on the key part of the resource name as well, but it is definitely a bigger issue than that to give a full plan

so i get it…was just curious if they advanced on that front

the only “progress” i’ve seen has been some minor discussion on allowing some kind of “partial” plan/apply, or “convergence” via multiple applies… tracking issue is here: https://github.com/hashicorp/terraform/issues/30937

I see Jeremy was in there discussing too

there is one weird situation where you can kinda use an attribute list to feed for_each, but still only if the length of the list can be fully determined in advance. and, if that is true, you can do something like {for index, value in list_of_things : index => ..... }

so basically, if you give a resource/module a known list of inputs, and the length of that list directly maps to the length of an attribute of that resource/module, then you can use that trick in a for_each expression to build up a map of values using the index of the list

but i really don’t recommend it

( i mention it only because i’m using that hack right at this moment working with a module that outputs a list of route-table-ids. the ids are unknown, so i can’t use them directly. but the length of the list is deterministic based on the number of AZs input to the module. so i can depend on the length. and that means i can use this hack to add routes to the route tables…. )

LOL. so long as it works.

that’s pretty much it. if the change requires users of the module to modify their configs or their state, then it’s a major version. if the change is done in a way that users can simply use the new version with no changes, then it’s a minor version

logic but what about changes in outputs? ie. if you change output of the module form list of strings to list of objects

consumers of that output would need to change their configs, so that’s backwards incompatible, and a major version change

have you heard of semver, Piotr? https://semver.org/ It includes descriptions of when an update should be considered minor or major (or patch)

yeah, this is what I think everyone is using however, I just wanted to hear “real life” opinions

Please post in atmos

The same approach we take at Cloud Posse with Atmos and Spacelift will work for TFC. Everything we do is based on layers.

But “compute-prod” is what we would call a terralith. That would be further broken down into workspaces.

Happy to share more on #office-hours

Tfe_output or a combination of resource tagging and datasources both work. The latter would make your code more portable should you decide to move away from TFC in the future

Happy to share more on #office-hours brilliant, thank you Erik. I’ll try to attend coming Wed.
a combination of resource tagging and datasources both work @Fizz interesting but i guess for this you will need to pass the workspace names as well in the output, have a standardise naming convention which is easily discoverable ? if you have a pseudo example maybe that will help visualising the benefit

I mean don’t use the outputs or the state at all. Use the datasources from the provider e.g. aws to query the resources that got created

Ah sorry, i’m with you, thank you!

variable myvar declared with type of map(string)

Though I believe it’s first interpretted as JSON?

I know it’s possible to pass maps and lists this way too. It’s not clear what your myvar variable is, but I think you have a string.

So I think you want the RHS of the = to be a string. But as it is, I think it’s getting decoded to a map.

just a hunch

@Erik Osterman (Cloud Posse) Yes, it is a string and it should be passed as a string to the module, without interpretation. I also tried to cover it with single quote, but terraform notified that I should use only double quotes. So I need to disable interpretation of the content, but looks like it is a feature to pass maps, as you mentioned

You can still escape it, the problem is it was only escaped for shell, now you need to escape it as a string for terraform. Basically, you need more backslashes :-)

The end result will look like a monstrosity

Thanks, got it!

What do you mean you had to use quotes?

im trying to pass in the ${rendered} json file into ec2 user-data.sh, but when i print ${rendered} in user-data.sh, it strips out every "

instead if my original json has \" instead of ", the ${rendered} json shows up with " when passed into user-data.sh

edit: added snippet back into main thread

Looks like its not just me : https://news.ycombinator.com/item?id=35295216

Thanks! We just saw that too but hadn’t dug into it yet

@Zinovii Dmytriv (Cloud Posse)

Might be related to this

https://www.bleepingcomputer.com/news/security/githubcom-rotates-its-exposed-private-ssh-key/

looks like just a coincidence that SSH key rotation and SSL cert expired happened on the same day

https://github.com/cloudposse/terraform-aws-glue/blob/main/modules/glue-catalog-table/main.tf#L29

partition_keys is a map

yes, i seen it, so it can accept only one partition ?

thanks!

Used helm_release to deploy and then kubectl resource to provision karpenter providers

And the IAM/OIDC/SQS bits?

With a module. Terraform-aws-modules/eks/aws/modules/karpenter

Please use atmos

Architecture is entirely flexible. We work with lots of enterprises with very custom requirements.

Everything can be adjusted using https://atmos.tools/cli/configuration

The only thing rigid right now is our context parameters of (name, namespace, tenant, etc). We plan to eliminate these discrete parameters sometime this year.

…and support arbitrary naming conventions, while still requiring naming conventions.

Thanks @Erik Osterman (Cloud Posse)

In this case, yeah I don’t think it’s actually an issue with atmos specifically. These context parameters are used everywhere by Cloud Posse so atmos is just using them too. It seems in this case, they actually weren’t specific enough.

Please use atmos

let’s move to atmos

Our implementation of our modules is what we call the refarch

The root modules (aka components) are defined here.

Our root modules for EKS are here https://github.com/cloudposse/terraform-aws-components/tree/master/modules/eks

All of the docs are centralized here https://docs.cloudposse.com/components/

We integrate with OIDC and support RBAC. https://docs.cloudposse.com/components/catalog/aws/eks/cluster/

@Dan Miller (Cloud Posse) looks like an example for this is not in this doc: @Linda Pham (Cloud Posse)

Looks like detaching then re-attaching the ec2 instances of an ASG might be better, has anyone tried that? Or setting the instances of an ASG to go on standby, but it’s not clear from the docs whether exit from standby will update tags, seems unlikely

hmmm I might be able to use resource aws_ec2_tag with a for_each that is empty when var.update_asg_instances is false, and is list of ASG instances otherwise. Then if I see that tags of an ASG are planned for update, I abort plan and I also set var.update_asg_instances=true, giving the resource the ASG’s tags; after the apply, I terraform state rm on the aws_ec2_tag, and rerun apply with var.update_asg_instances=false . Sounds complicated but it can actually be done in 3 lines of bash script (using --var). Not bad, beats using the aws cli.

this sounds like a time when using IaC is a bad idea

it’s a one-off task. Update the ASG configuration so new instances are tagged properly, and remediate any existing instances by hand (retag OR terminate)

I would use the Tag Editor to find the target resources (hopefully they have some other tag in common) and then add the new tag in bulk. I would only update the TF to apply the tag as needed to new resources vs trying to remediate existing resources.

https://docs.aws.amazon.com/tag-editor/latest/userguide/find-resources-to-tag.html

I agree that current tf + aws provider does not seem good place to handle this;

BUT my desired state is well defined: I want all instances of an ASG to have the tags of the ASG that manages. A custom provider that uses the AWS SDK would have no problem ensuring that this is true, without stepping on aws provider toes, since the tags of the instances created by the ASG are not tracked by terraform aws provider.

It’s unfortunate that the AWS provider does not provide a resource that supports that notion.

I think you misunderstand the goal of the AWS provider. It’s not to provide all features to everyone, but to be a relatively thin and unopinionated wrapper around the underlying AWS API.

If you want this feature, direct your feature request to AWS to support “updating the tags of ASG instances when I update the ASG’s tags”. Terraform’s provider will implement support for that.

I have two locals, first is the output of formatlist("%s/32", var.ips) and the second is concat(local.first_local, ["127.0.0.1", "127.0.0.2"]), feeding the former as cidr_blocks: local.first_local works fine, but cidr_blocks: local.second_local and I’m boggled on which part it thinks isn’t the same type. Debug output shows:

  got:  {{{} [{{{} map[cidr_blocks:{{{} [{{{} %!s(cty.primitiveTypeKind=83)}}]}} description:{{{} %!s(cty.primitiveTypeKind=83)}} from_port:{{{} %!s(cty.primitiveTypeKind=78)}} protoco
l:{{{} %!s(cty.primitiveTypeKind=83)}} to_port:{{{} %!s(cty.primitiveTypeKind=78)}} type:{{{} %!s(cty.primitiveTypeKind=83)}}] map[]}} {{{} map[cidr_blocks:{{{}}} description:{{{} %!s(
cty.primitiveTypeKind=83)}} from_port:{{{} %!s(cty.primitiveTypeKind=78)}} protocol:{{{} %!s(cty.primitiveTypeKind=83)}} to_port:{{{} %!s(cty.primitiveTypeKind=78)}} type:{{{} %!s(cty.
primitiveTypeKind=83)}}] map[]}}]}}
  want: {{{} {{{}}}}}

But that’s way to many brackets for me to grok.

So wrapping the concat in a flatten seemed to work, but I still dunno where the empty values are coming from

I’m curious too!