#terraform (2020-07)

these days, i believe the required_providers block, an attribute of the terraform block…

https://www.terraform.io/docs/configuration/terraform.html#specifying-required-provider-versions

I’ll look into it, thank you!

I thought that pinning provider versions in modules was advised against?

yes there may be breaking change.

this will allows you to roll out update version per module

im using tf to do this now

look at resource aws_cloudfront_distribution and look at its lambda_function_association

     lambda_function_association {
      event_type   = "origin-request"
      include_body = false
      lambda_arn   = "${aws_lambda_function.request.arn}:${aws_lambda_function.request.version}"
    }

Cool

Just need that and the cloudfront trigger?

you need the cloudfront distribution resource, you need the lambda resource, and then you can use that block with the cf distribution resource to tie it in

i slightly cheated tho. i created my cf distribution and lambda via the aws console, then i backported it into terraform. i went through a couple iterations of updating the lambda code in my repo and redeploying with terraform and it worked as expected

Yeah fair. I did the same with my cf distribution

I just discovered this though which is pretty cool

data "archive_file" "this" {
  for_each = local.lambdas

  type        = "zip"
  source_file = "${path.root}/lambdas/${each.key}_viewer_request.py"
  output_path = "${path.root}/lambdas/${each.key}_viewer_request.zip"
}

yep that’s what im using to zip up lambda

nice and easy

Is it possible to add the cloudfront trigger? None of the lambda resources seem right? Perhaps this isn’t needed if you’re telling the cloudfront dist about the lambda in the tf code?

(please use code blocks) e.g.

subscription_id=$1
sleep $2
echo "{ \"subscription_id\": \"$subscription_id\" }"
data "external" "subscription_id" {
  program = ["./install.sh", "<subscription_id>", "120"]
} 

:wave: FWIW It helps everyone to first run terraform fmt in the source folder as it makes it a bit more readable. Secondly, you can put code into Slack Codeblocks by either typing 3 backticks or find codeblock in the comment menu at the ellipse.

If you look closely at the double quotes you see something is off. I’m sure you’re on a Mac and sometimes if you copy paste double quotes from online, your mac makes it a weird double quote which has a direction left or right like “ or ” , it needs to be “

I have looked at those and replaced the all now it looks like my script is not being read. so it will not load the external data stanza. is there a special way for this to be called

TBH I think you should take a little time to go through a few of the online terraform courses, that will help you a lot

did you remove the quotes around client_secret and client_id ?

Yes see later response

can you post me the current code ?

Give me 10 minutes, just not at my machine at the moment.

Removing those quotes is wrong as they aren’t a variable, the previous code was right, I’ll respond there

Possibly not supported yet - PRs welcome! we’ve ramped up our review capacity.

Are you talking about ECS?

Yes. ECS Fargate

I guess you’re using the default ECS deployment mode which is a rolling update. That results in old and new containers running at the same time during the deployment.

Yes

There is a CodeDeploy blue green deployment option that should help

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-types.html

The “but” to the Blue Green using CodeDeploy is that service autoscaling is not supported.

Ok. Thanks

When you get into situations like this with terraform a good technique is to make a temporary folder and add a few of the bare minimum things to get an example working, preferably with no remote resources. Usually I’ll just declare a locals block and a few output resources to show me what’s going on.

In your case here, the problem you’re having is that you have a complex map defined and lookup expects the key to be at the top level. So to lookup up the port you’d actually need to pass the ‘block 1’ map to the function.

@Zach thanks for your valuable input. Can you give me a quick fix for the problem, meaning how do I pass block 1 map to function

you can stick that into a .tf and run an apply, you’ll get ‘443’ as the output.

So do I have to do this in every lookup function, such as in protocol

Anywhere you have nested maps, yes

great thanks:)

or figure out if there’s a way to do it using the ‘key’ part of the lookup function, but either way its going to mean you have to know the structure of the map

You may just want to step back and think if you’re taking the right approach in that case

exactly - this is a fundamental limitation

we wanted this too - for using jenkins (docker in docker), but had to stick with workers for that node group.

note, it’s possible to mix and match. you can use both modules in concert under the same EKS cluster

Got it makes sense. Yeah thats too bad. I love the simplicity of managed node groups but not being able to add kubelet args is a blocker for us

what flags in particular?

Specifically I need to be able to set the --cluster-dns flag so we can use node-local-dns caching for the cluster

aha, ok

fwiw, for others following along, this is the feature request issue: https://github.com/aws/containers-roadmap/issues/596

You can use a for loop on the attributes, no?

https://github.com/cloudposse/terraform-aws-ec2-autoscale-group/blob/26a90eff8cedd89fb50f039b8b26b2af09ff5c6e/main.tf#L26

more examples here https://github.com/search?q=org%3Acloudposse+for_each&type=Code

I’ll give that a try now!

Thanks Erik, I was able to use the examples to create a dynamic block with a for_each. First time I’ve used a dynamic.

Excellent! glad that worked out for what you needed.

HCl2 for the win!

quite possibly - but we’d accept PRs and are actively reviewing contributions

Does it take the parameter group settings of the default parameter group (docdb3.6) then?

Please see https://github.com/cloudposse/terraform-aws-documentdb-cluster/blob/b61b7fca3fbd934b52af16b5d8979871538e12b9/main.tf#L94-L110

im a little late to the party, but we can’t tag the underlyign ec2 instances w/ Name, etc?

• Stop using alb-ingress?

–

[alb.ingress.kubernetes.io/security-groups](http://alb.ingress.kubernetes.io/security-groups)

Instead, provision the security groups and setup the trust relationships. Then add pass the new blessed security group to the ingress.

sorry i’m not talking about kubernetes ingress. i was talking about the cluster security group ingress rule and node group security group ingress rule. when provisioning the nodegroup, i don’t seem to have a reference to its security group in terraform? going the traditional worker ASG route, the security group is provisioned manually by us in terraform

Maybe I’m still misunderstanding…

You don’t have control over the rules themselves added by AWS. That’s what you’re paying them for

but you have the security groups that you can add/remove rules to

forgive me @Erik Osterman (Cloud Posse) for not being specific enough. I don’t believe the EKS node group returns the automatically provisioned security group for the node group.

https://www.terraform.io/docs/providers/aws/r/eks_node_group.html

So, you’re right this does not appear readily available, but it seems there is a way:

1. https://www.terraform.io/docs/providers/aws/r/eks_node_group.html#autoscaling_groups
2. https://www.terraform.io/docs/providers/aws/d/autoscaling_group.html
3. https://www.terraform.io/docs/providers/aws/d/launch_configuration.html#security_groups

terraform exposes the autoscale group of the managed nodes

look up the autoscale group using the aws_autoscaling_group data provider

then from there, lookup the launch configuration using the aws_launch_configuration data provider

then from there you can access the security_groups

This seems like a nice addition to our module - if you end up implementing it, we’d accept it

@Erik Osterman (Cloud Posse) I may revist nodegroups in the future if there are new features. Right now, for those of us that had been doing things the traditional ASG way, I feel like the terraform module for eks workers is basically just as “managed” (because its automated provisioning) but allows for more configurability.

obviously if you were implementing the EKS workers terraform from scratch today without any prior context, doing it via nodegroups would’ve been way faster

What are you trying to achieve here?

Actually I am trying to setup aws credentials directly as env value during terraform execution..this would be used by terraform for AWS resource creation..Actually My plan is to run this terraform files anywhere and I will do the aws credential configuration..and initiate AWS resource creation..

Can you use aws-vault exec?

Till now I’ve been running it on ubuntu machines..where .bashrc file update made sure aws credentials are used during runtime..

oh ok..i never knew about this..any link u would suggest?

You could then use tfenv

https://github.com/cloudposse/tfenv

https://github.com/99designs/aws-vault

There’s also cloudposse/geodesic if you want to use a docker image as your standard environment for running Terraform things

oh ok..i never thought about running as docker image..interesting..let me chk out..

this will solve all my problem..if I am able run it as docker image..

It works well

You can try this repo https://github.com/cloudposse/testing.cloudposse.co

Pass your account id, etc as a docker run env var

See dockerfile

Then docker exec and run assume-role

sure really looks great..i never thought so many repo on cloudposse..really cool stuff..let me go over and see if i can set it up..

Thank you very much @Joe Niland

Good luck!

You’re welcome

Sounds like your pipeline resources should be a module and you should invoke usage of that module for your X many pipelines and pass the corresponding variables at that level.

Generally, it sounds like reading up on Terraform state and lifecycle would be useful to you as terraform doesn’t act like a typical bash script would. You invoke terraform once to make your resources and then upon any subsequent invoking it’s either updating, adding, or destroying those same resources.

Thanks @Matt Gowie! So if I have a module for pipelines I can then create x number of pipelines by changing the variables is it?

Yeah, you can create your project’s main.tf file, enumerate each pipeline you want by including a module block for it, and then pass your associated pipeline variables to their corresponding module.

Got it thanks again @Matt Gowie appreciate the help!

Np, good luck with it!

How about using terraform workspaces and having multiple state files for multiple executions?

I typically use workspaces for environments i.e. dev, stage, prod. You could use them for what you’re looking to accomplish, but I’d suggest against it considering what I know so far.

Yes, best practice is use workspaces for env variations, but not different deployments of unique stack. Good advice Gowiem.

Does this mean it becomes easier to publish/use custom providers in tf then? The hackery involved in using a community provider found out in the wild is little painful to automate.

yep, this should be easy - as long as the maintainers register their providers

it will continue to be a pain for legacy providers disitrbuted via github.

I wrote up on my experience here if it’s of interest….https://www.sheldonhull.com/blog/compiling-a-custom-provider-and-including-for-terraform-cloud/

Turns out I wasted some time on this. I totally missed that creating a bundle was only for terraform Enterprise and so I had to backtrack at the end of my exploration

I’d verify the S3 bucket and DynamoDB table were actually created via the AWS Console just to rule the simple stuff out.

This is our way https://github.com/cloudposse/terraform-aws-tfstate-backend

@Geordan @Erik Osterman (Cloud Posse) so much folks. I’ll check that out :)

@Erik Osterman (Cloud Posse) here we go mate. Should I rely on the conditional operator?

terraform apply -auto-approve
var.region
  AWS Region the S3 bucket should reside in

  Enter a value: yes

provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-west-2

data.aws_iam_policy_document.prevent_unencrypted_uploads[0]: Refreshing state...
module.terraform_state_backend.data.aws_iam_policy_document.prevent_unencrypted_uploads[0]: Refreshing state...
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=eg-test-terraform-state-lock]
aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=terraform-state-lock]

Error: Error in function call

  on main.tf line 255, in data "template_file" "terraform_backend_config":
 255:       coalescelist(
 256: 
 257: 
 258: 
    |----------------
    | aws_dynamodb_table.with_server_side_encryption is empty tuple
    | aws_dynamodb_table.without_server_side_encryption is empty tuple

Call to function "coalescelist" failed: no non-null arguments.


Error: Error in function call

  on .terraform/modules/terraform_state_backend/main.tf line 234, in data "template_file" "terraform_backend_config":
 234:       coalescelist(
 235: 
 236: 
 237: 
    |----------------
    | aws_dynamodb_table.with_server_side_encryption is empty tuple
    | aws_dynamodb_table.without_server_side_encryption is empty tuple

Call to function "coalescelist" failed: no non-null arguments.

Sorry about stupid question guys. Hopefully, you don’t mind too much. I’m still on a learning curve and quite a bit in my slow-mode regime lol

I went to this, was very disappointed.

Files called terraform.tfvars are loaded first.

*.auto.tfvars are loaded in alphabetical order after terraform.tfvars so they can be useful as overrides.

Files specified by -var-file are loaded last and can override values from the other two.

https://www.terraform.io/docs/commands/apply.html#var-file-foo

Also tfvars doesn’t work with Terraform Cloud. You have to use auto.tfvars or load the input values via TFC variables

Didn’t know that @Chris Wahl - thanks!

ah thx @Chris Wahl that’s what i was looking for. I should have mentioned we are using TFC but this is good to know too Joe. thx you both

Take a look at https://www.terraform.io/docs/configuration/variables.html#variable-definition-precedence

It is possible to use tfvars with TFC/TFE Make an Environment Variable called TF_CLI_ARGS_plan and a value of -var-file=./environments/prod.tfvars

This client used open source workspaces with named tfvars for each workspace, and when moving to TFE they didn’t want to start managing variables in the UI and wanted to keep using tfvars for variables.

I don’t think so. Unfortunately. It would be awesome if you could. What you can do though is use yaml configuration and parse it in tf This is what I’m currently doing instead of tfvars

Basically you’ll want to do something like this: https://github.com/cloudposse/terraform-aws-ecr/blob/master/main.tf#L24-L34

It gets complicated quickly though if you depend on a lot of resources

thanks @Erik Osterman (Cloud Posse) I only have one resource dependency so I think this should work perfectly …

have a look at these modules: https://github.com/cloudposse?q=iam&type=source&language=

you will pretty much just need roles, policies and users

ok. let me have a look

Did you create your queues and dynamodb with Terraform too?

yes

in link you sghared, there are 3 different project folders for policy, roles and users.. how they connect to create users

You create policies that are attached to roles and you allow individual users to assume those roles.

If you already use something like Okta (or other IdP), I’d strongly consider setting up Federated SSO. Managing individual users in AWS with Terraform gets annoying and being too granular can also cause you trouble as there are limits to numbers and sizes of policies that can be attached to a user, and the user will inevitably want more and more.

Of course, it depends how many users you are talking about. If there’s 10, then Terraform will be fine. If there’s 1000 across 220+ AWS accounts then SSO / Self-Serve access request is a must.

ok.. we have less users so terraform is fine for us

hi Team,

I created code for user account creation but once I run terraform plan, it also destroy many things by its own.. This destroy things is not needed,, how to avoid destroys or am I doing something wrong in my new code for user account creation

This is related to your using local state (assumption). The state of the managed objects is kept in local state files (terraform.tfstate) or in a remote data store, such as S3. Running TF in the same dir as the state files for a previous run of TF will operate on the same objects as the last run, because it gets its state from the state files.

I have a similar set-up and I was advised to look into Terraform Cloud, to manage the state.

You could use workspaces to separate out the state, regardless of where the state resides, but i’m not really up on that approach.

In all of my code AWS tags have an equals sign tags = {}, and the Terraform AWS Tagging documentation backs that up by saying resource tags are arguments that accept a key-value map (not blocks that support for_each). I wanted to verify that so I googled it and found the Terraform 0.12 Preview for Dynamic Nested Blocks gives a contradictory example showing exactly what you’re trying to do. Finally I found official documentation clarifying this:

This is required in Terraform 0.12 since tags is an argument rather than a block which would not use =. In contrast, we do not include = when specifying the network_interface block of the EC2 instance since this is a block.

It is not easy to distinguish blocks from arguments of type map when looking at pre-0.12 Terraform code. But if you look at the documentation for a resource, all blocks have their own sub-topic describing the block. So, there is a Network Interfaces sub-topic for the network_interface block of the aws_instance resource, but there is no sub-topic for the tags argument of the same resource.

For more on the difference between arguments and blocks, see Arguments and Blocks.

the equivalent of “dynamic” for a map, as tags is, would be to use a for expression…

tags = { for k, v in <map> : k => v }

something like that, anyway, not able to test it at the moment…

 tags = flatten([
    for key in keys(local.common_tags) : merge(
      {
        key   = key
        value = local.common_tags[key]
    }, var.additional_tag_map)
  ])

You can replace var.additional_tag_map with {} if you don’t have other tags

sweet thank you all for the info i will test this out and let you know if it works

use terraform show -json and pipe to jq

Sounds like the 2nd way you have there is a good way to do it. I don’t believe you can feed in a terraform variable to a module source, since it is needed up front when you run terraform init

https://www.terraform.io/docs/modules/sources.html#generic-git-repository give some good ideas. You can use ssh instead of HTTPS, and it will use your SSH key, or you can set up your git credential helper to already be logged in

You are right, I tried over Terraform variable but it didn’t work:

There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Unsuitable value type

  on main.tf line 2, in module "resource_name":
   2:   source = "git::<https://oauth2>:${var.gitlab_token}@gitlab.com/user/repo.git?ref=tags/v0.1.2"

Unsuitable value: value must be known


Error: Variables not allowed

  on main.tf line 2, in module "resource_name":
   2:   source = "git::<https://oauth2>:${var.gitlab_token}@gitlab.com/user/repo.git?ref=tags/v0.1.2"

Variables may not be used here.

need to read more about “git credential helper”, thx @roth.andy

Used the following:

• Have a simple script at /usr/local/bin/git-credential-helper

#!/bin/sh

# Script used to authenticate against GitLab by using the GitLab token
# When using the GitLab token the assumed user is oauth2
# Example: git clone <https://oauth2>@gitlab.com/user/project.git

# Return the GitLab token 
echo "${GITLAB_TOKEN}"

• create environment variable with the token:

export GITLAB_TOKEN='<MY SECRET GITLAB TOKEN>'

• then just download the module with:

GIT_ASKPASS=/usr/local/bin/git-credential-helper terraform init

The [main.tf](http://main.tf) looks like this now:

module "resource_name" {
  source = "git::<https://oauth2>@gitlab.com/user/project.git?ref=tags/v0.1.2"
  ...
}

@roth.andy thx again for pointing me in the right direction

you’re welcome

I ran into a similar need and saw this thread. Really appreciate the content / ideas here, I built upon it to solve my problem. Shared the details here! https://wahlnetwork.com/2020/08/11/using-private-git-repositories-as-terraform-modules/

so does that mean I just write infrastructure and app both in typescript ?

so, pulumi out of business yet?

Yeah, a race with pulumi becomes even more interesting:)

btw, up to 0.3.1 now

This is neat, but still can’t understand a reason why would someone aggregate policy into one ?

What are some use cases ?

I’m working at a large media company right now that is clearly pushing the limits of what IAM is capable of, for better or worse. We need every byte of IAM policy possible, so it’s useful to concat many small policies into larger one when attaching to a role.

Please see the supporting details in that issue. It’s not a module problem, it’s a terraform problem.

Are slashes valid for key? Never tried that

I think it not a problem because I also have other tags with slashes.

What is the issue? What is the error message? What type of resource? You need to provide more details to get help

I cannot set 7 tags for a resource by Terraform.

Terraform just shave 6 tags in the Terraform stage

Are you saying that you believe your code should set 7 tags, but you are only getting 6 and no errors? If so, I suspect it is an issue in your code. You would need to share all of the related code for anyone to verify this

My thoughts from about a month ago, the module-level count/for_each will definitely simplify things for many modules… https://sweetops.slack.com/archives/CB6GHNLG0/p1592345793219500?thread_ts=1592345793.219500&cid=CB6GHNLG0

Also makes it easier to work with community modules and integrate them into your own work

We have a PR for encryption that was contributed, but needs rebasing by @vkhatri

https://github.com/cloudposse/terraform-aws-eks-cluster/pull/62

You can also enable https://www.terraform.io/docs/providers/aws/r/ebs_encryption_by_default.html

For draining, you might want to have a look at https://github.com/aws-samples/amazon-k8s-node-drainer

(haven’t used it)

https://github.com/aws/aws-node-termination-handler

I know @mumoshu has been working on a helmfile provider

Yes he has and I’ve so hapily downloaded the latest release and built it locally: https://github.com/mumoshu/terraform-provider-helmfile

But in his example he mentions external helmfile_release_set which I use

But I also need to send env variables with it

And sending env variables is shown only with inline releases

For release sets it should work. We are doing this the following way:

resource "helmfile_release_set" "common_stack" {
...
  environment_variables = {
    EXTERNAL_IP = data.google_compute_address.dev_ip_address.address
  }
...
}

ohhhh

using release 0.3 ?

Not yet, we are facing some issues with the most recent release

which one you have so far?

terraform apply just started failing with no visible reason. I’m investigating this right now

same here…

so 0.2.0 until then?

This is the issue I opened: https://github.com/mumoshu/terraform-provider-helmfile/issues/22

for me it just ran endlessly

creating on and on

There is something similar

https://github.com/mumoshu/terraform-provider-helmfile/issues/21

interseting

should i revert to 0.2.0 until further notice?

You can give it a try. But it also has some troubles like the spoiled state problem I described here: https://github.com/mumoshu/terraform-provider-helmfile/issues/16

so 0.1.0 is safest

probably a reason why it’s labeled as latest

It has almost the same issues with the state:)

Btw you might find this useful:

https://github.com/roboll/helmfile/issues/505#issuecomment-653684082

huh, seems broken a lot …

unfortunatelly, helmfile is great for helm/kustomize combo + patches…

Regarding the spoiled state problem, you might not face this. We’ve been using this provider successfully for quite some time. But when you encounter the problem it will bother you a lot)).

Did I understand correctly that there are two approches. One is using the tf plugin to execute the release AND pass variable data and one is just to output variable data from the tf infrastructure that you can later using with helmfile?

<ref+tfstate://path/to/some.tfstate/RESOURCE_NAME> and helmfile gets it from tf automatically by using output variables?

I would like to see a tf variant2 provider instead :X

Helmfile does not handle postdelete actions/hooks for deleting CRDs. So I have to use taskfile/variant2 for pre and post actions/hooks, but then I will lost a chance to use helmfile tf provider..

Did I understand correctly that there are two approches. One is using the tf plugin to execute the release AND pass variable data and one is just to output variable data from the tf infrastructure that you can later using with helmfile? I think so, haven’t tried the latter yet though:)

interesting nonetheless, happy i got new info thanks a lot for that!

i will try

What if tfstate is stored in AWS S3, Azure Storage ?

https://github.com/variantdev/vals#terraform-tfstate

Remote backends like S3 is also supported…

Oh noe, Azure is not there..

Would be nice to support AWS SSO and Azure MSI auth/authz ( + Azure Storage Backend ) like terraform does.

https://github.com/fujiwara/tfstate-lookup/issues/6

https://github.com/fujiwara/tfstate-lookup/issues/7

for me it just ran endlessly This is super interesting. I thought I have not changed any part of the provider that may end up something like that

So I fixed a potential deadlock issue https://github.com/mumoshu/terraform-provider-helmfile/issues/21#issuecomment-662178012

My issue was also resolved

Thanks a lot for the support and fixes @mumoshu, I can get the latest release and try again?

yes 0.3.2 is available for testing!

I might be doing something really wrong but when I do terraform plan with 0.3.2 I get Error: mkdir : no such file or directory

It has just: content = file(“../helmfile/helmfile.yaml”)

Sry my bad! Fixed in 0.3.3

ohh np. I just noticed your timezone, yikes sorry for bothering you

ah no worry! much better than leaving it until tomorrow morning. thx for testing

on it now

seems like the path to the binary is hardcoded to /usr/local/bin/helmfile but I have it installed via snap, even though the helm binary works on cli. And I think previous versions worked

I tried also to helm_binary = “/snap/bin/helm” but no success

wow really?

i hope i’m not doing something wrong though

actually, it’s not hard-coded. the default value for helm_bin is helm. For bin its helmfile

beginner with terraform and news with helmfile

do you have terraform.tfstate in your work dir?

yes

if you don’t have any kind of secrets in it, could you share it so that we can see what’s happening in the state

or perhaps your .tf file if seeing the state file doesnt help

let me see

ohh i had a file with a long hash generated there (probably from last tests) after removing it I just get error that it doesn’t find some file but that’s normal because they are relative to the helmfile not terraform

so appart from this it seems to be working I just need to change the structure a bit I assume

after removing it I just get error that it doesn’t find some are you talking about a file named like helmfile-8003d363e667d0a8cbc898f0866f02756c24328f56045aad1745a59deca0a14e.yaml generated by the provider?

aand I just saw the working directory directive and now after changin that it works, wow you thought about everything

yes sir tha’t’s the one I was refering to

and now I can see the diff, this is SO SO cool

glad to hear it worked for you!

and I can get values from terraform like ip address, sql instance address, domains from uptime checks and pass them on

this is exciting

thanks so much for your help and support

i’ll do my best to push to into production, test it and provide feedback

awesome! please feel free to ask me anything towards that, here or in gh issues, whatever works for you

@mumoshu hello again

I went ahead and tested forward the latest release and it seems the diff generated from terraform is correct indeed but it ends with This is a bug in the provider, which should be reported in the provider’s own issue tracker.

Everything seems fine and running the command manually directly with helmfile makes the deployment

Error: Provider produced inconsistent final plan

I’ve filed it already

https://github.com/mumoshu/terraform-provider-helmfile/issues/23#issuecomment-662831982

The initial one was: https://github.com/mumoshu/terraform-provider-helmfile/issues/22

Thanks for the update Andrey

I also tried the approach you suggested with tfstate and such

Under values I added

• “ref+tfstate://../terraform/terraform.tfstate”

no provider registered for scheme “tfstate”

oh wait, maybe I need to update my helmfile

oh that was it

now I get

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1172ece]

PANIC!

not sure what I’m doing wrong

@Andrew Nazarov are you using this by any chance?

Ah think I got it, I need to name variables separately, i included them like a .yaml, my bad

Sorry, haven’t tried yet this ref+tfstate approach

Would have went with the first one but seems it doesn’t go through

@Andrew Nazarov for me updating to the latest helmfile seems to have worked

ah nevermind it came back once I changed something else

Any luck with this Andrey or did you revert back to an older version?

Thanks. Error: Provider produced inconsistent final plan is really interesting. I’m now wondering why it doesn’t reproduce on my machine while I’m developing it

Maybe it’s only a warning rather than an error on my env?

2020/07/27 09:52:31 [WARN] Provider "helmfile" produced an unexpected new value for helmfile_release_set.mystack2, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .diff_output: was cty.StringVal("Comparing release=myapp2, chart=sp/podinfo\ndefault, myapp2-podinfo, Deployment (apps) has changed:\n...\n          prometheus.io/port: \"9898\"\n      spec:\n        terminationGracePeriodSeconds: 30\n        containers:\n          - name: podinfo\n-           image: \"stefanprodan/podinfo:1234\"\n+           image: \"stefanprodan/podinfo:12345\"\n            imagePullPolicy: IfNotPresent\n            command:\n              - ./podinfo\n              - --port=9898\n              - --port-metrics=9797\n...\n\nin ./helmfile-3adb8a9ba929668a0ea4b06dbeddd4cba9f0283b926b55c1c31f5e7176c497e1.yaml: failed processing release myapp2: helm3 exited with status 2:\n  Error: identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)\n  Error: plugin \"diff\" exited with error\n"), but now cty.StringVal("")

Maybe my tf is outdated?

$ terraform version
Terraform v0.12.13

Still unable to reproduce it even with tf v0.12.29

@Andrew Nazarov @Paul Catinean Could you share me exact config and steps to reproduce the error you’re seeing? I’ve tried this for an hour with various configs but have no luck so far. It works fine on my machine

also which resource type are you using, helmfile_release_set or helmfile_release, when you get the error?

Hi @mumoshu, yes sure I will try to give as much data as I can

• Terraform v0.12.26

• helmfile version v0.125.0

resource “helmfile_release_set” “staging” { content = file(“../helmfile/helmfile.yaml”)

working_directory = "../helmfile"

environment_variables = {
    TERRAFORM-TEST = "This came from terraform!"
}

environment = "staging" }

output “staging” { value = helmfile_release_set.odoo-staging.diff_output }

That’s the full output of the terraform apply command

Let me know if I can help with any extra details

@Paul Catinean Thank you so much! I’ll try reproducing the issue once again with it.

In the meantime, would you mind trying 0.3.5 which I’ve just released, to see if it gives you any difference?

For me, it did fix the warning I was seeing(https://sweetops.slack.com/archives/CB6GHNLG0/p1595811243157400?thread_ts=1595351319.054400&cid=CB6GHNLG0) which might relate to yours

Just wondering but how are you upgrading terraform-provider-helmfile?

That’s also a good question, since this is the first time I’ve been doing this and first time using terraform

This is 0.3.6 provider I think that I was using

I download the latest release available from github, unpack it

I installed go with snap

go version go1.14.6 linux/amd64

inside the unarchived release I do go build

and the resulting executable I place directly in to .tgplugins directory

and replace the other one, then run terraform init

I had no instructions on how to do it from the page so i assumed this would be the way

Same error i’m afraid using https://github.com/mumoshu/terraform-provider-helmfile/releases/tag/v0.3.5

and the process describe above

@mumoshu

@Paul Catinean Thanks a lot! The process looks fine. Probably there should be some unseen bug I’m not aware of. Will keep investigating..

If I can help with anything else like a screencast, helmfile description, even a call if needed I can do just let me know

Would you mind sharing me terraform.tfstate file that should have been created by terraform?

That was not included in the paste I give with terraform apply?

I have the .tfstate not sure how to get the one that should have been created by terraform? if not in the paste

Ah also i ommited my helm version if that has anything to do with it: version.BuildInfo{Version:”v3.2.4”, GitCommit:”0ad800ef43d3b826f31a5ad8dfbb4fe05d143688”, GitTreeState:”clean”, GoVersion:”go1.13.12”}

I thought terraform apply creates it on first successful run

it never went through I guess, so every time I do terraform apply I get the same diff

the tfstate diff and helmfile diff inside it

Got it! Thanks

Maybe I should try a simpler helmfile with less data inside? see if it’s from that or idk

Maybe. JFYI, this is what I’m using..

helmfile.yaml

releases:
- name: {{ .Values.name }}-{{ requiredEnv "FOO" }}
  chart: sp/podinfo
  values:
  - image:
      tag: foobar2abcda
  labels:
    labelkey1: value1

is this part of the automated test cases?

Not yet. Just that I’m still not sure how this can be (easily) automated. I’m running terraform init, plan, apply on this every time I change code(and manually tag/publish a new release)

could this have something to do with me trying multiple times to do a release through this that failed with previous versions?

Possibly. Maybe you have helm releases installed on your cluster even though terraform+tf-provider-helmfile considers it a fresh install on every terraform apply run?

i just removed the file from terraform and it killed the instance

So that’s working

Thank god the env variable works and killed only staging

Are you saying terraform apply after removing some helmfile_release from your tf file resulted in deletion? :slightly_smiling_face: you may already know but you’d better run terraform plan beforehand

ah yes, that was the plan to remove it from tf state and also the cluster so it’s “reset”

and now doing an apply from scratch

same issue

ah, could you share me your ../helmfile/helmfile.yaml?

Sure, i’ll just remove some sensitive data here and there

but it’s multi-tiered with some subfiles

it’s not a must if it takes too much effort! my current theory is this issue has something to do with complex helmfile.yaml and/or first install so just wanted to have more samples if i can have

Now that you mention, cert-manager has the installCrds hook which can add a lot of data and I did have issues in the past regarding the gitlab-runner

But at the same time I went to the respective file and did helmfile -e staging apply and it went through

I’ll check again

much appreciated! i need some sleep now but i’ll definitely report back tomorrow

thanks for you time!

I’ll also post feedback if I get anything new

Let me know if you need any more info

It seems that if I change the main helmfile and remove entire conditions it works

As soon as I change a value in the values.tmpl file and the diff shows diff of values vs diff of helmfile

that’s when it breaks

so the thing to try is to have a helmfile which has an external gotmpl values file

Quite some talk:) Don’t know where to start. Yeah, I’ve got a huge helmfile (about 2k lines of code) injected into fairly small helmfile

Indeed

This is the only distinction I could find. If I remove an entire release it works and it shows just the few lines of the release

If I change anything in the values.yaml.gotmpl and I change 1 char then I get that

I tried in multiple releases

I have one suspicion. Could it be these dots in diffs?

 ...
 bit, office365-editor, ConfigMap (v1) has changed:
 ...

These are taken from diff_output

could very well be, i also have changes in the configmap as well

How can we test?

Or this is the change I’ve got in the diff for the failed run

 - path                  = "helmfile.yaml" -> null

But actually these look like just a red herring

maybe there’s a standard go function that parses this where we can feed the diff into?

Found this string in the output. There is a mess in logs after this error, I’ll try to grab something sensible

but now cty.StringVal("Adding repo stable

I’ve tried to reconstruct the message from the mess

When expanding the plan for helmfile_release_set.mystack to include new values
learned so far during apply, provider "registry.terraform.io/-/helmfile"
produced an invalid new value for .diff_output: was cty.StringVal("Adding repo
...
a lot of text
...
but now cty.StringVal("Adding repo stable
...
a lot of text
...

So probably aforementioned dots matters))))

I’ll try to decipher more

think I have to start learning go so I can add some breakpoints and do some tests myself

I would like to debug too, but don’t have time right now at all

I’ve made some work to decipher the output and to me the value pointed in was cty.StringVal() is the same as the value from but now cty.StringVal()

so it’s essentially the same?

But that would happen in any diff

Doing changes in the main helmfile has no issue whatsoever

I can also see

• error = (known after apply)

Thanks! I think it’s fixed in v0.3.6 https://github.com/mumoshu/terraform-provider-helmfile/releases/tag/v0.3.6

// I’ll add some explanation on why it (may) fix the issue later but I don’t have time right now!

Testing right away

Seems to be working indeed, but the problem is I don’t see the diff for change of helmfille. Just the main one

While this works I won’t be able to sign off on the final changes though

Works for me too. @mumoshu huge thanks. And yes the diff_output is missing. Is this expected behaviour?

Was wondering the same

I’ve managed to fix it in v0.3.7. Sorry for back and forth but could you try it once again?

Will do right after coffee! :D

Same error i’m afraid

I think reproducing the error is key here to fix the issue possibly adding it to automation later. Did you try having an external gotmpl values file from the main helmfile.yaml ?

Sad

So I did reproduce the issue and managed to fix it on my env

The important point was that terraform calls plan twice and the two plan results must be equivalent

twice, interesting

actually you are right it does call it twice

I see twice the same output

Yeah. The first plan can optionally be done beforehand by calling terraform plan and storing the planfile to be reused later by terraform apply

if you didnt store/reuse the plan on apply, terraform apply seems to run plan twice, one for initial planning and the second for the verification

just to be 100% sure that nothing changed by the time you called plan to apply?

yeah

I guess that makes sense, I do plan once to check, and apply right after since it’s a small infrastructure that only I am managing

to me the two plan results werent really equivalent

helmfile diff runs a number of helm repo add and helm reop update in the very beginning, and the result of helm reop update isn’t reliable

ohhhhhhhhhhhh

I was just looking at that now

maybe an option on helmfile to output directly just the diff? that way it’s also backwards compatible idk

so i’ve tried to “erase” the messages from helm repo up in the last few commits, which solved the isseu for me

yeah that might be an option!

Strangely enough when I don’t change the vals yaml it always works

thats odd. but if it’s the same issue, you should see the error after a few more trials

what do you mean?

maybe i’ve misunderstood what you’ve said in
Strangely enough when I don’t change the vals yaml it always works

so you just said that terraform apply works when you have no change on your helmfile.yaml, right?

does the issue disappear if you set concurrency = 1 on your helmfile_release_set resource in your tf file?

let me check

nop, same error

not sure what else to try

Could you share me the full log obtained by runnign terraform apply with TF_LOG=TRACE TF_LOG_PATH=tf.log terraform apply ?

sure

Thank you so much for your cooperation It really helps!

more than happy to do so, it’s in my own interest as well, will help me a great deal in my future deployments

hmm I see there is also private info there like public keys and such

I can try to remove them just not sure if I get it all

you need it all or just the helmfile part?

I’ll join soon too. Not sure what to check though. I’ll grab the latest version

I was a bit worried that either gitlab runner or cert-manager with installCrds set to true might be the problem but seems not since it’s not included in the diff

Hey I was the discussion on the github issue, how has the color diff been resolved?

@Paul Catinean Thanks for sharing the TF_LOG!!

Apparently helm-diff result is unstable… From your log, I can see that the first helmfile-diff run shows changes on configmap and then deployment, where the second run shows changes on deployment and then configmap…

ohhh

I am using the latest helm diff plugin (which you contributed to in the upstream project)

I was even going to open a PR to propose updating the diff version on helmfile to a more recent one

As for me having the installCRDs parameters set to true on cert-manager resulted in huge diffs of essentially the exact same data just organized differently

what version of helm diff are you using yourself sir?

yeah. i thought helm-diff used a go map(hashmap) to collect resources to be diffed. afaik map entry order isn’t guaranteed/stable

ouch hmmmm

I’m using the latest version of helm-diff

number?

I have 3.1.2

3.1.2

hmmmm

then it’s related to my chart and how it produces the output or how helm diff parses that output

yeah probably… and it may even be due to that I’m so lucky that helm-diff somehow prints the changes in a stable order

probably i’d get the same error if add more releases and k8s resources to be diffed

It’s a plausable scenario

not sure how to proceed here

maybe i can “enhance” the provider to skip running helmfile-diff on the second plan

as it seems, if this is the case, there is too much volatility between repos added and diff plugin version that it can be wildly inconsistent

on the surface - yes

what about switching to kubectl diff as it was proposed in one helmfile issue? Will this help?

I’ve just faced the issue with the fixed version (https://sweetops.slack.com/archives/CB6GHNLG0/p1595423095095200?thread_ts=1595351319.054400&cid=CB6GHNLG0).

To me the most stable version is after it became two times slower. I’m sorry for finding this:))))))))

It seems this is the commit from which I built the image and I’ve been using it successfully since then

But yes, it’s slow

Thanks! So I’ve cut v0.3.8 https://github.com/mumoshu/terraform-provider-helmfile/releases/tag/v0.3.8

Starting this version, the provider should run helmfile-diff only once while apply, which should fix the issue…

Also since it reuses the output from the previous helmfile-diff run against same resource attrs + helmfile build result, terraform apply results in one less helmfile-diff run, which should make it a bit faster to run

to me it seems working (again).. the nature of this issue makes it very hard for me to reliably test. your confirmation is much appreciated as usual

I’ll double check. There were quite some commits that day

It now creates snapshots of helmfile-diff outputs under .terraform/helmfile/diff-*.txt. Ideally the provider should clean it up after a successful apply. I’ll add that later if we confirm it’s actually working

testing now sorry work came in

HAPPY DAYS IT WORKED!

And on top of that it was indeed faster

Thanks for sticking with me on this one @mumoshu

I’ll do some more testing on the staging environments and after some extensive testing will try to move them into production

woop woop it also sends environment variables properly so this is great

It did give a strange diff with some strange distribution of removed/added (a lot of red) but it did go through

Works for me too. I’ll keep an eye on it and test more cases

@mumoshu you rocks)

@mumoshu Do you have an article about your time management?))))))

true that

Just as info for future version, not urgent since this seems to work indeed. The diff shows addition/removals on top of the diff which shows the helmfile diff and can be confusing

Should I create an issue with this?

awesome! thanks a lot for your patience and support

re: confusing diff output, i got to think that it’s unavoidable due to the nature of terraform plan

it shows the diff for the latest helmfile diff output against the previous helmfile diff stored in the tfstate

maybe we’d better use helmfile template instead so that terraform can use that to show changes in manifests?

diff_output can be just removed, or renamed to changes so that it is clear that terraform is showing “diff of changes”, rather than changes in the manifests

after asking the question myself it did make sense that it shows this indeed

that’s not a bad idea, I didn’t even use helmfile template until now but it does make sense indeed

I’m facing a strange issue when tf apply for the env ends up with the error map has no entry for key "securityService" . Like there is no such key in values. But actually it’s there and all prior runs were successful. Cannot understand what went wrong. Probably I’ll file an issue when I grab more information

It seems that it doesn’t see the the values file somehow

Found it. A values file was malformed - a colon was missing after one key not related to the key from the error at all.

Re: diff_output, I was able to make helmfile diff output as the whole diff shown in plan, not “diff of diffs”. I’ll cut a release later

Re: the “map has no entry” error, glad to see you fixed it so to be extra sure, we don’t need to fix helmfile/provider for that?

no, everything is working great so far:)

Awesome! Thanks for all the feedbacks.

checking out 0.3.9 now

FYI: v0.3.9 has been released with the diff fix

I’m like a kid on christmas with each delivery

Removed the intial release and starting a new one

Quick question while I run this. How would one be able to transfer data structures such as hashes/maps/dicts or lists from terraform to helmfile ? some decode on the helmfile part in the gotemplate?

I’d suggest using string interpolation w/ jsonencode on the terraform side

Like:

content = <<EOH
values:
- whatever: ${jsonencode(locals.whatever)}
EOH

and then on helmfile part jsondecode I assume we have a gotemplate function

to transforms maps/lists passed from tf to helmfile?

yeah

I assume you can only pass values as env variables? wth an external helmfile?

How about this?

content = <<EOH
values:
- whatever: ${jsonencode(locals.whatever)}
---
{{ range $_, $item := .Values.whatever }}
...
{{ end }}

or

values = [
		<<EOF
{"whatever": ${jsonencode(locals.whatever)}}
EOF
	]

content = <<EOH
{{ range $_, $item := .Values.whatever }}
...
{{ end }}

have to try both

if they work with external helmfiles then great

the latter would work with external ones

so I think 0.3.9 seems to work, I see the initial output with helmfile repo adding etc, the main helmfile and then the diff in the actual values.yaml within the terraform diff

Doesn’t show with colors or just the original diff but seems to accurately show the difference

ah so I disable coloring recently cuz i thought it would conflict with terraforms’ own coloring

I’ll have to sit and think a bit what’s going on, today is one of the slowest days ever, need more coffee

But I’m inclined to (re)enable coloring by default, and add an option to disable it instead

https://github.com/mumoshu/terraform-provider-helmfile/issues/24

So when changing a value in the values.yaml file

So it somehow shows the diff before this operations, the version now and then the diff from now

The last part is totally accurate just not sure why I see the diff from before this deploy, or maybe I’m just confused

maybe you’re saying that apply_output is confusing?

Actually apply output that’s at the very end (all in green but np) that’s perfectly accurate

apply_output is showing that the previous apply_output(contains previously applied “diff”) to be recomputed after apply

So I made that deployment now all I’m changing is strictly LIMIT_REQUEST: “” and the configmap will be changed along with it

just changing 0 to 1 and backward

And i see the diff with 0 to 1 and now the 1 to 0

In theory shouldn’t it show just the latest change? - - - the current value + the changed value

ahhh….

I believe that’s due to stale cache. I used sha256 hash of helmfile build output in the name of cache files, which is used to run helmfile diff only once to avoid the error we had before

"registry.terraform.io/-/helmfile" produced an invalid new value for
.diff_output: was cty.StringVal("Adding repo stable

any changes to values.yaml won’t change the hash value so it would definitely happen

(working on the fix

but this is more then great and functional, thank you so much

This is just feedback if you want to push it to another level it’s more than helpful

when changing a value in the values.yaml file should be fixed in v0.3.10

🥰🥰

Will test soon!

Not exactly sure why I keep getting “Error: the lock file (Chart.lock) is out of sync with the dependencies file (Chart.yaml). Please update the dependencies”

Could be an issue in my chart but I didn’t change it between tests

Running helmfile alone seems to be working though

So maybe something related to the provider

Thanks! Interesting - could you try running helmfile diff, template, build and apply alone? The provider just delegates everything to those 4 helmfile commands. So as long as those commands work, the provider should work…

it seems helmfile template fails indeed i have to find out why

it’s most likely because the chart is being downloaded and needs to have the dependencies updated

yeah that may be relevant. helmfile template updates repos, run helm dep build on local charts only, run helm fetch on helm 2 only, and finally run helm template.

this is remote chart so not sure why it refers to a /tmp/ downloaded version of the chart

just to be extra sure - you are using helm3, right?

perhaps helmfile v0.125.1 should work differently https://github.com/roboll/helmfile/issues/1377

but not sure why it doesn’t fail on other helmfile commands for you

helm 3 yes sir

is it a public chart, or private one?

helmfile version v0.125.0

public

could you share me the chart location? i’ll try reproducing it myself

I did, it’s public yet not publicized so to speak so pm-ed

it has a few dependencies which granted I need to update if I use it locally but remotely i did not have to, not sure

Updated the latest helmfile and still the same thing

Thanks. I reproduced it! So the root cause seems to be due to that the chart has outdated dependencies stored in Chart.lock

which needs to be updated by running helm dep up before publishing the chart, i think

but this is a chart issue so Helmfile should anyway have a way to graceful handle it

Maybe helmfile can just warn about the outdated chart deps and proceed

instead of failing and forcing you to block until the chart is fixed in upstream

rethinking it for a while, i’ve fixed it as a part of https://github.com/roboll/helmfile/pull/1400

so it doesn’t run helm dep build on fetched charts anymore, which turns out to be unnecessary and safely skipped

wow that sounds great, also thanks for the help on the chart that’s outside of helmfile and the provider. I’m getting free consultation now

I need to document on how to handle those dependencies, maybe remove the .lock or update the dependencies as you said or pin an exact version

It’s hard to follow this thread. But it seems with not the most recent version for the subsequent runs if nothing is changed tf apply prints out the apply result from the previous run and not related to the current one. Will investigate further

@Andrew Nazarov AFAIK terraform apply outputs are computed from the latest tfstate. The helmfile provider does not “reset” the apply_output when no run so next terraform apply would still show the previous apply_output. Is this what you’ve observed?

Maybe I can try to make the provider “reset” apply_output on read, so that it’s still shown in terraform output on successful apply, but will reset to an empty string on next terraform apply.

Yeah. For the vast majority of outputs this behaviour seems natural. But I’m still not sure about apply_output though. Maybe we can keep it as it is. Let me file an issue and we can think about it for a while and continue discussing there

It seems diff output is also shown

ah that’s too bad. the provider does have some code to “reset” diff_output on read. so this strategy won’t work for apply_output as well

next best workaround should be to let the provider show diff that “sets diff_output and apply_output to empty on terraform plan after the successful apply`

https://github.com/mumoshu/terraform-provider-helmfile/issues/26

Not an urgent thing though:)

thx!

Maybe I’ll find time to play with it by myself. But I need to understand how this provider works now since you’ve made a lot of changes recently. And I’m not quite sure I understand all tf internals right.

im still learning it, too. i’ll try to answer questions if you have any also, i recently started adding comments on code where i find difficult to understand at glance. so that may help

I’m holiday for the time being but I’ve seen the commit message of the latest helmfile and it works along with the latest provider thanks so much @mumoshu

this? https://www.terraform.io/docs/configuration/expressions.html#for-expressions

Yes. I’ve read that before. I’m going to reread it again for fresh insight

Very Go inspired it seems, but it’s own functionality I guess.

The source value can also be an object or map value, in which case two temporary variable names can be provided to access the keys and values respectively:

[for k, v in var.map : length(k) + length(v)]

Finally, if the result type is an object (using { and } delimiters) then the value result expression can be followed by the ... symbol to group together results that have a common key:

{for s in var.list : substr(s, 0, 1) => s... if s != ""}

the => operator in my .net world is a lambda typically, so also was a bit confused on that here

The type of brackets around the for expression decide what type of result it produces. The above example uses [ and ], which produces a tuple. If { and } are used instead, the result is an object, and two result expressions must be provided separated by the => symbol:

{for s in var.list : s => upper(s)}

the flatten and setproduct pages have detailed examples on how to use them with for_each…

• https://www.terraform.io/docs/configuration/functions/flatten.html#flattening-nested-structures-for-for_each

• https://www.terraform.io/docs/configuration/functions/setproduct.html#finding-combinations-for-for_each

IMO, the most important paragraph in all the docs is this:
However, unlike most resource arguments, the for_each value must be known before Terraform performs any remote resource actions. This means for_each can’t refer to any resource attributes that aren’t known until after a configuration is applied (such as a unique ID generated by the remote API when an object is created)

can get yourself into all sorts of trouble if you do not carefully consider how all the attributes in the for_each expression objects may be specified

Definitely aware of that

Gone done that rabbit hole. I’m just more confused on advanced expressions to manipulate an objects structure

Basically per that blog draft I showed I previously had to do key based lists, which is clunky. I figured out how to change a list of objects by using this to provide the key, but still not fully versant on using the flatten, nested for etc. trying to wrap my head around it as there is no “schema” doc + examples, it’s kinda all mixed with a general doc like what you showed which i find a bit confusing

the flatten example in the docs is doing exactly that

    for subnet in local.network_subnets : "${subnet.network_key}.${subnet.subnet_key}" => subnet

the left side of => is the key, and becomes the ID for the resource. it must be unique or you have duplicate resource IDs, which does not work

you can dynamically construct that key, exactly as they show there

your yaml is a bit confusing to me, it’s a list of maps of maps, where with terraform it would be cleaner if it were just a list of maps. i.e.

users:
  - name: foobar1
    email: [email protected]
  - name: foobar2
    email: [email protected]
  - name: foobar3
    email: [email protected]

and you’d then loop over it with for_each:

for_each = { for user in <users> : user.name => user }

you could of course mimic that exact structure in your data source, and just give for_each the map of maps… note there are no lists here, so no need to use for to build up the map

users:
  foobar1:
    name: foobar1
    email: [email protected]
  foobar2:
    name: foobar2
    email: [email protected]
  foobar3:
    name: foobar3
    email: [email protected]    

and the for_each:

for_each = <users>

i tend to prefer the former, a simple list of maps

in both cases, each.key is the user name, and each.value is the map of user attributes. and you can dot-index into the map, e.g. each.value.name and each.value.email