#terraform (2022-12)

which input list are you exactly referring to?

What do you mean?

you say “if I use the same as input to the list”

which list is that ?

I tried to use the same for loop as for the output, where I got “LIST OF UPLOAD BUCKETS”

But it generates a map, not a list with only the buckets id’s

are you saying this line doesn’t work for you?

bucket_name = random_pet.upload_bucket_name[each.key].id

if yes you can use something like

values(module.my_module.upload_bucket_ids)

of course change the my_module

I tried it like this with and without the tolist()

av_scan_buckets = concat([
  module.temp_bucket.bucket_id,
  module.db_objects_bucket.bucket_id
  ],
  tolist(values(module.upload_bucket.bucket_id))
)

Got this error:

│ Error: Unsupported attribute
│ 
│   on antivirus.tf line 49, in module "s3_anti_virus":
│   49:     tolist(values(module.upload_bucket.bucket_id))
│     ├────────────────
│     │ module.upload_bucket is object with 2 attributes
│ 
│ This object does not have an attribute named "bucket_id".

Makes sense module.upload_bucket got 2 objects as I’m creating 2 buckets with for_each = toset(var.upload_buckets)

Solved, this works

av_scan_buckets = concat([
  module.temp_bucket.bucket_id,
  module.db_objects_bucket.bucket_id
  ],
  [ for k in toset(var.upload_buckets) : module.upload_bucket[k].bucket_id ]
)

I think they have drift detection and some other features but afaik it’s a saas Atlantis offering

i thought atlantis was a saas… you have to install atlantis on your own machines? (real or virtual)

Atlantis is self hosted

Ah good to know

Im not aware of a centralized place. Depending on why you’re looking for this info, another piece to consider is that you may be using Name-Prefix for some resources (which can have a much smaller length limit, for example).

“can have a much smaller length limit” if prefix is used. do you have a docs link and/or example for this?

Im on mobile right now, but if you look at AWS load balancers you’ll find right now in Terraform the prefix can be at most 6 characters (last I checked).

It’s useful to supply a prefix instead of a complete name because LB names are unique (per account?), and you typically want to spin up a new load balancer before tearing down your old one with Terraform lifecycle create-before-destroy.

Yes. I do like prefixes. Thanks for the tip.

I wonder how Cloud Posse and null/label deal with these restrictions. Where do you make sure that the null/label output id fits as name for a resource and do you use prefixes or do they become irrelevant with null/label? @Erik Osterman (Cloud Posse)

no, sadly

3.5” floppies

dont have floppies

consul?

gitlab?

I’ve never used consul. sounds new to me. I’ll try that

thanks

some context, thats for my personal lab. I’m creating vms on libvirt/kvm and installing rke2 on that. still learning terraform

i mean, if it’s just you and you’re keeping it all local, you can encrypt the state with sops and commit it to your repo lol

Terraform Cloud state storage is free

on-prem? It probably depends what technologies you have available. If you have something that can emulate a local filesystem, the default backend would be simplest

Terraform Cloud state storage is free Is it faster/slower than S3? S3 feels really slow to me

some context, thats for my personal lab. oh, right, TFC is not on-prem, but it’s totally suitable for a personal lab, provided it’s not airgapped.

We don’t use TFC for stage storage, so I cannot comment. We use S3 and any slowness seems to just be more closely tied to the number of resources under management within and given root-module and the number of parallel threads.

Seems like a bug with our example. Our tests use the terraform version minimum set in the examples versions.tf file.

I wonder if you’re seeing this issue since you may be using a newer version locally?

Try using 0.13.7 terraform, just to see if you can reproduce this issue the way our tests would

https://github.com/cloudposse/terraform-aws-vpc-peering/blob/c9316cb2f9cd6e472808f0c95ab930e62f18de34/examples/complete/versions.tf#L2

Hey, I haven’t used this module before so I’m not a 100%, but what you’re after may be configured through the providers: https://github.com/cloudposse/terraform-aws-transit-gateway/search?q=provider&type=code

https://github.com/cloudposse/terraform-aws-transit-gateway/search?q=aws.prod

Seems to be related to this - https://github.com/cloudposse/terraform-aws-lb-s3-bucket/blob/master/main.tf#L1

But module.this.enabled value is available in the defaults so I’m not sure why it’s complaining.

That is odd. This error is everyone’s least favorite…

Could you open a ticket with all of your inputs and a sample of your hcl? It would help if it was possible to reproduce it

This issue comes up when you pass in another uncreated resources attribute to a module. It’s the first time I’ve seen it for the module.this.enabled flag tho

thanks, I’ve raised a bug here: https://github.com/cloudposse/terraform-aws-alb/issues/126 hope that’s OK.

Yeah, it works if the bucket is created before you run terraform apply with enable logging and the custom bucket ccess_logs_s3_bucket_id = aws_s3_bucket.alb_s3_logging.id

@RB this works when the literal name of the S3 bucket is used. I’ve updated the issue.

This issue is common and one of the most hated in terraform.

one reason we haven’t hit it is because we create a new logging bucket per alb by not specifying the access_logs_s3_bucket_id and allowing the module to create the bucket for you

And if we want a shared s3 bucket across albs and pass it in, then we would create that s3 bucket in its own root terraform directory and then it’s created, then go to the root terraform directory for our alb, retrieve the s3 bucket from a data source, and then the alb would apply correctly

One question that comes up, if you’re creating an s3 bucket for logging and passing it in to the alb in the same root terraform directory, then why not let the module handle it?

Thanks, @RB. Basically, I’d like to have fine-grained control over the bucket name. I did intend to let the module handle it but have 2 environments named prod in different regions and ended up with a name clash. Yes, hindsight… :)

You should take advantage of our null label. All of the modules use it.

https://github.com/cloudposse/terraform-null-label

{namespace}-{environment}-{stage}-{name}-{attributes}

Namespace=acme
Environment=use1
Stage=dev
Name=alb
Attributes=[blue]

These will then get joined by a dash to form a unique name

acme-use1-dev-alb-blue

i haven’t seen a “great” solution yet… i feel things like sops are the best way, if only because the secrets remain encrypted in terraform state. especially since remote state is synced to local disk in plaintext…

otherwise, if you can, push secrets to a secret store out of band of terraform, use the “path” to the secret in terraform, and pull the secret from the store in your app

Hmm i did not know you can keep secrets encrypted in tf state using sops.

Do you have an example or docs i could look over?

i didn’t mean to imply it was “integrated”… to keep them encrypted, you have to reference only the encrypted strings in tf configs. so basically tfstate becomes the secret store, and you pass the encrypted string to your app, and decrypt the string in the app

https://github.com/segmentio/chamber

@David Karlsson chamber saves to ssm/secrets manager but if you use a data source, it will still grab the secret and put it in plain-text in (an albeit encrypted) tfstate

edit: I use chamber today and like it a lot. Did not mean to sound dismissive. Thank you for sharing.

@loren how do you structure your sops today ? do you save all your secrets in ssm/secrets manager, then pull it down, encrypt it with sops, and then save the encrypted string into terraform? then when you deploy your app, your app knows to retrieve the key (kms?) to decrypt the sops key for the app?

I havent done it personally but segment describes a slightly different way of using chabmer in prod, at least if you run containers… … 1 sec

In order to populate secrets in production, chamber is packaged inside our docker containers as a binary and is set as the entrypoint of the container

https://segment.com/blog/the-right-way-to-manage-secrets/

no no, sorry, i haven’t had cause to use sops myself in this manner. i keep waiting for something native in terraform. i guess in general the only idea i’ve seen is to keep the secrets out of terraform, by encrypting them with something like sops, or to use an external secret store and manage only the path to the secret in terraform. either way, you are handling the secret itself in your app, instead of passing the secret value to the app with terraform

@Michael Dizon thank you. I’ve left a comment if you an take a look at it

on it

@Andriy Knysh (Cloud Posse) do the tests need to be kicked off manually>

started the tests

strange, it’s still not picking up that ami

i updated the PR to use an ami from us-east-2

approved and merged, thanks again

np!

Do you need to change length(var.dns_servers) > 2 to length(var.dns_servers) <= 2 ?

nope, i think im mistaken.

I always get the condition confused.

looks okay according to this …

https://developer.hashicorp.com/terraform/language/values/variables#custom-validation-rules

What is the current default value youre sending as an input ?

the exact one above

like I didnt add another dns server or different ones… if thats what you mean?

sorry, I mean I see that the default is a single dns, but what are you passing as the input ?

are you just using the default and the default isnt working?

yes, thats it

the other issue i see is that the validation should probably be length(var.dns_servers) <= 2

haha, i think I had it right in my first comment. You want to allow only dns servers with a count of 1 or 2

correct yes!

try the suggested change for the validation and it should work

ok… lemme try that

Worked! oh man I feel so dumb… ok… ty so much!

it’s tricky. I suppose we can think of the validation needs to be what we expect. If it’s true, continue. If it’s false, then the error condition will be thrown.

right… the end state we want in other words ya

I often think I need the validation to be what I do NOT expect in order to get the error condition. I have to keep reminding myself that the validation is the if (true) {} block and the error_condition is the else {} block

check out this tutorial, the TLDR is that they use the -var flag to switch between blue and green apps https://developer.hashicorp.com/terraform/tutorials/aws/blue-green-canary-tests-deployments

Thank you!

sounds like a connectivity issue and less a terraform issue, have you tried in the AWS channel?

security groups what ingress and egress do you have

the farthest I’ve gone in this direction is setting the account and region in the cloud provider config block in terraform. And terraform errors out if you are running it anywhere else.

Use state locking.

@Chris Dobbyn state locking doesn’t prevent state from the wrong account from being written to state storage, it just prevents collisions that would otherwise occur when concurrent state commits were happening.

Yep I miss read, denis’s answer is correct.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs#allowed_account_ids

Obviously this is dependent on aws provider.

This is a different ask.

@Simon Weil thanks for the PR, please see the comment

Thank you, will do it tomorrow

Updated the PR as required, although it did nothing And opened a new PR for the deprecation warnings

Is there anything the PRs are still waiting for? can they get merged?

commented on https://github.com/cloudposse/terraform-aws-sso/pull/33

Thanks, will attend to it next week

I tried the requested name change but it failed, please tell me what is the next step, what do you want to do next?

see my comment in the PR

please address the last comment and it should be ok, thank you

Done, tested and pushed, tell me if anything else is needed

approved and merged, thanks

Thank you!

Next PR is ready for review/merge

Any thoughts on the second PR?

this one https://github.com/cloudposse/terraform-aws-sso/pull/35 merged, thanks again

Great, thank you

data "aws_ami" "latest" {
  most_recent = true
  owners      = [var.owner]

  filter {
    name   = "name"
    values = ["${var.default_ami[var.ami]["name"]}"]
  }
  filter {
    name   = "image-id"
    values = ["${var.default_ami[var.ami]["ami_id"]}"]
  }
}

may be this https://stackoverflow.com/questions/65686821/terraform-find-latest-ami-via-data

Thanks but that part works (see “all good so far”) The problem is in paragraph 2.

Sorry, I misunderstood. Maybe you can manually taint the instances, forcing a replacement, but it’s not the most correct solution for sure

yes well that’s what I describe in paragraph 4

thanks for trying though

Is there any chance your EC2 instance is in an ASG? If so you are probably only updating the launch template.

No @Soren Jensen no ASG in this stack!

Can you upload your code and specify which version of the aws provider you are using?

I’ll try to pare it down

preventing the canceling of the CI job from potentially borking TF state what does this mean? you mean you cancelled your ci job and then how did your tfstate get borked? if you version your tfstate, then couldn’t you go back to the last version?

@RB this wouldn’t be me canceling the job.

our terraform is often embedded into the service repos and may run with application pull requests and I’m a big fan of this approach. developers may occasionally kill a job and not realize that a tf apply is in the middle of it. if whatever tf was doing at the time was very involved, like building a database, or doing something in production, this could actually cause a serious issue. and apparently we’re not the only people to have encountered this issue.

state is versioned, but rolling back state sounds scary. I’m not convinced that will be something to rely on and a seamless workflow that wouldn’t ultimately require manually blowing a bunch of stuff away. just like using destroy is not a straightforward operation.

what would be better is if there was a way, as TFC does, that I could implement a graceful shutdown or something. that doesn’t need to be the solution, but it’s one thing that comes to mind. but of course I don’t control the CI’s backend so I’m assuming if a user hits Cancel on the UI that the container runtime is going to send a KILL to every container running, of which terraform apply will be one.

developers may occasionally kill a job why do they do this?

because they’re human. why does anyone make mistakes. you make mistakes lol.

no no, I understand that they occasionally kill a job, but I’m wondering what is the reason they feel compelled to ssh into a server and run kill -9 on the terraform process?

is terraform taking too long?

do they think they made a mistake and want to save time?

there must be a reason other than that they are human

oh – ha, sorry.

because they realize that something is wrong with the job.

for example it could be in a lower environment.

and they might realize “oh crap – actually that’s [the wrong value]”

and cancel the job. I’ve done this before, although usually it’s with crappy development workflow.

they might not be aware that there is a step in the CI workflow that is running terraform

like for example they may be running something in a lower environment and iterating over and over again

how long does the terraform apply take for the jobs that developers are likely to kill ?

I have no way of tracking that, although I wish that the CI had a graceful shutdown option.

this is circleci

in terraform automation, you can usually exit terraform runs early, even gracefully, and that is similar to a kill <task> instead of a kill -9 <task>

if the reason they are doing this is that they mess up and the feedback loop is too long so they try to kill the process in order to rerun it with the correct param, then the solution is to reduce the size (number of resources managed) of your terraform root directories

it wouldn’t be on the terraform end

other than that, all you can do is educate your developers to not do this or put in a policy within circleci to prevent any shutdown of the terraform apply task.

why they are canceling

they’d be canceling for some reason due to their application I think, not realizing that tf is running as a sub-workflow

not the only ones that have this issue:

https://circleci.canny.io/cloud-feature-requests/p/prevent-a-job-from-being-cancelled-when-the-workflow-is-cancelled

I could use a dynamic step to trigger a centralized pipeline maybe

but they only allow one dynamic step per pipeline lmfao so it’s like if I use that one dynamic step for my little terraform hack that’s a pretty poor reason for doing so

We deploy all terraform code to prod accounts from GitHub actions. We haven’t taken action on avoiding a deployment to be cancelled. So far we haven’t had any issues.

@Soren Jensen Thanks!

so you handle linting, plans and everything from github actions?

What are the costs like? Are you deploying multiple times across a good number of services daily or infrequently on a small number of things or?

We do use Gitlab. Few times where we had to cancel a deployment in the middle, we had TF state issues….which we simply unlock using force unlock from our laptops or simply go into DynamoDB and delete the lock

Nice one @Sudhakar Isireddy thanks.

https://developer.hashicorp.com/terraform/language/functions/values

https://developer.hashicorp.com/terraform/language/expressions/splat

also, why do you need resource "null_resource" "output_ids" and print it to the console?

you can use another output which would output key = value pairs

Thank you! That worked. I most definitely do not need the output_ids null resource. I was just using that to illustrate what I was trying to do.

The tags are defined in the yaml

yes but how can i re-use the same tags over many monitors without duplication

u mean the catalog/monitor.yaml

https://github.com/cloudposse/terraform-datadog-platform/blob/8ff840e0d166e82bd0c88b2099a161db472fea87/modules/monitors/main.tf#L66

You can use yaml anchors

After looking at the code, it does look like it reads from var.tags. if that doesn’t work, i would file a ticket with your inputs

module.this.tags is filled by var.tags

hmm i see, looking for this i found: https://github.com/cloudposse/terraform-datadog-platform/pull/36

I just tried, if i remove the tags: from my monitor.yaml it inserts the tags from the tf.. Thats not very clear from the docs

Agreed. We’re always looking for contributions. Please feel free to update our docs :)

Where are you expecting to find the cluster? Host;port. Can you post the config for your providers?

Related issue https://github.com/cloudposse/terraform-aws-eks-cluster/issues/143

You probably want to use the 2.x version of the eks cluster module to get around that issue

@RB you were right! using 2.00 version and setting create_security_group = true fixed the issue. Now using latest version of the module. Thanks for quick support!

There’s many (slightly) different ways to go about this. When comparing any of them, I would think about

• keeping your IaC DRY (typically resources are same/similar across environments)

• how do you add a new account?

• how do you add a new service/project?

• how do you avoid drift?

The current environment is managed through aws console. That is one of the reasons why I’m moving to IaC using Terraform. I’m currently running an application using the following resources (Beanstalk, RDS, Route53, Cloudfront). Now do I create a separate directory for these services and another directory for the modules (VPC, Subnets, Security)?

Not sure what you are trying to do but here are some tutorials on getting started with terraform cloud. https://developer.hashicorp.com/terraform/tutorials/cloud-get-started

Hi @Fizz, thanks for your support, actually im planning to automate the tf deployment, I have my code in my local machine, which i want to push it to tf cloud and provision the resources in aws.

this is my requirement

So assuming you will upload your code to a source code repository like GitHub, the tutorials cover your use case. I’d suggest going through them.

And even if you plan to keep your code local, the tutorials get you most of the way there.

@Fizz thanks, i am going through the document, hope it may help me,. i will check and come back if i stuck

maybe share the task def json?

You may want to use the module to generate the container JSON

https://github.com/cloudposse/terraform-aws-ecs-container-definition

Then you can use it in your ecs task definition like this

  container_definitions = module.container_definition.json_map_encoded_list

I’ve seen this before and haven’t met anyone using it yet. Have you tried it?

I see very few forks and stars so I’d be hesitant to use in production.

https://github.com/AzBuilder/terrakube/network/members

It does look like possibly bridgecrew uses it according to the list of forks

It does look pretty nice. It hit our radar back on May 27th, but haven’t had a chance to look at it.

https://terrakube.org/wp-content/uploads/2022/04/Runs.png

you can quickly test it with docker-compose and Postman here:

• https://github.com/AzBuilder/terrakube-docker-compose seems promising for people who use k8s to manage their operations AKA operations k8s cluster

There is a desperate need for a feature-rich, production-grade taco, I have absolutely no doubt about that. And in fact I’ve committed to deprecating the taco at my current shop because the benefits are simply not justifying the cost. I’m not crazy about the idea of running k8s for terraform because I’d rather do it with a much simpler container scheduling platform like ecs fargate, but I genuinely hope terrakube enters CNCF.

Have you considered the other TACOS? We’ve had a lot of good experiences with spacelift

(Jonas was at Bread)

We’ve had a lot of good experiences with spacelift @RB As Erik mentioned I was at Bread and worked with Andriy to implement spacelift there. There are nothing wrong per sé with the tacos it’s just that aside from state and workspace management I’m not sure what benefit they truly bring? Yes they can sequential and parallel stacks, yes they can run arbitrary shell commands against jobs, yes they can do drift detection and they give a convenient hook for governance and auditing, but actually these features aren’t really worth that much right now. And they’re definitely not worth 500K - 1M USD which is about what the average big corporate TFC contract is.

I should reach out to env0 and spacelift and see what a contract with them for between 10K - 15K applies/month would be.

@Jonas Steinberg happy to chat again with a previous customer. Let’s connect offline.

We don’t charge on the number of applies/month so it won’t be apples to apples, but I can tell you the existing customers that we have who migrated from TFE and TFC are quite happy.

It definitely won’t be 500K - 1M USD.

Grab some time with me at https://calendly.com/ryan-spacelift or anyone else interested. Happy to provide transparent and simple pricing and discussion on this.

Thanks Ryan.

The attributes are a list. Wouldnt you have to do this?

  attributes = [each.value]

You know, thats a good call. …curious as to why its not mad at that.

Ah, because I was actually using tenant.

I changed it for this because I was thinking about doing attributes. That said, am I going about this the right way?

I think you’re doing it correctly. You just need to pass in the subnet AZ as an attribute

I think that’s where I’m struggling. How would I go about grabbing the AZ w/ the subnet from the dynamic subnets module?

Check the outputs first by using this

output subnets { value=module.subnets }

Then see where you can collect the AZ

If it’s not outputted then you could use a data source

Ahh, great call on the data source. I typically forget about using them. I’ll give that a whirl.

Thanks a bunch.

That worked like a charm. Thanks for that suggestion!