#terraform (2022-11)

Is the sqs queue ephemeral or persistent?

Persistent

if it’s persistent, then it sounds like terraform would make more sense

you may want to ask why your manager wants the app to manage sqs. I’d pull on that thread to see what problems he’s trying to workaround.

To avoid having to run terraform before a deployment is his concern.

how does he feel about vpcs, subnets, eks clusters, peering, s3 buckets, etc

or what about shared infra vs application infra

i suppose it could work to build it into the app, but it’s more logic to bake in

it would be easier to just deploy the infra required once and then deploy the app as many times as you like

less code to maintain, less likely a chance to break something, and now you no longer need sqs:create perms on the iam role of the app

also less code to unit test

The flip side to this question is how do you version/track the updates when breaking changes are introduced?

I am a firm proponent of using release tags in module repos and then using those tags to install the module. I’ve been burned in the past by using main or something similar and then having a breaking change bring plan and apply to a halt.

Thanks everyone, appreciate the answers.. For those who keep up to date at all times, do you dependabot or a similar service?

ManagedKaos, we got release tags on everything, that’s why we are falling behind, at the moment I’m not notified on new releases

Dependabot

Renovate here: https://github.com/renovatebot/renovate

awesome write up, that 200 connects/sec limitations seems interesting. I also wanted to mention that you should take a look at AWS session manager so that you don’t have to create a bastion open to the public internet

i generally avoid squash merge, and just accept the merge commit so git sees the same commit is present in both the pr branch and in the main branch, but i don’t think that detail is the primary question here

we currently run a plan as part of the pr review, then merge, then run apply

but i’ve seen enough problems that only occur on apply, that i definitely see the value in the way atlantis does it by default, and run the apply just before merging the pr

that way you have a chance to fix any apply-time issues before merging the work

Running apply before merge is what i’m leaning towards because of that very reason. It’s fairly often that we have to do some amount of tinkering before the apply is successful.

In your current flow, are there situations where other developers working within the same project are blocked because the terraform code on the main branch has not been cleanly applied? How do you manage this problem?

sure, you can get blocked by a broken apply either way. in both cases, it is important to keep the “contributor” (developer or otherwise) engaged in the result and any follow-on actions

terraform apply is almost inherently serial. so in either approach, before applying, any pr needs to be up-to-date with fully resolved conflicts, and have a clean, approved, understood plan

Yeah that’s a fair point about it being serial. I guess the apply before merge route makes that more intentional.

Did you all ever experiment with using a merge queue for applying changes?

I have not, couldn’t get my brain around them, seemed too complicated

You may be able to limit the impact of the serial nature by splitting things into lots of states… One state may be blocked due to an apply-time issue, but devs working other, independent states may not be

agreed, the smaller the tf projects the less of a problem this becomes. Unfortunately we have a fair number of large, legacy projects in a monorepo that need to be broken down

Heh, it’s hard to find a good balance between small states and a workflow that makes sense and is manageable for your team

I was just reading a blog by slack on how they do it, and it sounded like a lot of large states lol, and still needed a lot of home-built tooling to manage it all

https://slack.engineering/how-we-use-terraform-at-slack/

I read this one last week as well, it sounded pretty painful actually

but that’s probably because I can’t help but to think of my experiences with Jenkins while reading their blog

Yes, I noped pretty hard. But also, I feel a lot of their pain lol. These workflows are hard, no matter the tool!

So true!

Probably use terraformer

Best thing to do is to use terraformer but dont commit its output and wipe your hands.

For example, if you want to terraform all of your hosted zone records, you do not want to dump all the records and put it in main.tf and call it done

You want to put each record with its respective service and use the output of terraformer as a template

@Arash Bahrami does that make sense?

@RB wow thanks I’ll try it out also thanks for the advice, yeah I’d rather a clean and understandable directory with each service in different files

You should understand, though, that using terraformer only gives you a representation of existing resources as code.

it does not do an import right out of the gate. (at least not that I am aware of).

So if you want to manage/control your existing resources with TF, you will still need to import them into the TF state.

In my experience, terraformer is excellent for:

• getting a copy of something that is already running, perhaps built in the console, so you can customize it as a module and deploy it elsewhere (vs managing the thing that is already deployed.)

• getting a copy of existing code that is not backed up as IaC just in case you have to build it over again. Essentially the generated code becomes a backup or a reference.

looks like you could submit a pull request to add a variable and pass the value through to auto_minor_version_upgrade around here: https://github.com/cloudposse/terraform-aws-elasticache-redis/blob/master/main.tf#L115

Thanks Alex, Will try that

Yes please, we always encourage contributions!

I post some weeks ago about I can help and do some PR to suggest this fixes. But I got no attention. I was looking just to collaborate with this amazing CloudPosse infra.

What is the ecs repo

ecs_cluster

Link please?

https://github.com/cloudposse/terraform-aws-ecs-cluster

Please put in a pr if you see an issue. We love pull requests!

Ohh, sure, thanks.

Also why are you using an ancient version of terraform

If you use the latest or at least 0.12 you wouldnt see the issue

Laziness

dont be lazy, use the non-beta terraform

That is very interesting. I figured the role_arn was used to initialize the tfstate specifically for the bucket.

how can i double confirm ? i’m running a terragrunt init now with log-level set to debug

hopefully that shows me which role it uses for state lookup?

id use native terraform to verify it, that way you can reduce the number of additional variables in the equation

do you have an assume_role_policy or just a role_arn?

just a role_arn

please scrap what i said earlier.

it is definitely using the role_arn but the init fails unless i give my local aws creds permissions to read the bucket

to clarify, i have an atlantis pod in EKS in AWS account A.

the remote_state config has a role_arn specified which exists in Account B.

the init fails unless Account A has access to the bucket, even though Account B has full access to the bucket.

is there documentation on this?

this is pure assume role stuff, no Atlantis related

and assume role policy requires the Trust relationship in account B and a explicit allow policy to assume:role action on the Role used to assume in account A

plus a trust policy to assume the service role

then it should work

definitely not atlantis related, i was asking if there was terragrunt docs available

and assume role policy requires the Trust relationship in account B and a explicit allow policy to assume:role action on the Role used to assume in account A Yes this is all done. but still fails unless i add a bucket policy that allows Account A perms to the bucket. even though the role_arn is specifying Account B

I’m doing this right now and I’m using Account B role in it

but I had and issue like you using ECS and I was using the exec role instead of the Task role(which is the one that assumes roles)

maybe you have a similar issue?

but I had and issue like you using ECS and I was using the exec role instead of the Task role(which is the one that assumes roles) can you explain this further please?

in ECS you have two roles : one for the task execution (init), one for the running task ( task role)

the exec role is for thing like giving access to pull from a registry, or s3 or something at the START/INITIALIZATION of the container lifecycle

the TASK role is the one that take place AFTER the container is up

so once Atlantis is up ,then it uses the TASK role to assume role , run terraform or access other things

i do this pretty regularly. though instead of “live” i call them “stacks”. each stack is essentially a root module and has terraform state

the modules/ directory lets me centralize business logic around one or more modules, as well as gives me a single place to update versions of external modules

@Jonas Steinberg in a real system each live env (ie “stack”) will be slightly different. Eg dev might have 1 EC2 instance, qa 3 and prod 30. These would be defined in the my_mod/variables.tf, and passed as args in the my_mod module block of the stack.

If that doesn’t help maybe you can be more specific?

I don’t know. I guess it’s the same with anything else. You keep interfaces, abstract classes and the like in separate directories than main app src.

Thanks for the responses!

The first error makes sense. We usually always set a name because that’s tied to the null label in context.tf and every resource is named based on that

The second error is unfamiliar to me

One thing you could do is reach out to the examples and see if you can use a similar approach because thats the one thats tested on every new PR

https://github.com/cloudposse/terraform-aws-ec2-client-vpn/blob/b48fc7a4114d1192fc9792a5d0fe7d77452b80a2/examples/complete/main.tf#L39

It would also help to create a written ticket in that repo with the following

• Reproducible examples containing your hcl and exact inputs

• Errors that you found and worked around

• References to similar errors thrown found on the web

A workaround could be to just disable logging for now since logging_enabled=true is whats creating the kog group

https://github.com/cloudposse/terraform-aws-ec2-client-vpn#input_logging_enabled

Thx, so just to clarify. For the first error I think it is due to the fact that in the complete test you pass the context to the module. In the Readme however you don’t so by default “name” is use and is refused as a name for the log-group.

The second error appear due to the fact that I try to correct it without passing a full context (since we use a different model in my company)

I just do:

module "client_vpn" {
  source  = "cloudposse/ec2-client-vpn/aws"
  version = "0.13.0"

  name                = module.tags.id

Which override the name and works until I have the duplicate tags error which I think is caused by:

      + tags                  = {
          + "Attributes" = "log-group"
          + "Name"       = "my-custom-name"
        }
      + tags_all              = {
          + "Attributes"      = "log-group"
          + "Name"            = "my-custom-name"

I don’t know if that clarify the issue for you. I am just wondering now if I can use it without using the context

it’s more clear now. please create an issue and then we can work towards resolving it

for now you can either try disabling the logging, if that’s the only duplicate tag issue, or try passing in a context with only the name filled

we also love PRs on our documentation and fixes to our modules

Ok I am going to try to see what I can do

managed to escape around this by running the tf apply cycle and before pressing apply, deleting the parameters manually before allowing the apply run to proceed.

The below resources seem to constantly want to update/recreate on every apply run

• module.ec2_client_vpn.module.self_signed_cert_ca.aws_ssm_parameter.certificate will be updated

• module.ec2_client_vpn.module.self_signed_cert_ca.aws_ssm_parameter.private_key[0] will be updated in-place

• module.ec2_client_vpn.module.self_signed_cert_server.aws_acm_certificate.default[0] will be updated in-place

• module.ec2_client_vpn.module.self_signed_cert_server.aws_ssm_parameter.certificate[0] will be updated in-place

• module.ec2_client_vpn.module.self_signed_cert_server.aws_ssm_parameter.private_key[0] will be updated in-place

• module.ec2_client_vpn.module.self_signed_cert_server.tls_locally_signed_cert.default[0] must be replaced

what is it failing with?

task_instance_group_bid_price = “OnDemandPrice” i used this.

Error: ValidationException: The bid price is invalid. Revise the configuration and resubmit. │ status code: 400, request id: c07e612b-fd9d-4324-a9ab-522276c0cee8

OnDemandPrice is an invalid value

you have to express a number

bid_price - (Optional) If set, the bid price for each EC2 instance in the instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/emr_instance_group

https://github.com/cloudposse/terraform-aws-emr-cluster/blob/master/variables.tf

yes, but i don’t want to specify number. i want to use_on_demand_as_max_price. AWS Management console gives that option.

terraform aws provider doesn’t currently support it

https://github.com/hashicorp/terraform-provider-aws/issues/7155

i see this issue is closed. but why

?

the issue is not closed

yeah sorry.Just checked.

So, what can i do to solve this problem?

best you can do is probably the issue, add a comment, or attempt to add support to the provider by opening a PR.

Sure. Thanks for your time.

no problem

Can you describe how you’re trying to change the name?

I thought I could take advantage additional_tag_map to overwrite the “Name” tag generated by terraform-null-label by a value I provided as a string.

@Erik Osterman (Cloud Posse)?

@Jeremy G (Cloud Posse)

additional_tag_map is for a special use case, the details of which I forget but you and find in Git history. In any case, it does not do what you want, nor is it intended to. As a general rule, you cannot overwrite the Name tag because Name is special and gets the generated label id which is the main point of null-label.

Here is an example:

module "alb_log_s3_bucket" {
  source                             = "cloudposse/lb-s3-bucket/aws"
  version                            = "0.15.0"
  name                               = "my-test-alb-bucket-rchen"
  stage                              = var.environment
  namespace                          = var.namespace
  attributes                         = [var.region]
  lifecycle_rule_enabled             = true
  enable_glacier_transition          = false
  expiration_days                    = 31
  noncurrent_version_expiration_days = 30
}

Just tested version 0.12.0. It doesn’t have this issue.

0.14.1 also works. So it must be broken in 0.15.0, which upgrades cloudposse/s3-log-storage/aws from 0.24.0 to 0.26.0

Submitted an issue: https://github.com/cloudposse/terraform-aws-lb-s3-bucket/issues/57

I wonder if there anything that could speed it up, or literally just wait for them to fix the bug

open a ticket with aws support i guess? been an issue for years now

it takes exactly

ws_security_group.lambda_sg: Still destroying... [id=sg-00c13d8760cf5ced7, 27m50s elapsed]
aws_security_group.lambda_sg: Destruction complete after 27m51s
Releasing state lock. This may take a few moments...

every time

that’s awful

state rm first lol

or maybe suggest a flag on security groups to skip waiting, similar to the flag on cloudfront distributions

let. me try state rm…….

what is that flag in cloudfront?

well, it’s the opposite, since it’s on creation, but the deployment takes forever, so you can set wait_for_deployment = false and it will just continue on and let aws do its thing in the background… https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudfront_distribution#wait_for_deployment

so, for security groups, maybe wait_for_destroy?

yes, that would work

my workflow look like this:

workflows:
  deploy-all:
    description: Deploy terraform projects in order
    steps:
      - command: terraform deploy fetch-location -s stackname
      - command : terraform deploy fetch-weather -s stackname
      - command : terraform deploy output-results -s stackname

it’s kind of a pain to use on files. but otherwise it worked fine for me

what do you mean by “to use on files”, what files?

you can manage files with the provider

but it’s a mess if you have branch protection, or use a pr-based workflow

https://registry.terraform.io/providers/integrations/github/latest/docs/resources/repository_file

i was trying to use it to template an initial repository with a bunch of common files. but i ended up not doing that, because i couldn’t get it to work very well

did you try using “bypass branch protections” permission on the role/account used by terraform?

it was a couple years ago, i don’t remember what i tried or didn’t. all i remember is it wasn’t worth it

Thanks for the info @loren

We use it a bit too manage teams and repositories belonging to different teams, as well as populating repos with templates on creation.

If I change a file that is used in a terraform module and trigger an apply, it can create a lot of noise in all the repos that are managed by terraform, and including that module. (similar to what loren mentions about: pain on files )

Yeah, I was just thinking that maybe using lifecycle ignore_changes on file contents might work, for at least the initial file contents of the repo

Makes sense. Once a file is managed by terraform, it should be modified only through terraform, which would be easy to forget in a repo.

But I might rather just create a template repo, and create new repos from that template, using the github provider mostly just to create the repo from that template, and to manage repo settings

lifecycle and github-provider-to-create-repo-from-template is worth trying

I think the bigge st challenge is if you’re using it on a large organization with hundreds of repos, you can easily exhaust rate limits if you don’t architect the terraform with that in mind.

e.g. doing a plan on 600+ repos is basically guaranteed to exhaust rate limits.

how does this look like in your atmos.yaml

how does your workflow file looks like? that error could be a yaml issue

I didn’t change anything from git clone

workflows:
  # Can also be set using 'ATMOS_WORKFLOWS_BASE_PATH' ENV var, or '--workflows-dir' command-line arguments
  # Supports both absolute and relative paths
  base_path: "stacks/workflows"

workflows

workflows:
  deploy-all:
    description: Deploy terraform projects in order
    steps:
      - job: terraform deploy fetch-location
      - job: terraform deploy fetch-weather
      - job: terraform deploy output-results

✗ . [none] (HOST) 02-atmos ⨠ atmos version v1.9.1

@Dan Miller (Cloud Posse)

(also note, we’re moving these atmos tutorials to https://atmos.tools/category/tutorials)

I checked this location too…same instructions and the workflow command doesn’t work Is there a particular version of atmos I need to pin to get this working? There are also few other command lines that doesn’t work based on the document For example This command doesn’t work

atmos terraform backend generate component tfstate-backend --stack ue2-root

it should actually be

atmos terraform generate backend tfstate-backend --stack ue2-root

is it atmos terraform backend generate in the docs?

yes

can you share the link, because In the docs I see it correct

do you ming creating an issue in the atmos repo about this?

https://github.com/cloudposse/atmos/issues/262

Any suggestions on the workflow command issue?

A plan is returned as stale when you try to apply it a second time. The exact mechanism i don’t know.

I’m guessing an apply against the plan file would yield no changes and would provide an exit code that corresponds to no changes

Well as the error says the resource "aws_iam_policy" "lambda_forwarder_rds" { has a malformed Policy Document because the Resource must be in ARN format or “*”. Maybe share your code, and we can see what you have?

Yes:

module "datadog_integration" {
  source  = "cloudposse/datadog-integration/aws"
  version = "1.0.0"

  context      = module.datadog_label.context
  name         = "datadog"
  integrations = ["all"]
  host_tags    = local.datadog_host_tags
}

resource "aws_ssm_parameter" "datadog_key" {

  name        = "/datadog/datadog_api_key"
  description = "Datadog key"
  type        = "SecureString"
  value       = var.datadog_api_key
}

module "datadog_lambda_forwarder" {
  source  = "cloudposse/datadog-lambda-forwarder/aws"
  version = "1.0.0"

  forwarder_rds_enabled = true
  depends_on            = [aws_ssm_parameter.datadog_key]
}

the upper one, datadog/integration/aws is already working correctly

you need to populate the value for variable dd_api_key_source.

the default for dd_api_key_source.identifier currently is just "" which needs to be either * or an actual ARN

that was it. Thanks a lot Denis.

Yes prs are always welcome! We may not get to it right away but we will eventually

And if I make changes and raise the PR then after how much time i might be able to use it in my project?

DNS misconfigured? It looks like eks_control_plane_endpoint is being resolved to 0.0.0.5 or that a DNS request (since port 53 is mentioned) is being made to that address.

Are all the failures showing a similar error, like same domain?

thanks for your reply; I have masked the DNS and yes all the failures are showing the same DNS. let me know if you need any more details.

Did you solve this? I have a module for this somewhere.

No I didn’t… I ended up postponing it as I’m on a very tight deadline to make improvements to the infra before an audit..

If you can share the module you wrote it will be much appreciated

Here you go: https://github.com/deploymode/terraform-aws-modules/tree/master/modules/s3-replication

I pulled it out of another project and made a few untested tweaks, so let me know if you have any issues.

Great, thanks a million I will let you know how it goes

maybe share your terraform code as well?

Oooh, great point- I may have not imported the liatener into terrafom, that’s why it doesn’t know to recreate it

Honestly I can relate. I can’t actually write a complex for_each without google, and I’ve contributed code to the terraform aws provider. But with a few examples from CloudPosse from relevant modules and from internal modules, I can usually infer the syntax that I need to write. So, my suggestion is look for examples.

Of course there is terraform’s docs, but I haven’t found them helpful with complex objects (for the simpler ones I can do it by myself). For the books, I’ve seen praises for Terraform Up & Running, but I can’t speak wether the complicated stuff has enough coverage. So I still go with looking for examples, and adjust them to my needs.

Thanks @Denis i have read up and running on oreilly.com it has simple one liners for loop examples just with key value stuff, easy ones so not really helpful and examples from modules that other people use, including cloudposse is also something i’m using, but figured, i want to understand it and be able to write them without looking for examples as it takes time and got tired of it But anyway, thanks for your suggestions!

Understood. Personally, I’m not writing complex for_each loops often, so my cost benefit ratio for understanding it deeply is reduced, that’s where my suggestions are coming from But yeah, it’s good to dive deep on it if it makes a difference.

This was pretty good… https://brendanthompson.com/posts/2022/10/terraform-for-expression

Thanks! This is useful

I’ve seen something similar when a provider is configured for only one specific region. But this may be something different. Maybe share the terraform code?

It was nested providers in the module.

it was orginally built in the wrong region, i changed the module along with the provider to a new region and then it barfed… with that non-descriptive error. I actually found the error by logging to a file with error logs

Set the min and desired qty to 0

that’s…too easy. thanks