#terraform (2022-01)

one thing that i think helps is to always run a refresh after apply

Hmm, don’t think that is gonna cut it in a lot of cases…

perhaps, but it helps an awful lot of the time

Won’t help in places we have ignore_changes

Currently seeing

Not helpful at all

well that’s a very specific case, and not the majority

I’ve run 5 plans, 4 have confusing “Objects have changed outside of Terraform” , some for “whitespace changes” some not

Folks will just start to ignore all plans if they don’t understand what is going on

the whitespace changes is something the aws provider has been working on pretty actively the last few releases. it’s getting better, but there is still work there

lol

oof, the tags_all stuff is implemented so poorly. very annoying

ruby

We use it

@matt

The AWS v4 provider allows you to delete the default VPC…

Obviously haven’t tested it, but that’s what the blog says

It used to be best practice to not delete the default VPC. Is that no longer true?

It is still a best practice to remove the default VPC. In fact, AWS Security Hub will flag default VPCs in the standard compliance results as a threat to remediate.

yeah, someone just commented that in hangops (i assume they’re a Hashicorp employee)…
Hey all, those are just release notes updates that add some scare text to the top, to try to keep folks off those versions - nothing new to see here.

is there any usable, open-source baseline ruleset yet?

Hey @Moritz we have setup some common rego files that we use for gcp, azure, and aws. recording is here where we talk about it https://www.youtube.com/watch?v=V12785HySYM

recently added see https://github.com/hashicorp/terraform-provider-aws/releases/tag/v3.67.0 https://github.com/hashicorp/terraform-provider-aws/blob/main/ROADMAP.md#amazon-appstream

fore more details https://github.com/hashicorp/terraform-provider-aws/issues/6508

thanks but i can’t make it work even if i use the aws 3.70 provider - any idea what is wrong?

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "3.70.0"...
- Finding latest version of hashicorp/appstream...
- Using previously-installed hashicorp/aws v3.70.0
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/appstream: provider registry registry.terraform.io
│ does not have a provider named registry.terraform.io/hashicorp/appstream
│ 
│ All modules should specify their required_providers so that external consumers will get the correct providers when using a
│ module. To see which modules are currently depending on hashicorp/appstream, run the following command:
│     terraform providers
╵

> terraform providers    

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws] 3.70.0
├── provider[registry.terraform.io/hashicorp/appstream]
├── module.iam
│   └── provider[registry.terraform.io/hashicorp/aws]
└── module.tags

Providers required by state:

    provider[registry.terraform.io/hashicorp/aws]

this is our config @ismail yenigul if you have any idea please

clean your .terraform caches etc.

$ cat main.tf 
terraform {
  required_version = ">= 1.0.2"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.70.0"
    }
  }
}

$  terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "3.70.0"...
- Using hashicorp/aws v3.70.0 from the shared cache directory

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

try

$ rm -rf ~/.terraform.d
$ rm -rf .terraform

I haven’t tested it yet btw. I am using cloudformation over terraform for appstream

i tried the delete comands but still the same error

Providers required by configuration:
.
├── provider[[registry.terraform.io/hashicorp/appstream](http://registry.terraform.io/hashicorp/appstream)]

something in the config is asking for the [registry.terraform.io/hashicorp/appstream](http://registry.terraform.io/hashicorp/appstream) provider but i cant figure out what is this…..

 Finding latest version of hashicorp/appstream..

can you check your all providers. this should be not there

do you have something like following?

provider "appstream" {
  # Configuration options
}

not really

can you search for provider string in the codes

sure, but only the screenshots that i provided before exist

remove .lock file and try again or create a new directory and copy only .tf files there

once i delete the resource “appstream_stack” then error dissapears……

because it comes from https://registry.terraform.io/providers/arnvid/appstream/latest/docs/resources/stack

it should be resource "aws_appstream_stack" "test"

for official aws module

https://github.com/hashicorp/terraform-provider-aws/pull/21484/files#diff-aeb8eef9063476397264ddd023da6937e709ad6679cde4815d1aa843924d1812R143

indeed :tada: mate thank you so much!!! just a small aws_ keyword created so much confusion!!!

This is what I get in logs.

Connection closed by authenticating user azizzoaib786 127.0.0.1 port 59712 [preauth]
Connection closed by authenticating user azizzoaib786 127.0.0.1 port 59730 [preauth]

Issue opened - https://github.com/cloudposse/charts/issues/269

We’re not really using this anymore

we’ve moved to using SSM agent with a bastion instance

So that means its deprecated project?

pending customer sponsorship, we’re not prioritizing investment into it

We do have https://github.com/cloudposse/terraform-aws-ec2-bastion-server

this is actively maintained

The problem with the k8s bastion is if your k8s is truly hosed, even the bastion is unavailable.

This is our setup for the Lambda:

data "archive_file" "edge-function" {
  type = "zip"
  output_path = "function.zip"
  source_file = "function.js"
}

data "aws_iam_policy_document" "lambda-role-policy" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type = "Service"
      identifiers = [
        "lambda.amazonaws.com",
        "edgelambda.amazonaws.com"
      ]
    }
  }
}

resource "aws_iam_role" "function-role" {
  name = "lambda-role"
  assume_role_policy = data.aws_iam_policy_document.lambda-role-policy.json
}

resource "aws_iam_role_policy_attachment" "function-role-policy" {
  role = aws_iam_role.function-role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

resource "aws_lambda_function" "function" {
  function_name = "headers-function"
  filename = data.archive_file.edge-function.output_path
  source_code_hash = data.archive_file.edge-function.output_base64sha256
  role = aws_iam_role.function-role.arn
  runtime = "nodejs14.x"
  handler = "function.handler"
  memory_size = 128
  timeout = 3
  publish = true
}

This is the CF assignment of the Lambda:

lambda_function_association {
      event_type = "origin-response"
      lambda_arn = aws_lambda_function.function.qualified_arn
      include_body = false
    } 

I bet there’s a module for that

Thanks @omry. I am doing - more or less - the same yet the function doesn’t seem to work.. Just to check: do you see CF under Triggers in your Lambda function?

But perhaps I’m looking in the wrong place and is such a trigger not needed. Haven’t needed Lambda@Edge until now

Why do you need it?

One of our dev teams is working on a new application and they are using NextJS for that.

This uses some kind of proxy running within Lambda@Edge to determine whether to serve static content (from an S3 bucket) or pass it along to the API gateway and subsequent Lambda’s attached to it to handle that particular request

According to the error it looks like the lambda is calling some other AWS services but might not have permission to access them

It appears that it is not published as a Lambda@Edge function, just as a plain Lambda. That’s why CF can’t find it

But in your L@E function, do you see anything at the Triggers page?

Not sure, but it’s presented in CF

Let take a look at my TF code as I have a origin-request L@E function to handle the CSP headers for both my static site and API Gateway/Lambda function

For me Ansible is super usable for creating machine images (together with packer), but other than that I never fall back on Ansible anymore.

I use Terraform or Ansible, but try to avoid using both. I lean towards using Terraform myself, not very good at writing YAML.

os-level system management is all i do with ansible, e.g. files, configs, packages, services, etc. nothing specifically cloud

That ^^

which has its place. i think both are worth learning, but i wouldn’t focus on the cloud portions of ansible

I get the sense that Ansible is strongest in configuration management. One concern though, when applied to OS, is updates: it does not reset the state of the OS, eg if you did something manually to the OS, this change will remain.

But I could see creating a fresh OS image (like an AMI) by creating a temp machine via Ansible, configuring it, capturing it as image and tearing it down. Then when you need to update a system, you treat it as stateless and use terraform to apply the new AMI (ie there is no post-startup saved state on there except in data volumes that can be remounted after changing to the new AMI).

Ansible is a configuration management tool, mutable and stateless by design. Definitely worth learning. Re your concern on OS updates; Ansible will just run the playbook to check for updates and install them as required.. if someone manually installs updates after this, no problem; the next time Ansible runs, those updates won’t be available to install so it’ll come back with nothing to do

In an ideal world, you should be using Terraform and Ansible together. For e.g. you’d use Terraform to deploy your ec2 instance and Ansible to configure it after the fact (change hostname, make local changes to the host, join domain etc;). Whilst you can use things like SSM documents in Terraform to achieve some of those things (for e.g. joining the domain), they become immutable and when you do things like a Terraform destroy, Terraform won’t disjoin the machine from the domain so you’ll end up with orphaned objects in AD. As the saying goes; if a hammer is your only tool, every problem begins to look like a nail.. hope this helps

Thanks @IK exactly what I was looking for, in particular your example about using ssm via terraform leading to orphaned objects in AD.

If you (or anyone else!) has any other examples where there is a clear advantage of using ansible over the equivalent in terraform, it would help me make a case for it.

if your requirement is to build images, i would use packer+ansible. packer is purpose built for that, and addresses a ton of use cases and scenarios around interacting with the cloud provider. and it has a packer plugin to make it easy to use them together

i think the advantages are pretty clear.. i mean ultimately, terraform is not a configuration management tool and when you try to make it one, it quickly falls over (as per the SSM example i gave).. the way i see it, terraform is a day-0 tool (provisioning) and ansible is a day1+ tool (configuration management)

terraform is fantastic at continuing to manage the things it provisions. defining “configuration management” only as what happens inside an operating system is a very very limited and restricted definition of the term!

@IK Yeah but it’s the concrete examples that are hard to come by. I’ve seen the exact same statements about day 0 and 1 on the web, mutabilty, configuration management vs provisioning, but it’s the substantive examples that give weight to these statements

@loren I agree.. would be interested to hear your thoughts on what more configuration management is

oh i just feel like most everything has configuration. the tags applied to a vpc are configuration. the subnets associated to a load balancer are configuration. i use terraform to manage and update that configuration, and if somehow it was changed i often use terraform to tell me and to set it back

#office-hours

Here’s a snippet of my argument:
I do not believe that Datadog Monitors (and other configurations) are suited to be configured from API.

In more detail, I believe that despite the advantage of being able to use Terraform’s for_each or module functionality to templatize certain Datadog resource patterns, that doing so introduces an interweaving of dependencies that invokes the question of updating the larger pattern every time an incremental improvement is made. I believe this linking to templates reduces the ability of engineers to tune the Datadog Monitors and Dashboards to their needs, especially as Slideshare’s infrastructure is being heavily refactored. I do not believe that this advantage outweighs the need for monitoring to be continuously updated with the latest context about operational concerns. Reducing the toil associated with adding notes, and adjusting thresholds, etc. should be minimized with the utmost priority.

Using terraform directly for configuring Datadog severely curtails the usability of Datadog, since part of the workflow for configuring Datadog is to contextualize the monitors and dashboards with historical data. Doing so via a manifest would require actually using the web interface to produce the configuration, then exporting it to JSON, then translating that to HCL.

Having two places where changes may be introduced creates a sort of dual-writer problem, would require periodic reconciliation, and would result in confusion and non-beneficial work.

I’d rather not introduce a release process to a SaaS service, especially one that benefits greatly from using the web interface.

I disagree. We’ve been configuring datadog monitors using yaml and terraform and it’s wonderful. We can recreate standard monitors for specific services and we can create generic monitors with standardized messages using code.

if we used clickops, it would be a lot more challenging for consistency

@Nathaniel Selzer interesting thread re: our recent internal conversations

this might help make it easier to use.

so we dont have to mess around with custom hcl per monitor, we have a generic module that reads it from yaml. here’s an example yaml catalog.

https://github.com/cloudposse/terraform-datadog-platform/blob/master/catalog/monitors/amq.yaml

we also allow deep merging of this yaml to setup default monitor inputs.

We also use TF to deploy DD monitors. We deploy the same monitors to multiple similar environments (client A prod, client B prod, Client a UAT, etc.)j, across multiple services. We do this rather than use multi-monitors, because we need different thresholds in each environment/service and that’s not supported.

I broke it out so each monitor is in its own yaml file, and once you create the monitor, we configure environments.yaml so the monitor is deployed to either selected or to all environments. In my last job, we used Python and JSON to persist monitors, which was awful for multi-line messages.

This was my initiation to flattening() environments, services, and monitor defs into a data structure suitable for for_each processing.

We have several kinds of monitors that monitor different types of thigns at different levels. Currently, our TF only supports deployment of monitors at the client environments/service level. We have plans to add one-offs and enhance the inclusion/exclusion functionality.

I could see the use case if Datadog Monitors are being deployed consistently across multiple client environments. This essentially turns Datadog configuration into part of the product. Makes sense to version control then, since delivering updates to the monitoring suite becomes more important.

To elaborate, at my organization there is only a couple environments, and I found that some engineers were using terraform to codify Datadog Monitors and Dashboards to be applied once. This was hampering their ability to make changes without invoking a terraform release process, so we’re moving away from that.

@RB, I’m curious, what kind of environment(s) are you working on that work well with the terraform-datadog pattern?

It’s all tradeoffs. Picking which battles you want to fight based on what resources you have available. e.g. the same argument can be made for so many things. It’s so easy, afterall, to clickops an EKS cluster and deploy mongodb with a helm chart. Maybe < 30 minutes of work. Now “operationalize it”, takes 10x more effort.

I think it’s okay if teams want to click ops some datadog monitors to get unblocked, but as a convention, I don’t like it. Maybe for a small company, with 1-3 devs with small scale infra.

We just revamped our entire strategy for how to handle datadog/opsgenie. Major refactor on the opsgenie side. On the datadog side, we did the refactor last year some time, moving to the catalog convention.

When we develop a new monitor, we develop it still in the UI, but then codify it in via the YAML as a datadog query.

Does this UX suck? ya, it’s a tradeoff, but one that we can always walk back. We know how to deploy any previous version. We have a smooth CD process with terraform via spacelift.

the reality is very few companies I think practice good gitops when it comes to monitoring and incident management. when going through our rewrite, i reflected on this. i think it’s because it’s so hard to get it right with the right level of abstraction in the right places. i think now we finally have the patterns down to solve it (literally we’re rolling it out this week). check back with me in a couple months to see how it’s going.

To elaborate, at my organization there is only a couple environments, and I found that some engineers were using terraform to codify Datadog Monitors and Dashboards to be applied once. This was hampering their ability to make changes without invoking a terraform release process, so we’re moving away from that.

is the terraform release process cumbersome?

the other thing to think about with monitoring is how easy it is to screw it up. someone can go change the thresholds and make an honest mistake of the wrong unit (E.g. 1s vs 1000ms)

not every engineer will have the same level of experience. the PR process helps level the playing field making it possible for more people to contribute.

with the spacelift drift detection, if changes are made in the datadog UI, that’s cool, but we’ll get a drift detected in our terraform plans within 24 hours. Then we can go remediate it.

lastly, we’ve probably spent 2000 hours (20+ sprints) on datadog/opsgenie work in terraform just in the past year. so is it easy to get it right? nope. but is it possible, believe it is which is why we do what we do.

you can certainly manage some subset of your monitors with clickops. don’t get in the way of developers, if you don’t have the resources in place to handle it in better ways. let them develop the monitors, burn them in and mature. then document them in code. tag monitors managed by terraform as such, so you know which clickops and which are automated.

@Erik Osterman (Cloud Posse) It's so easy, afterall, to clickops an EKS cluster and deploy mongodb with a helm chart. Maybe < 30 minutes of work. Now "operationalize it", takes 10x more effort. can you expand on this please? Particularly the 10x more effort part

Operationalizing it is about taking it over the line into a production-ready configuration.

1. DR, Backups, restores
2. Environment consistency (e.g. via GitOps)
3. Logging
4. Monitoring & incident management
5. Upgrading automation of mongo (not all helm upgrades are cake walk, if the charts change it can lead to destruction of resources)
6. Upgrading the EKS cluster
7. Detecting environment drift, remediation
8. Security Hardening
9. Scaling or better yet autoscaling (storage and compute) for monogodb
10. Managing monogodb collections, indexes, etc. …just some things that come to mind

Great points, thanks for that. I totally agree.. i’m actually in 2 minds about deploying a simple ECS cluster with TF vs clickops.. will take me about 15mins via clickops but likely longer via TF.. was wondering if the effort using TF was worthwhile.. after thinking about some of the points you raise, i can see why i’d spend the extra time doing it in TF

Yup, it sort of hurts when you think about it. You’re right, it takes just 15 minutes with clickops. And it seems stupid sometimes that we spend all this time on automating it. With IAC, there’s this dip. You are first much less productive, until you become tremendously efficient. It’s not until you have the processes implemented that you start achieving greater efficiencies. So if you’re building a POC that will be thrown away, think twice about the IAC (if you don’t already have it). And if it’s inevitably going to reach production, then push back and do it the right way. Also, as consultants, what we see all the time is that something was done quick and dirty, but now we need to come in and fix it, but it’s a lot more gnarly because nothing was documented. There’s no history to understand why something was set up the way it was setup.

let’s discuss on #office-hours

please open an issue

Thanks, @Andriy Knysh (Cloud Posse). Already started the ticket, but the message on that page suggests reaching out here, so…

fwiw, terraform supports pure json

so no need to convert anything

we exploit this fact to generate “HCL” (in json) code, but prefer to leave it in JSON so it’s clear it’s machine generated.

If you want to test it locally, you’ll need to set the workspace to local, or change the backend to something else and use an alternate state file.

In remote, tf plans on tf cloud, and streams the output to your local machine

Thanks, confirms my suspicions. If I change execution mode to “local” , it breaks my CI/CD since I now have to supply variables somehow. So I’ve come up with :

1. Ensure tf apply runs clean
2. Create feature branch
3. Tfenv install 0.13.7
4. Tfenv use 0.13.7
5. Terraform 0.13upgrade
6. Fix syntax
7. In TFE UI, make sure auto apply = OFF. Set version to 0.13.7 and branch to our feature branch
8. Run plan . Do NOT APPLY.
9. If any syntax errors , fix in feature branch, git push.
10. Run plan in TFE UI again until it plans clean .
11. After this, point of no return.
12. Change workspace back to original (master) branch
13. Do a PR to master with new 0.13 syntax
14. Run tfe apply , should run clean. Fix errors and iterate until done

.. am I off track here ?

Seems reasonable.

the kafka module has an input called properties

https://github.com/cloudposse/terraform-aws-msk-apache-kafka-cluster#input_properties

have you tried using that? @Florin Andrei

I ended up passing the parameters I need as a map via the properties input, and that seems to work. AWS Support told me I only need to declare the non-default values there.

those are a lot of issues

the first 2 are solved with terraform deploy services like spacelift and atlantis

3rd is solved with appropriate usage of compliance and aws services like guardduty, shield, inspector, config, etc

4th is solved with git

5th, if related to terraform, can be solved by rightsizing and using the tool infracost

#office-hours today!

i gotta watch the recording, interesting topic !

they seem to be created by the permission sets: https://docs.aws.amazon.com/singlesignon/latest/userguide/using-service-linked-roles.html

far as i know, the role name is not returned by the permission set api calls…

i think you could use the iam_roles data source, with a regex pattern specific enough to always match just the one role… probably need to use depends_on to force the dependency. and also, it can take a bit of time for the permission set to populate the role to the account, so might need to use the time provider to delay the call a bit…

instead of depends_on, you could parse the name from the arn attribute of the permission set. that would be somewhat better, in terms of letting terraform manage the dependency graph

Ya, I think @loren’s suggestion is correct. When we wrote the module, the IAM data source didn’t support a regex pattern, but that is a nice addition.

The trouble is the account where the permission set is created is not necessarily the account(s) where the role is created, which means likely need a second provider alias to perform the lookup in the correct account

what OS are you using?

macOS Monterey

are you using tunnelblick? if yes, disable it

FWIW: I am only using the OpenVPN connect client. I don’t think it has anything to do with the VPN tools. The endpoint is just not accessible from the net somehow.

sorry, ya, just realised that after I typed it

how are you setting up the cvpn endpoint? https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/troubleshooting.html

I used the Terraform module provided by the Cloud Possee to set up the VPN. But that help article properly contains the answer to my problem

I’ll try modifying the DNS as suggested in the article.

ya I get it… this is “close” to me because Ive been fighting with a bunch of issues on my cvpn deployments… its always had nothing to do with terraform and almost always had to do with missing things in aws documentation tbh

Yeah — the AWS Client VPN hostnames are not resolve-able by all DNS servers AFAIR. I remember having to have clients add additional DNS servers to properly resolve the VPN hostname when I last set one up. An annoying issue.

@Leo Przybylski might have some ideas

@Jens Lauterbach You cannot use that address of the VPN endpoint as is. It must be a subdomain like acme.cvpn-endpoint-.....prod.clientvpn.eu-central-1.amazonaws.com

In the client config, you will see something like

remote cvpn-endpoint-.....prod.clientvpn.eu-central-1.amazonaws.com 443
remote-random-hostname

remote-random-hostname is not supported by openvpn AFAIK, so the client tries to connect directly rather than using a remote random hostname. You will need to add this yourself. FYI, you could just use the AWS VPN client which does recognize it.

Disclaimer: It was difficult getting openvpn to work. I don’t believe the AWS client vpn client configuration is fully supported by openvpn; therefore, I don’t recommend it. If possible, use https://aws.amazon.com/vpn/client-vpn-download/ instead

@Matt Gowie I also encountered what you were running into in separate instances. One was when I finally got my VPN client to connect, I could not resolve DNS entries. The reason for this is that unless you are using AWS for DNS service, it will not work. For this, I had to enable this option https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_client_vpn_endpoint#split_tunnel

What split_tunnel does is it allows traffic outside the VPN. In this case, it allows traffic to travel to DNS servers outside AWS. If you are using DNS servers elsewhere or your application needs to communicate with services outside AWS infrastructure, this will be important.

@Jens Lauterbach Regarding the remote-random-hostname in the client config, are you using the client configuration exported from the terraform module or from AWS console? I ask because the terraform module automatically produces a client configuration that will connect to openvpn. If it is not, I’d like to discuss further and resolve that. (See: https://github.com/cloudposse/terraform-aws-ec2-client-vpn#output_client_configuration)

Wow, that is surprising

What problem are you trying to solve?

Security appears to need the information for some kind of corporate security control.

no you can’t so thing like imports or such

Cheers, Pepe

I don’t think you’d use Terraform to do something like that. An playbook with Chef/Ansible/Puppet might be better.

you can do it in user_data, but terraform will not have any visibility into the results - it just blindly executes the script. Probably a better solution would be to bake an AMI with Packer first, then use terraform to spin up an instance based on it?

Yes, agree with the recommendations above

@Jim G Thank you. Now I have an AMI ready. From here how can I use that AMI to spin up using terraform? Is that to create a new ec2 based on that AMI?

@DevOpsGuy try not to double post

Ansible is much better for this task.

@jose.amengual These are two different issues and this is terraform group and I posted in that context only.

yes but now I know you are using an instance instead of a RDS instance, that is what I mention it

I got something helpful here https://thepracticalsysadmin.com/create-windows-server-2019-amis-using-packer/ In case it may help others cheers

hi @Jas Rowinski, fork, branch -> PR is the correct way of doing it

we all do branch -> PR at Cloud Posse (we don’t need to fork since it’s our own repo)

thank you for your contribution

no problem, just put a PR in. Tested on my own cluster and it worked as intended

Next up is scheduling capacity with on-demand and spot types

thanks, I reviewed, a few comments

Thanks for the review and merge!

you should use a template

https://github.com/cloudposse/terraform-aws-ec2-bastion-server/blob/master/user_data/amazon-linux.sh

https://github.com/cloudposse/terraform-aws-ec2-bastion-server/blob/master/main.tf#L64

wow, this is almost like if the libcurl maintainers said they don’t think they can maintain it anymore, … sops is integral to many tools

Looks like Mozilla isn’t going to let this die just yet. Ref: https://github.com/mozilla/sops/discussions/927#discussioncomment-2183834

Correct me if I’m mistaken, but I think you are saying you are currently using terraform to manage the GitHub configuration of your git repositories, and now you wish to use terraform to add files to the git repository.

Could you explain in more detail why you would wish to do this?

What I am trying to achieve is when developers need to create repo, run terraform we have to enter repo name, it will then create repo, required access, events, set webhooks, labels, etc. Say if it is java project using gradle all default files I want to create/upload using terraform.

so template will be ready for dev they just need to add their code to start CI

Do template repositories serve this purpose? There is the ability to create the repository from a template: https://registry.terraform.io/providers/integrations/github/latest/docs/resources/repository#template-repositories

I don’t think there is a way to manage template updates afterward, so this is only a solution for the start of the lifecycle.

Thanks @Jim Park, will have a look.

check that you are using the latest versions of the modules

yes, version 0.25.0 of module cloudposse/label/null

the error above is in some subnets module

I am trying to deploy atlantis in ec2 https://github.com/cloudposse/terraform-aws-ecs-atlantis/tree/0.24.1

I use the latest subnet module version 0.39.8

https://github.com/cloudposse/terraform-aws-dynamic-subnets/releases/tag/0.39.8

terraform-aws-ecs-atlantis was not updated for a while and prob uses the map() function

uff yes that module is very very old

and it was created for a forked version of atlantis

Do you have any recommendations for deploying atlantis in HA mode?

There is no HA mode.

@joshmyers is right, there is no HA for atlantis

you can have multiple atlantis and have WAF or something in between to match the repo URL from the body request of the webhook call and send it to the appropriate atlantis server, but there is no concept of load balancing or anything

(just to be clear, the HA atlantis is an atlantis limitation; fault tolerance can be achieved with healthchecks and automatic recovery)

Thank you all for the information!

(I foot-gun’d months ago, but neglected to share with y’all. Recommend bookmarking just in case =))

not sure about that one but have you looked at this one

https://github.com/antonbabenko/pre-commit-terraform#terraform_tflint

you can use args to configure it

Thanks! I hadn’t seen that

see the review dog version too

https://github.com/reviewdog/action-tflint

Cool, I’m gonna go with the Anton Babeko stuff for now

the difference is that reviewdog comments the pr inline whereas the anton one will just fail if a rule is broken

nice feature

A lot of the standard CI/CD tools these days can execute TF, but you’re better off looking at the TACoS platforms (Terraform Automation and COllaboration Software). These platforms are purpose built for IaC lifecycle Automation. env0, Terraform Cloud, Spacelift, and Scalr.

Disclaimer, I’m the DevOps Advocate for env0

The Spacelift docs (which I recommend) have a great explanation of why dedicated systems for Terraform are worth it: https://docs.spacelift.io/#do-i-need-another-ci-cd-for-my-infrastructure

• env0

• scalr

• spacelift

• TFC ( enterprise is pricey)

• DIY Pipeline - I prefer this, as I can use Github actions or Gitlab or Bitbucket or ( my best choice ) CircleCI to implement all needed stages with fine grain control.

now with options like OIDC between AWS and Github Actions runners, no need to share credentials or worry about access

I used most of these, and fount that Terraform Cloud is best for new comers, and small team < 5

Thanks everyone.

@Mohammed Yahya among the DIY pipeline which ones the best.

there is no best one, only I found CircleCI feature rich and speedy

But depends on organisation selective tools

This looks slick https://github.com/suzuki-shunsuke/tfcmt (via weekly.tf)

Maybe it can help: https://youtu.be/Jy-fBSS8-sA?t=2646

Thanks. Have heard of spacelift but wasnt sure if it was mature enough. Ill have a look

Seems to be an open ticket for AWS for this: https://github.com/aws/containers-roadmap/issues/1297

I’ve switched to using a small Managed Node Group to bootstrap the cluster and then using Karpenter to provision instances for the rest of the workload. Karpenter can have multiple profiles for the nodes and can easily mix spot/on-demand types

That’s cool - you’re already using karpenter?

…in production too?

No just dev, we’re transitioning to k8s

is this the same IAM role/user that created the cluster? If not, is that role/user in the auth config map? (by default, only the role/user that created the cluster has access to it until you add additional roles/users to the auth config map)

It should’ve been. However, I’ve added the user to the mapUsers and the associated roles to the mapRoles. Maybe I’m missing a role.

I’ll dig a bit more here.

Do you happen to know what the appropriate groups should be?

would - system:masters be viable?

yea, system:masters is for admins

The user here can authenticate to the cluster, it can access the cluster via the roles, but it cant do anything via the terraform due to the error above. Do you see anything I’m inherently missing?

  mapRoles: |
    - groups:
      - system:masters
      rolearn: arn:aws:iam::123456789012:role/DevRole
      username: DevRole
    - groups:
      - system:masters
      username: DepRole
      rolearn: arn:aws:iam::123456789012:role/DepRole
    - groups:
      - system:masters
      rolearn: arn:aws:iam::123456789012:role/OrgAccountAcc
      username: OrgAccountAcc
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::123456789012:role/dev-cluster-workers
      username: system:node:{{EC2PrivateDNSName}}
  mapUsers: |
    - userarn: arn:aws:iam::098765432123:user/deploy.user
      username: deploy.user
      groups:
      - system:masters

I found the issue. It was this, kube_exec_auth_role_arn = var.map_additional_iam_roles[0].rolearn Lets not taco bout it.

Thanks for your sanity checks. I appreciate you.

maybe try #terragrunt

If you’re running inside of geodesic amd64 it should “just work”

(with the emulation under the m1)

I didn’t realize I could run this repo inside geodesic… I guess I will try that

I guess that makes sense though. Thanks!

yep! use geodesic as your base toolbox image for all devops. that way it works uniformly on all workstations

No luck getting geodesic built… bumped into a build issue.

Did a fresh clone and check out the latest version muck when running make all I end up with a build error

 => ERROR [stage-2  7/30] RUN apk add --update $(grep -h -v '^#' /etc/apk/packages.txt /etc/apk/packages-alpine.txt) &&     mkdir -p /etc/bash_completion.  6.1s
------
 > [stage-2  7/30] RUN apk add --update $(grep -h -v '^#' /etc/apk/packages.txt /etc/apk/packages-alpine.txt) &&     mkdir -p /etc/bash_completion.d/ /etc/profile.d/ /conf &&     touch /conf/.gitconfig:
#13 0.223 fetch <https://dl-cdn.alpinelinux.org/alpine/v3.15/main/aarch64/APKINDEX.tar.gz>
#13 1.483 fetch <https://dl-cdn.alpinelinux.org/alpine/v3.15/community/aarch64/APKINDEX.tar.gz>
#13 3.843 fetch <https://apk.cloudposse.com/3.13/vendor/aarch64/APKINDEX.tar.gz>
#13 4.584 fetch <https://alpine.global.ssl.fastly.net/alpine/edge/testing/aarch64/APKINDEX.tar.gz>
#13 5.378 fetch <https://alpine.global.ssl.fastly.net/alpine/edge/community/aarch64/APKINDEX.tar.gz>
#13 6.044 ERROR: unable to select packages:
#13 6.044   awless (no such package):
#13 6.044     required by: world[awless]
#13 6.044   aws-iam-authenticator (no such package):
#13 6.044     required by: world[aws-iam-authenticator]
#13 6.044   chamber (no such package):
#13 6.044     required by: world[chamber]
...
...

Calling an evening for now and will have to pick this up tomorrow. Thanks for the suggestion and I will keep picking away at this

Thought: maybe because I am on M1 and not emulating correctly…

Realized I may not need to build it and can just do a docker run going to try that first

You can just use our geodesic public image

use FROM cloudposse/geodesic:debian-latest

We use github actions to build the docker image and there could be other things needed to get it to build

those packages come from our cloudposse package host. They must be installed after configuring the cloudsmith registry

I don’t know much about the module, but looking at the error, it wants a map of strings - perhaps it doesn’t like the null value.

try an empty string or "null"?

Thanks. I’ve tried that, along with "", and even omitting the value key all together. Though that one threw the error I expected.

Update…I learned that my version for the node group was not up to date with the docs I was looking at.

You mean this https://github.com/GoogleCloudPlatform/terraformer ?

Thank you for your response, no, it was more like terraform checker

name of the tool is driftctl if someone is intrested

(its not driftctl) @Almondovar you mentioned in the original message that it isn’t driftctl

true!! it was indeed

can you try setting one of those to null

actually, I just did that just after posting

it worked

it was not clear from the documentation

nice!

it’s a terraform thing

do you accept PR’s for such things?

it’d be great to document that!

if we documented it here, we’d have to document it in every module

oh, a pain

it’s a restriction in terraform when we explicitly set the object definition in the variable type

one way around it is to set the type to any and use lookups to retrieve the value then we can document the appropriate keys in the description

the other way, eventually, when terraform allows the optional argument without the experiment enabled

I see, that’s great when that happens

if you want to put in a pr, you can put one in to switch the type definition to any

you’d also have to define the object body in the description

i can’t promise that it would be merged but it would spark discussion and would be reviewed and maybe merged

thing is, I think it’s more like to know the expected behaviour. As the object definition does not clearly imply that either should be null in order for this to work properly. The error messages do not help either

more like, I would like to add an example rule, for each case, so others can refer to that!

there is also type validation that can be added

you could put in a pr to update the readme yaml and update the example module so terratest runs it accordingly

https://github.com/cloudposse/terraform-aws-ec2-client-vpn/blob/0e4f0d9dc660a945a6fafc84284d447e3104ca06/examples/complete/fixtures.us-east-2.tfvars#L34

https://github.com/cloudposse/terraform-aws-ec2-client-vpn/blob/0e4f0d9dc660a945a6fafc84284d447e3104ca06/README.yaml#L106

@Leo Przybylski

that’s something I can try and do! so other avoid this pitfall that I did

We’re very interested in AFT. Haven’t yet kicked the tires.

I’ll give it a shot this weekend and share first impressions.

No. It’s like the only thing I did manually

BTW you can still achieve 95% automation using Terraform only, I’m not sure if this over complicated thing still be needed, may be I should test it.

95% automation using Terraform only this is true, however, 0% control tower

also, to be clear to others AFT has nothing to do about managing control tower itself with terraform, but rather provisioning of resources inside of an account managed by control tower

and AFT itself must be manually provisioned, like @Josh B. said. The terraform parts of it with TFC/Spacelift/scalr/env0/etc. But any accounts needed by AFT are still clickops in control tower.

Ah, I never reported back. So I did try it that weekend, it was frustrating. AWS really wants you to use CodeCommit. I tried several times to do it using GitHub and it would work up to the point that mattered, which was vending an account. So I gave up and did it the official way with CodeCommit and that worked first time. It is indeed quite a manual process to setup, and you need to read the docs carefully, but I can see it being smooth sailing once you’re done with the initial setup.

Here’s the tutorial I followed for the GitHub attempt. Would really love to hear if anyone succeeded following it. I’d try again but I don’t want to create a whole new bunch of AWS accounts

@Shaun Wang

@Shaun Wang has been working on updating the tutorial for AFT and already fixed a lot of issues with it. But not sure if he tested the GitHub integration yet.

Hey! didn’t see this earlier Issac, will reach out to you

how are you trying this?

are you trying to reuse an existing access identify? if so i don’t think it’s possible with this module since it creates its own

https://github.com/cloudposse/terraform-aws-cloudfront-cdn/blob/9f0b0654a221131d2ab6b6d855c70748dd771f98/main.tf#L10

I’m not taring to use an existing access identify, i would like to use the one created by the module

when a CDN is deployed the default origin is of type Custom Origin and not S3 that uses Origin access identity

try the s3 module

https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/

@RB Thanks

see https://github.com/cloudposse/terraform-aws-eks-cluster#input_map_additional_iam_roles

list(object({ rolearn = string username = string groups = list(string) }))

ah, yes right - that’s helpful! I created the cluster and when I tried to refresh the state I got

│ Error: Get "<http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth>": dial tcp 127.0.0.1:80: connect: connection refused
│ 
│   with module.my_eks.module.eks_cluster.kubernetes_config_map.aws_auth_ignore_changes[0],
│   on .terraform/modules/my_eks.eks_cluster/auth.tf line 115, in resource "kubernetes_config_map" "aws_auth_ignore_changes":
│  115: resource "kubernetes_config_map" "aws_auth_ignore_changes" {

try using the same one from the example

https://github.com/cloudposse/terraform-aws-eks-cluster/blob/b745ed18d8832c7e8e53966264687d2ee1d64e1a/examples/complete/main.tf#L63

that’s what I am trying to use, however I am adding custom IAM roles

and users

to be allowed

I have only taken the bits (almost verbatim) from the node group and for the eks cluster - as I need to use a different VPC rather than create a new one.

the cluster is created and it shows correctly, however when I view it on AWS it does say that I have access (while I have provided the arn and the user name with empty groups for users I want to have access). And also caused this error in refresh.

I could solve this by using

tf state rm "module.my_eks.module.eks_cluster.kubernetes_config_map.aws_auth_ignore_changes[0]"

and then use targeted destroy on the module

however, recreating I do not think will solve the issue

seems like you don’t have access to the eks cluster api so it’s defaulting to localhost

yeah, I am using the template you’ve provided but I am also adding as I said additional IAM users.

through the provided map

nothing else…

try just using the example, does it work correctly?

are you using your own subnets or also creating the subnets and vpc from the example?

so, what user is that supposed to give access - just by running the example

in order to stay safe, I will use the example and peer the VPC later on with the one I want to target.

that should be a safe bet, right?

every pr validates the example using terratest so the full example should work

the test does a plan and apply and then destroy

ya that sounds like a safe bet

alright - cool

so, by running the example - which user owns the cluster/

the one used the AWS keys to create it?

Yes, the user that runs terraform apply

Or the role

right - alright, thanks! am modifying my code to first to the example verbatim, then add new IAM users, and then do the peering. Fingers that it works as expected!

Error: Get “http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth”: dial tcp 127.0.0.1 connect: connection refused

this is a networking issue

could be several reasons for that:

1. The cluster is deployed into private subnets only, and you can’t access the API server from your local computer

1. Terraform can’t access the cluster to get the kubeconfig. The AWS provider is written is such a way that if it can’t access the remote cluster, it tries localhost

either way, the error in almost all cases (maybe not in all) is about networking and access, not about IAM

right, however AWS said when I browsed EKS that I the user I am using (which was the one I used to create the cluster) did not have access - is this normal?

to be concrete, this is the message that I saw:

yes, this is related to IAM

there could be two diff issues

try our example first

aye, thanks so much regardless both your and Ronak’s help have been great!

it could be both networking and IAM issues b/c when you provision the cluster and you get

Error: Get "<http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth>": dial tcp 127.0.0.1:80: connect: connection refused

you are still the same user/role with all the permissions, so it should not be related to IAM

also pay attention to https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/variables.tf#L50-L60

so, upon further inspection it seems that map_additional_iam_users is either not doing something correctly, I am not setting it up properly.

is this a correct way to place it?

the user arn is the one that IAM shows in the dashboard

you need to add some

groups

oh, lol - I thought this was AWS groups

so what would be the groups to give full access to the cluster - would this be putting all the ones you provided as a list?

btw, even using the full example as you provided when I refresh something in the cluster (say change it’s name) I still get the same error as before.

wrt to Auth

system:masters

for admins

https://archive.sweetops.com/

Thanks!

(we have some bug right now in our archive where it’s only showing Y22)

who cares about 2019/2020/2021 erase them

lol

for real

we don’t support http backend in atmos

please open an issue, we’ll get to it

for s3 backend, atmos just generates a file like this

{
  "terraform": {
    "backend": {
      "s3": {
        "acl": "bucket-owner-full-control",
        "bucket": "xxx-ue2-root-tfstate",
        "dynamodb_table": "xxx-ue2-root-tfstate-lock",
        "encrypt": true,
        "key": "terraform.tfstate",
        "region": "us-east-2",
        "role_arn": xxxx,
        "workspace_key_prefix": "xxx"
      }
    }
  }
}

so atmos itself does not process any backends, it lets terraform do it

but we’ll review the http backend and what could be done there

@Mike Crowe

OK, thanks @Andriy Knysh (Cloud Posse) – let me try that

OK, based on that, this should work:

terraform:
  vars: {}
  backend_type: http
  backend:
    http:
      address: <https://gitlab.com/api/v4/projects/>...
      lock_address: <https://gitlab.com/api/v4/projects/>...
      unlock_address: <https://gitlab.com/api/v4/projects/>...
      username: ...
      password: ...
      lock_method: POST
      unlock_method: DELETE
      retry_wait_min: 5    
  

atmos does not process backend type http

So, is there a way to set variables like TF_HTTP_ADDRESS via the stack variables?

you can set ENV vars globally or per component. The env section is a first-class section like vars - it gets deep-merged in this order: globals -> terraform globals -> base components -> component

https://github.com/cloudposse/atmos/blob/master/examples/complete/stacks/catalog/terraform/test-component-override-3.yaml#L16

those ENV vars will be detected and set in the shell before executing terraform commands https://github.com/cloudposse/atmos/pull/77

@Andriy Knysh (Cloud Posse) – just to confirm, I can do the following, right? stacks/pinnsg/ue1/dev.yaml:

import:
  - globals/pinnsg-globals
  - globals/ue1-globals
  - catalog/terraform/sso

vars:
  stage: dev

terraform:
  vars: {}  
  env:
    TF_HTTP_ADDRESS: <https://gitlab.com/api/v4/projects/{gitlab_project}/terraform/state/{state_key}-{stage}>
    TF_HTTP_LOCK_ADDRESS: <https://gitlab.com/api/v4/projects/{gitlab_project}/terraform/state/{state_key}-{stage}/lock>
    TF_HTTP_UNLOCK_ADDRESS: <https://gitlab.com/api/v4/projects/{gitlab_project}/terraform/state/{state_key}-{stage}/lock>

where: • gitlab_project and state_key are defined in catalog/terraform/sso

Questions: • Do we persist/show environment variables somehow? • Terraform says the environment variable is TF_HTTP_ADDRESS. Atmos isn’t prefixing with TF_VAR by chance, is it?

@Mike Crowe we don’t replace tokens in ENV vars

then executing atmos commands, it detects the ENV vars, adds them to the shell that executes the commands, and show them in the output

https://github.com/cloudposse/atmos/pull/77

Using ENV vars: shows the detected ENV vars for the component and they are already in the executing shell

@Andriy Knysh (Cloud Posse) – are ENV vars used in the terraform init pass? I’m getting no ENV files when I run:

❯ atmos terraform plan sso --stack=pinnsg-ue1-dev 

Variables for the component 'sso' in the stack 'pinnsg/ue1/dev':

enabled: true
environment: ue1
namespace: psg
region: us-east-1
stage: dev
state_key: pinnsg
tenant: pinnsg

Writing the variables to file:
components/terraform/sso/pinnsg-ue1-dev-sso.terraform.tfvars.json

Executing command:
/usr/bin/terraform init

It pulls my vars fine – but I don’t get any environment variables

env section is processed on all commands and is printed in the console

show me your YAML config @Mike Crowe

also check that you are using the latest version of atmos

for the backend type http, can’t we just serialize it the way we do for s3?

I don’t think so. As best I can figure, it looks like the http back in does not support terraform workspaces.

(oh this was a question for andriy)

I get this error: Executing command: /usr/bin/terraform workspace select pinnsg-ue1-dev workspaces not supported

We’ll have to review the http backend and update atmos to support it

I guess the issue is that we dynamically manage the workspace parameter as part of the backend configuration

so that’s why it needs to be handled per backend

maybe we could generalize it, did not think about it since we used only s3 backend to generate backend files (we have other backends, but they are custom)

I’m going to move our back into S3 to avoid this whole issue

tfe-cloud-infra-automation is a pretty out of date / abandoned. TFE isn’t the best solution on the market today so Cloud Posse didn’t continue to invest into it + we don’t have many folks contributing to that particular module.

Unfortunately, I don’t know of another module out there that automates TFE, so don’t know where to direct you. I would read through the TFE provider resources and see if any of them do what you want to do.

I believe this is how you create Env vars btw — https://registry.terraform.io/providers/hashicorp/tfe/latest/docs/resources/variable

Thanks @Matt Gowie, appreciate your response.

Ya, unfortunately, we were’t able to get any customers on board for TFE so this is on pause until such a time

Instead see https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation

Do you have try()?

couldnt you use a 0 index? like myset[0]

@RB: I thought [0] indexes were only for lists.. sets don’t have any order, from what i’m reading? I’ll try now though, hold

## This will give me a set of roles, which there 
## should be only ONE element (ie. one role). Just wanT
## that role =]
data "aws_iam_roles" "readonly" {
  name_regex =  "AWSReservedSSO_AWSReadOnly_.*"
  path_prefix = "/aws-reserved/sso.amazonaws.com/"
}

… I think I solved it. first convert it to a list tolist() then get the 1st element of that list element()

# tf0.15 syntax
readonly_role_name = one(data.aws_iam_roles.readonly).name

# tf0.14 syntax
readonly_role_name = element(tolist(data.aws_iam_roles.readonly.names),0)

nice!

this would work too

tolist(data.aws_iam_roles.readonly.names)[0]

variable enable_iam_role {
  type = bool
}

resource "aws_iam_role" "this" {
  count = var.enable_iam_role ? 1 : 0
  # ...
}

?

Thank you sir!

Context: I’ve come up with an engine for my current client that does a large amount of what Atmos does but targeted squarely at multi-team hashicorp vault deployments (so like an internal SaaS for the many Vault secret/auth engines a team may want to use). I made the schema itself some yaml files as well but using Python and Yamale for validation.

we are constantly improving atmos and have a lot of new features to implement

and yes, schema validation is one of them

But as I add more features I find the manifest schema the most arcane part to manage. Additionally, maturation of the app could lead to parts of your manifest become REST or GraphQL queries. This leads down the line of openapi/swagger/et cetera and figuring out how declarative manifests can be created/managed at the start of a project so that the underlying schema threads through the component parts

we started w/o validation, but we need to add it since it will solve many possible issues with invalid config

thanks for pointing out, we’ll review Cuelang (this one looks good at the first glance )

I wish I were skilled enough to make Cuelang my b**ch but to really use it one needs more Golang acumen than I possess ATM

I did use it as a generic yaml schema validation task in a pipeline via its cli though, that wasn’t super hard.

(we were thinking about using Cuelang modules in our code, we’ll implement validation with Cuelang or w/o)

Perhaps integrating it for the validation bits would be something to look into as it claims to have automatic backwards compatible schema capabilities and such

I believe istio uses it

I was going to offer up my toddler level work in using it for basic schema validation but I’m certain istio has far for interesting examples….

do you have any istio links showing the examples of using Cuelang?

I think they only use it for schema definition for translation into OpenAPI, it was pretty complex as they have several repos working on concert

spacelift

https://cloudposse.com/faqs/why-do-you-recommend-spacelift/

https://spacelift.io/case-studies/cloud-posse

https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation

Thanks for all that!

Beyond the success story above — I’d be curious to hear how many (appx #) of clients that are having a good outcome with spacelift?

more than 10 of our clients are using Spacelift now

Thanks again!

in fact, we love it so much that it’s part of atmos, our command-line tool

https://github.com/cloudposse/atmos/tree/master/pkg/spacelift

https://github.com/cloudposse/terraform-yaml-stack-config/tree/main/modules/spacelift

https://github.com/cloudposse/terraform-yaml-stack-config/tree/main/examples/spacelift

I use spacelift, not part of cloudposse, and big fan

Yeah Spacelift gets two thumbs up from me as well

There’s a YouTube video that explains the contexts here https://youtu.be/V2b5F6jt6tQ

Check out the docs on Atmos as well — https://docs.cloudposse.com/

Thanks guys

import your existing zone & SOA record

usually dns primary is provisioned in the dns account and then dns delegated is provisioned in a member account with the same hosted zone as the one in the primary with a different subdomain

@RB Don’t you have to create the name server records in the dns primary zone for the delegated dns?

i believe that’s what dns-delegated does

https://github.com/cloudposse/terraform-aws-components/blob/e28ac31f7333843184c6996474d75b45cdca9c67/modules/dns-delegated/main.tf#L37-L39

you’d need a lambda to do this id imagine

unfortunately there isn’t an aws_tag resource to generically tag resources, if there was, we would be and to create a resource with an aliased provider to create these tags cross account

Yeah I figured. Thought there was a way of doing this by using an aliased provider, assuming a role in the owner account and making changes there but doesn’t seem to be a way to modify an existing TGW attachment

I’m creating a cloudposse vpc, dynamic-subnets and alb, as well as a aws_ecs_cluster. And then I have this ecs-web-app module:

module "web_app" {
  source = "cloudposse/ecs-web-app/aws"
  version     = "0.67.1"

  region      = var.region
  name        = var.project_name

  launch_type = "FARGATE"
  vpc_id      = module.vpc.vpc_id

  container_environment = [
    {
      name  = "LAUNCH_TYPE"
      value = "FARGATE"
    },
    {
      name  = "VPC_ID"
      value = module.vpc.vpc_id
    }
  ]

  # Container
  alb_container_name = module.this.name
  container_image   = "${var.aws_account_id}.dkr.ecr.${var.region}.amazonaws.com/${var.ecr_repository_name}:${data.aws_ecr_image.flask.id}"
  container_cpu     = 256
  container_memory  = 512
  container_port    = 80
  build_timeout     = 10

  # CI/CD
  codepipeline_enabled = true
  github_oauth_token    = "/Prod/GITHUB_OAUTH_TOKEN"
  github_webhooks_token = var.GITHUB_TOKEN
  repo_owner            = var.repository_owner
  repo_name             = var.repository_name
  branch                = var.repository_branch

  badge_enabled        = false
  ecs_alarms_enabled   = false

  # Autoscaling
  autoscaling_enabled               = false
  autoscaling_dimension             = "cpu"
  autoscaling_min_capacity          = 1
  autoscaling_max_capacity          = 2
  autoscaling_scale_up_adjustment   = 1
  autoscaling_scale_up_cooldown     = 60
  autoscaling_scale_down_adjustment = -1
  autoscaling_scale_down_cooldown   = 300

  # ECS
  aws_logs_region         = var.region
  desired_count           = 1
  ecs_cluster_arn         = aws_ecs_cluster.default.arn
  ecs_cluster_name        = aws_ecs_cluster.default.name
  ecs_security_group_ids  = [module.vpc.vpc_default_security_group_id]
  ecs_private_subnet_ids  = module.subnets.private_subnet_ids

  alb_security_group                              = module.alb.security_group_id
  alb_target_group_alarms_enabled                 = true
  alb_target_group_alarms_3xx_threshold           = 25
  alb_target_group_alarms_4xx_threshold           = 25
  alb_target_group_alarms_5xx_threshold           = 25
  alb_target_group_alarms_response_time_threshold = 0.5
  alb_target_group_alarms_period                  = 300
  alb_target_group_alarms_evaluation_periods      = 1

  alb_arn_suffix = module.alb.alb_arn_suffix

  alb_ingress_healthcheck_path = "/"

  # NOTE: Cognito and OIDC authentication only supported on HTTPS endpoints; here we provide `https_listener_arn` from ALB
  alb_ingress_authenticated_listener_arns       = [module.alb.https_listener_arn]
  alb_ingress_authenticated_listener_arns_count = 1

  # Unauthenticated paths (with higher priority than the authenticated paths)
  alb_ingress_unauthenticated_paths             = ["/", "/test"]
  alb_ingress_listener_unauthenticated_priority = 50

  # Authenticated paths
  alb_ingress_authenticated_paths             = ["/api/v1/*"]
  alb_ingress_listener_authenticated_priority = 100

  aws_logs_prefix   = module.this.name

  # authentication_type                        = "COGNITO"
  # authentication_cognito_user_pool_arn       = var.cognito_user_pool_arn
  # authentication_cognito_user_pool_client_id = var.cognito_user_pool_client_id
  # authentication_cognito_user_pool_domain    = var.cognito_user_pool_domain

  context = module.this.context
}

may be i guess you could associate the load balancer with the service for eg: like this

resource “aws_ecs_service” “nginx_deployment” { name = “nginx-deployment” task_definition = aws_ecs_task_definition.nginx_deployment_definition.arn cluster = data.terraform_remote_state.platform.outputs.ecs_cluster_id desired_count = “1” launch_type = “FARGATE”

network_configuration { subnets = data.terraform_remote_state.platform.outputs.private_subnet_ids security_groups = [ aws_security_group.ecs.id ] }

load_balancer { container_name = “nginx-deployment” container_port = “80” target_group_arn = aws_alb_target_group.nginx_deployment_green.id }

}

Hi, thanks for the reply! Doesn’t the ecs-web-app module already set up all the required connections? I thought it would make and hook up the service and any necessary target groups.

I definitely see that, even if I remove the target groups on AWS, the target groups get created. (They are both for 80/HTTP though, oddly.) It’s just that one of them isn’t associated with the load balancer. I am not sure which module is creating this target group but then not associating it with the alb.

OK, so I diagnosed this by adding attributes = ["webapp"] to the web app declaration and “alb” to the ALB. And that showed me that the errant target group was being created by the web app module. So then I found that creating the target group in the web app is configurable, so I turned it off:

  alb_ingress_enable_default_target_group = false
  alb_ingress_target_group_arn = module.alb.default_target_group_arn

And now there’s a single target group (which is what I wanted), and it’s properly connected to the ALB.

I wonder if ecs-web-app has changed, because it seems that this behavior means the web app examples as written don’t work.

for posterity if someone else is getting the same error: I had the same error using ecs-web-app module only (I did NOT use the other cloudposse modules, had created VPC, subnets, ECS cluster and ALB separetely), and my issue was that ecs-web-app creates the aws_lb_listener_rule in cloudposse/alb-ingress/aws child module conditionally based on the var.alb_ingress_unauthenticated_hosts, var.alb_ingress_unauthenticated_paths, etc. variables. If the relevant variable is empty, the listener rule is not created, so the target group will throwe the error with no ALB attached to it. spent more time with this then I care to admit, hope no one else have to.

What does the repository_url output from the ecr module look like?

Hi Nick … tks for your help … :slightly_smiling_face:

the content of repository_url is: [20xxxxxxxxx1.dkr.ecr.us-east-1.amazonaws.com/tiktok-front](http://20xxxxxxxxx1.dkr.ecr.us-east-1.amazonaws.com/tiktok-front)

Cool, so you should be able to do something like this:

container_image = "${module.ecr_tiktok_api.repository_url}:my_image_tag"

I’m not entirely sure why you have two ecr modules declared?

i am creating a template module based in the ecr module … to give to our devs … the ability to create their own “infrastructure” … based just in the template module the is simplified than the full ECR module !!! this is just a Test …

sounds good. hopefully that will work for you

thank you very much for your help

I suspect there can be subtle changes. The reason it’s probably not discussed is b/c terraform has promised all versions of 1.x will be backwards compatible, just not always forwards compatible.
You should be able to upgrade from any v1.x release to any later v1.x release. You might also be able to downgrade to an earlier v1.x release, but that isn’t guaranteed: later releases may introduce new features that earlier versions cannot understand, including new storage formats for Terraform state snapshots https://www.terraform.io/language/v1-compatibility-promises

Is your concern more about reading remote state from within 0.14 root modules to some modules provisioned with 1.1?

Just wondering if I can go from 0.14 -> 1.1 directly. The v1.0 upgrade notes say

(all signs point to - just try it )

yea, honestly - that’s your best course of action

Ah - they are explicit about it https://www.terraform.io/language/upgrade-guides/1-0#remote-state-compatibility
If you are upgrading from Terraform v0.14 or Terraform v0.15 to Terraform v1.0 then you can upgrade your configurations in any order, because all three of these versions have intercompatible state snapshot formats.

Thank you sir