#terraform (2022-06)

You can use an scp

https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps_examples_ec2.html#example-ec2-1

Thanks, there is some equivalent to SCPs in GCP ?

not sure to be honest. if you find the equivalent, please let me know

https://github.com/cloudposse/terraform-aws-transfer-sftp/pull/21#discussion_r886952836

Wow!

I wonder if this PR would kick it out of an experiment and make the feature GA

Oh yep, the PR removed the experimental function for defaults/optional so it could be GA once that’s merged and released

yeah, the discussion says it could make it into the 1.3 release

You should limit login and make sure to use Okta or something similar to protect effectively. As for plans, it is good to auto plan and require manual configuration to make sure plans look good and will not negatively impact a stack.

I am a zero trust type person. I believe it is best to restrict access to things people really need access to.

I’m see, good tips. Do you allow local-previews for your devs?

We try to keep things simple. Generally, anyone with read access to our GH org will have read access to Spacelift. A smaller group of devops/SRE staff will have access to make changes, most via PRs to a Terraform configuration using the spacelift provider. And these people have admin access to Spacelift to make runtime break-glass type changes

Thanks for sharing, will come in handy as we explore.

Good question for spacelift

(related, have you seen https://github.com/spacelift-io/spacelift-policies-example-library? this is a relatively new resource)

Oh, no I haven’t! Thanks for sharing, will come in handy. Speaking of Spacelift, I have a question for office hours!

ok, tee it up

#office-hours

I did, but just got dragged into a meeting :( I’ll try to get out as soon as I can but please do have the discussion without me

surprisingly, we manage several thousand assignments in a single configuration and don’t see this. Our SSO is homed in us-west-2.

You could try setting -parallelism=1 as a workaround

it’s not clear if you are talking about a list, map, object, or set here

for lists, you can use range https://www.terraform.io/language/functions/range

For sets, maps, and objects, you can convert to list and use range as above.

if you are working with a list of values, I think you want the slice() function: https://www.terraform.io/language/functions/slice

variable "test" {
  type    = list(number)
  default = [1, 2, 3, 4, 5]
}

output "first-three" {
  value = slice(var.test, 0, 3)
}

output "last-three" {
  value = slice(var.test, 2, 5)
}

gives:

first-three = tolist([
  1,
  2,
  3,
])
last-three = tolist([
  3,
  4,
  5,
])

btw, the a negative number is not a valid index error is coming from TF interpreting the [0-2] as a artithmetic operation… its subtracting 2 from 0 and then trying to use that as the index for the list.

That would be rad!

Promote it on your linkedin!

Yup, will add it to the queue

Ah this would be so nice. Nice find loren!

I’m jsonencoding this after

I need the commas?

example:

    {
      "title": "Sample Pet Store App",
      "description": "This is a sample server for a pet store.",
      "termsOfService": "<http://example.com/terms/>",
      "contact": {
        "name": "API Support",
        "url": "<http://www.example.com/support>",
        "email": "[email protected]"
      },
      "license": {
        "name": "Apache 2.0",
        "url": "<https://www.apache.org/licenses/LICENSE-2.0.html>"
      },
      "version": "1.0.1"
    }

speech marks too

this is HCL format

I was using this before :

that worked fine

both your first example and second example pass the sniff test https://www.hcl2json.com/

and it does not work

@matt

~Looks like in your first example the x-amazon-apigateway-integration is outside the GET method block~

Nevermind…my eyes were playing tricks on me…haha

Are you using the Cloud Posse API Gateway module? You shouldn’t be JSON encoding it before passing it in…

See the example here

I’m using the cloud Posse module

but I’m setting an example prive api

For those interested, we’re requesting feedback and bug reports in this post! Appreciate any and all alpha testing

@RB

https://github.com/cloudposse/terraform-aws-ecs-web-app#input_launch_type

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless-v2.html

Scroll down to RDS Serverless v2 Cluster you might see a little more context. You can have it scale with replicas or more compute units.

Serverless uses are there so you can have a better understanding of why you may or may not use it

AWS Docs require multiple rounds of reading and parsing (for me at least) as they condense ALOT of info in a small amount of text.

I read all that before posting the question Thanks anyways

Does this mean it automatically scales replicas in engine mode is serverless? <–yes if you setup scaling metrics

I think? that serverless automatically scales rather than you creating a scaling policy.

haha thanks @Josh

Well it can scale on two parts.

yes if you setup scaling metrics can you elaborate a bit on “setup scaling metrics” part?

I think? that serverless automatically scales rather than you creating a scaling policy. It definitely scales capacity units automatically

You can add a scaling policy to the cluster to scale replicas based on CPU or connection percentage.

scaling policy? Any link to docs or terraform argument?

Let me look.

Are you using the cloudposse module?

not using any module

You may have to use something like this https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appautoscaling_target

ok. that would indicate aurora serverless v2 is not scaling replicas automatically

Not really; without setting up a policy, then it is automatic, pretty much. If you were even setting the ACU to scale from .5 - 2.0 and capped, you would still need to manually up the scaling limit on that. If you want it automatic and do not have money worries, max it out

I am not sure how to target the specific module. use the -target flag. I pass the flag but TF builds everything.

I don’t believe you can.

you can

trying to find an efficient way. If terraform then less code without declaring all regions. if via script then a safe script that does the job

this has been my preferred method in the past https://github.com/gruntwork-io/cloud-nuke

we usually use this repo https://github.com/cloudposse/terraform-example-module as a template for all new terraform modules

Yep, my team is getting 404s and Bad Gateways

seems like it’s back now

There’s an outage: https://status.hashicorp.com/

sigh, why does this have to happen on a friday at lunch time

i had a lot of changes i needed to apply today

it’s intermittent

Heres my code and the error im getting…

data "aws_iam_policy_document" "config_sns_policy_doc" {
  statement {
    effect  = "Allow"
    actions = ["SNS:Publish"]
    principals {
      type        = "Service"
      identifiers = ["config.amazonaws.com"]
    }
    resources = [aws_sns_topic.config_sns_topic.arn]
    condition {
      test     = "StringEquals"
      variable = "AWS:SourceAccount"
      values = [
        "${slice(var.workspace_acct_number, 0, 8)}"
      ]
    }
  }
}


variable "workspace_acct_number" {
  type = list(string)
  default = [
    "123",
    "456",
    "789",
    "987",
    "765",
    "543",
    "169",
    "783"
  ]
}

│ Error: Incorrect attribute value type
│
│   on modules/config/data.tf line 132, in data "aws_iam_policy_document" "config_sns_policy_doc":
│  132:       values = [
│  133:         "${slice(var.workspace_acct_number, 0, 8)}"
│  134:       ]
│     ├────────────────
│     │ var.workspace_acct_number is list of string with 8 elements
│
│ Inappropriate value for attribute "values": element 0: string required.

Ive tried "slice(var.workspace_acct_number, 0, 8)" and that literally spits out that line as what TF wants to put in my policy. Im just not getting the syntax right here or something I think

slice returns a list, so you don’t need to wrap it in []

You also do not need interpolation, "${...}", since you aren’t doing any string concatenation here

So just values = slice(...)

yep…that worked. Thanks!

I knew it was something i was missing… [ ]

Sorry guys it was local issue - for resolving need to clean cache from ~/.terraform.d

In my experience just seems statefiles are such pain especially if youre not versioning them. Im not sure what the benefits are for TF Cloud over self init’d TF deployments… Ok just looked it up, here we go for the FREE version.

1. State management
2. Remote operations
3. Private module registry
4. Community support

this would seem to be only perk… Community support but I think we have that already for free, so there really aren’t any…

Yeah for #3 you can use git repos? For #4 hashicorp community forum and SO? Github actions or weaveworks terraform operator for #2?

I wouldn’t switch if you’ve already got everything setup and automated on S3. Locking yourself in for no additional benefit AFAIK.

@Matt Gowie I would definitely agree I see no benefit in doing so… @OliverS #2 you can use gitlab docker runners and for #3 https://docs.gitlab.com/ee/user/packages/terraform_module_registry/

I wonder if a private tf registry could be a good way to control access to in-house (ie non open source) tf modules…

Can tf cloud users outside of one’s tf cloud organization be given read-only access to an org’s private tf registry?

Now that I dont know, Im famiiiar with self hosted Gitlab and with that, yes via access tokens.

PATs cannot limit access to just the repo in read-only on specific branch, that I know of.

security should be built-in and not something you have to pay for

not sure what you mean by graphs, but terraform-docs on github will document your Terraform modules

Yeah, that I already use. I am thinking for more of graph tools. With terraform docs I use for doc modules in text format

So far I found diagrams.mingrammer (but you need to write code for it ) and rover (not exactly graphs)…

Do you mean something like using the output of terraform graph to create a more detailed version of your aws/azure resources? Have you consider using Lucidscale?

Have you tried https://github.com/cycloidio/inframap

It sounds like a bug in their module have you tried creating an issue on the repo? It would also help to see a minimum viable reproducible example.

I came here for asking first, just in case I missed something. I’m creating an issue in a minute

This is the issue: https://github.com/cloudposse/terraform-aws-eks-cluster/issues/155

@Jeremy G (Cloud Posse)

@Julio Chana The kubernetes_config_map_ignore_role_changes setting is a bit of a hack. Terraform does not allow parameterization of the lifecycle block, so to give you the option of ignoring changes, we implement the resource twice, once with ignore_changes and once without.

If you set kubernetes_config_map_ignore_role_changes to true, then the expected behavior is that you will not be able to add or change Roles later, because you have told Terraform to ignore changes to the Roles. However, if you change the setting from false to true, the old ConfigMap is deleted and new one is created, so the first time you will get your roles added.

I’m attaching an example when I change the variable (from false to true). The screen shot you provided looks like what would happen if you change kubernetes_config_map_ignore_role_changes from true to false and it looks like it includes the Roles you want. So I think you may just be understandably confused by Terraform’s and this module’s confusing behavior around how Role changes get ignored. I suggest you test it a little more with kubernetes_config_map_ignore_role_changes left at false (which is the only setting that will let you update the Roles after first creating the cluster) and report back if you still have questions.

Thanks for your answer.

My concern here is that, in the case that I let it to false, the worker roles will disappear, even though in the auth.tf file it’s concatenating them with

mapRoles    = replace(yamlencode(distinct(concat(local.map_worker_roles, var.map_additional_iam_roles))), "\"", local.yaml_quote)

This is why I’m more confused. I would love to manage all the authentication with terraform, but without breaking the regular configuration.

@Julio Chana just left a comment on the above issue. You can workaround it by first reading the existing config map using a data source then merge that with your settings

https://github.com/cloudposse/terraform-aws-eks-cluster/issues/155#issuecomment-1164123116

@Jeremy G (Cloud Posse) maybe we could implement this in the official module?

I’m happy to provide a PR

If you look at the Cloud Posse EKS Terraform component (think of it at this point as a work-in-progress), you see that the way we handle this is to output and then read the managed role ARNs to include them in future auth maps.

My concern about reading the map in the module before writing to it is that the map does not exist before the EKS cluster is created or after it is destroyed, and everyone has had a ton of trouble just keeping it updated as is.

You can make the datasource depend on the eks cluster creation so you will always have it.

@Sebastian Macarescu You are welcome to give it a try, but be warned: there be dragons. I suggest you read the comments in auth.tf and the v0.42.0 Release Notes before proceeding.

Hi @Jeremy G (Cloud Posse). I have implemented a PR that fixes this: https://github.com/cloudposse/terraform-aws-eks-cluster/pull/157

@Julio Chana You can take a look at the above

Feel free to post here and someone could help

@RB sorry updated the message

@Karina Titov have you seen this example ? https://github.com/cloudposse/terraform-aws-waf/blob/master/examples/complete/main.tf

yes, unfortunately, there is no example of the scope_down_statement or and_statement usage

This may be a limitation in the component

https://github.com/cloudposse/terraform-aws-waf/blob/cc930d80a0d388cf503426014b6f855e879ff129/rules.tf#L405-L417

looks like it. should i create an issue to include the scope down section?

Sure thing !

A pull request is also encouraged :)

sounds good! thank you

I just use the context and let it be defined. I define a label module in my IaC that calls cloudeposse/eks-node-group/aws and pass context = module.<label>.context with it

And this tags the EC2 instances with Name label?

Ah so you’re trying to get the EC2 instances from the node group to be tagged…

In order to get the instances to have the tagging applied to them you’ll need to be sure to set the resources_to_tag variable sent to the cloudposse/eks-node-group/aws module.

I set mine to resources_to_tag = ["instance", "volume", "elastic-gpu", "spot-instances-request", "network-interfaces"] to enable all possible tagging options but I could have probably left of “elastic-gpu” and “spot-instances-request” as we aren’t making use of either options currently

Thank you that works and prevents me from having to create a custom Launch Template.

no problem… I had to figure it out for our deployment as well

Here’s more information on why name isn’t used

https://github.com/cloudposse/terraform-aws-ec2-autoscale-group/pull/63

I previously added name because I wanted to import existing autoscaling groups into that module but there is a create_before_destroy lifecycle on it which causes issues. Also I accidentally caused one issue in PR 60 with that implementation .

Perhaps this can be revisited ? cc: @Jeremy G (Cloud Posse)

Okay, create_before_destroy could be set up via variable

And for some uses it could be okay

create_before_destroy is a lifecycle option and these are not parametrizable. To implement them (where we do) we have to provide 2 separate resources, identical except for the lifecycle rule, and choose which one to use. This is a maintenance problem and can cause problems when switching from one to the other.

For an Auto Scaling Group, the only advantage of destroy before create behavior is that you can set the name of the group to be exactly what you want. That does not seem to me to be compelling enough to warrant the additional effort and resulting issues that come with supporting 2 variants.

thank you, for answering.

You could simply invoke the module itself with the pure repository url and leave the configuration of your git client to your pipeline. Something like git config --global url."[https://oauth2:$](https://oauth2:$){{ secrets.GH_TOKEN }}@github.com".insteadOf [https://github.com](https://github.com) would in this case setup git access by PAT.

awesome thanks that’s what I was looking for

it’s also possible to use a .netrc file

https://gist.github.com/technoweenie/1072829

storing s3 in the shared account and use dynamo in the child account?

Is it possible to tell Terraform to use local dynamodb in the accounts where it is running plan and apply?

this is related to the backend definition

you want to tell it what role to use and if no role, it will assume the current role

note, providers and backends can both use different roles

so by the sounds of it, your provider might be correct, but your backend is not

@Erik Osterman (Cloud Posse) I added role_arn to backend.tf and it worked as expected. Thanks

I am kind starting to learn and use new relic at work for aws and I do have several questions

Would you be able to give me a hand ?

Sure. Please DM me.

Well, the fundamental problem IMO is you have too much state in one root module.

If editing one root module can affect 1500 sites, that seems like a massive blast radius

I would design it in partitions

Can’t help with the first question but can with the second question:
what is the best practice for passing environment variables to this web app’s task definition? How can it be done without hardcoding values to the code? My team uses the "secrets" section of the task definition with "valueFrom" to specify an location in parameter store. Looks like this:

"secrets": [
{
    "valueFrom": "arn:aws:ssm:${region}:${account_id}:parameter/shared/datadog/API_KEY",
    "name": "DD_API_KEY"
}
]

so you would be storing your secrets in parameter store and then giving permission to read the secret to the role assumed by the task. You can do the same thing with Secrets Manager if you’d prefer.

I should add that the task definition is defined as a template and gets called from terraform like this:

data "template_file" "ecs" {
  template = file("${path.module}/templates/container_definition_template.json")
  vars = {
    name = var.name
    environment = var.environment
    region = var.aws_region
    account_id = data.aws_caller_identity.id.account_id
  }
} 

Thanks, that’s very helpful! Could you elaborate on the task definition? I can’t find anything in the docs\examples about this data "template_file" "ecs"

https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file

Basically, you give TF a template and pass the variables in. It will fill in the template with the variable’s values and return the “filled in” template.

I use the template example from above to create a zip file with a appspect.yaml and a taskdef.json file for codepipeline to deploy to ECS:

 data "archive_file" "codepipeline" {
  type        = "zip"
  output_path = "${path.module}/templates/appspec_taskdef.zip"

  source {
    filename = "appspec.yaml"
    content  = data.template_file.appspec.rendered
  }

  source {
    filename = "taskdef.json"
    content  = data.template_file.taskdef_pipeline.rendered
  }
}

I might be leading you down a path that is not aligned with the Cloud Posse methods of deploying an ECS service. But just know there are multiple ways to get it done.

Thanks! I guess I’ll have to keep learning Terraform as this isn’t intuitive to me yet because I’m a bit confused about where this “template_file” came from and how everything connects together.

yes there are layers to it. dig in and you will find there is plenty to learn.

Have you looked into cloudposse/atmos (see atmos) for your environments? Its the tool we use to use yaml to auto create tfvars files per env which avoids the env/ directory structure

Also you may want to replace that data.template_file data source with the templatefile function

Ah I see the issue

there is no count logic on the aws_db_proxy resource

actually, it’s on zero of the resources

for now, if i were you, i would add a count logic on your module to enable / disable it

thanks! i’ll give that a try

I put this PR in for now https://github.com/cloudposse/terraform-aws-rds-db-proxy/pull/7

why not just use directly a templating system like handlebars, jinja, moustache or like even twig

aws-ia control tower uses jinja2 to template hcl

Im not recommending it by any means but it has been done

HCL is such an obscure format with so many strange syntactic aspects. Is it really wise to template YAML in it?

I’ve been impressed with gomplate for templating

i’m leaning towards the templating tools you all mentioned because I don’t want to use the wrong tool for the job. but just to answer @Alex Jurkiewicz, I needed to generate for multiple environments which using terraform workspaces makes that extremely simple as well as automatically showing terraform diffs of changes. I imagined the HCL to only use templatefile() function with a yaml template so i don’t think the syntax would be that big of an issue. I understand that terraform is technically not the tool for the job though

for those that are interested, HCL ended up being a bad idea in hindsight after doing a poc when compared to gomplate and jinja2

No document. If you see an issue or want to add a feature, the normal fork, branch, pull request model is the way we usually do it

Thank you