#terraform (2024-05)

Hrmmm… I have a theory. We recently rolled out some new workflows, maybe the artifact wasn’t produced.

@Igor Rodionov can you take a look?

Essentially, what the module does is a clever hack to download the artifact from S3 based on the commit SHA of the module version you are pulling

For some reason, the commit sha corresponding to that module version does not exist, which leads me to believe there’s a problem with the artfact and something wrong with the pipeline

cc @Gabriela Campana (Cloud Posse)

Thanks for getting back to me. Would this not effect all users then?

Yep

….all users using it at that version

For now, try an older version

ok… standby

Issue with an older version of the module is that AWS TF provider 5 has depreciated some calls.

"source_json" and override_json

I gotcha… so this is a problem, especially if you manage the terraform-aws-lambda-elasticsearch-cleanup in the same life cycle as, say, your elastic search cluster. While that’s not unreasonable, and what most probably are doing, it’s an example of why we like to break root modules out by lifecycle, reducing the tight coupling and dependencies on provider versions. That said, I totally get why this is a problem, just explaining why we (cloud posse) are less affected by these types of changes.

See https://atmos.tools/core-concepts/components/#best-practices

Makes sense . I just grouped things together that went together. Ending up in this situation.

Thanks for understanding… We’ll get this fixed, just cannot commit to when that will be.

No issues, this is just on my upgrade TF branch and not on master. So I am good for now

DEV-2134 : Fix issue with terraform-aws-lambda-elasticsearch-cleanup module. The commit sha corresponding to that module version does not exist

This is really more of a #kubernetes question, but I will take a crack at it here.

It seems to me you are confusing the EBS volumes associated with EC2 Instances as root volume, providing ephemeral storage (e.g. emptyDir) for Kubernetes, with EBS volumes associated with PersistentVolumes. The former have lifecycles tied to the instances: When new instances are created (e.g. when the AutoScaling Group scales up), new EBS volumes are created, and when the instances are deleted, so are the EBS volumes.

Kubernetes PersistentVolumes, which may be implemented as EBS volumes or something else, should persist until their PersistentVolumeClaims are deleted, and then only if the reclaim policy is set to “delete”.

Thanks @Jeremy G (Cloud Posse), though I’m using StatefulSet for my deployment (I have EBS CSI driver setup as well) so I thought this should be using EBS that from what I understood should be independent with my EC2 lifecycle?

In my StatefulSet deployment I have defined volumeClaimTemplates that from what I understood should be using the EBS volume? Thank you for the answer though should I post this to K8 (my bad for posting here because I was using Terraform to maintain our infra) and continue the discussion there? :o

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: authentication-postgres
  labels:
    app: authentication-postgres-app
  namespace: postgres
spec:
  serviceName: authentication-postgres-svc
  replicas: 1
  selector:
    matchLabels:
      app: authentication-postgres-app
  template:
    metadata:
      labels:
        app: authentication-postgres-app
    spec:
      containers:
        - name: authentication-postgres-container
          image: postgres:16.2-bullseye
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: authentication_db
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: authentication-postgres-secret
                  key: postgres-password
          volumeMounts:
            - name: data
              mountPath: /mnt/authentication-postgres-data
      imagePullSecrets:
      - name: docker-reg-cred
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: "5Gi"

What I’m saying is that on upgrade, some EBS volumes will get deleted and some will not. What leads you to believe that your PersistentVolumes, with active PersistentVolumeClaims, are the ones being deleted?

Ohhh I see! I came to this conclusion because the PostgresSQL database lost its data after I upgraded the nodes, which means it’s one of those EBS that unfortunately got cleared? Is there a way for me to avoid that so that when I upgrade nodes the EBS are safe?

I don’t know how you deployed PostgreSQL. It seems like you deployed it to use ephemeral storage rather than dedicated PersistentVolumes, despite having a volumeClaimTempate, but this gets into the details of your PostgreSQL deployment, and maybe Helm chart. Which is why I directed you to #kubernetes

If you are able to open a PR, post it in #pr-reviews and someone will review it promptly

PR sent. thank you so much @Erik Osterman (Cloud Posse)

I don’t see one from you in #pr-reviews

yep, because i was little bit sleepy last night. approves are welcome.

Looks like https://github.com/cloudposse/terraform-aws-elasticache-memcached/pull/79 was already merged

I have that problem a lot when using a security group created in another backend.

ie:

Backend A: contains a resource group Backend B: uses the resource group

One pattern I’ve been using to avoid problems is

Backend A: contains a resource group Backend B: attaches rules to the resources group and uses these.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group_rule

Don’t know if that helps ¯_(ツ)_/¯

thanks! I will look into it. I manage all the resources in the same backend but I have little control on the creation/destruction as the cloudposse modules are the ones managing the resources

maybe post a link to this message in #terragrunt

ok, just to cross-post? or am i in the wrong channel?

Hi @Veerapandian M welcome to the community!

Definitely feel free to ask pointed questions as you continue your journey.

@Jeremy White (Cloud Posse)

If you just want to get synthetics up and running, I think you can just copy the synthetics example and adjust it to use your own datadog api endpoints. After that, just start creating synthetics similar to what you see in the synthetics catalog within the same example

Please use #terragrunt

@Jeremy G (Cloud Posse)

It’s a long story. Does this help?

Hmmm….Sounds like the best workaround is to just del all the rules for a SG and then run apply.

@setheryops Upgrade to cloudposse/terraform-aws-security-group v2 and that should take care of it for you.

Are you using a resource or module?

resource through a local module

FYI - for anyone who needs it.. had to resort to the aws-CLI, unless someone has a better way

resource "null_resource" "this" {
  provisioner "local-exec" {
    command = "aws iam update-assume-role-policy --role-name OrganizationAccountAccessRole --policy-document '${data.aws_iam_policy_document.update_assume_role.json}'"
  }
  triggers = {
    always_run = timestamp()
  }
}

related https://github.com/hashicorp/terraform-provider-aws/issues/7922

hmm perhaps not exactly the same. You may want to create a new issue for your use-case

Ah, on further reading, I think I found my answer: https://github.com/cloudposse/terraform-aws-elasticache-redis?tab=readme-ov-file#input_parameter

Not AKS, but with EKS, you typlically need to wait for the control plane to come online and be ready before deploying helm

Also, now you have a requirement for Terraform to have direct connectivity to the control plane

my helm release just sits there for ages going round for like 10mins and then comes back failed

I have three helm charts that are all doing the same thing

Yes, that sounds like control plane is either a) not reachable for network reasons b) not ready yet and timing out

hmm

thanks for the helm

i mean help lol

Here’s what we used to do.

https://github.com/cloudposse/terraform-aws-eks-cluster/blob/c8a4adfc3c122074868dc60552950327610fe2e2/variables.tf#L153

Literally called curl until we had a successful response from the cluster

hitting the /healthz endpoint

I did not know you could do that

then how did you pass in the variable?

did you do depends on default?

https://github.com/cloudposse/terraform-aws-eks-cluster/blob/c8a4adfc3c122074868dc60552950327610fe2e2/auth.tf#L63-L74

nice

I consider this a dirty hack. But there was no other way.

I think your message from earlier has put me in the right direction.

The pub IP im assigning to the cluster is not responding to pings

That’s it probably

With Terraform there are only dirty hacks

Im trying to move more and more away from it

I want to be a dev im sick of devops

Hahaha! Yes, I empathize with that. But to quote Mike Rowe, “It’s a dirty job, but someone’s gotta do it”

Yeh been doing it for 7 years now no more

Too many late nights now and early mornings

I literally watch devs throw stuff over the fence to DevOps and then sign off. I’m like, yep, that’s what I want to be doing. Also, I want to create new stuff; I’m sick and tired of creating Kubernetes Infrastructure or VMs. I used to do web apps, and that was okay.

Yup - that’s what we felt at Cloud Posse and why we built our module ecosystem. We were tired of doing the same thing over and over again, having to fix the same things over and over again. No convergence to a solution. Unfortunately, we don’t do much with Azure.

I was able to get this kind of provider-level dependency working by using the terraform_data resource to create a dependency between the provider authentication attributes and the resources that need to be ready, using the input of the terraform_data. On vacation at the moment, but happy to share when I get back

Basically, “don’t initialize even the provider until all the required deploy-time dependencies are ready.” Most approaches initialize the provider as soon as the cluster auth attributes are ready, and that isn’t sufficient

As in dont do a terraform init on the provider at the same time as the other stuff?

I cant quite grasp what you mean

No. I’ll post code later when I have access to it. One init, one apply. It’s just careful management of the edges in the terraform graph, so it can map out the dependency order correctly

So, I might be understanding you, build the cluster, then data resource it and use the data resource in the Kubernetes provider setup, not the resource part? But for that to work, I would have to re-initialize as on the first init, the cluster does not exist…

Yes, the provider references the terraform_data resource attributes, rather than directly referencing the cluster resource attributes. But no, terraform handles that provider dependency chain naturally in a single init/apply

I do it with eks, and even alias a couple providers… Build the cluster, link an aliased “before compute” provider, resources that need to exist before node groups use that provider, chain the node groups to those resources, link another aliased “after compute” provider to the node group arns, resources that require compute use that aliased provider…

Okay, if you can share some code, that would be amazing. I need to turn this in today. I’ve been on this for far too long.

Sorry, on vacation like I said. Won’t have code until Monday.

Okay, thanks. I’d still be interested to see how you did this when you’re back. Thanks for your help.

I remembered I wrote up half of it on a PR for the eks module we used… This does the “after compute” provider… https://github.com/terraform-aws-modules/terraform-aws-eks/pull/3000#issuecomment-2059599312

Okay thanks and in the module thats just a normal eks resource right?

Yes, only trick there is that some of the module outputs have a dependency on the eks access resources to ensure cluster authentication is setup before the outputs become available. Check the module code and you’ll see what I mean

So you work for AWS?

Oh gosh no, not at all

Ohh I thought you did because this the Github for AWS Provider?

No not the AWS provider, just a community-maintained terraform module for eks

Ohh

How do you find the time?

Im always blooming working or looking after my little girl…

Umm, #newclient was already using the module, and I was brought in to fix/modernize some of their practices, and the module was broken so I fixed it as part of the client engagement

Ohh nice

Thats good work. Where I work, I am on this project with a tight deadline, then get moved to another tight deadline, then move on to another project. I never get to have time to give back, which Is something I want to do.

Yeah we try to use/publish open source terraform modules as part of all our engagements. Really helps improve reuse, and teaches our own folks good hygiene

Excellent, I code in Go and want to do some open-source stuff with Providers. That would be nice.

Follow apparentlymart on GitHub? He’s a terraform-core contributor, and frequently publishes niche providers. could be a good model to follow for working on your own…

I need help finding this bit data.aws_eks_cluster_auth.this.token on this repo. The aws_eks_cluster_auth part.

okay ill go look them up

Oh that’s just a data resource offered by the AWS provider. Check the docs. I don’t have that snippet handy

I think it takes the cluster id or something as input

okay is it essentially just the eks cluster giving out its token?

Yes

Cool ive seen something similar in AKS

right I think I might have this together in my head.

This guy apparentlymart is a G

Im reading through his Go proposal

totally distracted now lol

Yes, dude is sharp as F, super thorough, writes well, and is crazy humble. Great role model

Is there a terraform registry on this terraform_data as in what fields you can pass in as I can only find this on it: https://developer.hashicorp.com/terraform/language/resources/terraform-data#example-usage-null_resource-replacement

That’s all there is to it. The input argument takes any object or expression, and its value is available as an attribute

I couldnt get it to work with an AKS

The general idea should be agnostic to aks vs eks, I think. It’s only relying on terraform core features

I set a reminder for when I’m back to try and provide the equivalent setup for aks, based on your snippet above. I just can’t test it since I don’t have access to an azure environment at the moment

is this what you had?

# Use terraform_data inputs to create resources after compute
provider "helm" {
  kubernetes {
    host                   = terraform_data.aks_cluster_after_compute.input.host
    username               = terraform_data.aks_cluster_after_compute.input.username
    password               = terraform_data.aks_cluster_after_compute.input.password
    client_certificate     = base64decode(terraform_data.aks_cluster_after_compute.input.client_certificate)
    client_key             = base64decode(terraform_data.aks_cluster_after_compute.input.client_key)
    cluster_ca_certificate = base64decode(terraform_data.aks_cluster_after_compute.input.cluster_ca_certificate)
  }
}

# Force a dependency between the AKS cluster auth details, and the compute resources
resource "terraform_data" "aks_cluster_after_compute" {
  input = {
    host                   = azurerm_kubernetes_cluster.r21_new_prod_kubernetes.kube_config[0].host
    username               = azurerm_kubernetes_cluster.r21_new_prod_kubernetes.kube_config[0].username
	password               = azurerm_kubernetes_cluster.r21_new_prod_kubernetes.kube_config[0].password
    client_certificate     = azurerm_kubernetes_cluster.r21_new_prod_kubernetes.kube_config[0].client_certificate
    client_key             = azurerm_kubernetes_cluster.r21_new_prod_kubernetes.kube_config[0].client_key
    cluster_ca_certificate = azurerm_kubernetes_cluster.r21_new_prod_kubernetes.kube_config[0].cluster_ca_certificate
  }
}

Yeh something like that

Didn’t work

“didn’t work” isn’t exactly something that can be investigated. what config, what command, what error?

Sorry, @loren. I was tired when I replied to this. Basically, for the AKS, no matter what, it has to read from the Kubeconfig file, and that does not update on a first run. With EKS, you can get around this by using the Token reference, but for AKS, there is no Token. For username and password, it still goes to the Kubeconfig file to get the Username and Password.

kinda curious, can you close the loop for me? what about this reference is “goes to the Kubeconfig file to get the username and password”? it certainly appears to me like it gets it from the cluster resource attribute, not from a file…

azurerm_kubernetes_cluster.r21_new_prod_kubernetes.kube_config[0].username

The way I understand it to work is the part where it says kube_config[0] its reading from the kube_config file

Best to post PRs in #pr-reviews

Our opinionated refarch implementation of an SFTP root module is here: https://github.com/cloudposse/terraform-aws-components/blob/main/modules/sftp/main.tf#L69-L88

https://github.com/cloudposse/terraform-aws-eks-cluster?tab=readme-ov-file#input_addons

after adding i see issue related to iam role, “AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity”

am i missing anything

@Jeremy G (Cloud Posse)

@Narayanaperumal Gurusamy Yes, the EBS addon requires an IAM Role with a specific name and with a specific policy attached. See how Cloud Posse does it here.

Thanks @Jeremy G (Cloud Posse) its helped lot.

Hashi announcement https://www.hashicorp.com/blog/terraform-aws-cloud-control-api-provider-now-generally-available

what does gpt say

Haha

I fixed it if you want to see the anwser

no pinning of provider versions in reusable modules. just set the min version.

in the root module, pin everything and use .terraform.lock.hcl

For child modules, it’s best to specify the minimum required version of a provider: https://developer.hashicorp.com/terraform/language/modules/develop/providers#provider-version-constraints-in-modules

on the other hand, for module references, always pin exact versions

nice. I can do that. Just have to go look at the renovate config settings and i’ll have our module project look at provider max (main version) and min being set.

yes, otherwise I always do exact version pinning and use renovate to process

just thought i remembered some issue with resuable modules and provider versions being exact being a problem but it’s been so long

yes, any given terraform state can only reference a single version of a provider at a time. so if you have two modules in the same config trying to exact-pin different versions of the same provider, it fails

do you’ll normally do just the minimum version instead of also the max range allowed? I see that in Cloudposse’s repos, but curious if you’ve seen benefit to do “between min and max versions” like >=3 , <= 4.0 when it’s on 3? Otherwise I’ll stick with minimum version and we’ll give up a little testability

i don’t see a ton of value in restricting the max version of the provider in reusable modules, since i’m pinning exact versions in the root config and also using dependabot/renovate against the root config to test updates to provider versions.

sure, the update may fail, but the project is still “healthy” and we can update the module for compatibility on our own pace

ok. I don’t have control downstream for consumers. Taking over part of the internal resuable modules and making sure i updated renovate config to the correct setup, so I’ll look disabling provider version updates only in my Azure DevOps project.

should be able to use required_provider depType based on examples in github and override this one project to make sure it doesn’t try to pin those. great advice thank you!

what you can do in your reusable module to get an “early warning” of an issue, is write test configs as root modules and pin provider versions there, and use renovate to update the test configs

yes! Structure is aligned to normal cloudposse files in base, examples directory, so I can match this and ignore pinning anything in root directory which isn’t typical for root modules

of course, you need a test harness with credentials that can apply and destroy the test configs

yeah part of that is in place with Nuke and maybe I’ll replace with either the native new test features in terraform or just write it like I did before in Go with some terratest calls. Either way it’s on the list!

depending on the resources in use, localstack can be a reasonable option in tests to avoid real credentials and resources/costs

we have credentials for azure, but i’ll check. haven’t seen if localstack does azure.

What I want is to just use pulumi and move on I’m using it for all my independent projects, but it’s hard to get interest in it so i’ll keep grumbling at terraform