#atmos (2024-11)

Interesting. So we are working on supporting an enabled flag for components.

https://github.com/cloudposse/atmos/pull/756

The scope is not currently to destroy. However, it might be worth considering that.

In the current implementation, enabled would make the component “invisible”

an alternative to commenting it out.

I can see a component as having 3 states

• enabled

• disabled

• destroyed

I think if the components are commented out or deleted from the stack file, describe affected should know what to do with it or output a destroyed flag or something like that for another job to use that matrix to destroy it

Ok, that makes sense.

So enabled = true/false affects the visibility, but removal from the configuration is visible via atmos describe affected

@Andriy Knysh (Cloud Posse) what happens today in describe affected if the component is removed? is that surfaced?

I renamed a component from pepetest to pepetest1, describe affected saw the new pepetest1 component got deployed, but the old one is still there in the cloud environment

the key is do we have the information in describe affected JSON output

The right behavior might not be implemented, but maybe we have the data there to act on

if a component is removed, Atmos does not see it (it will not consider it affected) - this is the current implementation. This is b/c Atmos compares the current branch with a remote branch/tag/sha - if the current branch does not have the component, then it’s “not affected”.

Setting enabled: false is not “removal”, so describe affected sees that

but with that, you need a two-step approach, one to say enable: false and then another PR to remove the yaml from the stack file

yes

i would say we didn’t consider a complete component removal with describe affected - we need to revisit this

Added a task to the backlog

and this one is becoming really important, more than the vendoring issue

Is it that you want to destroy if a component in a stack if it is removed (deleted or no longer inherited)?

Yes

Ideally I can run describe affected and maybe have two matrices, one with affected and one with deletions

So that I can do something about it in my workflow

Also interested in this feature. @Erik Osterman (Cloud Posse) I checked the backlog but I don’t see a task - is this the right place? https://github.com/orgs/cloudposse/projects/34/views/1

Oh our internal tasks, are not published to GitHub issues

The public roadmap shows tasks that originated from Issues opened by users

Thanks!

I know docs are in a bit of flux right now - Do you know where the best practice docs ended up?

This URL does not resolve to anything useful https://atmos.tools/best-practices/terraform/#use-feature-flags-list-or-map-inputs-for-optional-functionality and https://docs.cloudposse.com/reference/best-practices/terraform-best-practices/ is dead

Im trying to find guidance on vars.enabled. I didn’t implement it properly in my components/modules and now I cant delete components using my CICD workflow. No worries, can delete locally for now while I convince team we need to go through our components to implement this.

https://docs.cloudposse.com/best-practices/terraform/

@jose.amengual another one for you

this feels like Xmas

Something is wrong with your name_pattern or name_template

i didn’t change those.

Share your atmos config, if you can

https://atmos.tools/cli/configuration/stacks/

Ok, I think I initially misunderstood.

Are you setting workspace key prefix anywhere?

nope

@Andriy Knysh (Cloud Posse) any ideas

is the workspaces are port of the terraform state right?

i’ve tried to clean all tf files and deleted this workspace, but still got back named as dev so maybe the default workspace holds some information?

Yes, in atmos we use one workspace for each instance of a component deployed.

Have you updated to the latest atmos? We just fixed a problem related to workspaces and changing backends

yes, i’m using the very latest

but i bet it’s atmos what is switching to the workspace so the name dev comes from atmos not from the state

Yes, atmos dynamically computes the workspace name and switches to it

i see.

this is my dev stack file

and interestingly for the project and the cluster the names are generated correctly

i have a component named nats and stack named dev and instead of dev-nats the workspace is simply dev

this is the correct behavior for Atmos components that don’t inherit from other components

in this case, the TF workspace is simply the stack name

only if you have a derived component (inherited from a base component), then TF workspace will be <stack>+<component>

hmm. what you mean by “inherit” the do cluster and project name is correct and didn’t inherit from anything. or i miss something here.

components:
  terraform:
    nats:
      metadata:
        component: nats
      vars:
        ...

vs

components:
  terraform:
    cluster:
      metadata:
        component: do/doks

regarding inheritance, please see https://atmos.tools/core-concepts/stacks/inheritance/

ok i read that before. but i didn’t us that in any catalog so far.

in the two examples above, both nats and cluster Atmos components do not inherit from any other Atmos components, so the TF workspaces for both of them will be dev

so what you have is 100% correct, the workspaces for these two components are just dev - the stack name

maybe those were generated wrongly because i made a lot changes back and forth since.

yes, looks like it

anyhow i’ fine with a single dev workspace named after the stack. as long as the states are correctly separated.

i’m not fully familiar with tf workspaces, but that means if all these components are in the same workspace they are sharing the state or not ?

each component has workspace_key_prefix - it’s usually generated by Atmos, but you can override it per component

workspace_key_prefix is, if you look at the backend s3 bucket, the top-level folder

so each component will have it’s own top-level folder in the bucket, and each stack, in a separate TF worksapce, will have its own subfolder in the folder

so yes, each component state is separated from any other component state (diff folders in the state bucket)

note that it’s still in the same backend (same S3 bucket). If you want to separate backends (e.g. per tenant/OU, per account, etc.), you need to create and configure multiple backends

sure. but i don’t see workspace_key_prefix generated anywhere.

it’s gcs, not s3 actually.

_defaults.yaml:

terraform:
  backend_type: gcs
  backend:
    gcs:
      bucket: mw-tf-state
      prefix: platform/infra
      encryption_key: '{{ env "GCS_ENCRYPTION_KEY" }}'

GCP has prefix

which is the same

https://atmos.tools/core-concepts/components/terraform/backends/#google-cloud-storage-backend

you don’t need to hardcode it here

backend:
    gcs:
      bucket: mw-tf-state
      prefix: platform/infra

in this case, all components will use the same prefix

hmm ok, let me check that

please review the doc, if you don’t specify prefix, Atmos will auto-generate it

thank you!

so if i understand it correctly, without setting the prefix atmos will generate it and because of that my component states will end up in separate folders in the gcs bucket, so even they share a workspace states are separated.

good to know that.:)

only problem is that i prefer to store them in some folder instead of the root of the bucket but i can leave with that

note that you can specify the prefix per component, in which case Atmos will just use it

i can review your config, let me know

it’s fine. this bucket is solely for tf states so it’s ok even in the root. i prefer to leave it to atmos to generate.

on a different topic while we chat.. any chance to add support for command like this in atmos.yaml :

components:
  terraform:
    command: xy command -- tofu

so support command with double dash

it would be perfect that way i could load secrets as env vars. and i won’t need custom commands.

i don’t remember if command: xy command -- tofu is supported now (need to look at the code). Did you test it?

yes. i just tested unfortunately it’s not working.

I have done that with asdf and it works

ok, we’ll create a task for this, thank you

did you try to quote it?

thanks a lot!!

@Erik Osterman (Cloud Posse) do you have an example maybe with asdf?

i’ve just tried with quote but not working.

trying with a small shell script:

command: ./optofu.sh

but still not working

i’ve created a PR: https://github.com/cloudposse/atmos/pull/762

thanks, i’ll review it today. Did you test it?

roughly. is there any go test(s) i can run?

strange but for some reason it’s not working with my atmos config. it’s working fine with atmos.yaml in the repository. i will dig into it.

problem is that when the template executed it uses tfexec

ah, yes, tfexec doesn’t understand those commands, it needs terraform

Of course… this is what we frequently do

you can create a catalog stack file for a solution.

E.g. here’s how we do “EKS” and all related components

ok. is there any related example in the repo?

Here you can see we created a default cluster config, that imports a bunch of other componetns

Those could be inline, but we chose to import them

thanks!

yes, please don’t use globals

there are a few ways to share vars b/w components

e.g. create a base abstract component (with the default values) and inherit it in the other components

see this doc for ref https://atmos.tools/design-patterns/abstract-component

And multiple inheritance

https://atmos.tools/core-concepts/stacks/inheritance#multiple-inheritance

thx!

It’s actually not required and we improved the demo and examples here

https://github.com/cloudposse/atmos/pull/753/files

https://github.com/cloudposse/atmos/tree/main/examples/demo-helmfile

This uses k3s

But if I want to use it with eks I have to set it, no? Your examples all set use_eks to false

Or other question: if I use use_eks: false how does atmos now to which kubernetes to deploy to? Will it use my current context? Can I dynamically change the context in atmos in the stack configuration?

that’s only if you want the automatic kubeconfig creation

use_eks could probably be better named. Its true, it’s if you use eks, but it’s not required to use EKS.

In the end, all you need is a kubeconfig. With use_eks set to false, it just means you need to manage the kubeconfig yourself.

Would love a bot that could auto answer with links to threads

and TL;DR

exactly, RAG can do with its metadata

and need an custom integration with slack, there should be existing SaaS service on the market to do the similar thing

And want to train it on atmos docs

and examples

Have you also checked out danswer?

just heard of it, saw some similar projects, will take a look into it

after a quick review, danswer uses langchain and fast api, should be a reliable project to be used

this looks a good use case, https://docs.danswer.dev/more/use_cases/support

dived into the project and gave it a test, seems it is not easy to make it fully up, e.g. I tried with a public #atoms web page, but QA failed, I used local LLM so I guess it may work with public LLM service. side note: the project has a big vision to be a platform, so the codes are abstracted very well.

On what way did the QA fail?

You had it index the atmos slack channel in the public web archive?

I used one of archived page, https://archive.sweetops.com/atmos/2024/08/

should be related to ollama python lib

litellm’s ollama lib

Aha understand.. so got stuck on even just indexing one page

indexing is ok, should be in the retrieve answer part

side note: found this one, https://ollama.com/blog/continue-code-assistant looks useful for code refactoring

Danswer got renamed, and found the other similar project, https://r2r-docs.sciphi.ai/introduction.

I find their new marketing materials confusing. I understood Danswer. If I landed on their new site, I would have bounced right away.

https://www.sciphi.ai/

this morning I got an email from R2R team about sciphi.ai product, but the links inside are not reachable, I hope I’m not phished, and then I found the above page and their project on Github

@Erik Osterman (Cloud Posse)

Yes, but in the context of atmos, we use our component

https://github.com/cloudposse/terraform-aws-components/tree/main/modules/tfstate-backend

ty thats perfect, never got a chance to walk through the process using automation. appreciate the quick response.

coming back here this is cool, sorry i was stuck in my head in the chicken/egg scenario of the backend and the cold start stuff helped alot.

Glad to hear!

I should have also shared https://docs.cloudposse.com/layers/accounts/initialize-tfstate/

ooo thank you.

I will say I think our internal Atmos is on an older ver

I’m kind of at a design decision with regards to a second updated atmos for my new region + new backend, idk yet

Hey @Derrick Hammer great to hear you’re checking it out.

We have an example mono repo structured here: https://github.com/cloudposse-examples/infra-demo-atmos-pro/tree/main

I intend to create modules in 1 repo and create environments in another Perfect, checkout vendoring: https://atmos.tools/core-concepts/vendor/

https://github.com/cloudposse/atmos/blob/main/examples/demo-vendoring/vendor.yaml

We also provide a well-maintained reference architecture for AWS (which is how Cloud Posse makes money). The docs are public here: https://docs.cloudposse.com

Oh, we also have some design patterns that might be helpful.

https://atmos.tools/design-patterns/organizational-structure-configuration

Not sure the question

The whole point is to output the components and stacks that changed

(so it’s always done that)

It also conveys what triggered the change as some metadata in the JSON

ok, for some weird reason I thought it was only yaml updates to stacks

it has been a coincidence that I have always done stack.yaml changes together with component changes

and since now is the first time I use describe affected on a pipeline, I just realized yesterday that a component change will show as a change

the command checks a lot of things: atmos stack for the component, terraform code in the component folder, and terraform modules in any other folder that the current component uses in terraform (atmos detects it by using terraform metadata about the component)

ahhh interesting

also, it checks the depends_on attributes in the stacks, if a component depends on an external file or folder, and the file or folder changes, the component will be considered affected

@Andriy Knysh (Cloud Posse) if we set the metadata.enabled to false will it also be ignore on CI/CD? or do we have to have both settings.github.actions_enabled to false?

related to the question: describe affected will show it as change?

i guess the second question relates to the first one

@Pulak Kanti Bhowmick can you please confirm what atmos describe affected will return if a component is disabled using metadata.enabled: false

if I add anything to metadata like sdfsfdsafdsdf: ff it shows on atmos describe affected output

using atmos latest

Hi @Andriy Knysh (Cloud Posse), let me check and get back here

it should be affected because it’s changed

it’s up to the caller to decide what to do with the disabled component

For additional context, do you mean these docs:

doc.cloudposse.com (cloud posse’s reference architecture)

Or https://atmos.tools?

That will help me address any confusion.

Hi, also there I see not the relationship made between let’s say a stage and the account ID.

It’s one of those pieces that would make Atmos click in my head. Architecture wise I can relate Catalogs/mixins/stacks etc but somewhere down the line you can’t apply to a ‘sandbox’ as long that ‘sandbox’ has an account id…right?

So just to disambiguate some things, in cloudposse’s refarch (docs.cloudposse.com), we by convention tie a stage (dev, staging, production) to an AWS Account.

In our refarch, we have something called the account-map https://docs.cloudposse.com/components/library/aws/account-map/

This is what handles that mapping

This is what allows us to refer to everything by name, instead of thinking of accounts account IDs

But it’s also what can make using the Cloud Posse refarch harder with brownfield environments https://atmos.tools/core-concepts/components/terraform/brownfield/

All that said, I want to point out that these are not Atmos conventions, these are Cloud Posse conventions in our reference architecture for AWS that uses Atmos

The account-map returns an IAM role that is used to access a given account

the IAM role will have the AWS account ID information

(that’s the missing gap, I believe in the understanding)

Thanks for your answers.

I saw on GH a file (see below) and it gives me some direction. But maybe it’s me and then I think “what are all the possible defaults?

ah..that last one.

# Global variables used for account maps, role maps and other global values
vars:
  account_map:
    dev: 222222222222
    staging: 333333333333
    automation: 111111111111
    prod: 444444444444

Yes, good question. I’ll answer that in a sec.

(Also, in a future release of our refarch, we intend to move away from the account-map convention to make it easier to use in brownfield environments)

Ah…that’s where my head indeed is: How on earth would I use this in brownfield situations? We have a lot of those; Customers that tried first themselves and then start looking for a partner.

Understandable… so what we have as our refarch was born from 99% of our engagements which are what we call “cold starts”, so brownfields considerations we seldom a concern. As we now reach a much wider audience, that’s coming up more and more often.

We’re doing some groundwork changes first, related to https://github.com/cloudposse/terraform-aws-components/issues/1177

And once we have that in place, we can start making more breaking changes to our components to support brownfield - while keeping our ecosystem stable.

Sounds really promising! I’m going to do some dishes over here (CET) and then just start with this afterwards.

Thanks for all your time!

@Andriy Knysh (Cloud Posse) @Jeremy G (Cloud Posse) where do we have a full definition of the static backend

I need to add it here so we can document it.

https://atmos.tools/core-concepts/components/terraform/brownfield/#hacking-remote-state-with-static-backends

I don’t think we have a full definition anywhere. In part, that is because it is too simple. See the examples here. You just set the remote_state_backend_type to static and set the outputs as a map under remote_state_backend.static. Although the example shows setting backend_type: static, that is a bit misleading because you cannot run terraform plan etc. with a static backend.

Also, what about a static account-map? @NotWillFarrell is working in a brownfield

My recommendation for brownfield is to use a static backend for account to create the list of accounts, and use the real account-map. Account map is an information collector and processor, it it does not manage any cloud resources. If account-map contains information for accounts you are not using, that should not cause problems.

The example @NotWillFarrell cited:

# Global variables used for account maps, role maps and other global values
vars:
  account_map:
    dev: 222222222222
    staging: 333333333333
    automation: 111111111111
    prod: 444444444444

is from before account-map got that information from account. Convert that to a static backend for account and I think account-map will work fine. If not, we should fix account-map so that it does.

here is an example of the static backend for account

components:
  terraform:
    account:
      backend:
        s3:
          role_arn: null
      vars:
        enabled: false
      # Use `static` remote state to configure the attributes (outputs) for the existing organization, OUs and accounts
      remote_state_backend_type: static
      remote_state_backend:
        static:
          account_arns:
            - "arn:aws:organizations::xxxxxxxxxxx:account/o-xxxxxxxxxxx/xxxxxxxxxxx"
            - "arn:aws:organizations::xxxxxxxxxxx:account/o-xxxxxxxxxxx/xxxxxxxxxxx"
            - "arn:aws:organizations::xxxxxxxxxxx:account/o-xxxxxxxxxxx/xxxxxxxxxxx"
          account_ids:
            - "xxxxxxxxxxx"
            - "xxxxxxxxxxx"
            - "xxxxxxxxxxx"
          account_info_map: {}
          account_names_account_arns: {}
          account_names_account_ids: {}
          organization_arn: "arn:aws:organizations::xxxxxxxxxxx:organization/o-xxxxxxxxxxx"
          organization_id: "o-xxxxxxxxxxx"

all the outputs returned from the account component https://github.com/cloudposse/terraform-aws-components/blob/main/modules/account/outputs.tf. can be added to the static backend and then used in other components as if they were returned from the account remote state

The only outputs of account used by account-map are • eks_accounts • non_eks_accounts • account_info_map The first 2 are just lists of account names, indicating which accounts are expected to have EKS deployments and which are not. Every account should be in one of those lists.

account_info_map is a map of account name to various pieces of information. In the real account output, there is more information, but I think you only need to fill in: • eks: boolean, account hosts at least one EKS cluster • id: account id (number) • stage: the account “stage” • tenant: (optional) the account “tenant”, or null if you are not using tenant Example:

account_info_map:
  artifacts:
    eks: false
    id: "123456789012"
    stage: "artifacts"
    tenant: null
  dev:
    eks: true
    id: "210987654321"
    stage: "dev"
    tenant: null

The other outputs of account are used by components that manage the accounts and organizations, but you can skip all that if you already that set up.

cc: @Erik Osterman (Cloud Posse)

@RB this closes something you asked for a long time ago

I saw that! Thank you very much

Our ECS components solve this using SSM parameters

https://docs.cloudposse.com/layers/software-delivery/ecs-ecspresso/

I’ll take a look. Thank you!

this is a regression from the many new PRs, we’ll fix it, thank you @Stephan Helas

That way if the plan was compromised without the ability to apply, no one could do any funny business with a local exec data source

This relates to the recent work by @jose.amengual

@jose.amengual where did we leave off with this?

Waiting for @Andriy Knysh (Cloud Posse) to have some time to go over the failing tests on my PR

in the convo we have in the other slack

Andriy should have some time now to look

Quick update: @Igor Rodionov will review the GH action PR (prob tomorrow), and @Andriy Knysh (Cloud Posse) will check what @jose.amengual said about Atmos (not related to the GH action)

@toka please show the current file system layout

you can use your TF modules as components (there is nothing special about components)

if you put your modules into components/terraform and define Atmos stacks in stacks, it should work. Atmos will generate the backend and varfile for the modules and execute Terraform

Yes, “components” are really just a philosophy. You don’t need to rewrite anything.

…it won’t be worse off than it is now, but it won’t benefit from the way of thinking about architectures as made up of components.

Also, components typically will comprise a solution, so they shouldn’t be as small as a small module.

https://atmos.tools/quick-start/mindset https://atmos.tools/best-practices/components

Ah thanks, I will respond later today

https://github.com/cloudposse/terraform-aws-components/issues/1177#issuecomment-2479089189

Are you familiar with the atmos provider generation?

https://atmos.tools/core-concepts/components/terraform/providers/

@Kalman Speier the question is, how do you handle the auth token. In terraform, you can get it from a secret storage (SSM/ASM, GCP vault, etc.), and send it as an input to the other resources/modules

the token should def not be hardcoded in Atmos stacks

I guess you are asking about how to do it in Terraform (get the token from data sources and then use it as input to other modules, per cloud)

what i’d like to achieve is simply like this: https://github.com/hashicorp/terraform-provider-kubernetes/blob/main/_examples/gke/main.tf

so, the kubernetes provider is configured outside of the component which is responsible to deploy k8s resources.

if the cluster outputs the token and i set as a variable to my k8s component that works fine, however these tokens are expiring, hence i’d like to use a provider specific datasource but i couldn’t place that in the component obviously.

1. create a cluster on gke for example

2. run cloud specific kubernetes provider configuration ``` data “google_client_config” “default” {}

provider “kubernetes” { token = data.google_client_config.default.access_token … } ```

1. deploy my k8s resources using the configured provider without the component knowing about which cloud provider in use

the #2 needs to run each time #3 is running, because outputs from state isn’t working as tokens expiring.

i would put all those providers and the data sources in separate files per cloud (obe for AWS/ECS, one for GKE) Then add a variable defining the cloud

variable "cloud" {
   description = "Cloud provider"
}

then in each provider "kubernetes", use count depending on the cloud variable

then in [main.tf](http://main.tf), use https://developer.hashicorp.com/terraform/language/functions/coalesce to read the token from all the data sources

similar to

token = coalesce(data.google_client_config.default.xxxx, data.xxxx.xxxx.xxxx)

too bad that the provider block still does not support count and for_each (people have been asking for that for many years). So oyu will need to make sure your TF code has access to all clouds at once for the provider blokcs to work and not throw errors

https://support.hashicorp.com/hc/en-us/articles/6304194229267-Using-count-or-for-each-in-Provider-Configuration#<i class="em em-~"</i>text=While%20a%20longtime%20requested%20feature,provider%20configuration%20block%20in%20Terraform>.

i would create a common TF module to deal with the clusters, then a few root modules (components) for the specific clouds

the common module would accept the token as a variable

the parent/root modules would read the token using the corresponding cloud provider and provide it in the variable to the child module

and if you are using Atmos, it’s easy to configure your components pointing to the diff root modules per cloud

ecs/xxxxx terraform component uses a data source to read the token from SSM/ASM

gke/xxxxx terraform component uses data "google_client_config" "default" {} to read the token from GCP

ok, thanks a lot, i will think about these.

tha main point is to create a common TF child module (with almost all the code except the code to read the token from the data sources), then reuse it in the root modules using diff providers

@Erik Osterman (Cloud Posse) any updates on this?

Our plan is leaning more towards implementing a pluggable way of storing/retrieving values from directly within Atmos, but supporting many of these same backends.

I’ll have a more conclusive answer by the end of this week.

oh nice thanks!

@Michael this should restore the behavior in #geodesic

We disable the prompt formatting by default, and instead allow it to be customized in atmos.yaml

Awesome stuff, thank you for such a quick turnaround on it! Excited to try it out

I’ve not yet had a chance to read through the entire message

Note, you will likely need to commit the varfiles for it, for it to work with TFC

Also, dynamic backend generation will not work well, if you use multiple backends for the same component (E.g. by region)

Currently, Atmos relies on the concept of workspaces for managing unique configurations and state per stack. However, with the introduction of OpenTofu’s Early Evaluation feature, there is potential to move away from workspaces and instead use unique keys per stack to manage state more flexibly.

While this functionality is not natively supported yet, I believe it could be feasible to implement. By leveraging Early Evaluation, we could dynamically configure the backend state storage, using variables to differentiate each stack’s state, rather than depending on separate workspaces. This approach would allow us to specify unique keys based on the stack name or environment and ensure proper isolation of state per stack.

In essence, although Atmos doesn’t currently support a workspace-less setup, utilizing unique keys per stack with Early Evaluation could be a similar concept and an effective alternative worth exploring.

Therefore, I don’t anticipate that the CloudPosse team will find it impossible to adapt to the deprecation of workspaces. They are an incredibly talented group, and I’m confident in their ability to develop a robust solution or an alternative approach. With their expertise, I believe they will be able to leverage features like Early Evaluation effectively to maintain or even improve the functionality of Atmos without relying on workspaces.

Actually, I think it’s already supported, depending on your backend. With S3, it’s just a different path. Since the backends are entirely configurable in Atmos, I think it’s just about configuring the right path to match the workspace path, and dropping the workspace parameter.

We might need to add a parameter in atmos to disable workspace operations, but the lift on that is trivial.

We talk about the file structure of the S3 backend here https://docs.cloudposse.com/layers/accounts/tutorials/terraform-s3-state/

So with that it should be as simple as updating the key to the fully qualified path to the workspace tfstate file

https://developer.hashicorp.com/terraform/language/backend/s3

Thanks, ill review!

Yes i think the workspace option would be needed.

You’re right, all the other stuff is there to override the backend key per component using some yaml magic (use stack name as the unique key, as you said)

Yes i think the workspace option would be needed. Just to confirm, an option to disable usage of workspace operations?

what I do is save the other third-party module in another directory and use that with CloudPosse context.tf file to create the naming.

For example: components/vendor/aws/vpc -> AWS Module compoents/aws/vpc -> custom module using the vendor aws vpc with CloudPosse [context.tf](http://context.tf) file for naming and enable ENV

and I will use the compoents/aws/vpc module in catalog

@Miguel Zablah Thanks! I understand, but to help, could you give me a simple example? I get the general picture, but the ‘custom module using the vendor aws vpc with CloudPosse context.tffile for naming and enableENV’ part doesn’t come to mind specifically.

If what you mean by custom module is ‘combine the newly attached context.tf with the “aws/vpc” component in the vendor directory to create a new Root Module (Component)’, this seems like it would be complicated to configure and maintain the component every time. Am I not understanding this correctly?

not really bc both are going to be manage by atmos vendor file, so using this example I use this two modules: https://github.com/cloudposse/terraform-null-label https://github.com/terraform-aws-modules/terraform-aws-vpc

so I add both of them to the Atmos vendor file and in a different directory on components in this example it will be something like this: components/terraform/vendor/cloudposse/tf-null-label -> https://github.com/cloudposse/terraform-null-label components/terraform/vendor/aws/vpc -> https://github.com/terraform-aws-modules/terraform-aws-vpc

than in I will create a new root module with this modules here: components/terraform/aws/vpc

where I will have a this files: [main.tf](http://main.tf) -> to call the components/terraform/vendor/aws/vpc [context.tf](http://context.tf) -> to call the components/terraform/vendor/cloudposse/tf-null-label

so essentially what I do is copy there context file but reference the module internally kind of what they do in this example: https://github.com/cloudposse/terraform-null-label/blob/main/examples/autoscalinggroup/context.tf

and then I can use module.this.id for naming, module.this.tag for tagging and module.this.enabled to enable/disable the module like CloudPosse dose on there modules all of this I will use it on [main.tf](http://main.tf) when calling the VPC module

hopefully this explains it a bit better

@Miguel Zablah Thanks for the detailed explanation, I understood it perfectly. My only further question is, so what I need to do is to actually create a components/terraform/vendor/aws/vpc module block in main.tf in the components/terraform/vendor/aws/vpc directory and declare the required variables one by one inside the module block so that they are assignable (ex: azs, cidr, private_subnes, etc…)?

so this is how the root component will look like: components/terraform/aws/vpc : • [main.tf](http://main.tf) -> here you will put the module block calling the components/terraform/vendor/aws/vpc • [context.tf](http://context.tf) -> use this example but reference your vendor module components/terraform/vendor/cloudposse/tf-null-label • [outputs.tf](http://outputs.tf) -> you will expose the vpc module outputs (you can use a loop for this) • [variables.tf](http://variables.tf) -> this will be almost the same as the aws-vpc module but removing name and stuff that is manage by [context.tf](http://context.tf) example [main.tf](http://main.tf) :

module "vpc" {
  count = module.this.enabled ? 1 : 0
  source = "../../vendor/aws/vpc"

  name = module.this.id
...
}

after this is setup correctly made you can use it as normally on your atmos catalog and what not

btw this just how I do it there might be a better way to do this haha

Some more context:

components:
  terraform:
    vpc:
      vars:
        enabled: true
        name: "compute"
        ipv4_primary_cidr_block: "10.1.0.0/16"
        vpc_flow_logs_enabled: false
        nat_gateway_enabled: true
        public_subnets_enabled: true
    vpc-flow-logs-bucket:
      vars:
        name: "vpc-flow-logs"
    internal-domain-and-cert:
      settings:
        depends_on:
          1:
            component: vpc
      vars:
        create_wildcard_cert: true
        vpc_id: '{{ (atmos.Component "vpc" .stack).outputs.vpc_id }}'

By default, templating is disabled. We originally implemented this as an escape hatch, but it’s become very popular.

It is (theoretically) super useful!

https://atmos.tools/core-concepts/stacks/templates/

Ooo…I probably just want inheritance, huh?

So at Cloud Posse, in our refarch, we almost never use templating, and instead use inheritance and imports 99% of the time.

However, we acknowledge the usefulness of the template functions. We’re also working on improvements, which involve moving towards what YAML calls “explicit types”, which are basically first-class functions in YAML.

Also, since you’re using atmos.Component make sure to read https://atmos.tools/core-concepts/stacks/templates/functions/atmos.Component#handling-outputs-containing-maps-or-lists

The docs imply this should “just work”. I’m clearly missing something obvious, and would love some help. Yes, that might be the case. We’re working on 2 things.

1. Warning if you’re using templates and have it disabled
2. Improving the docs to call it out

If you could share a screenshot or link to the page/chapter that you encountered it, I’ll fix it

https://atmos.tools/core-concepts/share-data/#using-template-functions

Jumps right out at you in the docs. Root page and all…

Yup, that could definitely need some TLC

Thanks!!

I appreciate the context, but do you have any tips on how I can reference the vpc_id from vpc in internal-domain-and-cert in my example above? All the links you’ve passed along seem to talk about sharing data between stacks, and I just need to pass between components in a single stack here.

This, no?

https://atmos.tools/core-concepts/share-data/#function-atmoscomponent

Yep. That is what isn’t working.

But do you have templating enabled in atmos.yaml

That looks like what I was missing!

In the flow of the documentation, we have this

Awesome. Thanks, Eric.

Definitely just saw that root page before I read more deeply about templating.

I’m going to update “Share Data Between Components”, to reference back to Template Configurations, to help avoid this snafu

and call out that templating needs to be enabled.

That’s great. The docs are generally pretty robust, which may have lulled me into a false sense of “well…if they say this just works…”

A workflow is one way to do that. We haven’t looked into solving ordered dependencies in a way that provides first-class support for Atlantis. But to your point, you could create a custom workflow in atmos that you call via Atmos. The issue is you really want to review/approve each individual component’s plan. So for that, a more elaborate approach is necessary.

@jose.amengual is one of the #atlantis maintainers and might have some other ideas

Thanks for the reply. I was initially confused with Atmos stacks thinking that I could deploy in an ordered way the components listed in it. But reading the documentation I realised is probably a workflow. Another option I’ve been thinking is to do this via CI/CD and build the dependencies in each stage of the CI/CD pipeline. So, what is atmos solution to be able to orchestrate multiple Terraform root modules that depend on each other?

Atlantis supports dependencies

we could potentially ask CloudPosse to add component dependencies to the automatic workflow generation that atmos can create

That would be great. We’re not using Atlantis dependencies at the moment. It would be good to find a away to orchestrate the plan/apply flow of multiple components (TF root modules) that depend on each other.

Oh that’s interesting, I think I forgot about that

Since we already represent dependencies in stack configs, it’s maybe a simple mapping

@karel_alfonso did you see the atmos Atlantis generation?

It’s templatized so it might already be possible

https://atmos.tools/integrations/atlantis

https://www.runatlantis.io/docs/repo-level-atlantis-yaml.html#example-using-all-keys (depends-on)

depends_on:
    - project-1

@karel_alfonso did you see the atmos Atlantis generation? No, haven’t looked into that yet. Will the generation take into account the dependencies in the stack config?

no, I do not think it will do that

But the entire Atlantis config is one big template

And atmos stacks define dependencies

The templates have the full context of the stack configs, so it should be possible to define the Atlantis dependencies

The way to think about it is you are transforming the shape of one YAML configuration to the shape of another

https://atmos.tools/core-concepts/stacks/dependencies

Look at the example here: https://atmos.tools/integrations/atlantis

that section, I believe is not free form Erik

What do you mean by free form?

I think if you add a line that is not predefined, atmos does not add it to the generated atlantis,yaml

Aha

Ok, that is plausible- but and easy change

yes, I remember Andry adding a few lined when I was testing the integration pretty quickly

Ok, so @karel_alfonso if you end up pursuing this route, we may need to make some tweaks, but I don’t think it would be anything radical.

Atmos supports configuring the relationships between components in the same or different stacks. You can define dependencies between components to ensure that components are deployed in the correct order. Is there a way to apply an entire stack with its dependencies using atmos CLI (without Atlantis) for demonstration purposes? I want to show to my team that a tool like atmos can help organise a large Terraform codebase with multiple tenants and AWS accounts. I can then look into Atlantis

All examples I’ve seen deploy a specific component in a stack

A concrete example of what I want to achieve is that I have a TF component that provisions a Kafka cluster. Whenever I change the number of brokers I want to deploy/update two other components that require the bootstrap servers returned after applying the first component. I thought/wished that atmos could help achieve something like that

In the CLI we have not implemented that. The main reason is that it’s dangeous to “apply all”, however, we do have it on our roadmap but no ETA.

However, it is possible using custom commands. Others have done the same.

https://atmos.tools/core-concepts/custom-commands/

excellent! thanks so much for the help and support. I’ll proceed with a demo I’m preparing using custom commands and later on will look into the integration with Atlantis

https://atmos.tools/integrations/github-actions/atmos-terraform-drift-detection

Drift detection will pick up the changes in the scenario you described, but the way it works is not based on dependencies

It work based on replanning all components

Oh, hadn’t seen that feature. It’s something we can run in a scheduled CI/CD job

Exactly!

Ultimately, I want to prove that we don’t need TACOS, just Terraform, best practices framework and existing tools to improve all the issues you’ve listed that teams run into when scaling TF to a large org.

Nothing helpful to add here, but want to just say thank you for asking the question and folks for answering swiftly. I was about to go on a rabbit hole reading atmos docs for the same exact use case (just no atlantis requirement). @karel_alfonso Curious on what you come up with.

Before reading this, I was also under the impression atmos can deploy all components in the stack in order of dependency. I’ll start investigating custom command and play around with our use case as well

Seeing HCP Terraform stacks (and deferred changes/plan capability) got me wanting to see what is available out there outside of the usual TACOS. Saw Terragrunt Stacks RFC, exciting, but timing of its release is unknown

you can create an atmos workflow to deploy components in order using the CLI

@Bob it would be helpful to know if you are primarily going to use this on the terminal/console, and if the expectation is to review each plan before apply, or want to automatically “apply all” in dependency order without reviewing the plan.

Primarily going to review the plan before applying, but I know that’s a complicated ask especially if components have dependents that had not been applied yet within the stack. I don’t believe atmos has mocking/placeholder outputs for components today?

@Dan Miller (Cloud Posse) @Ben Smith (Cloud Posse) maybe an easy one for you

your backend configuration is likely configured at a higher level so that it can be reused. It’s most likely under stacks/orgs/cs/_defaults.yaml

you can also always check the final result of stack configuration with atmos. For example,

atmos describe component tfstate-backend -s your-stack-name

https://atmos.tools/cli/commands/describe/component/

@Samuel Than in this stack name cs-core-gbl-root

cs - is namespace core - is tenant gbl - is environment (region, global region in this case) root - is the stage (account)

as Dan mentioned, you probably have these defined in the higher level stacks

search for tenant: core and environment: gbl and stage: root in your stacks folder

also check the account-map component and what values are provided to the module (see the [context.tf](http://context.tf) file for the context variables like namespace, tenant, stage , environment

if still not working, DM me your config to take a look

I’ve DM you @Andriy Knysh (Cloud Posse) with some more details.

why the change? like I don’t think is a big deal but I think the default to UI is good since well almost all cli tools will have the –help flag for this

That’s a good point

Maybe if there are no stacks configured, we should show help and don’t load the UI, but otherwise keep the current behavior

yeah that sounds great

I didn’t realize that was a feature, neat. Usually when i run cli commands without args, it returns a help screen so I’ve learned to expect that.

Would it be easier to allow the default behavior to be modified in the yaml config?

I think this applied more for CLI that do not have a UI or required at least 1 arg for it to work, for CLI that have UI or do not required args to work it always have a default run for example atmos will run UI at least this is the case in my experience but I can be wrong hehe

Yea, maybe we make it configurable. I think we’ll not change anything for the time being.

@Dan Miller (Cloud Posse) @Ben Smith (Cloud Posse)

a few roles, such as the gitops role, are meant to be assumed by GitHub Actions. These require the GitHub OIDC provider to be deployed first, since the policies will attempt to look up that ARN (which is what you see as empty).

You can deploy that with the github-oidc-provider component: https://docs.cloudposse.com/layers/github-actions/github-oidc-with-aws/

As a TLDR of that page to deploy gitops in particular, deploy github-oidc-provider into your identity account (core-gbl-identity). Then try again

The other steps will be used for GitHub Actions later

Awesome thank you

I suspect I am doing something wrong. I have checked the stacks/catalog/ has the github-oidc-role component enabled (as per this), that the repo’s are configured as per option 2 in the doc you linked to, and I have run (with no errors) atmos workflow deploy/github-oidc-provider -f gitops & atmos terraform apply github-oidc-provider -s core-gbl-identity but the error still persists when running atmos workflow deploy/all -f identity Running atmos terraform plan github-oidc-provider --stack core-gbl-identity shows there’s nothing outstanding

Hmm I’m not sure. Could you paste a larger snippet from Terraform plan of the policy with the empty federated identity ARN?

Here’s one of the roles that are erroring. I suspect the issue is the OIDC provider is missing in in IAM Identity Centre, and I think I’ve read through the doco on the TF modules and the ref architecture and it doesn’t seem to which modules to create it via TF, so does it have to be click ops?

 # aws_iam_role.default["gitops"] will be created
  + resource "aws_iam_role" "default" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = [
                          + "sts:TagSession",
                          + "sts:SetSourceIdentity",
                          + "sts:AssumeRole",
                        ]
                      + Condition = {
                          + ArnLike      = {
                              + "aws:PrincipalArn" = [
                                  + "arn:aws:iam::491085392849:role/sit-core-gbl-identity-devops",
                                  + "arn:aws:iam::491085392849:role/sit-core-gbl-identity-managers",
                                  + "arn:aws:iam::491085392849:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityDevopsTeamAccess_*",
                                  + "arn:aws:iam::491085392849:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityManagersTeamAccess_*",
                                ]
                            }
                          + StringEquals = {
                              + "aws:PrincipalType" = "AssumedRole"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = {
                          + AWS = "arn:aws:iam::491085392849:root"
                        }
                      + Sid       = "RoleAssumeRole"
                    },
                  + {
                      + Action    = [
                          + "sts:TagSession",
                          + "sts:SetSourceIdentity",
                          + "sts:AssumeRole",
                        ]
                      + Condition = {
                          + ArnLike = {
                              + "aws:PrincipalArn" = [
                                  + "arn:aws:iam::491085392849:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityViewerTeamAccess_*",
                                  + "arn:aws:iam::491085392849:role/sit-core-gbl-identity-viewer",
                                  + "arn:aws:iam::*:user/*",
                                ]
                            }
                        }
                      + Effect    = "Deny"
                      + Principal = {
                          + AWS = "arn:aws:iam::491085392849:root"
                        }
                      + Sid       = "RoleDenyAssumeRole"
                    },
                  + {
                      + Action    = [
                          + "sts:TagSession",
                          + "sts:SetSourceIdentity",
                          + "sts:AssumeRoleWithWebIdentity",
                        ]
                      + Condition = {
                          + StringEquals = {
                              + "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
                            }
                          + StringLike   = {
                              + "token.actions.githubusercontent.com:sub" = "repo:Sustainabil-IT/infrastructure:ref:refs/heads/main"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = {
                          + Federated = ""
                        }
                      + Sid       = "OidcProviderAssume"
                    },
                ]
              + Version   = "2012-10-17"

Yes the OIDC Provider ARN is missing, but it will never need to be click ops with refarch. Something is missing, but it’s hard to determine. Can you please walk through these steps to debug?

First check core-identity account in the web console. Do you see the Federated Identity there? If not then that’s the issue. Otherwise the issue is passing the output between Terraform.

If the Federated Identity doesnt exist:

1. (You already did this before, but for the sake of thoroughness) Check the component itself. Make sure this returns no changes: atmos terraform apply github-oidc-provider -s core-gbl-identity. Also confirm that the output, oidc_provider_arn exists.

   oidc_provider_arn: "arn:aws:iam::491085392849:oidc-provider/token.actions.githubusercontent.com" 
   

2. Make sure github-oidc-provider is “enabled” with enabled: true

If the Federated Identity does exist:

1. Ensure the component exists where aws-teams expects it to exist. Is the component called exactly “github-oidc-provider” and is it in the the gbl environment of the same stack, core-identity?
2. Check the aws-teams stack configuration for trusted_github_repos. Is gitops mapped to your infrastructure repo’s name? (this shouldnt be related, but it’s worth checking)

There’s most likely something minor missing. We create this role very frequently, and there is never any click ops necessary.

My sincere apologies for the long delay in response but I decided to start from the start re-read everything and double check I did it all right, it would appear there is either me not reading issue, a doc issue or unexpected depedency in the workflow, but running atmos terraform output github-oidc-provider -s core-gbl-identity produced no outputs, so reading through all the workflows etc I noticed that it appears that atmos workflow deploy/github-oidc-provider -f github needs to be executed prior to the atmos workflow deploy/all -f identity , as once I ran the github-oidc workflow I had the output you mentioned and the atmos workflow deploy/all -f identity went through fine

no worries at all. I’m glad you were able to resolve it!

Wonderful! Thanks for these. @Igor Rodionov will review

Cc @Gabriela Campana (Cloud Posse)

this will need my PR to be merged before hand I think

but https://github.com/cloudposse/github-action-terraform-plan-storage/pull/35 can be reviewed first

@jose.amengual your PRs are next top priority in my backlog

@shirkevich bear with us, we will need to merge @jose.amengual ‘s PRs that add/improve Azure support first. That should be this week or early next week.

@Igor Rodionov

• DEV-2787: Review PRs that add/improve Azure support

• DEV-2788: Review several PR’s to create Google usable workflow in GitHub

@shirkevich hello. I’v started reviewing your PRs.

great to hear that! should I merge master? you can ping me if you want me to change something

I’ll message you if there will be any actions required

@shirkevich I check your PRs overall they looks good.

1. I’ll merge https://github.com/cloudposse/github-action-terraform-plan-storage/pull/35 today. I merged the PR to an intermediate branch, but you can think this is done.

2. When I release GitHub-action-terraform-plan-storage, we will have to pin the new version in the plan and apply actions.

https://github.com/cloudposse/github-action-atmos-terraform-plan/pull/93 https://github.com/cloudposse/github-action-atmos-terraform-apply/pull/64

1. Also these two ^ have conflicts that have to be solved. The main difference is that yesterday, we released useful feature - you can now specify integration configs per stack level, So, the settings we used to get with

    - name: config
      shell: bash
      id: config
      run: |-
        echo "opentofu-version=$(atmos describe config -f json | jq -r '.integrations.github.gitops["opentofu-version"]')" >> $GITHUB_OUTPUT
        echo "terraform-version=$(atmos describe config -f json | jq -r '.integrations.github.gitops["terraform-version"]')" >> $GITHUB_OUTPUT
        echo "enable-infracost=$(atmos describe config -f json | jq -r '.integrations.github.gitops["infracost-enabled"]')" >> $GITHUB_OUTPUT        
        echo "backend=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].backend')" >> $GITHUB_OUTPUT
....

now we get https://github.com/cloudposse/github-action-atmos-terraform-plan/blob/main/action.yml#L88

    - name: Get atmos settings
      id: atmos-settings
      uses: cloudposse/github-action-atmos-get-setting@v2
      with:
        settings: |
          - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: settings.github.actions_enabled
            outputPath: enabled
          - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: component_info.component_path
            outputPath: component-path
          - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: atmos_cli_config.base_path
            outputPath: base-path
          - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: command
            outputPath: command
          - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: settings.integrations.github.gitops.opentofu-version
            outputPath: opentofu-version

....

while the old way is still true for affected-stacks action

1. affected-stacks action PR still have one comment https://github.com/cloudposse/github-action-atmos-affected-stacks/pull/55

1. We probably need PR for <https://github.com/cloudposse/github-action-atmos-terraform-select-components/blob/main/action.yml> with equivalent changes that @shirkevich have to affected-stacks

1. When we cut a new release for the apply action, we will need PR for this action https://github.com/cloudposse/github-action-atmos-terraform-drift-remediation/blob/main/action.yml#L83 to pin the new version

@shirkevich I released https://github.com/cloudposse/github-action-terraform-plan-storage/releases/tag/v2.0.0 you can use [github.com/cloudposse/github-action-terraform-plan-storage@v2](http://github.com/cloudposse/github-action-terraform-plan-storage@v2) in plan and apply actions

thank you, updating plan and apply now

have 2 questions:

• do you find word backend to choose between aws and google in plan and apply correct? https://github.com/cloudposse/github-action-atmos-terraform-plan/pull/93/files#diff-1243c5424efaaa19bd8e813c5e6f[…]6e63761421b3e5f5c8ced9a36e6b6R286 https://github.com/cloudposse/github-action-atmos-terraform-plan/pull/93/files#diff-1243c5424efaaa19bd8e813c5e6f[…]6e63761421b3e5f5c8ced9a36e6b6R308

• I copied step with lockfile creation from aws, is it really needed? can you please describe how it is used https://github.com/cloudposse/github-action-atmos-terraform-plan/pull/93/files#diff-1243c5424efaaa19bd8e813c5e6f[…]6e63761421b3e5f5c8ced9a36e6b6R336

• do you find word backend to choose between aws and google in plan and apply correct? Nice catch. really no. https://github.com/cloudposse/github-action-atmos-terraform-plan/blob/main/action.yml#L143

this is the setting we should rely on

and https://github.com/cloudposse/github-action-atmos-terraform-plan/blob/main/action.yml#L155

I copied step with lockfile creation from aws, is it really needed? can you please describe how it is used The plan result is plan file and lock file with hashes of all providers and terraform. The apply step needs a lock file to pull binary equals providers and terraform to guarantee the protection of bugs because of different versions. How where to store lock files is an opinionated theme. Some teams commit lock files into their repos, but we decided to store them the same way as we store plan. Because the provider’s binary depends on OS. Lock file created on Mac could lead to failure on Linux (in theory, I did not test that)

@shirkevich I added couple few comments to resolved code https://github.com/cloudposse/github-action-atmos-terraform-plan/pull/93

addressed your comments

somehow get-settings wasn’t working without tofu installation when I was testing, maybe it was something broken in my atmos.yaml config, now it is green again

yep it is failing again:

template: all-atmos-sections:176:46: executing "all-atmos-sections" at <atmos.Component>: error calling Component: exec: "tofu": executable file not found in $PATH
Error: Error: Command failed: atmos describe component jobs -s foo-stg --format=json --process-templates=true
template: all-atmos-sections:176:46: executing "all-atmos-sections" at <atmos.Component>: error calling Component: exec: "tofu": executable file not found in $PATH



Error: Error: Command failed: atmos describe component jobs -s foo-stg --format=json --process-templates=true
template: all-atmos-sections:176:46: executing "all-atmos-sections" at <atmos.Component>: error calling Component: exec: "tofu": executable file not found in $PATH

hm…

intresting

I wil research that today

the problem is in dependency on the output of the other module:

this foo module have this:

        env_vars: '{{ toRawJson ((atmos.Component "bar" .stack).outputs.env_vars) }}'

looks like chicken and egg problem ))

yea. That was my guess

we need tofu version before get-config

@Igor Rodionov do you need something from my side to proceed with PRs? have some spare time

@Igor Rodionov bumping this up

any news on those PRs?

@shirkevich I will check tomorrow. hard week

@shirkevich Hello. This PR https://github.com/cloudposse/github-action-atmos-terraform-apply/pull/64 looks good. The only comment is that we need 2 lines to example in readme.

Need conflict resolution. it would lead to the same updates you did for the plan action https://github.com/cloudposse/github-action-atmos-terraform-apply/pull/64

thanks, will fix those, what about problem with get-settings in terraform-plan workflow?

oh,,, you mean templating chicken-egg problem

yeeep

try https://github.com/cloudposse/github-action-atmos-get-setting/blob/main/action.yml#L20C3-L20C20 process-templates to false

also we should probably call the cloudposse/github-action-atmos-get-setting@v2 two times

1. get terraform and opentofu version with process-templates: false

1. After install terraform / opentofu call cloudposse/github-action-atmos-get-setting@v2 once again to get all other settings with process-templates: true

if it will work with process-templates: false will do as you suggesting

was thinking to switch to parsing the old way, but it will be ugly…

@shirkevich I’d like to discuss this PR https://github.com/cloudposse/github-action-atmos-affected-stacks/pull/55#pullrequestreview-2475918955

I’m not sure we need google auth here

possibly yes let me test this with my repo

it is failing without github auth

Writing the backend config to file:
infra/atmos/components/terraform/cloudrun/backend.tf.json

Wrote the backend config to file:
infra/atmos/components/terraform/cloudrun/backend.tf.json

Deleting Terraform environment file:
'infra/atmos/components/terraform/cloudrun/.terraform/environment'

Executing 'terraform init cloudrun -s foo-prd'
template: describe-stacks-all-sections:47:35: executing "describe-stacks-all-sections" at <atmos.Component>: error calling Component: exit status 1

Error: storage.NewClient() failed: dialing: google: could not find default credentials. See <https://cloud.google.com/docs/authentication/external/set-up-adc> for more information

@Igor Rodionov :point_up:

The google auth is needed there as terraform init doing backend initialisation and failing without auth, or am I missing smth?

@shirkevich is that a local test you run? or that is on the action tests?

When I added the azure support none of the tests used google auth, only aws test will run correctly

nope it’s my local test with real env

how do you run tests? are you spawning real infra for that?

the test uses a test account with real infra

but this tests uses local resources

without credentials and when using google backend this is failing:

atmos terraform init cloudrun -s foo-stg

│ Error: Failed to get existing workspaces: querying Cloud Storage failed: googleapi: Error 403: [email protected] does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist)., forbidden

and therefore github-action-atmos-affected-stacks is also failing

how is it working for azure ?

or you’re storing state in aws?

no no

state is stored in blobstore

but I pass the credentials to the provider using ENV vars in my case

after the plan is successful the https://github.com/cloudposse/github-action-terraform-plan-storage/pull/35 should use the same way of authenticating to store the plan, which means that by the time the action is being called, the login to google from the pipeline is successful

so you need to use a cicd/automation/machine/githubactions user for the pipeline to be able to authenticate so the action can sore the plan

can pass credentials same way for sure, but why not to use GH oidc everywhere?

workload identity is also available on azure: https://github.com/marketplace/actions/azure-ad-workload-identity-federation

we use OIDC in azure

that is just one ENV variable to be setup

We are working on an interface to support this. It will be pluggable. We are not implementing 1Password in the initial release but it will be very easy to add other backends. At this time, can you use ENV vars instead? That is supported.

oh nice got it I will do that then thanks!

@Erik Osterman (Cloud Posse) another question is there a way to maybe run a cli cmd before a component runs? like a pre-hook for that component?

Hooks may be coming. Can you describe what you want to accomplish?

yes, on some clients sometimes we need to set and ENV that requieres som custom cli cmd to get it although sometimes is possible to do with datasource sometimes we need a cli cmd to run or it will just be simpler for us, so I will like a pre-hook to run so we can get that ENV and then I can set that or use it on the provider config or pass it as an ENV

That use-case will be addressed with the exec function

https://github.com/cloudposse/atmos/pull/810

oh yeah that is perfect! that PR looks awesome specially !terraform.output !! really nice work!

@Dan Miller (Cloud Posse)

Yes, you can deploy account-map wherever you’d like, but we have several places where the default values of where to discover the component are defined. Like s3-bucket as you point out

You can overwrite those for your org. For example, we have several customers that dont use the name core or have other unique organizational units. All you would need to do is to define the value for your own org in your stack catalog for the component. Or you can define that variables once at a higher level in stacks as a common value for all Terraform

For example with s3-bucket in the README: https://github.com/cloudposse-terraform-components/aws-s3-bucket/blob/e19d3e7adb38805553246e740627fbef6c8b52a6/README.md?plain=1#L49

Note, this is more a refarch topic than an atmos topic.

Thanks @Dan Miller (Cloud Posse) for help in clarification and the suggestion on high level declaration of variables.

@Andriy Knysh (Cloud Posse)

Also, here it specifies “settings.templates.settings” as configurable in atmos.yaml which other documentations are invalidating: https://atmos.tools/core-concepts/stacks/templates/datasources

@Andriy Knysh (Cloud Posse)

You might need to mess with the template evaluations, as your are using templating with the template data sources

I’m not sure what you mean by that. I use 2 evaluations and already tried 1 and 3. The error stays the same

This is my current atmos.yaml

base_path: "."

components:
  terraform:
    base_path: "components/terraform"
    apply_auto_approve: false
    deploy_run_init: true
    init_run_reconfigure: true
    auto_generate_backend_file: true
  helmfile:
    base_path: "components/helmfile"
    use_eks: false
    cluster_name_pattern: "{environment}-cluster"

stacks:
  base_path: "stacks"
  included_paths:
    - "deploy/**/*"
  excluded_paths:
    - "**/_defaults.yaml"
  name_template: "{{.vars.tenant}}-{{.vars.account}}-{{.vars.account_id}}"

workflows:
  base_path: "stacks/workflows"

schemas:
  jsonschema:
    base_path: "stacks/schemas/jsonschema"

templates:
  settings:
    enabled: true
    evaluations: 2
    sprig:
      enabled: false
    gomplate:
      enabled: true

settings:
  list_merge_strategy: merge

logs:
  file: "/dev/stderr"
  level: Trace

@Dennis Bernardy please run the following command (it will show you the final config for the component in the stack, and if the templates got evaluated)

atmos describe component <your-component> -s <your-stack>

the command also accepts the ----process-templates flag to enable/disbale template procesing (so you can see the values with and wo template evaluation)

https://atmos.tools/cli/commands/describe/component/#flags

(note that templates are not evaluated in atmos.yaml, so if ou want to use url: "aws+smp:///ai/infra/acm/certificate/{{ .vars.account }}.url", it needs to be in Atmos stack manifests)

The command does not work:

-> % atmos describe component -s stack argocd

Found stack manifests:
- deploy/accounts/dev.yaml
- deploy/accounts/sandbox.yaml
- deploy/accounts/tools.yaml

panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/cloudposse/atmos/internal/exec.ProcessStacks({{0x108fc0ef0, 0x1}, {{{0x14000c3f620, 0x14}, 0x0, {0x1400092eec0, 0x31}, 0x1, 0x1, 0x1, ...}, ...}, ...}, ...)
	github.com/cloudposse/atmos/internal/exec/utils.go:426 +0xd4c
github.com/cloudposse/atmos/internal/exec.ExecuteDescribeComponent({0x16ba9f592?, 0x10641a958?}, {0x16ba9f57c?, 0x1063c369c?}, 0x1)
	github.com/cloudposse/atmos/internal/exec/describe_component.go:75 +0x174
github.com/cloudposse/atmos/internal/exec.ExecuteDescribeComponentCmd(0x1091ac2a0, {0x14000cb0e10, 0x0?, 0x0?})
	github.com/cloudposse/atmos/internal/exec/describe_component.go:46 +0x1c8
github.com/cloudposse/atmos/cmd.init.func5(0x1091ac2a0, {0x14000cb0e10, 0x1, 0x3})
	github.com/cloudposse/atmos/cmd/describe_component.go:21 +0x54
github.com/spf13/cobra.(*Command).execute(0x1091ac2a0, {0x14000cb0db0, 0x3, 0x3})
	github.com/spf13/[email protected]/command.go:989 +0x81c
github.com/spf13/cobra.(*Command).ExecuteC(0x1091aedc0)
	github.com/spf13/[email protected]/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:1041
github.com/cloudposse/atmos/cmd.Execute()
	github.com/cloudposse/atmos/cmd/root.go:107 +0x32c
main.main()
	github.com/cloudposse/atmos/main.go:10 +0x24

The templating happens inside a catalog file. for my understanding this is part of the stack and should work, no?

yes, templates work in the catalog files

please show the config for argocd (the error is not related to the templates, something wrong with the tack configs)

also , is your stack name stack?

No, the name is based on the template I have in atmos.yaml. I just replaced it here for something generic as it contains our aws account id

sorry, not sure I understand

atmos describe component -s stack argocd.  - the command needs a real stack name

Yea, sure. I run it on my machine with the real stack name. I just copied it over here with a generic name to not expose anything

This is the full catalog file of argocd (with redacted values)

settings:
  templates:
    settings:
      gomplate:
        timeout: 5
        datasources:
          certificate:
            url: "aws+smp:///ai/infra/acm/certificate/{{ .vars.account }}.url"
          argocd-iam-role-arn:
            url: "aws+smp:///ai/infra/argocd/iam"

components:
  terraform:
    argo-sso-secret:
      metadata:
        component: secrets
      vars:
        name: "name"
        secret_value:
          client_id: ""
          client_secret: ""
    argo-gitlab-secret-token-name2:
      metadata:
        component: secrets
      vars:
        name: "name"
        secret_value:
          url: "url"
          type: "git"
          username: "gitlab-bot"
          token: ""
    argo-gitlab-secret-token-argocd:
      metadata:
        component: secrets
      vars:
        name: "name"
        secret_value:
          url: "url"
          type: "git"
          username: "gitlab-bot"
          token: ""
    argocd-base:
      metadata:
        component: argocd
      vars:
        cluster_name: "{{ .vars.account }}-cluster"
        secrets:
          - "name1"
          - "name2"
          - "name3"
  helmfile:
    argocd:
      metadata:
        component: argocd
      vars:
        log_level: "info"
        sso_provider_url: "<https://sso>"
        sso_argo_secret_name: "name"
        gitlab_secrets:
          - name: name
            secret: "name"
        certificate_arn: '{{ (datasource "certificate").Value }}'
        eks_role_arn: '{{ (datasource "argocd-iam-role-arn").Value }}'
        enable_redis_ha: false
    argocd-apps:
      metadata:
        component: argocd-apps

to debug this, I would replace name_template: "{{.vars.tenant}}-{{.vars.account}}-{{.vars.account_id}}" with something static (any stack name), and then run

atmos describe component argocd -s <stack>

make the command work and show you the evaluated template tokens

That doesn’t work either. Same error

1. Replace the stack name with a static value
2. Make sure the command atmos describe component works and shows the final values for the component in the stack with the templates evaluated
3. Then make sure certificate_arn: '{{ (datasource "certificate").Value }}' works and returns the the correct value (since it depends on the credentials to access the datasource, so we need to test those credentials)

also run atmos describe stacks to see what stacks do you have and all the components in them

please run this command first

If I comment this line “certificate_arn: ‘{{ (datasource “certificate”).Value }}’” in the catalog file it works and it also renders the templates correctly

But it still errors when describing only the component

Is the aws parameter store maybe fetched before the templating is done?

I also noticed that if I describe a stack, that is a terroform component it works just fine (but it also does not have a datasource) but the argocd one, which is a helmfile, does fail

i think this is what’s happening. The templates here

        datasources:
          certificate:
            url: "aws+smp:///ai/infra/acm/certificate/{{ .vars.account }}.url"
          argocd-iam-role-arn:
            url: "aws+smp:///ai/infra/argocd/iam"

and here

certificate_arn: '{{ (datasource "certificate").Value }}'

get evaluated at the same time (Atmos sends it to the Go template engine)

this one url: "aws+smp:///ai/infra/acm/certificate/{{ .vars.account }}.url" needs to be evaluated first (in the first evaluation step)

then this one certificate_arn: '{{ (datasource "certificate").Value }}' in the second evaluation step

and to do it, you need to set evaluations: 2 and the following:

https://atmos.tools/core-concepts/stacks/templates/#excluding-templates-in-stack-manifest-from-processing-by-atmos

Ah, that did the trick. Thank you very much!

I just noticed that the datasource is used in every component. Can I limit this somehow?

yes

this

settings:
  templates:
    settings:
      gomplate:
        timeout: 5
        datasources:
          certificate:
            url: "aws+smp:///ai/infra/acm/certificate/{{ .vars.account }}.url"
          argocd-iam-role-arn:
            url: "aws+smp:///ai/infra/argocd/iam"

does not need to be global section

settings is the same first class section as vars

so it can be defined at the component level

(it will be deep-merged with all the global settings sections and with the base component’s settings section, so you can define parts of it globally, and parts related to the component in the component’s settings)

FYI https://atmos.tools/core-concepts/stacks/inheritance/

Nice. That works