#atmos (2024-04)

I believe we’ve predominantly tested it on WSL2

Are you using WSL?

No but I can redo in wsl, kinda hacking away at improving the dev environment this morning.

I’ll come back in a bit after I give wsl a shot

@Ryan I’m not sure why Atmos can’t see the Terraform binary in the PATH on Windows (we’ll investigate that). It spawns a separate process to execute TF and other binaries. For now, maybe you can try something like this:

terraform:
  command: <path to TF binary>

the command attribute allows you to select any binary to execute (instead of relying on it to be in PATH), and also it aloows you to set the binary for OpenTofy instead of Terraform (if you want to use it)

this config

terraform:
  command: <path to TF binary>

can be set globally (per Org, tenant, account), or even per component (if, for example, you want to use diff TF or OpenTofu binaries for diff components)

components:
  terraform:
    my-component:
      command: <TF or OpenTofu binary>

let us know if that works for you and how it can be improved

Will play around with it today, I appreciate your responses. We’re in geodesic now and only really leverage atmos so I’d like to try to make it easier to access for my team directly from vscode in native os.

Ok, I’ll probably have to talk to you gentlemen friday about risk. the issue is we’re on 1.4.x and I was pulling that atmos version, when I grabbed the latest atmos it was fine. I could tell it was a separate process too but im like uhhh where the heck is that little guy going for env.

I dont know the risk involved in using the updated exe, we are very narrow in our atmos usage specifically around terraform and the name structuring

i would think thats all just atmos yaml but the updated exe is like 4x the size of the one I was using previously.

so the latest Atmos version works OK on Windows?

yea but now its complaining about my structure so new issues, its ok though

the terraform piece no longer complained when i updated

sorry i didnt want to present a bug before i checked that

no problem at all. We can help you with any issues and with the structure

Yea it all works happy now besides that it cannot find the stack based on atmos.yaml stacks: config, I’ll read deeper on this. From what I can see , my stacksbaseabsolutepaths seems to be pointing at my components /../.. directory. Somethings fishy with my atmos.yaml.

you can DM us the file, we’ll review

Hrm… let me see if I can help.

if I can define a single resource using plain terraform syntax on the deepest level of a stack without creating a component for it.

So let’s take a step back. What we call a “component” in atmos is something that is deployable. For terraform, that’s a root module. In terraform, it’s impossible to deploy anything outside of a root module.

So we’re confusing 2 concepts here. And it happens easily because many different vendors define “stacks” differently.

Cloud Posse, defines a stack as yaml configuration that specifies how to deploy components (e.g. root modules).

So if you have a file like [eks.tf](http://eks.tf), you will need to stick that in a root module.

In atmos, we keep the configuration (stacks) separate from the components (e.g. terraform root modules).

So inside the stacks/ folder you would never see terraform code.

All terraform code, is typically stored in terraform/components/<something>

So this is quite different from tools like terragrunt/terramate that I believe combine terraform code with configuration.

so if I need to deploy a single resource to a single environment only, I’d need to create a wrapper module in components and use in the stack?

Perhaps… That said, I’d have to understand more about what you’re trying to do. For example, at Cloud Posse, we would never “drop in” an EKS cluster.

An EKS cluster has it’s own lifecycle, disjoint from a VPC, and applications. It deserves it’s own component.

Many times, users will want to extend some functionality of a component. In that case, a it’s great to use the native Terraform _override.tf pattern.

For example, if there’s already an eks.tf file in a component, and you want to tweak how it behaves, you could create an eks_override.tf file

https://developer.hashicorp.com/terraform/language/files/override

an eks cluster is a bad example. But a lighter resource like ECR registry which does not require too many input parameters to configure is a better one. Having a module for every AWS resource would be infeasible to maintain with the small team. I checked out the library of TF components from Cloudposse and it looks impressive but probably require an orchestrated effort of the entire team to support. AWS provider changes very frequently and you have to keep up to update the modules. Another point is that our org comes from a very chaotic tf organization trying to implement a better structure. Replacing everything with modules in a single sweep is not realistic. So we would like to start using Atmos with our existing tf code organization gradually swapping out individual resources with modules. In our use case environments (AWS accounts) do not have much in common. Having a module for a resource that only used in a single place does not make much sense

@alla2 you don’t need to replace everything. If you want a new component, just put it into components/terraform/<new-component> folder (it could be one TF file like [main.tf](http://main.tf), or the standard config like [variables.tf](http://variables.tf) [main.tf](http://main.tf) , [outputs.tf](http://outputs.tf) etc.

then, in stacks, you provide the configuration for your new component (TF root module)

and in stacks, you can make it DRY by speccifying the common config for all accounts and environments in stacks/catalog/<my-component>/defaults.yaml

and then in the top-level stacks you can override or provide specific values for your component for each account, region, tenant etc.

this is to separate the code (TF components) from the configuration (stacks) (separation of concerns)

so your component (TF code) is completely generic and reusable and does not know/care about where it will be deployed - the deployment configs are specified in stacks

So we would like to start using Atmos with our existing tf code organization gradually swapping out individual resources with modules

that’s easily done - put your TF code in components/terraform and configure it in stacks

then in the future, you can create a repo with your TF modules, and in your TF components you would just instantiate your modules making your components very small

Thanks, Andriy and Erik for your input and the options you provided. I’ll have another look. I agree that having a module for a logical group of resources is a right thing to do. But it usually only works when you already have a good code organization and a somewhat large codebase. I’d argue that managing everything with modules is not productive at the start as you spend a lot of time creating modules which you don’t even know if you can reuse later. Even within a mature org parts of the configuration can be defined as plain resources as they are only useful in this particular combination for this particular environment and does not demand a separate module.

Yes, so I know where you’re coming from, and if you’re stuck in that situation - other tools might be better suited for dynamically constructing terraform “on the fly”. Right now, atmos is not optimized for that use-case.

I’d argue that managing everything with modules is not productive at the start as you spend a lot of time creating modules which you don’t even know if you can reuse later.

This is true… There’s a significant up front investment. Our belief though, is once you’ve created that investment, you almost don’t need to write any HCL code any more.

Everything becomes configuration.

(We’ve of course spent considerable time building that library of code, which is why we have so much)

You might have already come across this, but here’s how we we think of components for AWS: https://github.com/cloudposse/terraform-aws-components

These are all the “building blocks” we use in our commercial engagements

yeah, I was referring to it above, impressive piece of work. Right now we’re mostly using another modules library which I guess should also work fine with Atmos.

Yes, though keep in mind those are child modules, and not root modules.

That said, it would be interesting to show a reference architecture using those child modules. It would be easy to vendor them in with atmos vendor, and atmos can automatically generate the provider configuration.

Let us know if you need some guidance…

Hey, I would to add some follow up to this discussion under the @alla2 message. As far as I see that atmos doesn’t provide any option to pass output from one module as an input to another (with optional conditionals and iteration). It can only be achieved with instrumenting the module’s code by context with remote backend module, isn’t it? P.S. It probably could be achieved also with some scripting in workflows , but it would be more like workaround, then I’m not taking it into account.

Yes, we have intentionally not implemented it in atmos since we want as much as possible to rely on native Terraform functionality. Terraform can already read remote state either by the remote state data source or any of the plethora of other data source look ups. We see that as a better approach because it is not something that will tie you to “atmos”. It will work no matter how you run terraform, and we see that as a win for interoperability.

To make this easier we have a module to simplify remote state look ups that is context aware. This is what @Roy refers to.

https://atmos.tools/core-concepts/components/remote-state/

@Erik Osterman (Cloud Posse) okey, because I was hoping if there is possibility to have implementation-agnostic (then context-agnostic) building blocks (that could be vendored from i.e. Cloud Posse ,Gruntwork, Azure Verified Modules, etc.) and to glue things together with business logic with use of higher-level tooling (i.e. Atmos), without usage of any intermediate wrapper modules and without instrumenting the low-level blocks. But it would require capabilities to connect outputs to inputs in declarative way with some basic logic support (conditionals, iteration). Of course for more sophisticated logic some “logic modules” could be written, but after all it would be still the totally flat structure of modules with orchestration on the config management layer, i.e. Atmos. For now with Atmos if I want to, i.e. use output from one module in for_each for other module, I need to have additional module that: 1. grab the output from remote state and 2. implements this for_each for the second module. Am I right?

So what I tried to say, was we take that implementation agnostic approach. By not building anything natively into atmos it’s agnostic to how you run terraform.

We are also building some other things (open source) but cannot discuss here. DM me for details.

These are terraform mechanisms to further improve what you are trying to do across multiple clouds, ideally suited for enterprise.

For now I have 3-layer layout of modules. First is set of context-agnostic building blocks (like Gruntwork modules ). The second is the orchestrator layer that carries the context with use of external config store and implements business logic. The last is environment layer that just calls the second with exact input. I’m looking for some solution that could eliminate the 2. and 3. layer and Atmos deals with the 3. in the way that I loved

now I’m looking at layer 2. and try to figure out how to deal with it :P

Hrmmm think there might have have been a punctuation problem in the previous message. You want to eliminate layer 2 and 3?

yes, I’m wondering about eliminating layer 2 and 3

Do you mean calling “child” modules as “root” modules by dynamically configuring the backend? Thus being able to use any module as a root module. Then needing to solve the issue of passing dynamic values to those child modules, without the child modules being aware of its context or how it got those values.

sometimes it would require “logic modules”, that would act as a procedure in general purpose lang (some generalisation of locals), but after that it would be simple as

resource_component:
  vars:
    var1: something
    var2: something
roles_logic_component:
  vars:
    domain_to_query: example.com
{{ for role in roles_logic_component.output.roles }}
{{ role.name}}-role-component:
  name: {{ role.name }}
  scope: {{ resource_component.output.id }}
  permissions: some-permissions

Of course it would require to build dependency tree on the tool level, what is duplication of tf responsibility, but.. it would be not only for tf

As You can see, the context is carried by tool itself then

A forth coming version of atmos will support all of these https://docs.gomplate.ca/datasources/

Will that solve what you want to do?

depends on final implementation, but sounds like it would

did You foresee this use case I described?

It makes sense, although it’s not a use case we are actively looking to solve for ourselves, as we predominantly want nice and neat components that are documented and tested. The more dynamic and meta things become, the harder to document, test and explain how it works.

correct, templating can become difficult to understand and read very fast. Having said that, we’ll release support for https://docs.gomplate.ca/functions/ in Atmos manifests this week

(Sprig is already supported https://masterminds.github.io/sprig/)

then what is for You the recommended way of gluing things together? Let’s take a resource <-> roles example from my para-yaml-file.

do You have some ETA for datasources?

recommended way of gluing things together? We take a different approach to glueing things together, which is the approach you want to eliminate We design components to work together, using data sources. This keeps configuration lighter. We are weary of re-inventing HCL in YAML.

Keep modules generic and reusuable (e.g. github.com/cloudposse)

Then create opinionated root modules that combine functionality. Root modules can use data sources to look up and glue things together.

Segment root modules by life cycle. E.g. a VPC is disjoint from an EKS cluster which is disjoint from the lifecycle of the applications running on the cluster.

The way we segment root modules, is demonstrated by our terraform-aws-components repo.

ETA for gomplate - we’ll try to release it this week (this will include the datasources)

but probably it won’t have such wonderful UX as {{ component.output }} reference? :D

@Roy Sprague maybe you have some thoughts on this one https://github.com/cloudposse/atmos/issues/598

@Roy Sprague this is now supported!

https://sweetops.slack.com/archives/C031919U8A0/p1718495222745629

Morning,

@Erik Osterman (Cloud Posse) one good think (among many others!) that i find using vendoring is, that i can simply vendor versioned context.tf files in every component. Also i can change the component locally before pushing the change, if i use a local version of a component.

is there a way to cover those use cases using the metadata.url option?

@Stephan Helas can you provide a configuration example of what you would ideally be able to do?

what i’m trying to say is: i like the idea to use the url metadata, but i don’t know how to develop components with this configuration

I think my confusion stems from which part of this thread you are referring to. It’s partially my fault for re-awakening it months later. So we just added the ability to pass outputs between components, which was something @Roy Sprague suggested above (with a different syntax), and we came around to the idea, and recently released it. I am very bullish about this now.

Since we don’t support a URL in the metadata of component configuration in a stack, I am scratching my head.

sorry, that was my fault you totally right, the topic about remote state vs passing outputs is much more important. my coworker struggle to grasp the concept for remote-state as module. so i will make time to look into the new approach right away

the metadata.url is nice to have and will be put into my backlog for later

imports, deps and deps_all are not related to the component, but rather to the stack

https://atmos.tools/cli/commands/describe/component/#difference-between-imports-deps_all-and-deps-outputs

they are added to describe component as an additional info that might be useful in some cases (e.g. to find all manifests where any config for the component is defined)

Thanks much @Andriy Knysh (Cloud Posse). I think my main intent to understand these things are related to a project that am trying to work is by creating a single file which will serve as a registry for bootstrapping new accounts with foundational infra (such as account dns , region dns , vpc , compute layer , and preferably storage , and some global component need for each account (each product gets an account) . So the registry file i think will be a stack in the atmos file with values in it and a template file preferably to create all the files and put in the path that atmos needs . So this way if I need to bring up an account , all I need is to add the contents to the registry file , run a cli tool probably to generate the file needed for atmos

Preferably have the account related meta data config to be stored in a json in a S3 to start with( relational db in the future ) an api to for the the platform team so devs can use platform cli to get the values to manipulate their configs etc .

Does it makes sense? Or I am approaching this wrong?

it does make sense. In fact, we will be working on atmos generate functionality to generate diff things (projects, components, stacks, files, etc.) from templates, and what you are explaining is exactly what the new Atmos functionality will do cc: @Erik Osterman (Cloud Posse)

Interesting! Including the metadata store with an api?

When is the generate feature set to release ?

Interesting! Including the metadata store with an api?

we’ll be leveraging the beautiful Go lib

https://docs.gomplate.ca/

https://github.com/hairyhenderson/gomplate

(embedded in Atmos)

so all these Datasources will be supported

https://docs.gomplate.ca/datasources/

and functions

https://docs.gomplate.ca/functions/

this is a multi-phase process, phase #1 will be to add gomplate support to Atmos stack manifests. Currently https://masterminds.github.io/sprig/ is supported, soon (this or next week) gomplate will be supported as well

https://atmos.tools/core-concepts/stacks/templating/

This looks nice . What do you think about how we could implement the metadata config store with atmos in picture? For storing environment specific metadata , say like I want to look logs for one environment , atmos somewhere does the deep merge and have the information , (other examples could be anything that folks need that information but don’t need to look at the tfstate or log into AWS account , cluster url , sqs used by the service (non secrets) . Is this possible ? If so a general idea would help . Thanks again Andriy

we’ll be implementing that by using https://docs.gomplate.ca/datasources/ - whatever is supported by gomplate will be supported by Atmos (and that’s already a lot including S3, GCP, HTTP, etc.). So at least in the first version it will not be “anything” b/c it’s not possible to create a swiss-knife tool that would do everything. Maybe later we might consider adding a plugin functionality so you would create your own plugin to read from whatever source you need (we have not considered it yet)

@Shiv if you are asking how you could do it now, you can do it in Terraform using the many providers that are already supported by Terraform (https://registry.terraform.io/browse/providers). Create Terraform components using providers you need (to get the metadata), and use Atmos to configure and provision them

you can do one of the following:

• exclude all defaults.yaml • exclude the entire catalog folder

see https://atmos.tools/quick-start/configure-cli

depending on the structure of your stacks folder, you can use included_paths to include everything and then excluded_paths to exclude some paths, or just included_paths to only include what’s needed

oooh, now I see, thanks!

So this is where the OPA policies come into play.

@Andriy Knysh (Cloud Posse) might have an example somewhere.

Since you’ve probably become familiar with atmos describe (per other question), anything in that output can be used to enforce a policy.

See:

# Check if the component has a `Team` tag

here: https://atmos.tools/core-concepts/components/validation#opa-policy-examples

also https://atmos.tools/quick-start/configure-validation/

I see , so even if the stack’s vars section is missing the necessary tags the opa policies can add the tags during the runtime ? Don’t think so yes? Because there is not state for the atmos to have that metadata context? Because the devs could be adding incorrect tags and then tags pretty much becomes useless if it doesn’t fit the company’s tagging scheme

opa policies can add the tags during the runtime Policies are simply about enforcement, but don’t modify the state.

However, since with atmos, you get inheritance, it’s super easy to ensure proper tagging across the entire infrastructure.

those tags can be set at a level above where the devs operate. They basically don’t even need to be aware of it.

Based on where a component configuration gets imported, it will inherit all the context. This is assuming you’re adopting the terraform-null-label conventions.

Without using something like null-label it will be hard to enforce consistency in terraform across an enterprise.

However, there’s still an alternative approach. Since we support provider config generation, it’s possible to automatically set the required tags on any component also based on the whole inheritance modell.

Enforcements makes sense . Yes we have context.tf in all our components . What do you mean by
tags are set at a level above where the devs operate

Yes… let me try to explain that another way.

Have a look here: https://github.com/cloudposse/atmos/tree/master/examples/quick-start/stacks/orgs/acme/plat

So we can establish tags in the _defaults.yaml that get imported in to the stacks. Stacks are at different levels

What makes it hard to describe, is we don’t know how you organize your infrastructure.

In our reference architecture (not public), we organize everything in a very hierarchical manner. By setting tags in the appropriate place of the hierarchy, those tags natuarlly flow through to the component based on where it’s deployed in the stack.

So when I say “above where the devs operate”, I just mean higher up in the hierarchy of configuration.

Here’s a more advanced hierarchy https://atmos.tools/design-patterns/organizational-structure-configuration

So at each level of that hierarchy, we can establish the required tags. Developers don’t need to be aware of that, so long as they use the [context.tf](http://context.tf) pattern, they’ll get the right tags.

I was also thinking say if we have a tagging scheme for the company . And the tagging scheme is purely account for cost model for teams . So it goes something like say domains -> subdomains -> service repos . And probably product tag as well .

My thought is I just want to ask for information once. If we ask for it in a standard place such as the backstage , I would want to be able to reuse that data where we need it. So the tagging scheme would inform the data that we want to ask for.

I am thinking terraform provider is if we need as a way to grab information on a backstage or something similar to that

Possibly, but now you have configuration sprawl. It’s an interesting use-case with backstage. I’d be open to more brainstorming.

DM me and we can do a zoom sometime.

take a klook at this custom command https://github.com/cloudposse/atmos/blob/master/examples/quick-start/atmos.yaml#L177

atmos list components -s plat-ue2-dev -t enabled will skip abstract (and you can modify the command to improve or add more filters)

ahhh huzzah tysm

Good question. Cool that you’re kicking the tires.

So vendoring child modules and using them as root modules, would be more of an advanced use-case.

The way Cloud Posse typically uses and advises for the use of vendoring is with “root” modules.

It’s interesting, because what want to do I think is very similar to another recent thread we have. Let me dig it up.

Read the thread here, to get on the same page. I’d be curious if there’s some overlap with what you’re trying to do.
I’m missing the connection of how to fetch the remote state from each security group and reference the ids in the web1 component This is why I think what you’re trying to do is related.

https://sweetops.slack.com/archives/C031919U8A0/p1712062674605219?thread_ts=1711983108.729749&cid=C031919U8A0

TL;DR: we probably don’t support in Atmos (today) what you’re trying to do. That’s because we take a different approach.

You can see how we write components here. https://github.com/cloudposse/terraform-aws-components

The gist of it is, we’ve historically taken a very delibrate and different approach from tools like terragrunt (And now terramate), that are optimized for terraform code generation.

There are a lot of reasons for this, but chief amongst them is we haven’t needed to, despite the insane amounts of terraform we write.

In our approach, we write context-aware, purpose build and highly re-usable components with Terraform. In terraform, it’s so easy to look up remote state, like “from each security group and reference the ids in the web1 component”, so we don’t do that in atmos.

Instead, we write our components so they know how to look up that remote state based on declarative keys,

So if we look at your example here:

    web/1:
	  name: sampleapp
	  vpc_id: <reference remote state from core stack>
	  ecs_security_group_ids:
	    - remote_state_reference_security_group/1
	    - remote_state_reference_security_group/2 

The “web” component should be aware how to look up security groups, e.g. by tag.

Child modules, shouldn’t know how to do this. Because they are by design less opinionated, so they work for anyone - even those who do not subscribe to the “Cloud Posse” way of doing Terraform with Atmos.

Atmos is opionated, in that it follows our guidelines for how to write terraform for maximum reusability without code generation.

I wouldn’t write it off that we’ll never support it in atmos, just there are other things we’re working on right now before we consider how to do code generation, the pattern we don’t ourselves use right now.

Yeah, I think my ultimate goal is to eliminate rewriting the same structure over and over again for a technology. So if I have a CloudPosse module that knows how to write a subnet very well, already has the context capabilities, and can be reused anywhere. I’d like to be able to take one product, say a kuberenets stack, and build subnets and what not by reference, gluing the pieces together like what you linked in the previous article. The modules can be opinionated enough that they meet our needs….or we can write our own more opinionated root modules, and then reuse them across all of our business needs.

Then a new product release can just tie those things together via the catalog and tweaked as needed on a stack by stack basis.

So with this in mind and just to check my comprehension of the current state:

• Vendor in root modules that can be reused over and over again.

• Create my child modules that reference those root modules which are more opinionated and build core catalogs off of the child modules which can have a baseline configuration in the catalog and then tweak based on a stack basis.

please see https://atmos.tools/core-concepts/components/remote-state

in your component you just need to add this part of code (in remote-state.tf ) using the remote-state module:

module "vpc_flow_logs_bucket" {
  count = local.vpc_flow_logs_enabled ? 1 : 0

  source  = "cloudposse/stack-config/yaml//modules/remote-state"
  version = "1.5.0"

  # Specify the Atmos component name (defined in YAML stack config files) 
  # for which to get the remote state outputs
  component = var.vpc_flow_logs_bucket_component_name

  # Override the context variables to point to a different Atmos stack if the 
  # `vpc-flow-logs-bucket-1` Atmos component is provisioned in another AWS account, OU or region
  stage       = try(coalesce(var.vpc_flow_logs_bucket_stage_name, module.this.stage), null)
  tenant      = try(coalesce(var.vpc_flow_logs_bucket_tenant_name, module.this.tenant), null)
  environment = try(coalesce(var.vpc_flow_logs_bucket_environment_name, module.this.environment), null)

  # `context` input is a way to provide the information about the stack (using the context
  # variables `namespace`, `tenant`, `environment`, `stage` defined in the stack config)
  context = module.this.context
}

and configure the variables in Atmos

yeah, I was reading through this, and I my disconnect has been trying to incorporate this with root modules that don’t already have this in place.

So, in my example, I want to build multiple subnets by declaring the same subnet module vendored from CloudPosse. If I don’t need a child module, I don’t want to create one. However, it sounds like what I need to do is build these child modules with the remote state configuration with the exports listed that I’ll need in other stacks.

I’ve only been working with atmos for a couple of days, so my apologies if I missed something obvious in that document.

if you want us to review your config, DM me your repo and we’ll try to provide some ideas

Other things that work well with vendoring in this sort of situation can be using [_override.tf](http://_override.tf) files

You can vendor in the the upstream child module and essentially do monkey patching using override files

https://en.wikipedia.org/wiki/Monkey_patch https://developer.hashicorp.com/terraform/language/files/override

(what we want to avoid doing is reimplementing HCL in YAML )

Hrm, that’s an interesting thought. Currently what I’m thinking is that I need to evaluate a polyrepo structure for my highly opinionated modules which build the solution we’re looking for. There I can run through assertion testing which was added recently and pull in passing versions to my “monorepo” which helps controls different stacks/solutions and build the catalog/mixins for those opinionated repositories.

So I can still use the CloudPosse subnet module should I need it in a root module by simply calling and pinning it, and then build my YAML catalogs around those vendored root modules. That brings less imports into the control repository and less things to maintain in vendor files…I just need to make sure that we are consistent and clean with versioned repos of our own that we’re vendoring in.

That would then allow me to bring in the remote_state configuration that Andriy referenced to build out all of my “core” services and export anything that would need to be referenced in other modules.

Yep, I think that might work.

If it would be helpful, happy to do a zoom and “white board” on some ideas.

I really appreciate that, thank you. I’m going to take some time and rework a personal project I’ve been working on to learn Atmos and will report back once I have it a bit more fleshed out. Really appreciate your time and conversation today, thank you so much.

heh, wow. A lot of similar requests to this lately. If not workflows steps, then state between components.

I think using an intermediary file is your best bet right now.

For advanced workflows, you may want to consider something like Go task. You can still call gotask as a custom command from atmos.

We have improvements planned (not started) for workflows. If you could share your use-case, it could help inform future implementations.

yeah i mean ultimately i need to run a shell command as part of my cold start (azure) based on an ID that got generated by terraform… long term i wont’ have this probably because there’s another provider i can use to skip the shell command and just do it all in terraform but going through corp approvals

Ok, then avoid the yak shaving for a elegant solution and go with an intermediary command.

Or, a variation of the above. Ensure the “ID that got generated by terraform” is an output of the component

Then in your workflow, just call atmos terraform output.... to retrieve the output.

You can use that in $(atmos terraform output ....) to use it inline within your workflow

e.g.

workflow:
  foobar:
    steps:
    - echo "Hello $(atmos terraform output ....) "

Yep that’s true. Haven’t quite discovered how complex this morphs into but best to start small and simple

Thanks

heads up @RB this rolls back whaet we implemented in 515 and TF_WORKSPACE due to it behaving differently

Ah yes no worries, i saw the issue that Jeremy raised. Thanks for the ping

please see this doc https://atmos.tools/core-concepts/components/remote-state

in your component where you want to read a remote state of another component, add

module "xxxxxx" {
  source  = "cloudposse/stack-config/yaml//modules/remote-state"

is one approach

We’re working on refining another appraoch as well, but it’s not published. It’s also a native-terraform approach.

It’s important to address the heart of it: we try to leverage native terraform everywhere possible

Since terraform supports this use-case natively, we haven’t invested in solving it in atmos because that forces vendor lock-in

We try to not force vendor lock-in

@Ben in Atmos 1.68.0 we added gomplate to Atmos Go templates, and added the enabled flag to enable/disable templating, but the flag affected not only Atmos stack manifests, but all other places where templates are used (including vendoring and imports with templates), see https://atmos.tools/core-concepts/stacks/templating

we’ll release a new Atmos release today which fixes that issue

for now, either use Atmos 1.67.0, or add

templates:
  settings:
    enabled: true

to your atmos.yaml

today’s release 1.70.0 will fix that so you don’t need to do it

thanks! enabling templating in the config solved the issue

thanks. That flag should not affect templating in vendoring and imports, so it’s a bug which will be fixed in 1.70.0

currently Atmos does one pass at Go template processing, your example requires two passes. It’s not currently supported, but we’ll consider implementing it in the near future

Thank you for the quick response (not expected on a Saturday). I believe it’s helpful/convenient.

for now consider repeating the template (not very DRY, but will work)

yes, that’ll work, thank you

BTW really enjoying using Atmos (as a current Terragrunt user)

thank you

@Andriy Knysh (Cloud Posse) would know the specifics of this, but one word of caution - this couples more of the filesystem to the terraform state, making it more cumbersome to reorganize stacks in the future

It’s really nice to be able to move stack files around, and it doesn’t affect your terraform state.

(It was a design consideration that of configuration be derived from configuration rather than filesystem location)

That said, now with some of the go template/gomplate stuff, it could be possible.

So if you add something like this in your stack manifests (where depends on your import/inheritance chain), it will work as you describe

vars:
  stage: staging
  tags:
    stack_file: '{{ .atmos_stack_file }}'
    
import:
  - catalog/myapp

terraform:
  backend_type: s3
  backend:
    s3:
      workspace_key_prefix: '{{ .atmos_stack_file }}'

components:
  terraform:
    myapp:
      vars:
        location: Los Angeles
        lang: en

Note, that var.tags.stack_file is entirely optional and has no bearing on the workspace_key_prefix. I just added it as an example for how you could also tag every resource with the stack file that manages it.

For this to work, make sure you have the following enabled in your atmos.yml

# <https://pkg.go.dev/text/template>
templates:
  settings:
    enabled: true
    # <https://masterminds.github.io/sprig>
    sprig:
      enabled: true
    # <https://docs.gomplate.ca>
    gomplate:
      enabled: true

Or you might get an error like this:

❯ atmos describe stacks          
invalid stack manifest 'deploy/staging.yaml'
yaml: invalid map key: map[interface {}]interface {}{".atmos_stack_file":interface {}(nil)}

the templates support all values that are returned from atmos describe component <component> -s <stack>. So you can also do something like

workspace_key_prefix: '{{ .vars.app_name }}/{{ .vars.environment }}...'

oh if you configure the settings section with some metadata, you could you it in the templates as well, e,g, (just an example)

workspace_key_prefix: '{{ .settings.app_name }}/{{ .settings.workspace_key }}...'

the settings section is a free-form map, you can add any properties to it, including embedded maps/objects. Then it cam be used anywhere in the stack manifests in Go templates and Sprig Functions, Gomplate Functions and Gomplate Datasources

https://atmos.tools/core-concepts/stacks/templating

thank you both for the response. I see the value in disconnecting the path to the state-file and the file-system, I’ll give that more thought and thank you for informing about the template support of atmos describe component ... values.

Hey @toka this is the exact use-case we solve in our reference architecture and our customers use. I’ll take a stab at answering it.

Right know was thinking should I split the code between different orgs directories, or between different tenats, or both

So first to understand is how we achive this at the terraform layer. It’s important that your terraform is written to a) support parameterized region in [providers.tf](http://providers.tf) b) leverage something like our terraform-null-label module, c) use some field to connote the region.

We use env in the terraform-null-label to connote the region, and stage to connote where it’s in the lifecycle (e.g. dev, staging, prod, etc.)

So basically, to go “multi-region” requires first solving how you naming your resources, using a naming convention.

Are you already familiar with our cloudposse/terraform-null-label module?

Regarding naming, I have a naming-module implemented already, all module resources are named by sourcing the naming module first

Right know was thinking should I split the code between different orgs directories, or between different tenats, or both So, “code” might be a bit ambiguous here since “code” could refer to the stacks (YAML) or maybe the terraform (HCL).

We recommend organizing the stacks the way you describe but organizing the terraform much the way you organize any software application’s code. So for example stick all “EKS” related root module components into components/terraform/eks/<blah>

Ok, I’d need to know more about the naming module to answer specifically, but if it works similar to null label, you should be good. To be clear, atmos does not know anything about null-label, it’s just most of our documentation assumes usage.

https://masterpoint.io/updates/terraform-null-label/

Here’s how we organize it in our commercial reference architecture

How we organize components is largely the same as this (which is very different from how stacks are organized): https://github.com/cloudposse/terraform-aws-components/tree/main/modules

Note, components don’t have to live in a separate repo. Ours do because this is how we distribute them as open souce.

In general, component should be organized by “service” (e.g. EKS, vs ECS), or “app1” and “app2”, and if multi-cloud, but provider, so aws/eks and azure/aks

but if it works similar to null label, you should be good Yes, it works very similar, almost the same, though Cloud Posse implementation seems to be better since you can inherit with

 # Here's the important bit
    context = module.public_alb_label.context

avoiding code duplication - in my TF codebase, I need to instantiate naming_module for each and every resource.

So, “code” might be a bit ambiguous here since “code” could refer to the stacks (YAML) I’m referring to stacks/YAML files structure. For HCL TF files, I’m planning to re-use all TF modules existing know, such as common_vpc , common_compute_instance , common_iam etc.

I think my need is to not duplicate stack configs across regions

Ok, so far I think we’re on the same page.

Now, use inheritance to set vars at each level of your directory structure.

We use the convention of a file called _defaults.yaml that we stash at each level. This then sets the namespace (At the root of the directory), the tenant (aka OU) at the OU level, the stage in the stage folder, the env in the region folder, and so on. Each default file can import the parent default.

I think my need is to not duplicate stack configs across regions Then we use the “catalog” diretory to define the baseline configuration for “something”

• Something could be a service

• Something could be a region

• Something could be an application

• Something could be a layer What “something” is will depend on how you want to logically organize your configuration for maximum DRYness and reusability

Here’s a quick demo of how we organize it in our refarch

What “something” is will depend on how you want to logically organize your configuration for maximum DRYness and reusability Ok Erick I think now I get it, this seems to be it. This is very lit, very promising. I needed to see that example in order to understand that though catalog, I can set a baseline for regions, for services or a layer and not only a TF component which is how I saw it initially

Yes, exactly! It’s more than just for individual TF components, which makes it very powerful. You can define the configuration for how a set of components behave when in a “region” or in an OU.

@toka please take a look at https://github.com/cloudposse/atmos/tree/master/examples/quick-start/stacks, which is described in Atmos Quick Start

As Erik mentioned, the defaults go to _defaults.yaml at every scope (org, tenant, account, region)

https://github.com/cloudposse/atmos/blob/master/examples/quick-start/stacks/orgs/acme/_defaults.yaml

the mixins are the catalog/defaults/baseline for all defaults for tenants, accounts and regions

https://github.com/cloudposse/atmos/tree/master/examples/quick-start/stacks/mixins

(the mixins are imported into the top-level stacks to make them DRY)

the catalog is the catalog/baseline for the components config

https://github.com/cloudposse/atmos/tree/master/examples/quick-start/stacks/catalog

(the component catalog is imported into mixins if you use the same component in many tenants/accounts/regions, or is imported into the top-level stacks directly)

so the baseline for regions is in https://github.com/cloudposse/atmos/tree/master/examples/quick-start/stacks/mixins/region

the baseline for tenants is in https://github.com/cloudposse/atmos/tree/master/examples/quick-start/stacks/mixins/tenant

the baseline for accounts is in https://github.com/cloudposse/atmos/tree/master/examples/quick-start/stacks/mixins/stage

etc.

Thank you for you help! I understand you put a lot of work for the documentation and examples and I appreciate it a lot, but I won’t pretend its easy to switch the thinking

I’ll try my best to adopt atmos and prove it’s value, hopefully some day we’ll have some resources to contribute back to the project. Wish me luck

I also got a lot of value out of the design patterns documentation. https://atmos.tools/design-patterns/

Aha, so you’re trying to pull down some Cloud Posse managed components, which assume you’re using tenant

It’s a good question, I’m afraid right now that’s not possible without forking the components. Our components are designed/optimized to work with our commercial reference architecture. We give them away in the event they are useful for others, but they are not as generic as our child modules (e.g. https://github.com/cloudposse)

We intend to generalize this in future versions of our refarch, but for now that’s not supported.

But I can tell you how to work around it!

Use the monkey patching approach described here

https://sweetops.slack.com/archives/CB6GHNLG0/p1713271645909859?thread_ts=1713254557.118159&cid=CB6GHNLG0

In this case, you would create an overrides that alters the map_environment

This way you can preserve all the rest of the functionality, without forking.

In this case, you would use [main_override.tf](http://main_override.tf) to replace the local

locals {

  map_secrets = ....
}

Ah k, and the override is there so you can pull updates with getting it overwritten? From time to time you might need to update the override in order to match the new schema?

Yes, since the overrides are in a file named ....[_override.tf](http://_override.tf) they will never get overwritten by our upstream. This is a “contract” we have to ensure vendoring + overrides always works.

In otherwords, we will not have files named ....[_override.tf](http://_override.tf) in our upstream components repo.

k, thanks for the fast answer!

So there’s no strict enforcement on AWS or in atmos on what those names are. It’s purely self-inflicted

It’s not clear why you cannot change the account name in stack configuration

If we make the stage shorter, then new account names have to be learned when targeting a stack from the command line.

I did think maybe using stage aliases would help with this problem and wrote up https://github.com/cloudposse/atmos/issues/581

Aha, yes, I recall that

I think this is an interesting use case.

oh i didnt see you commented there. I was also opening this thread to see alternatives but ill read up the comments now, sorry

For example, a recent customer didn’t like that we abbreviated the stage name for production to be prod to conserve length. If we had support of aliases, the full length can still be named

Short term, I don’t see any solution besides using an abbreviation.

Longer term, there’s something else we’re about to release that could be where this is solved.

If we go with short term of an abbreviation, that would be the codified naming convention.

Another alternative is dropping namespace from the null label for accounts that hit these restraints…. but then we have inconsistency.

Both short terms are hard to back out from

Also, it predominantly affects specific resources in AWS.

You only need the exceptions in those cases.

We have an issue on our roadmap to implement this, but no ETA

Ah ok found the ticket https://github.com/cloudposse/atmos/issues/293

ty

currently, all paths are relative to the “stacks” folder

ya but i was trying to import relative to another catalog

you’re right, both are relatives

you are talking about paths relative to the current manifest file. This will save you from typing the path prefix (e.g. catalog/xxx), but at the same time will make the manifest non-portable (always some compromises/tradeoffs)

Hey @pv - this is the first report we hear of that.

I wonder if it could be a concurrency setting somewhere?

Are you using self-hosted runners?

@Erik Osterman (Cloud Posse) No, in this setup it is not self hosted

@Igor Rodionov any ideas on this one?

@pv can you share the logs of GHA failed run ?

There are no logs of value. On the plan or deploy stage, it just gets stuck and runs forever until the workflow is cancelled

Are you using the version of the GHA that uses atmos.yml or the “gitops” config?

weird

We have an atmos.yaml

Would you happen to be leveraging -lock-timeout anywhere?

@Erik Osterman (Cloud Posse) Nope, just checked the yaml and don’t see that value anywhere

Your integrations section is empty, which tells me it’s not configured for the current version of the actions

I am afk, but if you check the actions themselves the u have a migration guide

@Igor Rodionov @Gabriela Campana (Cloud Posse) we should error if the integration configuration is absent

What integration do I need? The documentation only shows the run Atlantis integration and I can’t find any list of other integrations

https://github.com/cloudposse/github-action-atmos-terraform-plan?tab=readme-ov-file#config

https://github.com/cloudposse/github-action-atmos-terraform-apply?tab=readme-ov-file#config

Thanks! It would be nice to have the list of integrations linked in the docs integrations page as well

Most of these arguments seem to be related to AWS so I don’t know if this will actually solve my issue. I am using GCP. The only thing I could really use from this is setting the Atmos version but we already have that set as latest in our pipeline

@Igor Rodionov any action item here?

https://linear.app/cloudposse/issue/DEV-1952/gha-with-atmos-should-error-if-the-integration-configuration-is-absent

https://linear.app/cloudposse/issue/DEV-1953/add-a-list-of-integrations-in-the-gitops-gha-with-atmos-docs

@Erik Osterman (Cloud Posse) @Igor Rodionov I did some digging and I found the issue. When deploying each project I need to pass a GHA variable for the project ID which is secured. When running this workflow, it freezes because each apply command is asked for the variable that is stored as a GHA var. So my question is how do I fix the workflow to apply the GHA variable for each plan and apply command?

Let me put this in my own words, reading between the lines.

In a GitHub repo, you’ve configured a variable (or a secret) that contains some sort of a “project id”.

When you run the GitHub Action Workflow (not an atmos workflow), it freezes during the atmos terraform apply because terraform apply is prompting for an unset variable (e.g. var.project_id). The value you want to pass to Terraform (e.g. var.project_id ) is available in the repository.

https://developer.hashicorp.com/terraform/tutorials/automation/automate-terraform

We might not be passing -input=false in our action. I’ll check. But that would explain why it’s hanging.

To solve the variable passing, are you using the TF_ENV_project_id format which corresponds to var.project_id in Terraform?

We do set -input=false which is good in the plan. Checking apply https://github.com/cloudposse/github-action-atmos-terraform-plan/blob/main/action.yml#L208

And we do set -input=false in apply. So I’m not sure why it would be hung on input. https://github.com/cloudposse/github-action-atmos-terraform-apply/blob/main/action.yml#L302

If you could share any more details, or logs that would be great. And if it’s sensitive, feel free to DM me

@Erik Osterman (Cloud Posse) I don’t know what logs I can give when the job is stuck. When I reduce the workflow file to one command, it works. When I add more than one, it hangs. When I run locally, with all commands enabled, it asks for the billing ID for each command. I am passing it as an env var as GH recommends and this was all working a month ago just fine. We have made no changes to the pipeline and the only change to this atmos workflow was adding additional plans and applies

Understood… that’s frustrating

I know it’s no consolation, but none of our customers are reporting any issues with this, so we don’t have any way to reproduce it. What happens if you pin to an earlier commit of the actions? For example, pin to a commit that corresponds to the last known date it was working

Then we could look at what changed between then and now.

Morning btw everyone.

Hrmmm so basically, are you trying to figure out the shape of the YAML for the component?

Ideally, our components all have a working example to make that easier.

In this case, you’re referring to our child module terraform-aws-network-firewall

yea but network_firewall has ALOT of variables

yea

here’s an example on how to configure terraform-aws-network-firewall in Atmos :

nice ty andriy, give me a few to read im split between a call and this.

Make sure you are using our component and not the child module

https://github.com/cloudposse/terraform-aws-components/tree/main/modules/network-firewall

ok maybe i pulled the wrong module

thats much closer in the example, i was trying to figure out the YAML objects myself

ooo likely interested in the zscaler module but with govcloud configs btw

im still very much learning terraform, so if there are better ways to handle inputs like this im all ears on learning it.

rule_group_config is a very complex variable with a LOT of diff combinations of data types. That’s why it’s set to any - this simplifies the variable, but complicates figuring out how to define it (either in plain Terraform or in Atmos stack manifests in YAML)

the example in the component repo (and the one I posted above) should help you to start (that’s a working example)

yea i started laughing out loud when i realized what i was doing was futile and then seeing how you handled it

it was complex either way but your way is much better, i appreciate the help this am

let us know if oyu need any help

if you vendor the https://github.com/cloudposse/terraform-aws-components/tree/main/modules/network-firewall component and use the YAML above to configure Atmos component, you should be able to provision. Pay attention to the VPC component and the remote-state (this needs to be already provisioned)

i might need help but i need to struggle through this a bit and learn, ill come back in a few hours.

and thank you, idk if ill have to modify based on dependency?

its a thought i hadnt had

please review the Atmos remote state docs (and ask questions if any)

https://atmos.tools/core-concepts/components/remote-state

https://atmos.tools/quick-start/vendor-components#remote-state-notes

yea we’re on remote state under globals i believe but we still need to have env review

in short, you provision the vpc component first (e.g. atmos terraform apply vpc -s <stack> )

yea i know it has a vpc dependency and i was kinda hacking vpc.id or vpc.arn

then in other components that need to get the remote state from the vpc component (e.g. to get the VPC ID), you define the remote-state Terraform module and configure a variable in Atmos to specify the name of the component

i was kinda hacking vpc.id

yea ive done that for my own modules in and out

you can def hack that for testing and to start faster with the network firewall

yea thats basically where i am now in testing

vpcid and subnetid as statics

theyre probably not going to care if my modules interact properly btw, i have an unreasonable deadline of making 5 weeks work in 2 1/2 weeks

ive made most of the work needed but im down to this piece

i can always try it in dev bc i believe were using vpc but unsure of integration

i hear you. The network firewall is def not an easy thing

yea when i got to the rules after saying it was easy im like oh…………….

anyway, adding remote state for vpc is much simpler than network firewall, and you can do it later

good lesson on data structures though

ok cool, i appreciate it

so a few thoughts - I had to edit providers.tf to include root_account_environment_name under module iam_roles - then plan fired up local.vpc_outputs and shared config profile do not exist per plan results. i could probably cut out reliance on the buckets and make them myself as annoying as that would be, otherwise i think I see defaults I can maybe configure in remote-state.tf and just point to resources. i appreciate the intergration btw i hadnt really seen tf talk across resources that way.

going to take a walk and get some fresh air.

appreciate your teams help on this.

Not sure if I follow…. need more context.

Sorry I should’ve provided context, I tried network-firewall out in the environment

to disable reading remote state for the log buckets

        # Logging config
        logging_enabled: true
        flow_logs_bucket_component_name: "network-firewall-logs-bucket-flow"
        alert_logs_bucket_component_name: "network-firewall-logs-bucket-alert"

set

logging_enabled: false

Ty

finally seeing green andriy but im unsure of my hacking consequences lol. i had to modify providers.tf as follows -

provider "aws" {
  region = var.region

  # Profile is deprecated in favor of terraform_role_arn. When profiles are not in use, terraform_profile_name is null.
  #profile = module.iam_roles.terraform_profile_name

  dynamic "assume_role" {
    # module.iam_roles.terraform_role_arn may be null, in which case do not assume a role.
    for_each = compact([module.iam_roles.terraform_role_arn])
    content {
      role_arn = assume_role.value
    }
  }
}

module "iam_roles" {
  source  = "../account-map/modules/iam-roles"
  root_account_environment_name = var.root_account_environment_name
  context = module.this.context

}

variable "root_account_environment_name" {
  type = string
  description = "Global Environment Name"
}

I also went into remotestate.tf and created a defaults = {vpc_id} and adjusted main.tf firewall_subnet_ids to a subnet map

what it does is it reads the terraform role to assume

not my final solution

i was kinda hacking it together to get it to show me green text

as long as you have it, and it can read it, and the role has the required permission, should be fine

i could probably pipe those back to variables in the yaml, im trying not to get too hacky as im still learning atmos structure

im going to run my first deploy now and see how she goes, half of what sucks about working with this resource is the coffee break each time you destroy/recreate

mmmmm

thanks again andriiy and erik, will come back again soon

hey andriy - do you have any examples of multiple rulegroups or stateful/stateless combo examples? Im swapping back and forth between the the tf code and yaml right now trying to get multiple rules going. the siracata parse of ip sets was very cool btw, i was trying to think of ways to deal with the massive rule lists

just the yaml example would be much appreciated

ill dig around otherwise, most of what i have works though, just trying to get syntax correct for more rules

this is what im struggling with but im going to take a break for a bit, i went back and forth between the main.tf and into the vars to be sure it was configuring correctly, but my yaml attempts have failed thus far for a stateless rule example -

              rules_source:
                stateless_rules_and_custom_actions:
                  stateless_rule:
                  - priority: 1
                    rule_definition:
                      actions:
                        - "aws:drop"
                      match_attributes:
                        protocols:
                        - "1"
                        source: 
                        - address_definition: $SOURCE
                        destination: 
                        - address_definition: $DST

Planned Result:

      + rules_source {
              + stateless_rules_and_custom_actions {
                  + stateless_rule {
                      + priority = 1

                      + rule_definition {
                          + actions = [
                              + "aws:drop",
                            ]

                          + match_attributes {
                              + protocols = [
                                  + 1,
                                ]
                            }
                        }
                    }
                }
            }

i feel like im giving it source and destination but my spacing or something is off

@Ryan I think it should be done like this

              rules_source:
                stateless_rules_and_custom_actions:
                  stateless_rule:
                  - priority: 1
                    rule_definition:
                      actions:
                        - "aws:drop"
                      match_attributes:
                        protocols:
                        - "1"
                        source: 
                        - $SOURCE
                        destination: 
                        - $DST

look at how it’s done in Terraform https://github.com/cloudposse/terraform-aws-network-firewall/blob/main/main.tf#L163

yea that is exactly what i was referencing but i am terrible still going from tf to yaml

let me know if it works for you

im glad i was referencing the right place

ok i see now and yea its planning

for some reason i thought i had to define address_definition, but looking now i was wrong

I understand. It’s just how it’s implemented in the TF module. We could have easily parsed address_definition in TF

content {
                        address_definition = destination.value.address_definition
                      }

and then you’d have to do

                        source: 
                        - address_definition: $SOURCE
                        destination: 
                        - address_definition: $DST

network firewall is complicated

yea thats what i was figuring in my head after you pasted and i went oh

and i appreciate the complication of it, its good practice for me

Unfortunately, Cloud Posse has not been able to prioritize this work over other work. Note, this relates more to refarch than Atmos.

@RB has success I believe doing this, and may have more guidance.

@Andrew Chemis see this prs. I was able to do it successfully.

https://github.com/cloudposse/terraform-aws-components/pull/945 (open)

https://github.com/cloudposse/terraform-aws-components/pull/943 (closed see comments)

At this point, we would recommend you use the code from the associated branches submitted by RB

We haven’t merged/incorporated it, because we (Cloud Posse) are responsible for LTS support and have no way to test the changes.

Also, since our plans involve signficant changes to these components, we don’t want to take on more than we can chew.

Excellent - thank you. This is useful

list of objects

vars:
  database_encryption:
    - state: DECRYPT
      key_name: xxx

That worked, thanks!

So there are a few observations here. And…. while we can directly address what you’re trying to do, I think there could be an xyproblem.info

When you say it’s not working, what’s the error or the observed behavior?

…is it a syntax error?

(note, that templating right now is a single pass, it’s not re-entrant. We plan to make this configurable, e.g. make it 3 passes, but it won’t be infinitely recursive)

@Stephan Helas it’s not working not b/c any errors in the template and not b/c it’s multi=pass (it’s a single pass)

it’s not working b/c you are overriding the stage variable, which is a context var which Atmos uses to find components and stacks in stack manifests

if you are using namespace, tenant, environment and stage as context vars, and in your atmos.yaml you have stacks.name_pattern, those context vars must be provided in stack manifests and known before any processing

but, what you are trying to do, it looks like you want to override the context vars. Here’s how to do it:

https://atmos.tools/cli/configuration#stacks

look at stacks.name_pattern and stacks.name_template

in your case/example, you can do this in atmos.yaml:

stacks:
  name_template: "{{.settings.tenant}}-{{.settings.region}}-{{.settings.accoint}}"

what i tried to do is to use settings instead vars as source of truth. if i have general purpose variables i don’t always want them as inputs for tf modules. but i also don’t want to duplicate this information. i’ll try adjusting the name_template. i think that is the solution to this.

Is this something I need to manually set in the Dockerfile ? or is this a bug in geodesic ?

https://github.com/cloudposse/geodesic/blob/a43fab19f924987964736a55ba3672ec4d402c3d/rootfs/etc/profile.d/atmos.sh#L3-L32

I don’t see the ATMOS_CLI_CONFIG_PATH defined in geodesic

we usually place atmos.yaml in rootfs/usr/local/etc/atmos/atmos.yaml in the repo, and then in Dockerfile

COPY rootfs/ /

which becomes /usr/local/etc/atmos/atmos.yaml in the container, and that’s a location which Atmos always checks

ohhhh i see

thanks!

that’s how we usually do it, but you can place atmos.yaml in any folder and then use the ENV var ATMOS_CLI_CONFIG_PATH (if that’s what you want)

if you don’t use the ENV var, Atmos searches for atmos.yaml in these locations. https://atmos.tools/cli/configuration#configuration-file-atmosyaml

my problem was that i want to run atmos from outside geodesic and inside geodesic so I have my atmos.yaml file in the repo root so I copied the above linked atmos.sh script and added the following to it

export ATMOS_CLI_CONFIG_PATH="${ATMOS_BASE_PATH}"

did it work?

yes

if you don’t use the ENV var, Atmos searches for atmos.yaml in these locations. https://atmos.tools/cli/configuration#configuration-file-atmosyaml this is odd because it says

Current directory (./atmos.yaml) and that is where my file is located. I get an error unless I also set ATMOS_CLI_CONFIG_PATH.

⨠ unset ATMOS_CLI_CONFIG_PATH
⨠ atmos version | grep -i atmos
Found ENV var ATMOS_BASE_PATH=/localhost/git/work/atmos
👽 Atmos v1.70.0 on linux/arm64
⨠ atmos terraform plan service/s3 --stack ue1-dev

...

module.iam_roles.module.account_map.data.utils_component_config.config[0]: Reading...

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error:
│ Searched all stack YAML files, but could not find config for the component 'account-map' in the stack 'core-gbl-root'.
│ Check that all variables in the stack name pattern '{tenant}-{environment}-{stage}' are correctly defined in the stack config files.
│ Are the component and stack names correct? Did you forget an import?
│
│
│   with module.iam_roles.module.account_map.data.utils_component_config.config[0],
│   on .terraform/modules/iam_roles.account_map/modules/remote-state/main.tf line 1, in data "utils_component_config" "config":
│    1: data "utils_component_config" "config" {

Seems like a bug. Should I write it up ?

no, this is not a bug The error is from the remote state module, see https://atmos.tools/core-concepts/components/remote-state#caveats

for the remote-state to work, atmos.yaml needs to be in a “global” directory (e.g. /usr/local/etc/atmos/atmos.yaml)

booo

lol

(all of that is b/c Terraform executes providers in the component folder, so any relative paths will not work, and atmos.yaml placed in one component folder will not affect another component)

so basically i should always have the ATMOS_CLI_CONFIG_PATH set

we always have it in /usr/local/etc/atmos/atmos.yaml (where both Atmos binary and the Terraform utils provider for remote state can find it)

and yes, you can use the ENV var

so then the alternative is

COPY atmos.yaml /usr/local/etc/atmos/atmos.yaml

I suppose that’s easier than overwriting the atmos.sh file

you can try that (there are many diff ways of doing it, copying, using ENV vars). The idea is to have atmos.yaml (or pointer to it in the ENV var) at some path where all involved binaries can find it

kk thanks andriy!

https://github.com/cloudposse/terraform-aws-components/blob/main/modules/account-map/account-info.tftmpl

Looks like it got added here https://github.com/cloudposse/terraform-aws-components/pull/496

It’s a handy script, just kind of painful to run manually since it’s so far deep into the components dir.

⨠ components/terraform/account-map/account-info/acme-gbl-root.sh "account-id" root
1234567890

Also curious how I can integrate it, otherwise i might gitignore it for now or copy it into /usr/local/bin in geodesic

We use this script with our aws-config script to generate a local AWS config: https://github.com/cloudposse/terraform-aws-components/blob/main/rootfs/usr/local/bin/aws-config#L47

Hi Dan! I was searching for this and didn’t find it. Thank you very much.

No problem!

how about using Go templates, see https://atmos.tools/core-concepts/stacks/templating#use-cases

templates can be used in the tag names and tag values

Bingo , that’s it . Thanks

currently, lists are replaced, not merged (for various reasons, one of which is it’s not possible to know in general how to handle the same values, and whether or not to delete items if one of the lists does not have them

in the next Atmos release (prob next week), we’ll add configuration (global or per component, in settings, to specify how to deal with merging of lists and to allow it

right now, you can use mixins (abstract base components) and inheritance to achieve what you want. (Note that type: abstract makes those mixins not-deployable so nobody could provision them by mistake, they are just blueprints). For example:

you can put principals-mixing-1 and principals-mixing-1 into catalog (e.g. catalog/mixins, and then just import them into the stacks. For example:

this will make those mixins reusable (you can import them into many stacks where you have the components that inherit from them), and the config DRY

@Andriy Knysh (Cloud Posse) thanks for the immediate support and details Even in this stack file var principal is not merging correct? my-component-1principal would just be [user1,user2,user3,user4] correct?

import:
  - catalog/mixins/principals-mixing-1
  - catalog/mixins/principals-mixing-2

components:
  terraform:
    my-component-1:
      metadata:
        component: # Point to the Terraform component
        inherits:
          - principals-mixing-1
      vars:
        principal: [user5,user6]
    my-component-2:
      metadata:
        component: # Point to the Terraform component
        inherits:
          - principals-mixing-2
      vars: # other vars 

would this give me final var principal to [user1,user2,user3,user4,user5,user6]?

no, the lists will not be merged (as I mentioned, we’ll add support for that). But if you create a map instead of a list, it will be merged (for this you will need to update your terraform variable to be a map type)

Ok thanks

you can use a map type and don’t change the terraform variable at all. Using Go templates. For example:

if you define principal in settings as a map, it will be inherited by the deried components and deep-merged

then using Go templates, you can convet the alredy deep-merged map into a list

for templating, see https://atmos.tools/core-concepts/stacks/templating

Thanks @Andriy Knysh (Cloud Posse) will follow up.. appreciate your help

Thanks for reporting - will get that fixed

I am so curious, are you planning to use the new backend encryption functionality of opentofu, to encrypt the state locally and commit it?

not yet first need to stop hating ibm for buying off hashicorp. until now i’ve never looked into opentofu

@Stephan Helas thanks for this. The change to the schema will be in the next Atmos release

Haha, I confess I didn’t realize how much animosity there was out there for IBM!

this is not currently supported as one command. We are working on adding this functionality to Atmos to show you the difference b/w all the components/stacks defined for the infra and what are actually in the Terraform state

for now, atmos describe stacks command shows all the stacks and all the components in each stack

refer to https://atmos.tools/cli/commands/describe/stacks

the command accepts many filters to filter the result

then, you can use a shell script and jq/yq to loop through all the stacks and components and run a TF command on each one (e.g. terraform state list)

you can also create a custom Atmos command to do so, see https://atmos.tools/core-concepts/custom-commands/

and you’ll be able to execute your custom command like this: atmos <my-command> <params>

this requires you to write some scripts, but as mentioned, we are working on adding more functionality to Atmos to list and show the diff b/w what is configured in Atmos manifests and what is actually deployed

ok. i understand. i need to look into custom commands.

but one thing: if i list components using atmos list components i got real and abstract compontest back. evan with --type = real is this normal?

and also show “stale” resources (e.g. those that were deployed in the past, but then removed from the stacks, and they are still in the TF state (or deployed)

this feature would be awesome

for a custom list command with filters for component type, see https://atmos.tools/core-concepts/custom-commands/#list-atmos-stacks-and-components

(you can copy it and def improve b/c it’s just a shell script with some jq/yq

so i’d create some shell script and the call it by using a custom comand, right?

yes

ok

and the shell script can be external to atmos.yaml (in some folder) and you just call it from the custom command. Or the shell script can be just embeded in the command itself in YAML

let us know if you need any help on this

and i could vendor this script from a git repo?

yes, you can vendor anything

yeah, thought so pretty cool

see https://atmos.tools/core-concepts/custom-commands/#advanced-examples for a some examples of custom commands

which components of which stacks are in sync with the infrastructure? Also, to be clear, we do support this today. Just not in the CLI. That’s because in practice, doing it on the command line is not how teams will succeed.

We call this “drift detection” and our GitHub Actions for Atmos support that today out of the box.

oh, yeah - i forgot about that. unfortunately i’m forced to use bitbucket without pipelining for the time beeing. but maybe i can adopt the github action somehow

Ahah, bitbucket

BitBucket / GitLab / GitHub / etc are polarizing. Teams pick them and their tradeoffs for various reasons. Often we speak to teams who are just looking for a reason to switch, something to justify it. Maybe this can be your golden ticket.

thanks @Erik Osterman (Cloud Posse), yes it’s called drift detection and it’s done in CI/CD (we support it out of the box uisng GHA), and it’s not practical to do it on command line. @Stephan Helas you need to consider that.

Whatever I mentioned above was for command line and to detect old/stale/non-used-anymore workspaces and resources (not drift detection on the existing resources) - same thing you are describing in the other thread with an old/stale TF workspace.

You should not do drift detection on the command line.

if you have hundreds or thousands of stacks, it’s not even feasible for people to do and review.

ok, do you have a tip for me how to handle drift detection, if we don’t use github (in fact its bitbucket )?

The problem with BitBucket and GitLab, et al, is they lack a marketplace like the GitHub Actions marketplace. So it’s not straight forward to distribute reusable actions.

The gist of it is that you’ll want to use atmos describe affected in concert with atmos terraform plan and atmos terraform apply. However, there’s quite a lot of work to implementing a proper CI/CD workflow for Terraform/OpenTofu.

this is what Terraform does when you use the same workspace for diff configurations

you can clean it before you switch the components, e.g. by using atmos terraform clean command https://atmos.tools/cli/commands/terraform/clean

ah, so its because i did use the component with a different stack before. ok understood.

did you override TF workspaces for the component? (if you used the default Atmos behavior to always calculate TF workspaces for the components, you would not see that behavior)

as far as i remember, i only overwrite the Bucket key_prefix.

can i define the workspace names for the components to inlude the stackname?

you can override TF workspaces per component https://atmos.tools/core-concepts/components/terraform-workspaces

ah, ok - so i do in fact overwrite the workspace, but for the remote state of a component in a different stack. a part from that i dont

so, if i use tf shell it looks like this:

▶ kn_atmos_vault terraform shell wms-base -s wms-it03-test

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Reusing previous version of hashicorp/aws from the dependency lock file
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/external from the dependency lock file
- Reusing previous version of cloudposse/utils from the dependency lock file
- Using previously-installed hashicorp/aws v5.46.0
- Using previously-installed hashicorp/local v2.5.1
- Using previously-installed hashicorp/external v2.3.3
- Using previously-installed cloudposse/utils v1.21.0

Terraform has been successfully initialized!

▶ tf workspace list
  default
* wms-it03-test-wms-base
  wms-it03-test-wms-wms

 ▶ kn_atmos_vault terraform shell wms-base -s wms-apt01-test

Initializing the backend...

The currently selected workspace (wms-it03-test-wms-base) does not exist.
  This is expected behavior when the selected workspace did not have an
  existing non-empty state. Please enter a number to select a workspace:

  1. default
  2. wms-apt01-test-wms-base
  3. wms-apt01-test-wms-wms

  Enter a value: 1


Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Reusing previous version of hashicorp/external from the dependency lock file
- Reusing previous version of cloudposse/utils from the dependency lock file
- Using previously-installed cloudposse/utils v1.21.0
- Using previously-installed hashicorp/local v2.5.1
- Using previously-installed hashicorp/aws v5.46.0
- Using previously-installed hashicorp/external v2.3.3

Terraform has been successfully initialized!
Switched to workspace "wms-apt01-test-wms-base".

▶ tf workspace list
  default
* wms-apt01-test-wms-base
  wms-apt01-test-wms-wms

so, it seems to create workspaces using <stack><component> but it clashes

if i ‘clean’ before the switch to another stack, it works. so for me it seems that tf tries to switch from a no longer existing workspace to another one. and thats why it is complaining

the current selected workspace is from the old stack, and therefore not existing (because of the workspace name template)

you prob need to delete the old/stale workspace from the state

ok, but the state is in s3 bucket, right

if yoiu are using s3 backend, then yes

in the bucket, the folders are worspace_key_prefixes , then the subfolders are workspaces

but terraform clean is not tf but atmos, right? so i can only delete something local

yes, atmos terraform clean deletes local files only. We don’t go to your TF state to delete things

yeah, i think i just live with it for the moment. in ci pipeline this should not be a problem, right?

i’m not sure about that, you have to test. You still should prob delete the stale TF workspace from the bucket

Expansion

• 25 = cloudposse context ◦ 5 = namespace is 4 chars + 1 for delimiter ◦ 16 = account names (stage) are 12 to 15 chars + 1 for delimiter ◦ 4 = fixed region is 3 chars + 1 for delimiter ◦ 0 = tenant • 20 = existing eks clusters have 20 char non-standard names • 25 = service-names can be between 15 and 25 chars Options

1. Trim the name using context’s id_length_limit a. not a good option because we can get multiple conflicting roles
2. Reduce the name of the account a. This is too painful to do
3. Reduce the name of the eks clusters by using shorter aliases in some kind of generic alias map ◦ Is it possible using go-templating to create a generic map of aliases to shorten the cluster name ?
4. Drop some or all of the cloudposse context
5. Any other option?

I’m leaning towards option 3 or 4 unless there is a better alternative

I recall load balancer names and target groups have a lower max of 32 as well.

I believe the length limit does not do a naive truncation.

https://github.com/cloudposse/terraform-null-label/blob/main/main.tf#L166

It uses a hashing strategy

Hmm interesting, but even so, I think it’s less than ideal

Haha, but the ideal is capped at 64 characters

yes, i wish amazon would lift the restrictions

maybe then aliasing account names and aliasing the eks cluster names may be a viable solution via go templating

How would another naming convention give you N dimensions of variation, and somehow not hit the 64 character limit, and if such a convention exists, why is it not possible with null label?

Atmos could still use full length names

The truncation happens in provisioning, so users don’t use the hashed ids

The users use the iam role names which need to be codified in argocd later on

If the ~users~oles are trimmed with a hashing strategy then the names would be less predictable by humans

What about making an exception for those and using a uuid()?

or more precisely https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/uuid

currently Atmos supports the Gomplate datasources https://docs.gomplate.ca/datasources/

what you are saying is to have remote-state in Atmos templates, e.g.:

Yes that would be nifty!

we discussed that and will implement the AtmosTerraformOutput function (no ETA yet)

Ah I figured it out, kind of. I can set privileged = false and then it will use the role_arn in the terraform_remote_state resource to get the information properly from the s3 bucket…

I did this a bit backwards. I reused an existing s3 bucket for the state, provisioned aws-teams using an Admin SSO role with role_arn: null for both s3 and remote_state_backend .

Now I have provisioned the tfstate-backend which gives me IAM roles to assume the appropriate tfstate role cross-account.

But now when I run AWS_PROFILE=admin-identity atmos terraform apply aws-teams --stack gbl-identity, it throws an error because the Admin SSO role cannot update the bucket directly.

I thought maybe the tfstate-backend would have a bucket policy to allow the PrincipalArn of my SSO permission set Admin to access it directly (similar to SuperAdmin) but that doesn’t seem to be the case.

Im at a bit of a loss.

1. On one hand, I want to update the s3 tfstate bucket policy to allow my SuperAdmin-esque Admin role in identity
2. and on the other hand I want to set privileged = false for all the remote state in aws-teams (and other root components) Both options seem incorrect

Final error since s3 bucket is not in the identity account

⨠ AWS_PROFILE=admin-identity atmos terraform plan aws-teams --stack gbl-identity

Initializing the backend...
Initializing modules...
╷
│ Error: Failed to get existing workspaces: Unable to list objects in S3 bucket

I did see the following here

For convenience, the component automatically grants access to the backend to the user deploying it. This is helpful because it allows that user, presumably SuperAdmin, to deploy the normal components that expect the user does not have direct access to Terraform state, without requiring custom configuration. However, you may want to explicitly grant SuperAdmin access to the backend in the allowed_principal_arns configuration, to ensure that SuperAdmin can always access the backend, even if the component is later updated by the root-admin role.

and since I’m using SSO, I do have this set in the backend

            allowed_permission_sets:
              identity: ["Admin"]

which allows SSO Admin from identity to assume it… but i still get Error loading state error

Here is the tfstate-backend component yaml

    tfstate-backend:
      vars:
        enabled: true
        name: tfstate
        enable_server_side_encryption: true
        force_destroy: false
        prevent_unencrypted_uploads: true
        access_roles:
          default:
            write_enabled: true
            allowed_roles:
              identity:
                - admin
                - dev
                - cicd
                - terraform
                - readonly
              devops-prod-01: ["admin"]
            denied_roles: {}
            allowed_permission_sets:
              identity: ["Admin"]
            denied_permission_sets: {}
            allowed_principal_arns: []
            denied_principal_arns: []

cc: @Andriy Knysh (Cloud Posse) if you have a sec

@RB not sure if you solved the issue, but take a look at https://github.com/cloudposse/terraform-aws-components/blob/main/modules/aws-teams/providers.tf

this component (if you are using it directly w/o any changes) will use the OrganizationAccountAccessRole role and needs to be provisioned as SuperAdmin

@Jeremy G (Cloud Posse) maybe you can provide more info

Oh interesting, thank you. How is the OrganizationAccountAccessRole created? Is it created in the same account as the tfstate bucket?

If i deploy aws-teams in identity, how does it get access to the tfstate bucket if the tfstate bucket is in the corp account (for example)?

So far the only way I’ve been able to connect it is by updating the tfstate bucket policy manually (since the tfstate-backend module doesn’t support adding additional principals directly)

regarding OrganizationAccountAccessRole

https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_access.html

he role is created in all accounts automatically by AWS, but only if the account is part of Org

if the account is not part of Org, or just invited to the Org, you need to create that role

I see this role exists. However it looks like the trust role needs to be updated in order to assume it per account.

Since my root tfstate is not in the root account, that’s probably the “root” issue. The remote state error is thrown when I use my superuser user (admin sso) in root to deploy identity and my tfstate-backend is in corp (because there is a rule to have fewer root resources and users).

I suppose then the need to update the bucket policy for the tfstate backend is a must or id have to create a new s3 tfstate bucket for root.