#atmos (2024-06)

So the irony there is these were actually developed for a customer using them primarily on Azure

They support Azure, but our docs might be lacking.

Any PRs improving that would be welcome.

@Matt Calhoun do we have any azure docs that weren’t published?

Oh really? Ha! Looking at the action.yml I don’t see a way to bypass the AWS steps and instead authenticate with Azure… https://github.com/cloudposse/github-action-atmos-terraform-plan/blob/2313b46bbad5da8a93177cd6d443f565e229906e/action.yml#L169

Hrmmm

Maybe I’m looking at the wrong actions?

No, so I think that the story goes a little bit different. They are using custom workflows on their end that basically do with the actions do. These actions were developed afterwards to do the same thing, but as you noticed, had a AWS. I’m pretty sure if you look at the plan file Storage action you will see specific logic for Azure. That action they are using.

https://github.com/cloudposse/github-action-terraform-plan-storage

Here you can see the docs as a relates to this action for Azure

So what we need to do is do the same thing for these other actions. If you wouldn’t mind opening a PR for that, that would be amazing.

Also, we have a PR open that’s updating the docs on the website which are out of date

https://github.com/cloudposse/atmos/pull/606

I’ll check it out, thanks @Erik Osterman (Cloud Posse). We’ll PR any changes, of course. @Zack Ferguson

@Andriy Knysh (Cloud Posse)

@Yangci Ou you can update the stacks.name_pattern in atmos.yaml to make it the same as you did for Spacelift using settings.stack_name_pattern

https://atmos.tools/cli/configuration#stacks

(there is no setting for that in the stack manifests because Atmos needs to know the stack name pattern before being able to process the stack manifest files, to be able to find the stack and component in the stack when executing CLI commands) - that’s why it’s in atmos.yaml only

stacks.name_pattern in atmos.yaml is used in these cases:

• it’s the pattern for Atmos stack names

• it’s the pattern for Terraform workspaces (by default, but you can override TF workspaces as well if needed, see https://atmos.tools/core-concepts/components/terraform-workspaces#terraform-workspace-override-in-atmos)

Hey @Andriy Knysh (Cloud Posse) - I did take a look at the name_pattern in atmos.yamlbut that sets it for the entire project. I’m more looking for what the TF workspace override is doing, but that’s doing it for the individual component

That’s what I want, but I’d like it for that particular entire .yaml stack

At the same time, the name_pattern for this project is already set, it’s more of just these couple of components* in this .yaml stack file that needs overriding

I’m thinking of something that does the same thing as terraform_workspace_pattern in that docs/pic, but on the terraform.settings level instead of the components.terraform.resource123.metadata level

currently, TF workspace can be overridden in the metadata section per component (not globally and not per file) (but it can be added to Atmos)

@Yangci Ou just currious, why do you want to override the TF workspaces for all components in a stack file, but don’t overide the Atmos stack name ?

@Andriy Knysh (Cloud Posse) it’s because we already have a name_pattern set for the stacks in the atmos.yaml. For all stacks and the components in them, all of the name patterns are as consistent between the workspace name to the Spacelift stack names.

But our situation calls for a special case where we’re creating a new .yaml file for a Atmos stack, but we want all the component that live in that to have a different Spacelift stack name, which in turns means that we want that different workspace name as well (hence overriding the TF workspace)

so essentially: these components are in the same Atmos stack as others, but we want different Spacelift name (which is implemented per file via stack_name_pattern) and looking to do the same for the TF workspace (which you mentioned only done per component)

@Yangci Ou i understand what you want to do. While currently you can override the TF workspace per component only (since metadata section is not inherited), we’ll consider adding a global settings to be able to do it on a set of components (we improve/update Atmos often when seeing real-life use-cases like this one)

having said that, I’d like to review with you what you are doing with overriding the Spacelift stack names for the components in a stack manifest file. I’ll put a wall of text below for you to better understand the difference b/w/ an Atmos stack (logical construct) and a stack manifest file (physical file where the components for the stack are defined)

let’s say you have an Atmos stack plat-use2-prod (the stack name is defined by the context variables tenant, environment and stage, and by the stacks.name_pattern in atmos.yaml (so Atmos knows which context variables are part of the stack name and in which order)

the components in the stack plat-use2-prod can be defined inline or imported in a single manifest file, e.g. orgs/acme/plat/prod/us-est-2.yaml - it’s called a top-level stack manifest

or the logical stack plat-use2-prod can be split into multiple top-level stack manifest files. e.g.

orgs/acme/plat/prod/us-east-2.yaml orgs/acme/plat/prod/us-east-2-extras.yaml

a set of components for the Atmos stack plat-use2-prod are defined (inline or imported) in orgs/acme/plat/prod/us-east-2.yaml manifest file, while another set of components for the same Atmos stack plat-use2-prod are defined (inline or via imports) in the second stack manifest file orgs/acme/plat/prod/us-east-2-extras.yaml

this pattern is called “Partial Stack Configuration” https://atmos.tools/design-patterns/partial-stack-configuration

while it’s a powerful pattern to use when you want to split your stack manifest files into smaller parts to manage them easily, you need to pay attention to the following:

• Regardless of how many stack manifest files you have, it’s still the same (logical) Atmos stack plat-use2-prod (just split into a few physical files)

• Any global config you defined in any of the files and any of the imported files will be applied to ALL components in the Atmos stack plat-use2-prod, not only to the components in one manifest file. For example, if you define something like this inline or via imports

      settings:
        spacelift:
          workspace_enabled: true
          # `stack_name_pattern` overrides Spacelift stack names
          stack_name_pattern: "{tenant}-{environment}-{stage}-{component}"
   

the settings.spacelift section is now a global section for the entire Atmos stack plat-use2-prod, and all components defined in the two stack manifest files orgs/acme/plat/prod/us-east-2.yaml and ``orgs/acme/plat/prod/us-east-2-extras.yaml will get that global config and the stack_name_pattern` will be applied to all of them

if you need to make sure that you provide or override some config just for a specific stack manifes file, Atmos supports the overrides section https://atmos.tools/design-patterns/component-overrides

it allows you to create a new separate scope per FILE (not per Atmos stack), and only the components in that file (defined inline or imported) will get the config defined in the overrides section. The other stack manifest files (even for the same logical stack) will not be affected. On the other hand, any config specific to a component (added into components.terraform.<my-component> section) will be applied only to that component in the stack and will not affect anything else (as we can currently do with the metadata section)

@Yangci Ou let me know if it makes sense and describes the use-case you want to implement

Hey Andriy! Just got around to reading this, yes this 100% makes sense and was a good explanation of the use-cases. Super helpful on the clear distinction of an Atmos logical stack and the physical stack manifest files. The partial stack configuration use case is basically what’s being implemented here with the idea of overrides.

Thanks for taking the time to provide that detailed and thorough explanation, reading up more examples on the docs for those links you provided and the organization/structures of those components looks very effective and clean

thanks. Having said that, we’ll consider adding the ability to override terraform workspace for a group of components, not only for one component at a time

@Andriy Knysh (Cloud Posse)

thanks @Stephan Helas , we’ll add it to Atmos

Big thumbs up on this! Saw the recent updates on atmos validate and I was looking into using it in our CI/CD pipeline as a check - was looking at the results and saw bunch of errors for the invalid assume_role because of the manifest

Fixed in

https://sweetops.slack.com/archives/C031919U8A0/p1717877618576239

What do you folks think

Are you just proxying the module as a component? then there’s another way.

Kind of. I think this approach could be used for any component even a component that composes multiple modules

What is the other way?

This is what I was thinking of https://atmos.tools/core-concepts/components/vendoring/#vendoring-modules-as-components

even a component that composes multiple modules Inevitably, users want “glue” to compose stateful root modules from multiple child modules. Do you think there’s a way without something more HCL based like what terragrunt and terramate do?

I want to be cognizant of reinventing HCL in YAML (not the end goal)

decode_tfvars and then pass it into modules would prevent having to define each var in each componen I have a mental block for how this would work. This allows for dynamic loading of tfvar files, which is nice, but the values still need to be passed. Also, if done at the component level, this method bypasses variable validation, which is very helpful for users. It’s a good thing to declare the variables, because it’s a documented interface.

without it, it’s like type any and that’s anything but helpful to understand what is accepted.

Many times the module inputs are copied and pasted into the component variables.tf so if the component uses 3 modules, each of their vars includng types, descriptions, validations etc are copied over. This also means if something changes in the module, that introduces drift, and so the component also needs to be updated.

We can side step this by allowing vars to pass through from yaml to tfvars with atmos and then decode using decode_tfvars. That should avoid having to copy and paste the module vars into component vars and reduce the redundant validation checks in the component and defer that to the upstream (or local) modules

Or maybe I’m mistaken here. I haven’t tested it out. It seems promising tho and may allow less module to component variable drift

So I get the gist of what you’re saying but the function you’re referring to is more like the YAML decode method so I don’t see the end solution coming together. The fact that function exist is no different than any of the other decoding method such as for JSON or YAML. Am I missing something?

If you could mock some thing either through YAML or something else that might help me connect the dots.

I’m on a flight at the moment but i will mock something up this week to illustrate it

@Andriy Knysh (Cloud Posse) do you have any clever ideas on how to have like a prioritized output of components?

We’re implementing ordered dependencies with with github actions, but due to technical limitations of GitHub matrixes, jobs, and concurrency groups, there’s no way to do it without also introducing a GitHub App.

We thought that may be the case. Are you able to share your timeline? Need any help with that?

< 1 month to working MVP.

as Erik mentioned, this is a multi-phase process. First step we added in this PR https://github.com/cloudposse/atmos/pull/616 - atmos describe affected --include-dependents=true. Second step is to use that functionality in GitHub Actions and GH App

Thanks @Andriy Knysh (Cloud Posse). In Azure, management of resource dependencies seems more fundamental - so hopefully this approach solves the problem. For now, I think I’ll create the resource group within the component, and adjust the component input vars to accept multiple things to create. I.e. the storage component creates a storage resource group, and then provisions n storage accounts within it. And if another storage resource group is required, have another storage component in the stack with a different config.

i’ve generally organized my components into their own resource groups. That kinda logically makes sense too but YMMV

Thanks Matt!

It should work. To our understand it just doesn’t work with local state backends.

Also, due to popular demand (and a change of heart), we are implementing the ability to directly pass state (terraform outputs) using atmos without relying on terraform data sources. This should be released soonish < 2 weeks.

@Amit the remote-state module is not configured to use the GCP backend, just (but it can be updated to support it). https://github.com/cloudposse/terraform-yaml-stack-config/blob/main/modules/remote-state/data-source.tf

what you can do is fork the repo, add the GCP backend to the map, use your branch as the source in your terraform code, test it, then open a PR in the Cloud Posse repo. This would be the fastest way to test it

Thanks, I will do it

@Amit you can now try this instead: https://sweetops.slack.com/archives/C031919U8A0/p1718495222745629

The indenting is weird. Not sure if that’s just a formatting issue. Have you run atmos validate? Also make sure you have defined a schema.

its just written from my head

the problem is, that i can only define values, not objects

There’s nothing special you need to do to pass the parammeters for an object. It works the same as a map.

Can you elaborate on what in particular is not working? in terraform? is the Deep merging not working as you expected?

so, this is my resource in the component

resource "hcloud_ssh_key" "this" {
  for_each = var.hc_ssh_keys

  name       = each.value.name
  public_key = each.value.filename
  labels     = var.tags
}

this is my stack

components:
  terraform:
    server:
      metadata:
        component: '{{ .settings.version }}'
        inherits:
          - 'server/_defaults'
      vars:
        hc_ssh_keys:
          shelas:
            name: [email protected]
            filename: '¨/.ssh/id_xxxxx.pub'

plan looks like this

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create

Terraform will perform the following actions:

  # hcloud_ssh_key.this["shelas"] will be created
  + resource "hcloud_ssh_key" "this" {
      + fingerprint = (known after apply)
      + id          = (known after apply)
      + labels      = {
          + "Managed-By"        = "Terraform"
          + "foo"               = "bar"
        }
      + name        = "[email protected]"
      + public_key  = "¨/.ssh/id_xxxxx.pub"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

but, if i change it to .settings:

settings:
  stage: test
  hc_ssh_keys:
    shelas:
      name: [email protected]
      filename: '¨/.ssh/id_xxxxx.pub'

components:
  terraform:
    server:
      metadata:
        component: '{{ .settings.version }}'
        inherits:
          - 'server/_defaults'
      vars:
        hc_ssh_keys: '{{ .settings.hc_ssh_keys}}'

i’l get this error:

You can apply this plan to save these new output values to the Terraform state, without changing any real
infrastructure.
╷
│ Error: Invalid value for input variable
│
│   on isa-zoom-test-server-server.terraform.tfvars.json line 3:
│    3:    "hc_ssh_keys": "map[shelas:map[filename:¨/.ssh/id_isa_hetzner.pub name:[email protected]]]",
│
│ The given value is not suitable for var.hc_ssh_keys declared at variables.tf:7,1-23: map of object required.
╵

It appears to me that settings is being confused with variables.

if i look into tfvars.json, in the first example, i got this:

{
   "hc_ssh_keys": {
   "shelas": {
   "filename": "¨/.ssh/id_isa_hetzner.pub",
   "name": "[email protected]"
}

but in the second, i’ve got this:

{
   "datacenter": "fsn1-dc14",
   "hc_ssh_keys": "map[shelas:map[filename:¨/.ssh/id_isa_hetzner.pub name:[email protected]]]",
   "namespace": "default",

Settings block relates to integrations. Think “integration settings” for spacelift.

ok

Settings for GitHub actions, settings for Atlantis

But I can probably guide you to what you want to accomplish. Let’s take a step back. What is the goal you were hoping to achieve by putting the settings at the top?

i try to configure a map containing ssh keys to be rolled out by the server component. i’ll then reference some of those keys to some server instances by using the name.

this map should be defined “further up” in my stack

I think what you’re looking for for then is the multiple inheritance pattern

Are you familiar with the design patterns page on Atmos docs?

yeah, more or less

the whole component looks like this

    server:
      metadata:
        component: '{{ .settings.version }}'
        inherits:
          - 'server/_defaults'
      vars:
        network_component: network
        datacenter: '{{ .settings.datacenter }}'
        network_zone: '{{ .settings.network_zone }}'
        server_type: '{{ .settings.server_type }}'
        hc_ssh_keys:
          shelas:
            name: [email protected]
            filename: '¨/.ssh/id_isa_hetzner.pub'
        servers:
          web1:
            image: 'debian-12'
            ssh_keys:
              - [email protected]
            public_ipv4_enabled: true
            public_primary_ip: false

i’d like to provision one or more ssh keys, and then attach one or more to a server

in that pattern, you’ll basically describe two abstract classes of component configurations. In each one of those you’ll put your vars. Then using inherit your pull from one of those. Overall, we discourage using templating as much as possible, when template free approaches exist

but i’d like to have the list of ssh keys outside the component

You’re taking a templated approach in the examples provided

ok. i think, i don’t understand. is templating good or bad?

With multiple inheritance, you can have the values live outside of the direct component configuration. For example, we almost never use templating in our reference architecture. To say that it’s bad, they’re definitely times when it makes sense. The simpler the templating, such as templating values like you are doing is the lesser evil.

@Stephan Helas Go templates can’t generate maops of objects by default, you can use a few techniques:

1. use range inside range (see this for example https://docs.dynatrace.com/docs/manage/configuration-as-code/monaco/guides/configuration-as-code-advanced-use-case)

1. Use Sprig toJson function (since JSON is a subset of YAML) https://masterminds.github.io/sprig/defaults.html

ah, i understand. go template just creates some string here, right?

Yep!

can you point me to the design pattern to change my design as you described?

Atmos essentially takes a two phased approach. This is typical with tools that use templates. The first treats the file as plain text unstructured. Then it processes, the templates, which results in a new file. It loads that resultant file as YAML, and then proceeds to the next file.

or a third way, use gomplate data.ToYAML https://docs.gomplate.ca/functions/data/#datatoyaml

But Andriy can’t just be done with multiple inheritance as well?

Then you have no templates and it’s a lot easier than string manipulation

Also, we have the new setting to control how lists are deep merged

can’t just be done with multiple inheritance as well?

yes, it can be refactored. All the above methods (range, toJson, data.ToYAML) are if you don’t want to refactor and want just use a map of objects

the data.toYaml generate this:

 '{{ data.ToYAML .settings.hc_ssh_keys}}'

{
   "datacenter": "fsn1-dc14",
   "hc_ssh_keys": "shelas: filename: ¨/.ssh/id_xxxx.pub name: [email protected] ",
   "namespace": "default",

yes, this is b/c Go templates are string-based, they don’t know anything about the shape of your data type

in Helm, they introduced toYAML and indent functions for that

toJson and data.ToYAML return a string representation of the data

you can always use range

{{- range ....

Sprig has indent as well https://masterminds.github.io/sprig/strings.html

you have to experiment with using toJson/toYAML and indent

(or just use range)

i tried, but i don’t get the syntax right

        server_type: '{{ .settings.server_type }}'
        hc_ssh_keys:
        '{{- range $i, $e := .itemList}}'
        '{{$i}}. {{$e.name}} is priced at {{$e.price}}. {{$e.description}}'
        '{{- end}}'
        servers:
          web1:

will not run

yaml: line 48: could not find expected ‘:’

review YAML multi-line syntax https://yaml-multiline.info/

all of this is jumping through many hoops. Since Go templates are string-based, they don’t know anything about the shape of the data. That’s why in YAML files with Go templates, you have to “create” the correct “shape”.

Note that if you changed the Terraform var type to string, then in the template you would just use toJson, and then in the TF code you would just do

locals {
  hc_ssh_keys = jsondecode(var.hc_ssh_keys)
}

oh yeah, that sounds good

toJson leaves me with an empty string

data.ToYAML worked for you, you can use it, and then in TF use yamlencode

yamldecode

hm, ok. i’ll tinker a bit with this.

oh, this seems to do the trick

         hc_ssh_keys: '{{ data.ToYAML .settings.hc_ssh_keys | toJson }}'

and then yamldecode. not really elegant solution

so, for reference, this works:

encode to string in atmos

hc_ssh_keys: '{{ data.ToYAML .settings.hc_ssh_keys | toJson }}'

decode in terraform

locals {
hc_ssh_keys = yamldecode(jsondecode(var.hc_ssh_keys))
}

does it work w/o | toJson?

@Stuart Martin yes, you can define those context variables in other Atmos sections, e.g. settings (in which case they will not be variables anymore)

https://atmos.tools/cli/configuration/#stacks

Look for the description of stacks.name_pattern and stacks.name_template

in short, you can define those in e.g. settimgs.context (context is just an example here, you can name it as you want)

settings:
  context:
    environment: usw2
    stage: dev
    tenant: plat

and in atmos.yaml define stacks.name_template like this:

you can use any names for the context, e.g. if you want to use account instead of stage , you do:

(settings is a free-form map, you can put anything in it, including other maps like context)

Amazing! This is exactly what I was looking/hoping for

I’ll give it a go, thanks so much

So to be clear, we are using Atmos to deploy Argo CD itself, and Argo CD itself is defined as a terraform component. Argo CD does not drive Atmos.

@Andrew Ochsner thanks for the question. We constantly improve Atmos from the users feedback like yours. We’ll think about “imports for components”. Having said that, you can achieve what you want by using base components, inheritance and

metadata:
  inherits:
    - base-component-1
    - base-component-2

here are a few docs that can help you:

ah that’s a good call

i mean i guess by definiton they’ aren’t part of the same stack just the same yaml file… because environment makes up part of my stack name

can you show some config that you want to create, we can prob point you into the right direction

well i keep going in circles in my own head… wehtehr it makes more sense for the “management group stack” to contain “subscriptions” or for “subscription stacks” to contain both the subscription itself and the resoures in that subscription….

please show your YAML config (you can DM me). I’m not very familiar with Azure, so will have to look at the code to understand the patterns

yep can do one sec

welp actualy that might be harder than i think… i’ll tryo t get back toyou today

ok here’s an example: mg-vdi:

import: #management group is global region global environment.. 
  - mixins/region/global
  - mixins/environment/global
  - catalog/subscription/default
components:
  terraform:
    subscription/vdi:
      metadata:
        component: subscription
        inherits:
          - subscription/default
      vars:
        name_base: vdi
        # the subscription needs the mixins/environment/eng & mixins/region/eastus values
    subscription/poc:
      metadata:
        component: subscription
        inherits:
          - subscription/default
      vars:
        name_base: poc
        # the subscription needs the mixins/environment/test & mixins/region/eastus values

i can only think of copy/pasting the values int he mixins in teh vars of the subscription comopnents (can have base components for the different options but still approach being the same)… is there any way not to duplicate taht sort of thing

i haven’t noodled around if templating would solve this but generally i know that adds another layer of complexity

what context variables define your stacks (stack name pattern), and where do you have them?

from the code above, it looks like you want to have components belonging to two diff top-level stacks but defined in one stack manifest

tenant-namespace-environment-stage… the environment == region (and defined int he mixins) & stage == environment (and defined int he mixins)

the two components belong to two diff top-level stacks?

yeah, so whats unique here is there are 2 “types” of stacks… management group stacks and subscription stacks…. and the twist is that management group stacks contain the subscriptions that get provisioned in those managementn groups…

no… 2 components (subscriptions) belong to 1 top level stack (vdi management group in this case)…

i think short term, i can create component specific mixins to override the top level management group values… again i think this only will happen in subscription components

ok, you should not do it like that. You should have manifests for your top-level stacks, then import the required mixins into them, then import the defaults for the components

this way, you don’t have to specify the context variables (which define your stack names) in the components. They will get them automatically from the corresponding imports in the top-level stack manifests

what’s’ the diff b/w mixins/environment/global , mixins/environment/eng and mixins/environment/test ?

yeah so effectively have component specific mixins for region & environment… that makes sense… just the values are the same as the top level mixins, just scoped to the component

the pattern here is to define your stacks by importing the global settings (e.g. from the mixins). Then in each stack, import the components that you want to provision in the stacks (and you can override some vars inline). The components should NOT know where they are provisioned, they get the context variables from the stacks

mixins/environment/eng.yaml

vars:
  stage: eng
  syslevel: eng
  tags:
    SysLevel: eng

so i’d also need to create catalog/subscription/mixins/environment/eng.yaml

components:
  terraform:
    subscription:
      vars:
        syslevel: eng
        tags:
          SysLevel: eng

but the context of the management group != the context of the subscription we are provisioning here… i don’t want to override the stage/environment of hte mgmt group w/ the imports of the subscription (which is what i’ve got now that doesn’t work)

i think short term, i can create component specific mixins to override the top level management group values… again i think this only will happen in subscription components

this is correct

my comment “ok, you should not do it like that.” was related to the previous messages

note that if you import global variables, they will affect all components in the stack

i think this is the best way to do it

components:
  terraform:
    subscription:
      matadata:
        type: abstract
      vars:
        syslevel: eng
        tags:
          SysLevel: eng

even if you repeat the vars there

a better way would be to create these components in the corresponding stacks

    subscription/vdi:
      metadata:
        component: subscription
        inherits:
          - subscription/default
      vars:
        name_base: vdi
        # the subscription needs the mixins/environment/eng & mixins/region/eastus values
    subscription/poc:
      metadata:
        component: subscription
        inherits:
          - subscription/default
      vars:
        name_base: poc
        # the subscription needs the mixins/environment/test & mixins/region/eastus values

and not override the context variables

they are separate components, just inheriting the default values from the common base component

as I described above, each component should belong to a specific too-level stack, we should not override the context variables for components in a stack

well, teh alternative approach would be to put the subscription component in the “subscription” stack… i played w/ that but it’s a bit messy because the backends are different then…

appreciate the back and forth btw

Something I noticed, in the examples the vendor.yaml file is looking like this:

  - component: eks
    source: "github.com/cloudposse/terraform-aws-components.git//modules/eks/cluster?ref={{.Version}}"
    version: "1.463.0"
    targets:
    - "components/terraform/eks/cluster"
    included_paths:
    - "**/*.tf"
    # If the component's folder has the `modules` sub-folder, it needs to be explicitly defined
    - "**/modules/**"
    excluded_paths:
    - "**/providers.tf"
    # Tags can be used to vendor component that have the specific tags
    # `atmos vendor pull --tags networking,storage`
    # Refer to <https://atmos.tools/cli/commands/vendor/pull>
    tags:
    - eks

(a modification I’ve made for EKS) The providers file is excluded

Why is that, I see that iam_roles module is there?

@Dan Miller (Cloud Posse)

The terraform-aws-components repository is a collection of opinionated root modules. These are all designed around a set of opinionated choices we have made around designing a fully functional AWS architecture. As such, many of these components are intertwined with each other.

Primarily, [providers.tf](http://providers.tf) for almost every component relies on the account-map component. That component establishes the foundation for planning and applying Terraform across many accounts in the AWS Organization and much more. That includes the iam_roles submodule, which in particular selects the correct IAM role to use to plan/apply terraform in the given account

This is actually incredible, output sharing from component to component in YAML has been my dream feature since I started using Atmos

Sometimes components don’t expose all the parameters of the child module

This is not “by design”, it’s just for the original use case we implemented it for, it was not a requirement. If this turns out to be the case, just open a PR to add the missing variable.

If you share an example of your stack configurations it will be easier to assist

yea i looked back at the code and it doesnt look exposed which is what i kinda figured.

Note, when this happens you should get a warning from terraform about a variable being passed that was not delcared

That would not be intentional

@Aleksandr Rozhkov templating was implemented recently (in the past few months). Since atmos generate backends is not used often, it was not tested with templating. We’ll look into that

(@Aleksandr Rozhkov I’m curious, are you using this strategy to integrate with another system?)

I make internal POC to choose toolset for TF states orchestration. using backends instead of backend per component reduces my initial steps in workflow.

and templating needed - to delimit state objects in the same storage.

vanila TF with scoped states + some orchestration of states(dependancy) + GH PR-driven UI(GH actions )

the fix will be in the next release

@Erik Osterman (Cloud Posse) POC: check some in-house implementation and compare with possible SaaS usage.

no “another systems” except GH Actions

@Aleksandr Rozhkov the Go template processing in atmos generate backends is fixed in https://github.com/cloudposse/atmos/releases/tag/v1.82.0

@Aleksandr Rozhkov were you able to validate?

@Andriy Knysh (Cloud Posse) Thanks. It works now.

@Erik Osterman (Cloud Posse) Was busy checking all cloudposse github actions usable for me. PS. We are multi-cloud with GCP as the main cloud and I use it for the backend. A lot of opinionated configurations.

It is important to realize that all you are doing is manipulating a text file using using templating, not YAML

It is similar to Helm and Helmfile

If you share your component configuration in the stack, I can provide an example.

Remember that YAML is a superset of JSON, so that should work for you

or use range

{{range atmos.Component("vpc", .stack).outputs.private_subnet}}
  - {{.}}
{{end}}

Thanks!

@prwnd9 did you get it working?

I was looking at this action to see

https://github.com/cloudposse/github-action-validate-codeowners

Is it possible

• to create a new check for codeowner approval validation

• then the gha for apply can check the other actions result ◦ if its green then it applies ◦ if its red, the apply should fail

I’m unsure if one gha can check the result of another gha

For example, if I only changed an s3 bucket in ue1-dev, i shouldn’t be able to deploy an eks cluster in uw2-prod

@RB are you asking about how to prevent users or machines from running apply on other components not related to the changes in a PR?

Yes!

so that is not related to the PR itself?

it’s related to the PR cause the apply is done on the PR itself, no ? other than drift detection

in a GH action? The apply will be done only on the affected components. The action should not attempt to apply not affected

is it what you are asking?

or you asking how to prevent it in general if the action tries to apply some other components (for any reason, e.g. misconfig or wrong code)?

So if we do atmos apply <component> -s <stack> and it’s of a component and stack outside of the PR’s changes, it will be applied, no ?

@Andriy Knysh (Cloud Posse)

@RB Atmos (the CLI) does not know anything about PRs, CI/CD and other external systems. It means you can always execute any command, e.g. atmos terraform apply <component> -s <stack> on the command line. Maybe I’m not understanding your question or use-cases completely, please provide the details

I meant that can be added to a comment in the pr to apply other stacks in order to target specific stacks. Id like the comment on the pr to only be able to apply the affected stacks

w/o implementation details, the idea would be to do the following in the PR (or in the comment handling action)

1. Execute `atmos describe affected` to get a list of affected components in the stacks (<https://atmos.tools/cli/commands/describe/affected/>)
2. Detect the component and stack from the comment in the command that the user wants to execute
3. If the provided component and stack is not in the list of the affexcted components/stacks, show an error to the user 

Great question - you would update the workflow it self

e.g. https://github.com/im-open/is-actor-team-member

I just did a quick google for that action, haven’t used it

You would put that in the drift remediation workflow

Thanks for sending, I’ll check it out

out of curiosity, is this problem solved already for clients ?

There’s no one way to solve it.

Are you on github enterprise?

i believe so

With GHE, you can use environment protection rules.

You would add this to the workflow, then use protection rules. Those can be as advanced so you would like.

https://docs.github.com/en/[email protected]/actions/deployment/targeting-different-environments/using-environments-for-deployment#cust[…]rules

environment: something

This is the most native way in GitHub to control who can do what.

We are leaning mostly on leveraging GitHub for as much as we can, which is why right now, we don’t prioritize a custom solution in our GitHub Actions, when GitHub already supports it.

Oh very cool! I’ll check that out, ty

it should be the same speed as executing terraform output ... command (which is not super fast). Atmos uses https://github.com/hashicorp/terraform-exec to execute terraform output programmatically, but the lib uses the same code as the regular terraform

Ah okay, thanks

Keep us posted.

But @Andriy Knysh (Cloud Posse) is right, that this is expected, since it’s basically a subprocess. Right now, we don’t do any caching.

It would be interesting to learn how you use it, and if you are retrieving exactly the same component outputs in multiple places

we can add caching for cases like that

Caching would be nice, for me - I’m just passing things like arn roles or instance profiles from an iam component to something else

Cool, this is a use-case we anticipated would be simplified using atmos.Component, so glad you are using it. Makes sense it would benefit from caching. I think @Andriy Knysh (Cloud Posse) will be adding it related to other work he is doing to optimize data sources.

Awesome! I’ve been using it on the GCP side for creating multiple cloud build triggers and a repo connection — passing the repo connection id to multiple build trigger components, as well as a few gcp project id’s between components. I don’t think the remote_state module you guys have supports GCP yet, so is great to have this new option

caching will be in the next release, ETA the end of the week