#atmos (2024-05)

Great question! I think we could/should add some guidance to our docs.

Let’ me take a stab at that now, and present some options.

First, make sure you’re familiar with the .gitattributes file which let’s you flag certain files/paths as auto generated, which collapses them in your Pull Requests, and reduces the eye strain.

Option 1: By default, Cloud Posse (in our engagements and refarch), vendor the everything in to the repositories.

Pros

• an immutable record of components / less reliance on the repo repositories

• super easy to test changes without a “fork bomb” and ensuring “PR storm” as you update multiple repos

• super easy to diverge when you want to

• super easy to detect changes and what’s affected

• much faster than cloning all dependencies

• super easy to “grep” (search) the repo to find where something is defined

• No need to dereference a bunch of URLs just to find where something is defined

• Easier for newcomers to understand what is going on

Cons

• Reviewing PRs containing tons of vendored files sucks

• …? I struggle to see them

Option 2: Vendoring components (or anything for that fact, which is supported by atmos) can be done “Just in time”, more or less like terraform init for provider and modules.

Pros:

• only things that change or are different are in the local repo

• PRs don’t contain a bunch of duplicated files

• It’smore “DRY” (but I’d argue this is not really any more DRY than committing them. Not in practice because vendoring is completely automated) Cons

• It’s slower to run, because everything must first be downloaded

• It’s not immutable. Remote refs can change, including tags

• Remote sources can go away, or suffer transient errors

• It’s harder to understand what something is doing, when you have to dereference dozens of URLs to look at the code

• Cannot just do a “code search” (grep) through the repo to see where something is defined

• In order to determine what is affected, you have to clone everything which is slower

• If you want to test out some change, you have to fork it and create branch with your changes, then update your pinning

• If you want to diverge, you also have to fork it, or vendor it in locally

Option 3: Hybrid - might make sense in some cirucmstances. Vendor & commit 3rd-party dependencies you do not control, and for everything else permit remote dependencies and vendor JIT.

I didn’t even give thought to someone making a cheeky ref update to a tag

Alright, this is supremely helpful, thanks a ton for the analysis. I’ll bring these points back to my team for discussion.

Great, would love to hear the feedback.

Also, not sure if this is just a coincidence, but check out this thread

https://sweetops.slack.com/archives/C031919U8A0/p1714484118739959

I really like the point of this making the configuration set immutable as well. We’re locked in with exactly what we have committed to the repository.

A lot of our “best practices” are inspired by the tools we’ve cut our teeth on. In this case, our recommendation is inspired by ArgoCD (~Flux) is the inspiration - at least the way we’ve implemented and used it.

Yeah, I was working with vendoring in different versions today to their own component path (component/1.2.1, component/1.2.2) and it took me a moment to realize this was changing the workspace prefix in the S3 bucket where the state for the stack was being stored.

it took me a moment to realize this was changing the workspace prefix in the S3 bucket where the state for the stack was being stored.

This is configurable

So now am vendoring in different versions to different environment paths to keep the component names the same as things are promoted. (1.2.1 => component/prod, 1.2.2 => component/dev)

This is a great way to support multiple concurrent versions, just make sure you configure the workspace key prefix in the atmos settings for the component.

So now am vendoring in different versions to different environment paths to keep the component names the same as things are promote This is another way. Think of them as release channels (e.g. alpha, beta, stable, etc)

Either way, its’ great to fix the component path in the state bucket, so you don’t encounter problems if you reorganize how you store components on disk.

So then my configs look sort of like,

components:
  terraform:
    egress_vpc/vpc:
      metadata:
        component: vendored/networking/vpc/dev
        inherits:
          - vpc/dev

    egress_vpc:
      metadata:
        component: vendored/networking/egress_vpc/dev
      settings:
        depends_on:
          1:
            component: egress_vpc/vpc
      vars:
        enabled: true
        vpc_configuration: egress_vpc/vpc

@Andriy Knysh (Cloud Posse) do we have docs on how to “pin” the component path in the state backend?

Note, I would personally invert the paths for release channels.

• vendored/dev/networking….

• vendored/prod/…. Since over time, some components might get deprecated and removed, or others never progressing past dev.

Yeah, that’s a very good point and something we’re still working through.

We also want to get to the point where we have “release waves”…..so, release a change to a “canary” group, and then carry out groupA, groupB, groupC, etc….

Which, version numbers would honestly help with that a bit more. How could I pin the workspace key prefix for a client if I did have the component version in the path?

Ahh, looks like towards the bottom of the page here: https://atmos.tools/quick-start/configure-terraform-backend/

Yea, I think our example could be better….

But this:

I can have my cake and eat it too. Delicious.

Yeah, just hugely beneficial, thank you so much for the help.

Our pleasure!

I have the remote-state stuff working that you were able to guide me through the other week…many thanks as well there, I think that’s really going to help us level up our IaC game.

Please feel free to add any thoughts to this https://github.com/cloudposse/atmos/issues/598

Yes, exactly one of the use cases for custom sub commands. This way you can call it with atmos to make it feel more integrated, documented (e.g. with atmos help). You can then automate it with atmos workflows. Alternatively you can skip the sub command and only use a workflow too. Up to you.

You can also access the stack configuration from custom commands

any examples on how to access stack configuration through custom commands?

Have you seen this? https://atmos.tools/core-concepts/custom-commands/#show-component-info

Instead of running echo, just run your python script instead.

@Gabriela Campana (Cloud Posse) we need a task to add example command invocation for each example custom command. cc @Andriy Knysh (Cloud Posse)

E.g. the docs only show how to define it, not how to call it. Of course, it can be inferred, but “as a new user I want to quickly see how to run the command because it will help me connect the dots”

@Kubhera we’ll improve the docs. For an example on “any examples on how to access stack configuration through custom commands?”, please see this custom command (as Erik mentioned above):

or this one https://atmos.tools/core-concepts/custom-commands/#set-eks-cluster

DEV-2129 : Add example command invocation for each example custom command

Or would gomplate, if a random function exists, generate a new random string upon each atmos run?

I suppose uuid does exist here

https://docs.gomplate.ca/functions/uuid/

Unfortunately cannot effectively use the uuid function in gomplate for this because it will cause permadrift

Just to confirm, aesthetically the automatic checksum hash is not option you are considering? …because it solves exactly this problem with the bucket length, but will chop and checksum the tail end

Sorry, I didn’t read carefully. You are not asking about controlling the length.

Regarding the post you linked, we discussed this on office hours. @matt mentioned that Jeff Barr in response to the post, said AWS is not going to charge for unauthorized requests and they are making changes for that. To be clear, they refunded the person who made the unfortunate discovery, and are making more permanent fixes to prevent this in the future.

Oh i didn’t realize that jeff bar responded with that. That’s great news. Thanks Erik. Then i suppose it’s a non issue once they makes changes for it

Is there a doc available that points to jeff barrs response? That may calm some nerves

Found it

https://twitter.com/jeffbarr/status/1785386554372042890

https://youtu.be/QNcSZwHgBEg?t=1441

Haha, @Kubhera my ears are ringing from the number of times this is coming up in questions.

TL;DR: we don’t support this today

Design-wise, we’ve really wanted to stick with vanilla Terraform as much as possible. Our argument is that the more we stick into atmos that depend on Terraform behaviors, the more end-users are “vendor locked” into using it; also, we don’t want to re-invent “terraform” inside of Atmos and YAML. We want to use Terraform for what it’s good at, and atmos for where it’s lacking: configuration. However, that’s a self-imposed constraint, and it seems like one users are not so concerned about and a feature frequently asked for, albeit for different use cases. This channel has many such requests.

So we’ve been discussing a way of doing that. It’s tricky because there are a dozen types of backends. We’ll not want to reimplement the functionality in atmos to read the raw state but instead make terraform output a datasource in atmos.

We’ve recently introduced datasources into atmos, so this will slot in nicely there. Also, take a look at the other data sources, and see if one will work for you.

https://atmos.tools/core-concepts/stacks/templating#configuring-templating-in-atmos-stack-manifests

Here are the data sources we support today: https://docs.gomplate.ca/datasources/

So if you write you outputs to SSM, for example, atmos can read those.

Alternatively, since you’re using a workflow, you can do this workaround.
File file Files can be read in any of the supported formats, including by piping through standard input (Stdin). Directories are also supported.

So in one of your steps, save the terraform output as a json blob to a file.

Then define a datasource to read that file. All the values in the file will be available then in the stack configuration.

As always, atmos is a swiss army knife, so there’s away to do it. Just maybe not optimized yet for your use-case.

What about switching from terraform remote state to aws data sources instead for each component?

This way we don’t have to depend on the output of another component to deploy a dependent component.

We are planning to switch to using a configurable kv-store model, so that things work better in brownfield.

Using data sources assumes too much: a) that a data source exists for the type of look-up on all resources, and is not consistent across providers b) frequently don’t support returning all values of a given type or error if not found c) is complicated when what you need to look up is in multiple different accounts, d) is messy if you depend on resources cross-cloud. The “kv” store pattern, allows one to choose from any number of backends for configuration, such as dynamodb, s3, SSM, etc. We’re building modules for multiple mainstream clouds (as part of an engagement), so expect to see aws, artifactory, gcp, and azure kv-store implementations.

In a brownfield, that obligation would be to populate the “well known path” in whatever backend you choose for kv-store with the values you need. However, @RB since you are doing a lot of brownfield, I’d like to learn if you think this could work for you.

Hmmm a kv-store model would then require to host dynamodb/s3/ssm. It sounds like a generic cross-platform generic remote-state.

You would still have the dependency issue unfortunately where component X needs to be deployed (or redeployed) for component Y to get the correct information even if it’s simply adding an extra output to X whereas if using a data source, the extra output may already be available.

Since the remote-state uses the context which includes all the contextual tags, I’m not sure I fully understand why it would be difficult to use data sources.

For example, a vpc created by cloudposse’s component, should have the Namespace, Stage, Environment, Name, etc added to the vpc resource as well as the subnets. Using just those tags, a data source for the VPC and subnets can retrieve the vpc id, private subnets, and public subnets which is mostly why the remote state for the vpc component is used.

Just chiming in…the big difference here, IMO, is that in order to use a data source you have to be able to authenticate to the account where the resource is defined. So imagine a component where you need to read some piece of data (like CIDR subnet blocks) for all of the VPCs in your organization. Now you need a ton of providers for that component just to be able to read a list of all your CIDR blocks across accounts and regions. Compare that with the k/v store pattern, where you just read the data directly in the k/v store in your local account and region where you’re deploying the component.

Hi Matt. Interesting use-case and agreed, if you’re deploying in us-west-2 and need to retrieve a resource in another region like us-east-1, then creating lots of providers is not scalable. That does seem like an edge case compared to same-region/same-account resources.

However, you could have both implementations depending on the use-case

1. the kv-store for the use-case you described, where you have resources in different regions/accounts a. Pros i. works for cross-region/cross-account without additional providers b. Cons i. requires chaining applies for components that rely on other components’ outputs
2. the data source method for resources that are in the same region and account a. Pros i. works for same region, same account without additional providers ii. no need for chaining applies for components b. Cons i. does not work well with cross-region/cross-account as additional providers are needed It doesn’t have to be one or the other.

We can use (1) for those edge cases and (2) for the common case of retrieving items like VPCs, EKS, etc via data source.

Which components affect you the most with this, @RB ?

Pretty much anything that currently depends on the vpc component for now but im sure I’ll hit it more with other common remote states too like eks

Yea, maybe we could support that use-case for a finite number of common data sources, and support a specific type of look up method, for example by tags and resource type.

Thoughts on this?

Wouldn’t that be an abstract component?

It doesn’t have to be. It could be a non abstract or abstract component.

non abstract or real

# non-abstract or real
components:
  terraform:
    vpc:
      metadata:
        # atmos refuses to run commands on it because this is false
        enabled: false
      vars:
        # if metadata.enabled is true or omitted, this enabled flag will enable all terraform resources
        enabled: true

@RB this sounds like a good idea. Currently we can use metadata.type: abstract to make the component non-deployable and just a blueprint for derived components. But you are saying that you’d like to “disable” any component (abstract or real). There could def be use-cases for that. We’ll create a ticket

Yes please! Thank you very much!

@Gabriela Campana (Cloud Posse) please create a ticket to add metadata.enabled to Atmos manifests

DEV-2150 : Add metadata.enabled to Atmos manifests

Wondering if I should place this provider in the [main.tf](http://main.tf) of the module and pass in the values via variables, or if there is a better way to do this.

Hi @Justin

i have the same problem. my current workaround is to use sops in the component to decrypt the secrets. but you should be able to use other providers as well.

So, I think what you have should work provided,

a) you’re using lower case environment variables (since your example uses lower case) b) you have templating enabled c) you’re aware of this warning

https://atmos.tools/core-concepts/stacks/templating/#atmos-sections-supporting-go-templates

Oh, I didn’t look closely enough. Your syntax is wrong. Use Go Template syntax.

if your envs were upper case, you would need to retrieve it that way too

   client_secret: '{{ env "EXAMPLE_CLIENT_SECRET" }}'

Amazing, I will give this a shot, thank you.

Atmos supports Sprig and Gomplate functions in templates in stacks manifests. Both engines have specific syntax to get ENV variables

https://masterminds.github.io/sprig/os.html

https://docs.gomplate.ca/functions/env/

Also, make sure that templating is enabled in atmos.yaml

https://atmos.tools/core-concepts/stacks/templating#configuration

@Andriy Knysh (Cloud Posse) what do you think about importing the docs for these functions? Since they are not in our docs, they are not searchable or discoverable without knowing to go to another site

i think it’s a good idea, we need to improve our docs. How do we “import” it?

I suspect it does this because I think null-label convention used to be to always downcase, but I think that option is now configurable.

@Andriy Knysh (Cloud Posse)

@Stephan Helas i think it’s not related to the remote-state provider (it does not change the case on its own), but as Erik mentioned, it’s the null-label module that always downcase, in which case, if you are using Upper case for any of the context variables, it will not work with Atmos commands

https://github.com/cloudposse/terraform-null-label/blob/main/variables.tf#L193

you can set it to none and test. Let us know if it’s working

@Andriy Knysh (Cloud Posse) worth double checking

@Stephan Helas I would like to see your config to better understand what you are doing and the issues you are facing. You can DM me your repo (or part of it with the relevant config) and I’ll take a look and help you with any issues

Since stacks.name_template is a new feature, it could be some issues in Atmos, which we can def fix. It was tested (and is used now) in some configs, but we def did not spend much time on it as on the other Atmos features. Without seeing what you are doing, it’s difficult to tell if there is any issues or it’s just a misconfig

I’ll try to make a small demo. essential, what happens is this:

atmos.yaml

stacks:
  base_path: 'stacks'
  included_paths:
    - 'bootstrap/**/*'
    - 'KN/**/*'
  excluded_paths:
    - '**/_defaults.yaml'
  name_template: '{{.settings.tenant}}-{{.settings.instance}}-{{.settings.stage}}'

remote-state.tf

module "vpc" {

  source = "...."

  component = var.vpc_component
  context   = module.this.context
}

atmos output:

 Terraform has been successfully initialized!
module.wms_vpc.data.utils_component_config.config[0]: Reading...
module.base.data.aws_availability_zones.available: Reading...
module.base.data.aws_availability_zones.available: Read complete after 0s [id=eu-central-1]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: stack name pattern must be provided in 'stacks.name_pattern' CLI config or 'ATMOS_STACKS_NAME_PATTERN' ENV variable
│
│   with module.wms_vpc.data.utils_component_config.config[0],
│   on .terraform/modules/wms_vpc/remote-state/main.tf line 1, in data "utils_component_config" "config":
│    1: data "utils_component_config" "config" {
│

after changing remote state, it works :

module "vpc" {

  source = "..."

  component = var.wms_vpc_component
  context   = module.this.context

  env = {
    ATMOS_STACKS_NAME_PATTERN = "{tenant}-{environment}-{stage}"
  }

  tenant      = var.tags["atmos:tenant"]
  environment = var.tags["atmos:instance"]
  stage       = var.tags["atmos:stage"]

}

Aha, @Andriy Knysh (Cloud Posse) looks like the provider is out of date, or not supporting the name_template

the latest version of the utils provider supports uses the Atmos code that supports name_temoplate

https://github.com/cloudposse/terraform-provider-utils/releases/tag/1.22.0

make sure it’s downloaded by terraform

also please make sure that your atmos.yaml is in a location where both Atmos binary and the utils provider can find it, see https://atmos.tools/core-concepts/components/remote-state#caveats

if all is ok and still does not work, ping me, I’ll review your config

i did a terraform clean and added the provider version, but it did not work the modul version used is 1.22.

 Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...
- base in ../../../../../../poc/modules/wms-base
Downloading registry.terraform.io/cloudposse/label/null 0.25.0 for base.this...
- base.this in .terraform/modules/base.this
Downloading registry.terraform.io/cloudposse/label/null 0.25.0 for this...
- this in .terraform/modules/this
Downloading git::ssh://..../aws-atmos-modules.git?ref=v0.1.0 for wms_vpc...
- wms_vpc in .terraform/modules/wms_vpc/remote-state
Downloading registry.terraform.io/cloudposse/label/null 0.25.0 for wms_vpc.always...
- wms_vpc.always in .terraform/modules/wms_vpc.always

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding cloudposse/utils versions matching "!= 1.4.0, >= 1.7.1, ~> 1.22, < 2.0.0"...
- Finding hashicorp/local versions matching ">= 1.3.0, ~> 2.4"...
- Finding hashicorp/aws versions matching "~> 5.37"...
- Finding hashicorp/external versions matching ">= 2.0.0"...
- Installing cloudposse/utils v1.22.0...
- Installed cloudposse/utils v1.22.0 (self-signed, key ID 7B22D099488F3D11)
- Installing hashicorp/local v2.5.1...
- Installed hashicorp/local v2.5.1 (signed by HashiCorp)
- Installing hashicorp/aws v5.48.0...

@Andriy Knysh (Cloud Posse) looks like he’s using the correct version

yes looks like it

i don’t know what the issue is, and we never tested a config like that with the remote-state module, will have to review in more details

@Stephan Helas if you could DM me your config (or the relevant part of it), that would save some time to test and figure out any issues. If not, I’ll recreate something similar. Thank you

yes, will do. will take till tomorrow.

Hi @Andriy Knysh (Cloud Posse),

i tried to build a simple hello-world using local backend, but i can’t get it to work. I’ve created two components, vpc and hello-world. vpc will simply output vpc_id. hello-world should then use the output and just output it again.

https://github.com/astephanh/atmos-hello-world/tree/main/hello-world-remote

after terraform apply vpc i have a locale state, containing the output.

 ▶ atmos terraform apply vpc  -s org-acme-test

Initializing the backend...
Initializing modules...

Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Using previously-installed hashicorp/null v3.2.2

Terraform has been successfully initialized!
module.vpc.null_resource.name: Refreshing state... [id=8866214659539567934]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no
differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

vpc_id = "vpc-123456789"

but outputs of remote_state is null ( i don’t know why)

▶ atmos terraform plan hello-world -s org-acme-test

Initializing the backend...
Initializing modules...

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/external from the dependency lock file
- Reusing previous version of cloudposse/utils from the dependency lock file
- Using previously-installed hashicorp/null v3.2.2
- Using previously-installed hashicorp/local v2.5.1
- Using previously-installed hashicorp/external v2.3.3
- Using previously-installed cloudposse/utils v1.22.0

Terraform has been successfully initialized!
module.vpc.data.utils_component_config.config[0]: Reading...
module.vpc.data.utils_component_config.config[0]: Read complete after 0s [id=e69aca3dce4ce3047ed5ff092291e5c4c02ee685]
module.vpc.data.terraform_remote_state.data_source[0]: Reading...
module.vpc.data.terraform_remote_state.data_source[0]: Read complete after 0s

Terraform used the selected providers to generate the following execution plan. Resource
actions are indicated with the following symbols:
  + create

Terraform planned the following actions, but then encountered a problem:

  # module.base.null_resource.name will be created
  + resource "null_resource" "name" {
      + id = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + remote_vpc = {
      + backend               = {}
      + backend_type          = "local"
      + outputs               = null
      + remote_workspace_name = null
      + s3_workspace_name     = null
      + workspace_name        = "org-acme-test-vpc"
    }
  + tags       = {
      + "atmos:component"         = "hello-world"
      + "atmos:component_version" = "hello-world/v0.1.0"
      + "atmos:manifest"          = "org/acme/hello-world"
      + "atmos:stack"             = "org-acme-test"
      + "atmos:workspace"         = "org-acme-test-hello-world"
      + environment               = "acme"
      + stage                     = "test"
      + tenant                    = "org"
    }
╷
│ Error: Attempt to get attribute from null value
│
│   on main.tf line 5, in module "base":
│    5:   vpc_id = module.vpc.outputs.vpc_id
│     ├────────────────
│     │ module.vpc.outputs is null
│
│ This value is null, so it does not have any attributes.
╵
exit status 1

@Andriy Knysh (Cloud Posse) have we tested remote-state with the local backend?

i don’t think we have (will check if the remote-state module supports that). I’ll check the Stephan’s repo as well

@Stephan Helas unfortunately, the module https://github.com/cloudposse/terraform-yaml-stack-config/tree/main/modules/remote-state does not support local remote state. looks like you are the first person that tried to use it with local backend

you can do any of the following:

1. Use s3 backend (local backend is just for testing, not for real infra)
2. Use data "terraform_remote_state" directly (https://developer.hashicorp.com/terraform/language/settings/backends/local). This is not an Atmos way of doing things, but you can test with it
3. Use remote state of type static https://atmos.tools/core-concepts/components/remote-state-backend

i have this ugly workaround for now

✗ atmos describe stacks | yq e '. | to_entries | .[].value.components.terraform | with_entries(select(.value.metadata.type != "abstract")) | keys' | grep -v '\[\]' | sort | uniq
- aws-team-roles
- aws-teams
- account
- account-map
- aws-team-roles
- github-oidc-provider
- vpc

See https://atmos.tools/core-concepts/custom-commands/#list-atmos-stacks-and-components

I think we should add native support, but you could imagine due to how much configuration is managed by Atmos that the number of possible ways of filtering is quite staggering. So, for now, the recommendation is to create a custom command to view the data the way you want to view it. Literally think about it like creating a view.

@RB what do you think about adding a --jq parameter like in the gh cli as a stop-gap measure?

Thanks for sharing Erik. I did not realize there was already documentation for this and it was already in my atmos.yaml (at least the list stacks command)

Yes, the --jq or --query (jmespath) would be handy so you do not need to depend on jq

I don’t know how much it would help me personally since I already have jq installed

I’m probably mistaken here but is the problem that .setting.region is not getting replaced by the region or that the list subcommand is not interpreting the region?

Or should it be changed to .settings.region or .vars.region instead?

Or what was the exact result you were expecting? Because if you change the metadata.component to a versioned component, you wouldn’t see that result in atmos list components command since that’s only returning the names of the real planable components and not their real component dir names (which is located in the metadata).

the way i understand atmos is, that .settings get the same treatment as .vars. difference is, that .vars will be converted to terraform inputs for components. thats the reason why i try to use .settings.

so, the documentation states, that metadata can be used in templating. but this is not working for me (templating is enabled)

atmos.yaml

stacks:
  base_path: 'stacks'
  included_paths:
    - 'bootstrap/**/*'
    - 'KN/**/*'
  excluded_paths:
    - '**/_defaults.yaml'
  name_template: '{{.settings.tenant}}-{{.settings.instance}}-{{.settings.stage}}'

templates:
  settings:
    enabled: true
    evaluations: 1
    sprig:
      enabled: true
    gomplate:
      enabled: true

stacks/foo.yaml

settings:
  tenant: accounts
  instance: sandbox
  stage: 5
  component: account-vpc/v0.2.0

components:
  terraform:
    vpc/eu-central-1:
      metadata:
        component: '{{ .settings.component }}'

atmos list components -s accounts-sandbox-5
global
vpc/eu-central-1

plan doesn’t start

▶ atmos terraform plan vpc/eu-central-1  --stack accounts-sandbox-5
'vpc/eu-central-1' points to the Terraform component '{{ .settings.component }}', but it does not exist in 'components/terraform'

setting

templates:
  settings:
    evaluations: 2 

makes no difference

ah I see, interesting. I haven’t used [templates.settings.xyz](http://templates.settings.xyz) yet. I did notice that in your stacks/foo.yaml, you omit the templates portion so should

stacks/foo.yaml

settings:
  tenant: accounts
  instance: sandbox
  stage: 5
  component: account-vpc/v0.2.0

be

templates:
  settings:
    tenant: accounts
    instance: sandbox
    stage: 5
    component: account-vpc/v0.2.0

you mean like this?

settings:
  tenant: accounts
  instance: sandbox
  stage: 5

templates:
  settings:
     component: account-vpc/v0.2.0

components:
  terraform:
    vpc/eu-central-1:
      metadata:
        component: '{{ .settings.component }}'

its the same result for me:

you may have better luck with a yaml anchor

but my goal is to use multiple instances of components with different versions. if i inherit something, only .vars and .settings are merged.

I’ve created a sample repo: https://github.com/astephanh/atmos-hello-world

[hello-world-settings](https://github.com/astephanh/atmos-hello-world/tree/main/hello-world-settings) is working, while [hello-world-settings-template](https://github.com/astephanh/atmos-hello-world/tree/main/hello-world-settings-template) is not.

the only difference i’ve made is using template for the component path

diff --color -r hello-world-settings/stacks/org/acme/hello-world.yaml hello-world-settings-template/stacks/org/acme/hello-world.yaml
7a8
>   component: v0.1.0
13c14
<         component: hello-world/v0.1.0
---
>         component: 'hello-world/{{ .settings.component }}'

Here is the setion of the documenation:

https://atmos.tools/core-concepts/stacks/templating#atmos-sections-supporting-go-templates

component:
  terraform:
    vpc:
       metadata:
        component: "{{ .settings.component }}"

I’m not sure. Perhaps it’s a bug?

Maybe @Andriy Knysh (Cloud Posse) may know more here

@Stephan Helas this looks wrong (wrong indent)

settings:
  tenant: accounts
  instance: sandbox
  stage: 5

templates:
  settings:
     component: account-vpc/v0.2.0

should be

settings:
  tenant: accounts
  instance: sandbox
  stage: 5
  component: xxxx
  templates:
    settings:
      evaluations: 2

component:
  terraform:
    vpc:
      metadata:
        component: "{{ .settings.component }}"

{{ .settings.component }} refers to the `component` in the `settings` section

I'm not sure what you are trying to achieve, or just playing/testing, let me know and we'll try to help

settings.templates.settings should be used. It might look “wrong”, but we’ll be adding more features to templates (e.g. specify some templates to use with other Atmos commands), so in the near future it might look like this

settings:
   templates:
     settings:
       evaluations: 2
       gomplate:
         datasources: {}
     definitions:
       readme: {}. # template config to generate READMEs
       component: {} # template config to generate components
       stack: {} # template config to generate stacks

i was following the documentation here https://atmos.tools/core-concepts/stacks/templating#atmos-sections-supporting-go-templates.

I try to reuse components in stack without the need to define the component version every time. for this i try to use .settings.component like here:

repo: https://github.com/astephanh/atmos-hello-world/tree/main/hello-world-template

https://github.com/astephanh/atmos-hello-world/blob/main/hello-world-template/stacks/org/acme/hello-world.yaml

settings:
  component: v0.1.0
  templates:
    settings:
      enabled: true
      evaluations: 2

vars:
  tenant: org
  environment: acme
  stage: test

components:
  terraform:
    hello-world/1:
      metadata:
        component: 'hello-world/{{ .settings.component }}'
      vars:
        lang: de
        location: hh
        region: hh

result:

atmos terraform plan hello-world/1 -s org-acme-test
'hello-world/1' points to the Terraform component '{{ .settings.component }}', but it does not exist in 'components/terraform/hello-world'

you need to enable templating in atmos.yaml

templates:
  settings:
    # Enable `Go` templates in Atmos stack manifests
    enabled: true
    # Number of evaluations/passes to process `Go` templates
    # If not defined, `evaluations` is automatically set to `1`
    evaluations: 2

see https://atmos.tools/core-concepts/stacks/templating#configuring-templating-in-atmosyaml-cli-config-file

remove this from settings in the stack manifest

  templates:
    settings:
      enabled: true
      evaluations: 2

enabling templates in stack manifests is not currently supported (we might add it in the future versions)

Templating in Atmos can also be configured in the settings.templates.settings section in stack manifests.

The settings.templates.settings section can be defined globally per organization, tenant, account, or per component. Atmos deep-merges the configurations from all scopes into the final result using inheritance.

The schema is the same as templates.settings in the atmos.yaml CLI config file, except the following settings are not supported in the settings.templates.settings section:

settings.templates.settings.enabled
settings.templates.settings.sprig.enabled
settings.templates.settings.gomplate.enabled
settings.templates.settings.evaluations
settings.templates.settings.delimiters
These settings are not supported for the following reasons:

You can't disable templating in the stack manifests which are being processed by Atmos as Go templates

If you define the delimiters in the settings.templates.settings section in stack manifests, the Go templating engine will think that the delimiters specify the beginning and the end of template strings, will try to evaluate it, which will result in an error

please see these examples in the Quick Start

https://github.com/cloudposse/atmos/blob/master/examples/quick-start/atmos.yaml#L246

https://github.com/cloudposse/atmos/blob/master/examples/quick-start/rootfs/usr/local/etc/atmos/atmos.yaml#L245

ok. did that, but still the same error:

https://github.com/astephanh/atmos-hello-world/commit/2b9706fde6f6bc6276ccfb40d3a62069b8b99434

or create a new component for dynamic-subnets and attach a new suite of public/private subnets to an existing VPC ?

@RB take a look at

https://github.com/cloudposse/terraform-aws-dynamic-subnets/pull/174

https://github.com/cloudposse/terraform-aws-dynamic-subnets/tree/main/examples/multiple-subnets-per-az

it allows you to create “named” subnets (e.g. database), and then use their names (using remote state and outputs) to deploy resources into

That looks promising. Thanks Andriy!

Is it possible then to create public and private subnets and additionally create a named subnet for databases in the same vpc component instantiation?

But if we did do multiple subnets in the same vpc, each component that reads from the remote state would only read the public and private subnet output. How would we get the remote state to show different outputs and then use them differently for components that depend on the vpc component? I imagine we’d need to update each component, right?

see the PR description, the module outputs maps of public and private subnets, you can use it from remote state

named_private_subnets_map = {
  "backend" = tolist([
    "subnet-0393680d8ea3dd70f",
    "subnet-06764c7316567eacc",
  ])
  "db" = tolist([
    "subnet-0a7c4b117b2105a69",
    "subnet-074fd7ad2b902bec2",
  ])
  "services" = tolist([
    "subnet-02c63d0c0c2f84bf5",
    "subnet-0f6d042c659cc1346",
  ])
}

we use that for Network Firewall since it always required a dedicated subnet in a VPC just for itself

Fantastic. Thank you. I’ll give this a try

Hmm the vpc component may be out of date because i don’t see the input exposed

https://github.com/cloudposse/terraform-aws-components/blob/a9252bc79d5fed90edec6ddb147bd61287de0620/modules/vpc/main.tf#L140

Missing input

https://github.com/cloudposse/terraform-aws-dynamic-subnets?tab=readme-ov-file#input_subnets_per_az_names

Ref network firewall usage https://github.com/cloudposse/terraform-aws-components/blob/a9252bc79d5fed90edec6ddb147bd61287de0620/modules/network-firewall/main.tf#L26

oh, the component is missing the inputs for the module

variable "subnets_per_az_count" {
  type        = number
  description = <<-EOT
    The number of subnet of each type (public or private) to provision per Availability Zone.
    EOT
  default     = 1
  nullable    = false
  validation {
    condition = var.subnets_per_az_count > 0
    # Validation error messages must be on a single line, among other restrictions.
    # See <https://github.com/hashicorp/terraform/issues/24123>
    error_message = "The `subnets_per_az` value must be greater than 0."
  }
}

variable "subnets_per_az_names" {
  type = list(string)

  description = <<-EOT
    The subnet names of each type (public or private) to provision per Availability Zone.
    This variable is optional.
    If a list of names is provided, the list items will be used as keys in the outputs `named_private_subnets_map`, `named_public_subnets_map`,
    `named_private_route_table_ids_map` and `named_public_route_table_ids_map`
    EOT
  default     = ["common"]
  nullable    = false
}

@RB we’d appreciate it if you opened a PR and add it

https://github.com/cloudposse/terraform-aws-components/pull/1032

thanks @RB please format TF code

I’m not sure what’s happening. Looks like a lot of the checks failed because they took 6 hours and maybe timed out?

I added the question here too

https://sweetops.slack.com/archives/CUJPCP1K6/p1715820136355889?thread_ts=1715820136.355889&cid=CUJPCP1K6

Thanks @Stephan Helas!

(small typo in PR)
support atmoskj from

yeah was late in the night

fixed it

I think they talk about sign-off here https://github.com/tofuutils/tenv?tab=readme-ov-file#signature-support

ok, i signed it off. one more thing i need to read about….

So, first and foremost, we are starting to come around to the idea of doing what the other tooling in the Terraform ecosystem does: directly making outputs available from one component as inputs to another.

For the longest time, we’ve been adamant about not doing this, as to forces reliance on the tooling. We always say Atmos lets you eject and Terraform handles remote state just fine, so why should the tool do it?

But this is a self-imposed limitation, and how we’ve implemented our reference architecture, and it seems many would prefer just to do it in the tooling layer.

Therefore we are discussing how to best support this concept in atmos, such as with the newly released Atmos data sources.

So. you have a few options.

1. The remote state implementation, which works very well once you have it set up, but it takes some configuration to get it working.

1. Use data sources and lookups as you would normally, however, not all information is available via data source lookups, not all providers have sufficient data sources, etc.

1. Consider using a KV store for your cloud platform. For an enterprise customer, for example, we implemented a kv store for artifactory and plan to do the same for GCP, Azure, and AWS.

https://github.com/cloudposse/terraform-artifactory-kv-store

We recommend the KV store approach b/c it’s provider agnostic and cloud agnostic. The same pattern can be used however makes best sense.

Different systems can read/write from it and populate it. It’s not restricted to the information provided by data sources, so it’s more generalizable.

Thanks. Is any of that guidance/pattern/tradeoffs documented anywhere? woudl hate to lose this useful info ot the Slack ether

I know. I reallllly hurt to downgrade.

We also have https://archive.sweetops.com but slack changed also how exports work. So they are now limited to 90 days too. That did not used to be the case.

Is any of that guidance/pattern/tradeoffs documented anywhere? woudl hate to lose this useful info ot the Slack ether You’re absolutely right. It would be beneficial to post this and the trade offs.

Also, here’s a recent thread on the topic as well. https://sweetops.slack.com/archives/C031919U8A0/p1714829174944729

Noting I think I could build the change on top of it? idk. Just brainstorming based on new reqs this week.

We’ve accomplished this I know previously, since we’ve implemented centralized organizational egress over the transit gateway

I don’t know the specifics. Would need to defer to @Andriy Knysh (Cloud Posse) or @Jeremy G (Cloud Posse)

Yea we’re on Transit but then we also have Ciscos SDWAN product as the interchnage to make it tougher to understand.

Yea that’s cool, we’re literally just in planning phases.

I believe we implemented something similar to https://d1.awsstatic.com/architecture-diagrams/ArchitectureDiagrams/inspection-deploy[…]els-with-AWS-network-firewall-ra.pdf?did=wp_card&trk=wp_card

Yea I REALLY wanted that subnet firewall design

instead now I have to route to ip i think

ty for that diagram

im imagining doing some ugly static vars in to fix this lol

I think it can be hard to do fully dynamically

we have implemented the network architecture depicted in the diagrams https://d1.awsstatic.com/architecture-diagrams/ArchitectureDiagrams/inspection-deployment-models-with-AWS-network-firewall-ra.pdf, but that’s a completely separate/new Terraform components to do all the routing b/w diff VPCs (ingress, egress, inspection). The components is configured and deployed by Atmos. Since it’s complex and custom-made for specific use-cases (even in the diagrams in the PDF there are many diff use-cases), we have not open-sourced it

No I wasn’t expecting open source, I was moreso just brainstorming on how to handle modifying those route tables for my case

it’s prob not possible to make such a component to be able to define all possible networking configurations in such complex architectures as using ingress/egress/inspection VPCs and TGWs

yea i was kinda like uhhhhh when they said they wanted to go this direction

def preferred network fw

yep, I know what you feel

anyway, regarding “I was moreso just brainstorming on how to handle modifying those route tables for my case”

what we did, we implemenbetd all of that in Terraform (as a few components), including defining all the VPCs (using our vpc components), and the following:

• Route tables (they are not default)

• All TGW routes b/w all those ingress/egress/inspection VPCs

• All EC2 subnet routes b/w all those ingress/egress/inspection VPCs

that’s why the Terraform components are custom, we did not try to make it universal for all possible combinations of such network architectures (which would be a very complex task)

@Ryan I hope this will help you

I recommend not trying to make a universal component, but do what you need in Terraform and then deploy it with Atmos

thats prob most of what im doing now, i def do not dig into root/child cloudposse structure besides what was previously deployed

are you going to use the network firewall and the inspection VPC?

asking because network firewall complicates the design even further, e.g it requires a separate dedicated subnet in a VPC where it deployed

No we currently have a pretty flat network design but everything routes through a network acct

Besides the igws the vpc module deploys

Well somewhere in that module stack lol

that’s why, if you already have VPCs with subnets and definitely can’t destroy them to add another subnet, you can use a separate inspection VPC with a dedicated Firewall subnet and route traffic through it

That would make cost lower too

I think

see this PR https://github.com/cloudposse/terraform-aws-dynamic-subnets/pull/174 where we added the ability to deploy multiple “named” subnets in a VPC per AZ

it was done to support Network Firewall’s dedicated subnet

@Ryan if you have questions, feel free to ask. It’s not possible to explain everything at once since it’s too many moving parts, but we’ll be able to answer questions one by one on diff parts of that network architecture

@Ryan We recommend using terraform-aws-dynamic-subnets to deploy and configure the VPC’s public and private subnets. (We have a bunch of similar modules, but this we are trying to consolidate them down to just this one, so it has the most features and best support.) It has a so may options even I have forgotten them all, and I wrote most of them. The options I would recommend first considering: • If you do not want a route to the internet gateway, you do not have to supply the IGW ID, and no route to it will be created. You can then add your own route, using the aws_route resource, to the route tables specified by the public_route_table_ids module output. • You can suppress the creation and modification of route tables altogether by setting public_route_table_enabled to false, and then you can create and configure the route tables yourself.
If neither of those suit you, you can probable figure out yet another way. I suggest reviewing the code as the best documentation.

yes, that’s what we did: use the dynamic-subnets component to create subnets and disable some routes, then used the aws_route resource to create subnet routes, and aws_ec2_transit_gateway_route resource to create TGW routes

appreciate it gentlemen

@Dan Miller (Cloud Posse) @Jeremy G (Cloud Posse)

We can probably further tune it too cause we probably don’t need AddRoleToInstanceProfile or RemoveRoleFromInstanceProfile

Also should allow

        "sts:SetSourceIdentity",
        "sts:TagSession",

Hrm… that’s odd. I don’t believe symlinks are used anywhere in the quick start. Are you using any symbolic links with your project (e.g. ln -s)

sometimes people symlink their home directory to google drive or dropbox, or onedrive

Not consciously, as in I did not run any symlinking

we’re using the latest version of the components: (vendor.yaml)

- component: "tfstate-backend"
      source: "github.com/cloudposse/terraform-aws-components.git//modules/tfstate-backend?ref={{.Version}}"
      version: "1.433.0"
      targets:
        - "components/terraform/tfstate-backend"
      included_paths:
        - "**/*.tf"
      excluded_paths:
        - "**/providers.tf"
      tags:
        - core

To be clear, this is not an atmos error. This is terraform

│ Unable to evaluate directory symlink:

Atmos does not create any symlinks as part of vendoring.

What operating system are you on?

OSX

Hrm…

Im running atmos in Docker though

Are you using it in geodesic or a custom image?

custom image based on cloudposse/geodesic:${GEODESIC_VERSION}-${GEODESIC_OS}

Aha, ok, Geodesic does have symlinks. So that’s probably wherein we have a problem.

@Jeremy G (Cloud Posse) or @Jeremy White (Cloud Posse) any ideas on this one?

Thanks so far!

Our cold start procedure has changed over time, and you may be looking at an older version. In the current cold start, you need to have in your repository the source code and stack configuration for: • account • account-map • account-settings • aws-teams • aws-team-roles • tfstate-backend before you start running Terraform.

Then, to start, you run

atmos terraform apply tfstate-backend -var=access_roles_enabled=false --stack core-usw2-root --auto-generate-backend-file=false

(assuming you use our defaults, where the org root is core-root, the primary region is us-west-2, and the region abbreviation scheme is short).

Then, if you have not already configured the Atmos stacks backend , you configure it now to use the S3 bucket and Role ARN output by the component.

Then, you move the tfstate-backend state to S3 by running

atmos terraform apply tfstate-backend -var=access_roles_enabled=false --stack core-uw2-root --init-run-reconfigure=false

Later, after provisioning all the other components, you come back and run

atmos terraform apply tfstate-backend --stack core-uw2-root

@Marvin de Bruin Please post links to the documentation you are looking at.

@Jeremy G (Cloud Posse) Thanks for looking in to this.

These are all the docs Im looking at apart from some google-fu results that didnt bring me far

Main: https://atmos.tools/quick-start/ TF backend docs: https://atmos.tools/quick-start/configure-terraform-backend#terraform-s3-backend provisioning section: https://atmos.tools/quick-start/configure-terraform-backend#provision-terraform-s3-backend (this step is where Im currently blocked, but I havent followed your steps yet)

I’ve also been looking at the tutorials, but they come with outdated components https://atmos.tools/tutorials/first-aws-environment (repo: https://github.com/cloudposse/tutorials) The component: https://github.com/cloudposse/tutorials/tree/main/03-first-aws-environment/components/terraform/tfstate-backend

I’ve also looked at https://github.com/cloudposse/terraform-aws-components/tree/main/modules/tfstate-backend which also does not mention the cold start dependencies

I’ve now added all the components you mentioned, pulled them down (atmos vendor pull) and gave them a minimal config (based on the usage docs of each component). I’m still seeing that Terraform symlink issue though when I run the command you mentioned (I modified it slightly):

 ⨠ atmos terraform apply tfstate-backend -var=access_roles_enabled=false --stack core-gbl-root --auto-generate-backend-file=false

Initializing the backend...
Initializing modules...
- assume_role in
╷
│ Error: Unreadable module directory
│
│ Unable to evaluate directory symlink: lstat ../account-map/modules: no such file or directory
╵

╷
│ Error: Unreadable module directory
│
│ The directory  could not be read for module "assume_role" at iam.tf:30.
╵

exit status 1

Btw, Im not expecting a response in the weekend, Im just very excited working on this Have a great weekend.

@Marvin de Bruin I don’t know where or why you have a symlink. Our standard directory structure looks like this:

interesting, I indeed do not have that modules folder

You do not need to have components/terraform exactly, but you do need to have all the components in the same directory. In particular, all the modules reference ../account-map/

And these are in the components pulled down by the vendor pull script, I also see them in the repo. I think I might have a misconfig in my vendors file

I might have missed a step, but is the recommended way to manually modify the vendors.yaml or do this automatically somehow (like with NPM for javascript, Composer for php)?

Vendor config for account-map

Our vendoring system has quirks and kinks still to be worked out. The glob patterns do not quite work as expected, for example. I’d expect **/*.tf to pull in all the *.tf files, in all the subdirectories, but it does not do that if you do not include the **/modules/** as well.

account-map is a special case because of the sub-modules

Ah! This was indeed the cause of the blockage! Thank you very much! Is there a repo somewhere with all the default vendor configs for each module? It could be a fun project to create an installer make module add x

@Erik Osterman (Cloud Posse)

Lets move future refarch questions to refarch

Re: installer for components, we plan to make it easier to do via CLI.

Re: vendor configs, those are part of the typical jumpstart engagement and generated based on the type of architecture you need.