#atmos (2024-10)

@Andriy Knysh (Cloud Posse)

@Miguel Zablah Atmos caches the results of the atmos.Component functions only for the same component and stack, and only for one command execution. If you run atmos terraform ... again, it will not use the cached results anymore

Thanks it looks like I had an error on another file that is why I though it was maybe cache, thanks!

run validate stacks to see if you have a wrong yaml

all successful

and make sure the base_path is correct, and atmos cli can find the atmos.yaml

for backends to be auto-generated, you need to configure the following:

• In atmos.yaml

  components:
  terraform:
    # Can also be set using `ATMOS_COMPONENTS_TERRAFORM_AUTO_GENERATE_BACKEND_FILE` ENV var, or `--auto-generate-backend-file` command-line argument
    auto_generate_backend_file: true
  

• Configure the backend in YAML https://atmos.tools/quick-start/advanced/configure-terraform-backend/#configure-terraform-s3-backend-with-atmos

you need to have this config

terraform:
  backend_type: s3
  backend:
    s3:
      acl: "bucket-owner-full-control"
      encrypt: true
      bucket: "your-s3-bucket-name"
      dynamodb_table: "your-dynamodb-table-name"
      key: "terraform.tfstate"
      region: "your-aws-region"
      role_arn: "arn:aws:iam::<your account ID>:role/<IAM Role with permissions to access the Terraform backend>"

in the defaults for the Org (e.g. in stacks/orgs/acme/_defaults.yaml) if you have one backend for the entire Org

What is strange it was working, this is the error.

atmos terraform generate backend wazuh -s dev –logs-level debug template: all-atmos-sections35: executing “all-atmos-sections” at <atmos.Component>: error calling Component: exit status 1

Error: Backend configuration changed

A change in the backend configuration has been detected, which may require migrating existing state.

If you wish to attempt automatic migration of the state, use “terraform init -migrate-state”. If you wish to store the current configuration with no changes to the state, use “terraform init -reconfigure”.

try to remove the .terraform folder and run it again

Same error with the .terraform folder removed

Ahh ran with log_level trace, and found the issue

what was the issue?

The component had references for var in other components

removed .terraform in those other components

TY for trace

I should make a script to just clean-slate all those .terraform directories

Ya, we should have that as a built in command.

@Dennis DeMarco for now, instead of a script add a custom command to atmos

https://atmos.tools/core-concepts/custom-commands/

@Igor Rodionov

@Andriy Knysh (Cloud Posse)

@Miguel Zablah by “disabled component” do you mean you set enabled: false for it?

Yes

i see the issue. We’ll review and update it in the next Atmos release

That will be awesome since this is a blocker for us now

@Andriy Knysh (Cloud Posse) any ETL on this?

Hi @Miguel Zablah We don’t have an ETA yet. But should be soon

we’ll try to do it in the next few days

thanks!!

@Andriy Knysh (Cloud Posse) @Jeremy G (Cloud Posse) @Dan Miller (Cloud Posse)

I think it could be related to the path of your atmos.yaml file, since you deal with remote-state and terraform-provider-utils Terraform provider. Check this: https://atmos.tools/quick-start/simple/configure-cli/#config-file-location

I thought it might be that but it is clearly registering my atmos.yaml file as it is in the main directory of my repo and i’ve found atmos to respond to settings there (such as the include/exclude paths and log levels) but the other terraform providers aren’t picking it up. (EDIT! Just saw the bit at the bottom… exploring now)

@Drew Fulton Terraform executes the provider code from the component directory (e.g. components/terraform/my-component). We don’t want to put atmos.yaml into each component directory, so we put it into one of the known places (as described in the doc) or we can use ENV vars, so both Atmos binary and the provider binary can find it

Got it! Is there a best practice for selecting where to put it to prevent duplication?

for Docker containers (geodesic is one of them), we put it in rootfs/usr/local/etc/atmos/atmos.yaml in the repo, and then in the Dockerfile we do

COPY rootfs/ /

so inside the container, it will be in usr/local/etc/atmos/atmos.yaml, which is a know path to Atmos and the provider

perfect! Thanks so much for the help

and it works!

Yes, centrally defined components is best for larger organizations that might have multiple infrastructure repositories

However, it can also be burdensome when iterating quickly to have to push all changes via an upstream component git component registry

yeah, thanks for the input. We’re not a large organization, there will be some trade off we’ll have to consider, if we try to limit the devs to only work with yaml. Still early stages of exploring the capabilities of atmos.

there is a lot of options recently added that atmos might now recognize

although

 plan_requirements: [undiverged]
        apply_requirements: [mergeable,undiverged] 

I think they can only be declared in the server side repo config repo.yaml

mmm no, they can can be declared in the atlantis.yaml

FYI is not recommended to allow users to override workflows, so it is much safer to configure workflows and repo options on the server side config and leave the atlantis.yaml as simple as possible

Yeah it make sense to have these values in server side repo config… I was trying some test scenarios and encountered the issue. in related question, the field ‘when_modified’ in atlantis.yaml .. I was trying to find some documentation about how it determines the modified files, just determined on if the file is modified in the PR?

file modified in the PR yes

is a regex used for autoplan

@jose.amengual another issue I am running in to with atlantis Have a question about the ‘when_modified’ field in the repo level config, we are using dynamic repo config generation and in the same way generating the var files for the project. The var files are not commited to the git repo, since it’s not commited I beleive the when_modified causes an issue by not planning the project. Removing the when_modified field from config doesn’t seem to help becuase of the default values. Do we have way to ignore this field and just plan the projects, regardless of modified file in the project

I answer you on the atlantis slack

please DM me your config, i’ll review it

Yes, they would open a PR and add the necessary components to the stack configuration.

…and prior to performing any application deployment that depends on it.

Cool, i think i begin to have clearer understanding on the Atmos mindset … thnks !!

I just now tried something that seemed to work. Since I’m using the “{stage}” name pattern inside atmos.yaml, I set a “vars.stage: dev” inside my dev.yaml stack file and it seemed to do the trick. Is this a correct pattern? Thanks!

@Andriy Knysh (Cloud Posse)

Can you confirm that you see a valid zone_id as a terraform output of keycloak_route53_zones

and please make sure the component is provisioned

atmos.Component calls terraform output, so it must be in the state already

atmos terraform output keycloak_route53_zones -s aws-dev-us-east-1

gives me

zone_id = {
  "redacted" = "redacted"
}

which is redacted but it is the zone subdomain as key and the id as value

aha, so it’s returning a map for the zone id

ah!

so I guess it should be

zone_id: '{{ (atmos.Component "keycloak_route53_zones" .stack).outputs.zone_id.value }}'

value is not correct

you should get the value by the map key using Go templates

there the key is the "redacted" in your example

Thanks! I’ll test this out, I appreciate the help

It didn’t seem to work and for this case I just passed the zone id directly (like the actual value from the output). The error was that terraform complained that it should be less than 32 characters, which means that terraform understood my go template as an actual string and not a template. I run it with atmos terraform apply component -s stack after a successful plan creation

I passed the value directly because I don’t think the zone id will change in the future, but for other components this might be an issue for me. Maybe I should run it differently? I set the value exactly as suggested by Andriy

please send me your yaml config, i’ll take a look

Sure!

yes

settings is a free form map

its the same as vars, but free form, participates in all the inheritance, meaning you can define it globally, per stack, per base component, per component, and then everything gets deep merged into the final map

cool, that is awesome and exactly what I need

imagine if this was added to something like atmos.yaml and CODEOWNERS only allow very few people to modify it

like having Sec rules for all components

no, metadata is not inherited, it’s per component

ok

because in metadata you specify all the base components for your component

understood

Note that atmos never uses this backend. We generate a backend.tf.json file used by terraform

Could it be a GCP permissions issue

I am a organisation admin, have full access to the bucket, and besides Ive tested access to the bucket itself with cli.

Based on your screenshot the YAML is invalid.

the hint is it’s trying to use a “local” backend type.

https://atmos.tools/core-concepts/components/terraform/backends/#google-cloud-storage-backend

Note in your YAML above, the whitespace is off

Terraform is attempting to use this: https://developer.hashicorp.com/terraform/language/backend/local

which indicates that the backend type is not getting set

Thanks Erik. I’ve double checked again everything. I’ve tried to delete backend.tt.json file, disable auto_generate_backend_file and after that added backend configuration directly into module - same result. That got me thinking something is not right with auth into GCP. Re-logged again with gcloud auth application-default login and now state is migrated into the bucket. Honestly no idea, I’ve logged in multiple times this day already. Even before posting here

maybe this is the kind of tax for changing whole workflow to atmos

@jose.amengual from where you want to vendor all the components?

from my iac repo

to another repo ( using atmos)

let me check

I did this :

apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
  name: iac-vendoring
  description: Atmos vendoring manifest for Atmos-iac repo
spec:
  # Import other vendor manifests, if necessary
  imports: []

  sources:
      - source: "github.com/jamengual/pepe-iac.git/"
        #version: "main"
        targets:
          - "./"
        included_paths:
          - "**/components/**"
          - "**/*.md"
          - "**/stacks/sandbox/**"

I was able to vendor components just fine

but not sandbox stack files for some reason

you need to add another item for the stacks folder separately

that’s how the glob lib that we are using works

we don’t like it and will revisit it, but currently it’s what it is

you mean something like :

 sources:
      - source: "github.com/jamengua/pepe-iac.git/"
        #version: "main"
        targets:
          - "./"
        included_paths:
          - "**/components/**"
          - "**/*.md"
      - source: "github.com/jamengual/pepe-iac.git/"
        #version: "main"
        targets:
          - "./stacks/sandbox/"
        included_paths:
          - "stacks/sandbox/*.yaml"

try this ^

so this pulled all the stacks

       - "**/stacks/**"

but it looks like I can’t pull a specific subfolder

hmm, so this "**/stacks/sandbox/**" does not work?

no

i guess you can use

- "**/stacks/**"
- "**/stacks/sandbox/**"

and exclude the other stacks in excluded_paths:

I could do that….but it makes it not very DRY

can you verbose the pull command?

atmos vendor pull --verbose

Atmos 1.88.1 on darwin/arm64

oh sorry, try

ATMOS_LOGS_LEVEL=Trace atmos vendor pull

@jose.amengual also check out this example for using YAML anchors in vendor.yaml to DRY it up

https://github.com/cloudposse/atmos/blob/main/examples/demo-component-versions/vendor.yaml#L10-L20

could you pass an ENV variable as a value inside of the yaml?

like

 excluded_paths: 
          - "**/production/**"
          - "${EXCLUDE_PATHS}"
          - "${EXCLUDE_PATH_2}"
          - "${EXCLUDE_PATH_3}"

currently not, env vars in vendor.yaml are not supported

ok, Thanks guys

I was thinking

Option 1) a unique namespace via a separate dir

e.g.

# cloudposse components
components/terraform
# internal components
components/terraform/internal

Option 2) a unique namespace via a prefix

# upstream component
components/terraform/ecr
# internal component
components/terraform/internal-ecr

Option 3) Unique key in component.yaml and enforce this file in all components

Option 4) Vendor internal components from an internal repo

This way the source will contain the other org instead of cloudposse so that can be used to distinguish

We’re actually looking into something imilar to this right now related to our refarch stack configs

For stack configs we’ve settled on stacks/vendor/cloudposse

For components, I would maybe suggest

components/vendor/cloudposse

The alternative is a top-level folder like vendor/cloudposse

which could contain stacks and components.

Thats for stack configs, what about terraform components ? Or do you think it would be for both stack configs and terraform components?

What our team dose this: Vendor: components/terraform/vendor/{provider, ej. cloudposse} Internal: components/terraform/{cloudProvider, Ej. AWS}/{componentName, Ej. VPC}

Thats for stack configs, what about terraform components ? https://sweetops.slack.com/archives/C031919U8A0/p1728683594309549?thread_ts=1728675666.706109&cid=C031919U8A0ß https://sweetops.slack.com/archives/C031919U8A0/p1728683609704999?thread_ts=1728675666.706109&cid=C031919U8A0

I have my vendor.yaml

I tried git:// but then tried to use ssh git, and this is running from a reusable action

I tried this locally and it works but in my local I have a ssh-config

if I run git clone using that url it clones just fine

I’m hitting this error on the go-git library now

// <https://github.com/go-git/go-git/blob/master/worktree.go>

func (w *Worktree) getModuleStatus() (Status, error) {
    // ...
    if w.r.ModulesPath == "" {
        return nil, ErrModuleNotInitialized
    }

    if !filepath.IsAbs(w.r.ModulesPath) {
        return nil, errors.New("relative paths require a module with a pwd")
    }
    // ...
}

I do not know if this is because vendir sources on http are not compatible with git?

How log does it take to get the linux x64 package?