@Andriy Knysh (Cloud Posse)

@Miguel Zablah Atmos caches the results of the atmos.Component functions only for the same component and stack, and only for one command execution. If you run atmos terraform ... again, it will not use the cached results anymore

Thanks it looks like I had an error on another file that is why I though it was maybe cache, thanks!

run validate stacks to see if you have a wrong yaml

all successful

and make sure the base_path is correct, and atmos cli can find the atmos.yaml

for backends to be auto-generated, you need to configure the following:

• In atmos.yaml

  components:
  terraform:
    # Can also be set using `ATMOS_COMPONENTS_TERRAFORM_AUTO_GENERATE_BACKEND_FILE` ENV var, or `--auto-generate-backend-file` command-line argument
    auto_generate_backend_file: true
  

• Configure the backend in YAML https://atmos.tools/quick-start/advanced/configure-terraform-backend/#configure-terraform-s3-backend-with-atmos

you need to have this config

terraform:
  backend_type: s3
  backend:
    s3:
      acl: "bucket-owner-full-control"
      encrypt: true
      bucket: "your-s3-bucket-name"
      dynamodb_table: "your-dynamodb-table-name"
      key: "terraform.tfstate"
      region: "your-aws-region"
      role_arn: "arn:aws:iam::<your account ID>:role/<IAM Role with permissions to access the Terraform backend>"

in the defaults for the Org (e.g. in stacks/orgs/acme/_defaults.yaml) if you have one backend for the entire Org

What is strange it was working, this is the error.

atmos terraform generate backend wazuh -s dev –logs-level debug template: all-atmos-sections35: executing “all-atmos-sections” at <atmos.Component>: error calling Component: exit status 1

Error: Backend configuration changed

A change in the backend configuration has been detected, which may require migrating existing state.

If you wish to attempt automatic migration of the state, use “terraform init -migrate-state”. If you wish to store the current configuration with no changes to the state, use “terraform init -reconfigure”.

try to remove the .terraform folder and run it again

Same error with the .terraform folder removed

Ahh ran with log_level trace, and found the issue

what was the issue?

The component had references for var in other components

removed .terraform in those other components

TY for trace

I should make a script to just clean-slate all those .terraform directories

Ya, we should have that as a built in command.

@Dennis DeMarco for now, instead of a script add a custom command to atmos

https://atmos.tools/core-concepts/custom-commands/

@Igor Rodionov

@Andriy Knysh (Cloud Posse)

@Miguel Zablah by “disabled component” do you mean you set enabled: false for it?

Yes

i see the issue. We’ll review and update it in the next Atmos release

That will be awesome since this is a blocker for us now

@Andriy Knysh (Cloud Posse) any ETL on this?

Hi @Miguel Zablah We don’t have an ETA yet. But should be soon

we’ll try to do it in the next few days

thanks!!

@Andriy Knysh (Cloud Posse) Is this the fix? Release - v1.89.0 If so I have try running it but I still get the same error

what error are you getting?

must be something wrong with the Go template

@Andriy Knysh (Cloud Posse) the problem is that vpc component is not applied yey

to there is no output

atmos.Component "vpc" .stack).outputs.vpc_private_subnets

I do not know how we should handle that

yes but I was thinking that since it had the:

terraform:
  settings:
    github:
      actions_enabled: false

it should get ignore no?

Github actions would

but atmos still have to process yaml properly

so that would not solve your problem

my issue is that github actions is not doing this since it dose a normal:

atmos describe affected --include-settings=false --verbose=true

atmos need valid configuration of all components at any call

have you tried that on local?

yes and I get the same error bc that stack is not applied

but that is intencional since that stack is mostly for quick test/debug

atmos needs to parse valid all stack configuration then it operate on stack you need

if there is any error or misconfigration on yaml atmos will fail at any command

well there is no error on configuration the issue is that the stack is not applied

but that’s the error

but I guess maybe I can mock the values with some go template or something to ignore this

will that work?

@Andriy Knysh (Cloud Posse) do our templating support sprig functions?

@Miguel Zablah try this one

{{ concat (default (list) (atmos.Component "vpc" .stack).outputs.vpc_private_subnets) (default (list) (atmos.Component "vpc" .stack).outputs.vpc_public_subnets) | toRawJson }}

?

I get the same error:

template: describe-stacks-all-sections:74:26: executing "describe-stacks-all-sections" at <concat ((atmos.Component "vpc" .stack).outputs.vpc_private_subnets) ((atmos.Component "vpc" .stack).outputs.vpc_public_subnets)>: error calling concat: runtime error: invalid memory address or nil pointer dereference

ok

let’s try another one

like another version of this?

armos templates support sprig and gomplate functions

it’s not supposed to work on a component that is not applied yet , and Atmos doesn’t know if it was applied or not

what “terraform output” returns for such a component, we need to handle that in the template

same error:

template: describe-stacks-all-sections:78:82: executing "describe-stacks-all-sections" at <concat (get . "vpc_private_subnets") (get . "vpc_public_subnets")>: error calling concat: Cannot concat type string as list

@Andriy Knysh (Cloud Posse) how about mocking or default values?

that’s different error

sec

aah that is right

try this one ^

and we are back:

template: describe-stacks-all-sections:78:82: executing "describe-stacks-all-sections" at <concat (coalesce (get . "vpc_private_subnets") (list)) (coalesce (get . "vpc_public_subnets") (list))>: error calling concat: runtime error: invalid memory address or nil pointer dereference

just to test

I get a different error now:

template: describe-stacks-all-sections:78:82: executing "describe-stacks-all-sections" at <concat (coalesce (get . "vpc_private_subnets") (list "test1")) (coalesce (get . "vpc_public_subnets") (list "test2"))>: error calling concat: Cannot concat type map as list

@Miguel Zablah is that not provisioned component enabled or disabled? If it’s enabled, set enabled: false to disable it and try again

sorry had to run an errand

@Igor Rodionov I fot this error:

template: describe-stacks-all-sections:74:26: executing "describe-stacks-all-sections" at <concat ((atmos.Component "vpc" .stack).outputs.vpc_private_subnets) ((atmos.Component "vpc" .stack).outputs.vpc_public_subnets)>: error calling concat: runtime error: invalid memory address or nil pointer dereference

so same error

@Andriy Knysh (Cloud Posse) where I set this?

you add enabled: false in the YAML config for your component

components:
   terraform:
     my-component:
       vars:
         enabled: false

but this will only work of components that support that right? or will atmos actually ignore it?

yes, the enabled variable only exist in the components that support it (mostly all CloudPosse components)

Atmos sends the vars section to terraform

@Andriy Knysh (Cloud Posse) I’m using aws vpc module so I don’t think it will do much here also this will only return empty outputs right? so this might still not work right?

@Andriy Knysh (Cloud Posse) @Jeremy G (Cloud Posse) @Dan Miller (Cloud Posse)

I think it could be related to the path of your atmos.yaml file, since you deal with remote-state and terraform-provider-utils Terraform provider. Check this: https://atmos.tools/quick-start/simple/configure-cli/#config-file-location

I thought it might be that but it is clearly registering my atmos.yaml file as it is in the main directory of my repo and i’ve found atmos to respond to settings there (such as the include/exclude paths and log levels) but the other terraform providers aren’t picking it up. (EDIT! Just saw the bit at the bottom… exploring now)

@Drew Fulton Terraform executes the provider code from the component directory (e.g. components/terraform/my-component). We don’t want to put atmos.yaml into each component directory, so we put it into one of the known places (as described in the doc) or we can use ENV vars, so both Atmos binary and the provider binary can find it

Got it! Is there a best practice for selecting where to put it to prevent duplication?

for Docker containers (geodesic is one of them), we put it in rootfs/usr/local/etc/atmos/atmos.yaml in the repo, and then in the Dockerfile we do

COPY rootfs/ /

so inside the container, it will be in usr/local/etc/atmos/atmos.yaml, which is a know path to Atmos and the provider

perfect! Thanks so much for the help

and it works!

Yes, centrally defined components is best for larger organizations that might have multiple infrastructure repositories

However, it can also be burdensome when iterating quickly to have to push all changes via an upstream component git component registry

yeah, thanks for the input. We’re not a large organization, there will be some trade off we’ll have to consider, if we try to limit the devs to only work with yaml. Still early stages of exploring the capabilities of atmos.

there is a lot of options recently added that atmos might now recognize

although

 plan_requirements: [undiverged]
        apply_requirements: [mergeable,undiverged] 

I think they can only be declared in the server side repo config repo.yaml

mmm no, they can can be declared in the atlantis.yaml

FYI is not recommended to allow users to override workflows, so it is much safer to configure workflows and repo options on the server side config and leave the atlantis.yaml as simple as possible

Yeah it make sense to have these values in server side repo config… I was trying some test scenarios and encountered the issue. in related question, the field ‘when_modified’ in atlantis.yaml .. I was trying to find some documentation about how it determines the modified files, just determined on if the file is modified in the PR?

file modified in the PR yes

is a regex used for autoplan

@jose.amengual another issue I am running in to with atlantis Have a question about the ‘when_modified’ field in the repo level config, we are using dynamic repo config generation and in the same way generating the var files for the project. The var files are not commited to the git repo, since it’s not commited I beleive the when_modified causes an issue by not planning the project. Removing the when_modified field from config doesn’t seem to help becuase of the default values. Do we have way to ignore this field and just plan the projects, regardless of modified file in the project

I answer you on the atlantis slack

please DM me your config, i’ll review it

Yes, they would open a PR and add the necessary components to the stack configuration.

…and prior to performing any application deployment that depends on it.

Cool, i think i begin to have clearer understanding on the Atmos mindset … thnks !!

I just now tried something that seemed to work. Since I’m using the “{stage}” name pattern inside atmos.yaml, I set a “vars.stage: dev” inside my dev.yaml stack file and it seemed to do the trick. Is this a correct pattern? Thanks!

@Andriy Knysh (Cloud Posse)

Can you confirm that you see a valid zone_id as a terraform output of keycloak_route53_zones

and please make sure the component is provisioned

atmos.Component calls terraform output, so it must be in the state already

atmos terraform output keycloak_route53_zones -s aws-dev-us-east-1

gives me

zone_id = {
  "redacted" = "redacted"
}

which is redacted but it is the zone subdomain as key and the id as value

aha, so it’s returning a map for the zone id

ah!

so I guess it should be

zone_id: '{{ (atmos.Component "keycloak_route53_zones" .stack).outputs.zone_id.value }}'

value is not correct

you should get the value by the map key using Go templates

there the key is the "redacted" in your example

Thanks! I’ll test this out, I appreciate the help

It didn’t seem to work and for this case I just passed the zone id directly (like the actual value from the output). The error was that terraform complained that it should be less than 32 characters, which means that terraform understood my go template as an actual string and not a template. I run it with atmos terraform apply component -s stack after a successful plan creation

I passed the value directly because I don’t think the zone id will change in the future, but for other components this might be an issue for me. Maybe I should run it differently? I set the value exactly as suggested by Andriy

please send me your yaml config, i’ll take a look

Sure!

yes

settings is a free form map

its the same as vars, but free form, participates in all the inheritance, meaning you can define it globally, per stack, per base component, per component, and then everything gets deep merged into the final map

cool, that is awesome and exactly what I need

imagine if this was added to something like atmos.yaml and CODEOWNERS only allow very few people to modify it

like having Sec rules for all components

no, metadata is not inherited, it’s per component

ok

because in metadata you specify all the base components for your component

understood

Note that atmos never uses this backend. We generate a backend.tf.json file used by terraform

Could it be a GCP permissions issue

I am a organisation admin, have full access to the bucket, and besides Ive tested access to the bucket itself with cli.

Based on your screenshot the YAML is invalid.

the hint is it’s trying to use a “local” backend type.

https://atmos.tools/core-concepts/components/terraform/backends/#google-cloud-storage-backend

Note in your YAML above, the whitespace is off

Terraform is attempting to use this: https://developer.hashicorp.com/terraform/language/backend/local

which indicates that the backend type is not getting set

Thanks Erik. I’ve double checked again everything. I’ve tried to delete backend.tt.json file, disable auto_generate_backend_file and after that added backend configuration directly into module - same result. That got me thinking something is not right with auth into GCP. Re-logged again with gcloud auth application-default login and now state is migrated into the bucket. Honestly no idea, I’ve logged in multiple times this day already. Even before posting here

maybe this is the kind of tax for changing whole workflow to atmos

@jose.amengual from where you want to vendor all the components?

from my iac repo

to another repo ( using atmos)

let me check

I did this :

apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
  name: iac-vendoring
  description: Atmos vendoring manifest for Atmos-iac repo
spec:
  # Import other vendor manifests, if necessary
  imports: []

  sources:
      - source: "github.com/jamengual/pepe-iac.git/"
        #version: "main"
        targets:
          - "./"
        included_paths:
          - "**/components/**"
          - "**/*.md"
          - "**/stacks/sandbox/**"

I was able to vendor components just fine

but not sandbox stack files for some reason

you need to add another item for the stacks folder separately

that’s how the glob lib that we are using works

we don’t like it and will revisit it, but currently it’s what it is

you mean something like :

 sources:
      - source: "github.com/jamengua/pepe-iac.git/"
        #version: "main"
        targets:
          - "./"
        included_paths:
          - "**/components/**"
          - "**/*.md"
      - source: "github.com/jamengual/pepe-iac.git/"
        #version: "main"
        targets:
          - "./stacks/sandbox/"
        included_paths:
          - "stacks/sandbox/*.yaml"

try this ^

so this pulled all the stacks

       - "**/stacks/**"

but it looks like I can’t pull a specific subfolder

hmm, so this "**/stacks/sandbox/**" does not work?

no

i guess you can use

- "**/stacks/**"
- "**/stacks/sandbox/**"

and exclude the other stacks in excluded_paths:

I could do that….but it makes it not very DRY

can you verbose the pull command?

atmos vendor pull --verbose

Atmos 1.88.1 on darwin/arm64

oh sorry, try

ATMOS_LOGS_LEVEL=Trace atmos vendor pull

@jose.amengual also check out this example for using YAML anchors in vendor.yaml to DRY it up

https://github.com/cloudposse/atmos/blob/main/examples/demo-component-versions/vendor.yaml#L10-L20

could you pass an ENV variable as a value inside of the yaml?

like

 excluded_paths: 
          - "**/production/**"
          - "${EXCLUDE_PATHS}"
          - "${EXCLUDE_PATH_2}"
          - "${EXCLUDE_PATH_3}"

currently not, env vars in vendor.yaml are not supported

ok, Thanks guys

I was thinking

Option 1) a unique namespace via a separate dir

e.g.

# cloudposse components
components/terraform
# internal components
components/terraform/internal

Option 2) a unique namespace via a prefix

# upstream component
components/terraform/ecr
# internal component
components/terraform/internal-ecr

Option 3) Unique key in component.yaml and enforce this file in all components

Option 4) Vendor internal components from an internal repo

This way the source will contain the other org instead of cloudposse so that can be used to distinguish

We’re actually looking into something imilar to this right now related to our refarch stack configs

For stack configs we’ve settled on stacks/vendor/cloudposse

For components, I would maybe suggest

components/vendor/cloudposse

The alternative is a top-level folder like vendor/cloudposse

which could contain stacks and components.

Thats for stack configs, what about terraform components ? Or do you think it would be for both stack configs and terraform components?

What our team dose this: Vendor: components/terraform/vendor/{provider, ej. cloudposse} Internal: components/terraform/{cloudProvider, Ej. AWS}/{componentName, Ej. VPC}

Thats for stack configs, what about terraform components ? https://sweetops.slack.com/archives/C031919U8A0/p1728683594309549?thread_ts=1728675666.706109&cid=C031919U8A0ß https://sweetops.slack.com/archives/C031919U8A0/p1728683609704999?thread_ts=1728675666.706109&cid=C031919U8A0

Interesante, I was trying last week to use remote-state between two components with GCP backend and cannot make it, it forced me to re-write my module to switch from remote state to using outputs and atmos templating

This should work now as we merged the PR

Version 1.8.0 should give you full support for using remotes-state with any backend Terraform or OpenTofu supports.

@Jeremy G (Cloud Posse) were you also able to resolve that deprecated warning? I think you maybe mentioned you had an idea

Yes, 1.8.0 should fix deprecated warnings from remote-state. Or, more accurately, if receiving deprecation warnings from remote-state, they can now be resolved by updating your backend/remote state backend configuration to match the version of Terraform or Tofu you are using. For example, change

terraform:
  backend:
    s3:
      bucket: my-tfstate-bucket
      dynamodb_table: my-tfstate-lock-table
      role_arn: arn:aws:iam::123456789012:role/my-tfstate-access-role
  remote_state_backend:
    s3:
      role_arn: arn:aws:iam::123456789012:role/my-tfstate-access-read-only-role

to

terraform:
  backend:
    s3:
      bucket: my-tfstate-bucket
      dynamodb_table: my-tfstate-lock-table
      assume_role:
        role_arn: arn:aws:iam::123456789012:role/my-tfstate-access-role
  remote_state_backend:
    s3:
      assume_role:
        role_arn: arn:aws:iam::123456789012:role/my-tfstate-access-read-only-role

@kevcube

@Michael Dizon

relates to https://github.com/cloudposse/terraform-yaml-stack-config/pull/96 https://github.com/cloudposse/terraform-yaml-stack-config/pull/93

@Gabriela Campana (Cloud Posse) please create a task to fix this in our infra-test and infra-live repos

@Jeremy G (Cloud Posse) can you please suggest the task title?

@Gabriela Campana (Cloud Posse) Update backend configurations to avoid deprecation warnings and add a note in the task that says all remote-state modules must be updated to v1.8.0

DEV-2681: Update backend configurations to avoid deprecation warnings

I have my vendor.yaml

I tried git:// but then tried to use ssh git, and this is running from a reusable action

I tried this locally and it works but in my local I have a ssh-config

if I run git clone using that url it clones just fine

I’m hitting this error on the go-git library now

// <https://github.com/go-git/go-git/blob/master/worktree.go>

func (w *Worktree) getModuleStatus() (Status, error) {
    // ...
    if w.r.ModulesPath == "" {
        return nil, ErrModuleNotInitialized
    }

    if !filepath.IsAbs(w.r.ModulesPath) {
        return nil, errors.New("relative paths require a module with a pwd")
    }
    // ...
}

I do not know if this is because vendir sources on http are not compatible with git?

@Andriy Knysh (Cloud Posse) this is when I was trying to debug the pwd error

How log does it take to get the linux x64 package?

@Andriy Knysh (Cloud Posse)

@jose.amengual please use this release

https://github.com/cloudposse/atmos/releases/tag/v1.90.0

it’s already in the deb and rpm Linux packages

https://github.com/cloudposse/packages/actions/runs/11320822457

(release 1.89.0 was not published to the Linux packages b/c of some issues with the GitHub action)

awesome, thanks

@Malte would you mind reviewing this? https://github.com/cloudposse/atmos/pull/723

It addresses templating in vendor files. Let me know if something else would be helpful to add.

A bit late but: LGTM! And I just facepalmed because in hindsight the missing git:: Prefix is obious

Just for completeness sake: We actually use private repos as well (BitBucket but the idea is similar) and our Jenkins (don’t ask…) uses a small Git Credential Helper which pulls the variable from the env and outputs the required format (cf. https://git-scm.com/docs/gitcredentials)

I think is because this:

 sources:
      - source: "<https://x-access-token>:${{ secrets.TOKEN }}@github.com/pepe-org/pepe-iac.git"
        #version: "main"
        targets:

I can’t template inside the vendor file

I think its related to https://github.com/cloudposse/atmos/pull/712

Cc @Andriy Knysh (Cloud Posse)

@jose.amengual can you open an issue so I can tag the other author

@jose.amengual what is this token {{ secrets.TOKEN }}?

it’s a GH action token not Atmos Go template token?

this is a known issue with using Atmos templates with other templates intended for external systems (e.g. GH actions, Datadog, etc.)

since that PR enabled processing templates in all cases (even if the version is not specified as it was before), now Atmos processes the templates every time, and breaks on the templates for the external systems

we have a doc about that:

https://atmos.tools/core-concepts/stacks/templates/#excluding-templates-in-stack-manifest-from-processing-by-atmos

you have to do

{{`{{  ...   }}`}} 

there is no way around this

let me know if that syntax works for you

OHHHHHHHHH let me try that

ok, that seems to work but now I’m on this hell

Run #cd atmos/
  #cd atmos/
  # git clone [email protected]:pepe-org/pepe-iac-iac.git
  # git clone ***github.com/pepe-org/pepe-iac-iac.git 
  ATMOS_LOGS_LEVEL=Trace atmos vendor pull
  ls -l
  shell: /usr/bin/bash -e {0}
  env:
    ATMOS_CLI_CONFIG_PATH: /home/runner/work/pepe-iac/pepe-iac/atmos/config
    ATMOS_BASE_PATH: /home/runner/work/pepe-iac/pepe-iac/atmos
    GITHUB_TOKEN: ***
Processing vendor config file '/home/runner/work/pepe-iac/pepe-iac/atmos/vendor.yaml'
Pulling sources from '<https://x-access-token>:${{ secrets.TOKEN }} @github.com/pepe-org/pepe-iac-iac.git' into '/home/runner/work/pepe-iac/pepe-iac/atmos/atmos'
relative paths require a module with a pwd
Error: Process completed with exit code 1.

if I switch to a git url ( with ssh) this problem goes away and then I get permission denied because I do not have a key to clone that other repo in the org, which is expected

I was trying to use a PAT and change the url to https:// to pull the git repo

note: the git clone command with https in the scrip works fine, so I know my PAT works

you have a space after {{ secrets.TOKEN }} and before @

ohhh no, if that is the problem I quit

uff, that could have been embarrassing

so I changes the token for

- source: "<https://x-access-token:12345>@github.com/pepe-org/pepe-iac.git"

and I got bad response code: 404 which makes me believe that is trying to clone with the Token that I’m passing

@jose.amengual sorry, I don’t understand the whole use-case, so let’s review it step by step

before Atmos 1.90.0 release, this was working ?

<https://x-access-token>:${{secrets.TOKEN}}@github.com/pepe-org/pepe-iac.git 

I tagged you on the other thread I have, if you prefer that one that is cleaner

and the template {{secrets.TOKEN}} was evaluated by the GH action?

I tagged you on the other thread I have, if you prefer that one that is cleaner

this is the same issue

ok

The goal: clone pepe-iac repo which is on the same org ( pepe-org) from another repo ( the app repo)

the rendering of the token was definitely a problem and now that is resolved thanks yo your suggestion

<https://x-access-token>:${{secrets.TOKEN}}@github.com/pepe-org/pepe-iac.git. - was it working before?

no, it never did

never? even before Atmos release 1.90.0?

( using atmos)

correct, it did not worked before because I was not using the {{{{ … }}}} format

i don’t understand

you mentioned it was working before the Atmos 1.90.0 release

I thought it did because I was getting relative paths require a module with a pwd and I thought I had a ATMOS_BASEPATH problem

the error through me off to another direction, thinking the token was not an issue anymore

I broke something?

Then I upgraded to 1.90.0 and the error changed

let’s review this:

in vendor.yaml

this is what I have in vendor.yaml

apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
  name: iac-vendoring
  description: Atmos vendoring manifest for Atmos-iac repo
spec:
  imports: []
  sources:
      - source: "<https://x-access-token>:${{`{{secrets.TOKEN}}`}}@github.com/pepe-org/pepe-iac.git/"
        #version: "main"
        targets:
          - "atmos"
        included_paths:
          - "**/components/**"
          - "**/*.md"
          - "**/stacks/**"
        excluded_paths: 
          - "**/production/**"
          - "**/qa/**"
          - "**/development/**"
          - "**/staging/**"
          - "**/management/**"
          - "**/venueseus/**"

how does your GH action work? It needs to evaluate {{secrets.TOKEN}} and replace it with the value before atmos vendor pull gets to it

let’s focus on source: <https://x-access-token>:${{secrets.TOKEN}}@github.com/pepe-org/pepe-iac.git

ok

what I said beofre about {{{{secrets.TOKEN}}}} is not your use case

so

how does your GH action work? It needs to evaluate {{secrets.TOKEN}} and replace it with the value before `atmos vendor pull` gets to it

ohhh…

you should use

source: <https://x-access-token>:${{secrets.TOKEN}}@github.com/pepe-org/pepe-iac.git

and make sure the token is replaces in the GH action before atmos vendor pull is executed

that’s all that you need to check

this {{secrets.TOKEN}} is not Atmos Go template, it’s not related tpo Atmos at all

the GH action needs to evaluate it first

the token is available to the action as a secret

if I run on the action

git clone <https://x-access-token>:${{secrets.TOKEN}}@github.com/pepe-org/pepe-iac.git

before the atmos command, I can clone the repo( I’m just clarifying that the token works and is present)

I will change the vendor.yaml and try again

no no

the token ${{secrets.TOKEN}} will be replaced by the GH actions only if it’s in a GH workflow

the GH actions don’t know anything about that token in an external file vendor.yaml

I c what you mean

Is the token also available as an environment variable? Then you should (now) be able to do something like source: <https://x-access-token>:{{env "SECRET_TOKEN"}}@github.com/pepe-org/pepe-iac.git (note that there is no $ in this case, ie. that will be evaluated by atmos)

yes correct, thanks @Malte

ok, so within the GH workflow, I need a way to use the value of ${{secrets.TOKEN}}

my gh action is like this :

- name: Vendoring stack and components
        working-directory: ${{ github.workspace }}
        env:
          GITHUB_TOKEN: ${{ secrets.TOKEN }}
        run: |
           cd atmos/
           # git clone [email protected]:pepe-org/pepe-iac.git
           git clone <https://x-access-token>:${{secrets.TOKEN}}@github.com/pepe-org/pepe-iac.git 
           ls -l
           ATMOS_LOGS_LEVEL=Trace atmos vendor pull

in your GH action workflow file, you create an env variable from the secret, and then use the env variable in the vendor.yaml file

@Malte i might have jumped the gun. It might not be related to your PR, but thanks for jumping in!

(not related to @Malte PR at all)

@Erik Osterman (Cloud Posse) no worries, I assumed it was one of those cases where I broke some other weird legit use case of curly braces .-)

I think ``{{env “GITHUB_TOKEN”}` should do the trick now

instead of the secrets.TOKEN you mean?

instead of ${{secrets.TOKEN}, yes (ie. drop the $ as well

testing

ohhhh single brackets…..

ok I did this:

apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
  name: iac-vendoring
  description: Atmos vendoring manifest for Atmos-iac repo
spec:
  imports: []
  sources:
      - source: '<https://x-access-token>:{{env "GITHUB_TOKEN"}}@github.com/pepe-org/pepe-iac.git'
        #version: "main"
        targets:
          - "atmos"
        included_paths:
          - "**/components/**"

looks correct?

I got

Processing vendor config file 'vendor.yaml'
Pulling sources from '***github.com/pepe-org/pepe-iac.git' into 'atmos'
bad response code: 404

those stars, the token was replaced correctly?

GH action obscure tokens and such

I will add the token to the vendor file to just test it ( without using the ENV variable)

I wonder if go-getter supports this at all…

ok this is what I got :

Run cd atmos/
  cd atmos/
  # git clone [email protected]:pepe-org/pepe-iac.git
  git clone ***github.com/pepe-org/pepe-iac.git 
  ls -l
  ATMOS_LOGS_LEVEL=Trace atmos vendor pull
  shell: /usr/bin/bash -e {0}
  env:
    ATMOS_CLI_CONFIG_PATH: /home/runner/work/pepe-iac-service/pepe-iac-service/atmos/config
    ATMOS_BASE_PATH: /home/runner/work/pepe-iac-service/pepe-iac-service/atmos
    GITHUB_TOKEN: ***
Cloning into 'pepe-iac'...
total 32
-rw-r--r--  1 runner docker 15178 Oct 14 20:51 README.md
drwxr-xr-x 10 runner docker  4096 Oct 14 20:51 pepe-iac
drwxr-xr-x  2 runner docker  4096 Oct 14 20:51 config
drwxr-xr-x  4 runner docker  4096 Oct 14 20:51 stacks
-rw-r--r--  1 runner docker   760 Oct 14 20:51 vendor.yaml
Processing vendor config file 'vendor.yaml'
Pulling sources from '***github.com/pepe-org/pepe-iac.git' into 'atmos'
bad response code: 404

new token, I changed the secret in the repo, git clone command uses same token as

git clone <https://x-access-token>:${{secrets.TOKEN}}@github.com/...

and you can see it can clone

the vendor file has the same token but clear text now

Just to be sure: It doesn’t work with a cleartext password either? I quickly checked the go-getter source and it should work. Weird.

it does not

is it possible that the @ is being changes for something else?

I digged a bit. Can you try this please? `

source: 'https://{{env "GITHUB_TOKEN"}}@github.com/pepe-org/pepe-iac.git'

If that works I don’t understand why your plain git clone works unless maybe GH has some magic in their workflow engine to fix up this common error

Ah, wait. Why do you use secrets.TOKEN and not secrets.GITHUB_TOKEN? https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication

this is a reusable action, I’m passing the secret as an input

Ok, I must admit I have no idea why that isn’t working. go-getter should call more or less exactly the git clone command you execute. The only difference is that it will do a `

git clone -- <https://x-access-token>:${{secrets.TOKEN}}@github.com/pepe-org/pepe-iac.git

(note the double dash) but I doubt that this makes a difference…

I changed the git command to git clone – just to try it and it worked

so somehow with atmos this does not work

I even tried this:

git config --global url."<https://x-access-token>:${GITHUB_TOKEN}@github.com/".insteadOf "<https://github.com/>"

git command works, atmos 404s

https://github.com/hashicorp/go-getter/issues/449

So just to reiterate, atmos uses go-getter, which is what terraform uses under the hood, so the syntax should be the same.

Was this tried as well?

Adding the git::

that worked!!!

so if you want to avoid having to template the secret in the vendor.yaml you can do this :

git config --global url."<https://x-access-token>:${GITHUB_TOKEN}@github.com/".insteadOf "<https://github.com/>"

then run atmos vendor pull

super

so we need to use git:: if we provide credentials

and actaully, what you said above, if you execute the command git config --global url."<https://x-access-token>:${GITHUB_TOKEN}@github.com/".insteadOf "<https://github.com/>" first, then in vendor.yaml you don’t need to specify any credentials at all

and this works too

source: 'git::https://{{env "GITHUB_TOKEN"}}@github.com/pepe-org/pepe-iac.git'

but this does not

source: 'git::https://{{secrets.TOKEN}}@github.com/pepe-org/pepe-iac.git'

so the ENV is the way to go

this would be good to add to the docs

yep, this git::https://{{secrets.TOKEN}}@github.com/xxxxxxxx/yyyyyyyyy.git does not work b/c the template is not Atmos template, and the file is not a GH action manifest

@jose.amengual can you review https://github.com/cloudposse/atmos/pull/723

for those using github and wanted clone internal repos, you can do it without using org-level PAT and using Github App Tokens that expiry every hour, using this actions/create-github-app-token https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/generating-an-installation-access-token-for-a-github-app

Thanks for sharing!

Also, here’s how we’re using that action with GHE environments

https://github.com/cloudposse/.github/blob/main/.github/workflows/shared-auto-release.yml

you mean show commnad?

yea but im unsure how to daisy chain it in an atmos command against my s3 backend, a pbkac issue

my plans longer than the vscode buffer

Can you share the command you are running? We use -out in all our github actions and it works fine

yea apologies atmos terraform plan ec2-instance -s platform-platformprod-instance

i didnt play with it much to see if I can tee out at the end

I was expecting to see -out in your command

yea im just going to mess around today and read the cli guide, wasnt sure if I can just add an -out at the end with atmos

this is nested in a Github Action that Erik mentioned, but here’s an example https://github.com/cloudposse/github-action-atmos-terraform-plan/blob/main/action.yml#L218-L223

TLDR:

atmos terraform plan ${{ COMPONENT }} \
  --stack ${{ STACK }} \
  -out="${{ PLAN FILE }}" \

ooooh so thats how it gets the plan details into github? I honestly just took a different approach and increased my vscode buffers - i did get it to output without -out:\file\whatever, but it was in what looked like encrypted/plan format.

appreciate the multiple responses here

it was honestly something new to me, i hadnt run into a plan output that got cutoff in the console

storing plans in our github actions are entirely separate from running locally. In github actions, we store the plans with a combination of s3 and dynamodb, but locally we use native terraform - plans arent stored locally unless you specify so

no no sorry im not using correct technical terms

i mean the console output of whats going to happen

thats all i was struggling with, half my planned changes were cut off in the console so i couldnt tell if some stuff was going to be destroyed or not

im interested in actions btw but I am on an internal GHES server that I would have to work through compliance reqs to expose.

that output file is a planfile for Terraform. This is an special Terraform file that you can then pass to Terraform again with terraform apply to apply a specific plan. But you dont have to create a plan file to apply terraform. You can always run terraform apply without a planfile, then manually accept the changes

yea as soon as i saw the garbled text im like oh thats not what im looking for

An internal GHES should be fine for the actions too! But if you want to validate a planfile you can then use terraform show to see what that planfile includes. For example

terraform plan -out=tfplan
terraform show -json tfplan

ohhhhh thank you

cool to know

and yea with GHES but its internal and not exposed yet

and you dont need atmos to use terraform show. You could generate the planfile with the atmos command, and then just use native TF with show

im pretty sure the OIDC connection needs a name outside my perimeter

yup we use OIDC as well with our actions. Here’s a little bit about how that works to authenticate with AWS https://docs.cloudposse.com/layers/github-actions/github-oidc-with-aws/

the subdir regex does not work

I had to do that so I can only vendor the stacks/sandbox stack

the expression "**/stacks/sandbox/**" does not work

it would be nice if the **, could work at any level

@jose.amengual yes, in order to support **/stacks/sandbox/**, you have to also add **/stacks/** - this is a limitation of the Go lib that Atmos uses. But obviously you don’t want to use that b/c oyiu have to then exclude all other folders in excluded_paths. We’ll have to review the lib

This is in our backlog - I think we can probably get to it pretty soon

yes, please open a PR, thank you

https://github.com/cloudposse/github-action-atmos-affected-stacks/pull/54

and if you have time : https://github.com/cloudposse/github-action-atmos-terraform-apply/pull/62/ and https://github.com/cloudposse/github-action-atmos-terraform-plan/pull/90 that will allow people to use Azure

@Igor Rodionov

I think Igor was waiting for a review

this ones Erik

Ah yea, this just seemed untennable

We can’t keep that pattern. While it addresses the current problem it doesn’t scale code wise

if you can comment the pr and add your thoughts I could work on it if Igor does not have the time

https://github.com/cloudposse/github-action-atmos-terraform-plan/pull/90/files#r1805483290

like atmos terraform apply -all -s mystack

currently Atmos does not have that native command (we will add it).

You can do this using Atmos workflows (if you know all the components in the stacks) - this is static config

or you can create a custom Atmos command - this is dynamic. You can call atmos describe stacks --stack

or this :

atmos describe stacks -s sandbox --format=json|jq -r '.["sandbox"].components.terraform | keys[]'

and then use shell script and jq to loop over all components

yes, I will use that for now

@jose.amengual add that to a custom command

good idea

https://atmos.tools/core-concepts/custom-commands/

@Andriy Knysh (Cloud Posse)

So I don’t know that we would add something like atmos-integrations-scope, and we should probably move role assumption like this out of the action itself, as there are just too many ways it can work.

However, using the free-form map idea should still work for you, @jose.amengual

Then you can use this action to retrieve the roles https://github.com/cloudposse/github-action-atmos-get-setting

from the integrations section

the role, was just an example, for me the problem is: I have a bucket per account where I want to store the plans with the storage action

so I can’t use one github integration setting for all accounts

Integrations can be extended in stacks

…following inheritance

Pretty sure at least..

but the github actions do not do that, they look at the atmos.config.integrations

I think we just need to change this action slightly.

https://github.com/cloudposse/github-action-atmos-terraform-apply/blob/main/action.yml#L78

So today, it does this, like. you said:

    - name: config
      shell: bash
      id: config
      run: |-
        echo "opentofu-version=$(atmos describe config -f json | jq -r '.integrations.github.gitops["opentofu-version"]')" >> $GITHUB_OUTPUT
        echo "terraform-version=$(atmos describe config -f json | jq -r '.integrations.github.gitops["terraform-version"]')" >> $GITHUB_OUTPUT
        echo "enable-infracost=$(atmos describe config -f json | jq -r '.integrations.github.gitops["infracost-enabled"]')" >> $GITHUB_OUTPUT        
        echo "aws-region=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].region')" >> $GITHUB_OUTPUT
        echo "terraform-state-role=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].role')" >> $GITHUB_OUTPUT
        echo "terraform-state-table=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].table')" >> $GITHUB_OUTPUT
        echo "terraform-state-bucket=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].bucket')" >> $GITHUB_OUTPUT        
        echo "terraform-plan-role=$(atmos describe config -f json | jq -r '.integrations.github.gitops.role.plan')" >> $GITHUB_OUTPUT

correct

However, these are already required.

  component:
    description: "The name of the component to plan."
    required: true
  stack:
    description: "The stack name for the given component."
    required: true

So instead, we should be retrieving the integration config for the component in the stack.

Then it will do what you want.

or that yes

So most things in atmos.yaml can be extended in settings section of a component. Integrations is one of those.

I am not sure when @Igor Rodionov can get to it, but we could probably define it well enough so if you wanted to PR it, you could do it - since maybe it’s a blocker for you

the other PRs that Igor have inflight and mine to add the --stacks for describe-affected are a blocker for me

The problem is it should be using this get settings action.

( I mention them in the other thread)

that will handle all the lookups the right way

so you are suggesting to move away from using this

"terraform-state-role=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].role')" >> $GITHUB_OUTPUT

to use github-action-atmos-get-setting? are you ok with breaking changes, or I keep the current use of atmos describe config and I add the a new section to use the github-action-atmos-get-setting?

@Erik Osterman (Cloud Posse)

I am not sure that it would be breaking

I think it would work the same way, but add the ability for configurations to be inherited

@Igor Rodionov any ideas?

Have you set the atmos-config-path

?

See https://github.com/marsadle/github-action-atmos-terraform-apply?tab=readme-ov-file#migrating-from-v1-to-v2

Yep, atmos-config-path is set and I verified it’s the correct value.

This is the complete action execution:

Run cloudposse/github-action-atmos-terraform-apply@v2
  with:
    component: aurora-postgres
    stack: plat-euw1-prod
    sha: c60fd18fb31bea65c8bf5975913940623c3d98c6
    atmos-config-path: ./rootfs/usr/local/etc/atmos/
    atmos-version: 1.90.0
    branding-logo-image: <https://cloudposse.com/logo-300x69.svg>
    branding-logo-url: <https://cloudposse.com/>
    debug: false
    token: ***
  env:
    AWS_CLI_VERSION: 2
    AWS_CLI_ARCH: amd64
    VERBOSE: false
    LIGHTSAILCTL: false
    BINDIR: /usr/local/bin
    INSTALLROOTDIR: /usr/local
    ROOTDIR: 
    WORKDIR:

and the error message

exec: "terraform": executable file not found in $PATH

exec: "terraform": executable file not found in $PATH

Could this be related? read PR description https://github.com/cloudposse/atmos/pull/717

I don’t think so, since terraform shouldn’t be called at all, tofu should.

When I was running this on GitHub-hosted runners (but apparently that has changed according to the PR), I was actually getting a different error that makes more sense now and it was about schemas not being found because they were using the OpenTofu registry (registry.opentofu.org) but the plugins were downloaded from registry.terraform.io.

I can also see the difference between plans (OpenTofu has successfully been initalized) and applies (Terraform has successfully been initialized) in the output, now that I know what to look for.

Edit: going to rerun with ATMOS_LOGS_LEVEL=Trace and report back.

Okay, I feel stupid now. :face_palm: Apparently my two atmos configs (rootfs/... in the base directory) were out of sync and that caused the issues, because they are obviously not merged. I guess I misunderstood something in the last couple of years and not re-read the docs properly. Anyways, thanks for your help, Erik!

What still baffles me though, is that the -plan action worked perfectly fine.

Hrmmm why do you have 2 atmos configs?

I’ve just followed this: https://github.com/cloudposse/atmos/tree/main/examples/quick-start-advanced

To be honest, I’ve seen these 2 configs since the early examples in the Atmos repo and always thought that this was a common thing, so I never questioned it.

If it would help, checkout could be feature flagged

ok, I will create a PR for that

it does help but I will need to add a restore cache step too

using the same flag

basically something like this :

 - name: Checkout
      uses: actions/checkout@v4
      with:
        ref: ${{ inputs.sha }}
    
    - name: Restore atmos files cache
      uses: actions/cache/restore@v4
      with:
        path: atmos
        key: ${{ runner.os }}-atmosvendor
    

the checkout needs to exist for the job to see the files of the repo in the new runner

Are you also using {{ atmos.Component ...}} ?

Yes

I’m using some components that are depending on the vpc id of the vpc component

Ok, I think that could be a scenario not thoroughly tested.

We are also working on a replacement of the {{ atmos.Component ...}} implementation, which is problematic for multiple reasons, chief among them is performance, since all must be evaluated everytime - and since this is handled by the go template engine at load time, it’s slow. We’re working on a way to do this with YAML explicit types, and lazy loading.

However, I’m really glad that you bring this use-case up, as I don’t believe we’ve considered it. The crux of the problem is the component is invoked nearly concurrently in multiple “init” contexts, so we would need to copy it somewhere so it can be concurrently initialized with different backends.

So I can understand why you want to do this, and why it’s needed, but we need to think about how to do this. It’s further complicated by copying, since a root module (component), can have relative source = "../something" module statements

I thikn the problem may be mititgated by using terraform data sources (in HCL) instead of atmos.Component

@Andriy Knysh (Cloud Posse) for consideration…

I will try it with a datasource. Thanks for your explanation

Just another question, do you already have something in mind to replace the templating with? Just another templating or remote-state?

yaml explicit type functions - will probably be released this week, and will be used to define Atmos functions in YAML

the functions will include getting a terraform output from a remote state, getting secrets from many different systems, etc

Hey, just a quick catchup. This is not released yet, is it?

We have the branch published but the PR is not yet ready

We discussed this on Friday internally, and are considering introducing feature flags - as this should be considered experimental until we understand all the implications and edge cases.

We learned a lot from the template functions, and the way it was used was not the way we used it ourselves :-)

Ah, good to know. I would gladly appreciate such feature flag and try it out

I anticipate we should be able to get this wrapped up this week, but defer to @Andriy Knysh (Cloud Posse)

@Dennis Bernardy are you suing the latest Atmos release https://github.com/cloudposse/atmos/releases/tag/v1.100.0 ?

Please try it out and let me know if it fixes the issue with multiple backends

Clean Terraform workspace before executing terraform init. When using multiple backends for the same component (e.g. separate backends per tenant or account), and if an Atmos command was executed that selected a Terraform workspace, Terraform will prompt the user to select one of the following workspaces:

 1. default
 2. <the previously used workspace>
The prompt forces the user to always make a selection (which is error-prone), and also makes it complicated when running on CI/CD.

The PR adds the logic that deletes the .terraform/environment file from the component directory before executing terraform init. The .terraform/environment file contains the name of the currently selected workspace, helping Terraform identify the active workspace context for managing your infrastructure. We delete the file before executing terraform init to prevent the Terraform prompt asking to select the default or the previously used workspace.

Looks like the same error still:

template: describe-stacks-all-sections:56:24: executing "describe-stacks-all-sections" at <atmos.Component>: error calling Component: exit status 1

Error: Backend configuration changed

A change in the backend configuration has been detected, which may require
migrating existing state.

If you wish to attempt automatic migration of the state, use "terraform init
-migrate-state".
If you wish to store the current configuration with no changes to the state,
use "terraform init -reconfigure".

I see you are using atmos.Component (and multiple backends). This case is not solved yet, we know this is an issue and discussed it (will have a solution for that prob this week)

we will probably have a solution for that today, but no ETA

Ok, I’m looking forward to it

Just bumping this up, in case anyone knows something about it. I get a map[] when I don’t specify a key so I assume it’s correctly accessing Vault? But it’s weird that the map is always empty

How do you access that value?

Hey, I’m trying to get it like this:

- name: KC_SPI_REALM_RESTAPI_EXTENSION_SCIM_LICENSE_KEY
  value: '{{ (datasource "vault-dev" "infrastructure/keycloak").KC_SPI_REALM_RESTAPI_EXTENSION_SCIM_LICENSE_KEY }}'

And this is part of a ECS “environment” list

This works, for example:

gomplate -d vault=vault+http:/// -i '{{(datasource "vault" "infrastructure/keycloak").KC_SPI_REALM_RESTAPI_EXTENSION_SCIM_LICENSE_KEY }}'

Given the proper vault token and addr are set

And I’ve set the vault-dev data source in my atmos.yaml

And I’m setting an environment variable in my terminal with VAULT_TOKEN before running atmos

Ah. That looks like a different approach then ours. I only noticed recently that if you use .atmos.component to template outputs from a module you can not get lists or maps. Maps specifically only return no_value. Maybe that limitation also applies here and templating in general only is supported for strings, but that is better confirmed by someone as this is just a suspicion

Hmm, this is interesting, the KC_SPI… var should be a string, if I try to get the whole secret path, like not specifying the ).KC... part, I get a “map[]”, which seems to be an empty map. The worst part is not knowing what I’m doing wrong lol

I only noticed recently that if you use .atmos.component to template outputs from a module you can not get lists or maps. See this workaround

https://sweetops.slack.com/archives/C031919U8A0/p1727909658480679?thread_ts=1726147975.215489&cid=C031919U8A0

Also, we’re working on a better long-term solution using YAML explicit types. ETA this week.

This is interesting, do you think this applies to my issue with the vault gomplate data source?

Hrmm… probably less so, since you’re trying to retrieve a string.

Have you tried enabling ATMOS_LOG_LEVEL=Trace

It might shed more light

Are you able to share your datasource configuration in atmos.yml for

vault-dev

I currently have Trace set in my atmos.yaml, it doesn’t output any errors. Yes, sure:

templates:
  settings:
    enabled: true
    gomplate:
      enabled: true
      datasources:
        vault-prod:
          url: "vault+<http://vault.companydomain:8200>"
        vault-dev:
          url: "vault+<http://vault-dev.companydomain:8200>"

gomplate -d "vault-dev=vault+<http://vault-dev.companydomain:8200/>" -i '{{ (datasource "vault-dev" "infrastructure/keycloak").RANDOM_SECRET_KEY }}'

works on the same terminal, in case it helps

Can you give an example

I can get this no problem

 - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: settings.integrations.github.gitops.role
            outputPath: integrations-gitops

but then I thought about doing this :

import:


vars:
  location: west
  environment: us3
  namespace: dev

settings:
  github:
    actions_enabled: true
  integrations:
    github:
      gitops:
        region: "westus3"
        role: "arn:aws:iam::123456789012:role/atmos-terraform-plan-gitops"
        table: "terraform-state-lock"

components:
   terraform:
    ......

but I do not think that will work since the stack yaml is not free form as I understand it

But that is then inherited by all components, so no need to retrieve it from anywhere else

but this is on a stack file

Thats fine

this goes back to my questions about

 echo "aws-region=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].region')" >> $GITHUB_OUTPUT
        echo "terraform-state-role=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].role')" >> $GITHUB_OUTPUT
        echo "terraform-state-table=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].table')" >> $GITHUB_OUTPUT

Its still inherited right?

for the plan action

integration.config, lives in atmos.yaml ( we could call that global)

if I add overwrite the settings at the component level then I can retrieve it like this :

- component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: settings.integrations.github.gitops.role
            outputPath: integrations-gitops

from

cosmosdb:
      settings:
        github:
          actions_enabled: true
        integrations:
          github:
            gitops:
              region: "westus3"
              role: "arn:aws:iam::123456789012:role/atmos-terraform-plan-gitops"
              table: "terraform-state-lock"

I am on my phone so I cannot give examples

I was trying to avoid to have to do it per component

The point ianthst we should not be retrieving the value from integrations, we should be using the settings action but we are not

You are only applying 1 component at a time, so this is ideal - use settings action

I am pretty sure everything in atmos integrations is just deepmerged into .settings

@Igor Rodionov should not be calling jq on values in the atmos config, but instead using the final value. Then it works both the way you write it here, with putting it in a stack config, as well as putting it in atmos yaml

understand

I started implemented your suggestion as follows:

     - name: Get atmos settings
      uses: cloudposse/github-action-atmos-get-setting@v2
      id: component
      with:
        settings: |
          - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: settings.github.actions_enabled
            outputPath: enabled
          - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: component_info.component_path
            outputPath: component-path
          - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: atmos_cli_config.base_path
            outputPath: base-path
          - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: command
            outputPath: command
          - component: ${{ inputs.component }}
            stack: ${{ inputs.stack }}
            settingsPath: settings.integrations.github.gitops.role
            outputPath: integrations-gitops

and use that to get the settings for the gitops integration

On my phone that looks like the right trajectory

and with glasses?

I would have to compare what was there originally, with this new one and my mental cache scrolling up and down (or late night willpower) is depleted. The real question is, is that working? If so, I think its probably good

yes, it works

Sweet! Lets have @Igor Rodionov review.

Cc @Gabriela Campana (Cloud Posse)

I will add a few more things and I will create a PR

Thanks @jose.amengual !

https://github.com/cloudposse/github-action-atmos-terraform-plan/pull/92, needs doc changes but I want to see if that seems ok so far

@Igor Rodionov up

@jose.amengual why do we need to cache the whole repo while we can check it out?

not the whole repo, but just the stacks folder

I have been testing a lot , so I tried many things

somehow the cache is not getting created

what is the profit? hit rate would be low as key based on the sha

the problem is when using vendor files or in this case Templates stack files that are changed in another step but not commited to the repo

Any file part of the repo (in git) will be overwrited by action/checkout after the sed command runs

same happens with any untracked files

could you provide example of untracked files?

untracked files is a problem I have when I use atmos vendor since I vendor stacks files from another repo

for the test github-action-atmos-terraform-plan something similar happens

Keep in mind that Erik wanted to move the integration settings from atmos.yaml to each component

why do not you commit vendor dir into the repo?

we have vendoring to terraform components and commit them

let’s use the same for stacks vendoring

Lets focus on the test issue because if we solve that problem, my usecase will be solved too

sure

what you need to is

rollback changes like htis

https://github.com/cloudposse/github-action-atmos-terraform-plan/pull/92/files#diff-984de38d283d4b13e1884a9ae42013408db6a33688c4a3fdeaf3b9e255115e28

I suppose you would have to get rid of this part

here

https://github.com/cloudposse/github-action-atmos-terraform-plan/blob/f68610870bf040804c7e73a6af2e64d26fe1c798/tests/terraform/atmos.yaml#L67-L89

and replace it with

and add here

https://github.com/cloudposse/github-action-atmos-terraform-plan/blob/f68610870bf040804c7e73a6af2e64d26fe1c798/tests/terraform/atmos.yaml#L20

and so on

can you have settings at that level?

@Andriy Knysh (Cloud Posse) would be suggestion works ?

and what do we do with those files after the template runs ? this part here

 sed -i -e "s#__INFRACOST_ENABLED__#false#g" "$file"
            sed -i -e "s#__STORAGE_REGION__#${{ env.AWS_REGION }}#g" "$file"          
            sed -i -e "s#__STORAGE_BUCKET__#${{ secrets.TERRAFORM_STATE_BUCKET }}#g" "$file"
            sed -i -e "s#__STORAGE_TABLE__#${{ secrets.TERRAFORM_STATE_TABLE }}#g" "$file"          
            sed -i -e "s#__STORAGE_TABLE__#${{ secrets.TERRAFORM_STATE_TABLE }}#g" "$file"
            sed -i -e "s#__STORAGE_ROLE__#${{ secrets.TERRAFORM_STATE_ROLE }}#g" "$file"
            sed -i -e "s#__PLAN_ROLE__#${{ secrets.TERRAFORM_PLAN_ROLE }}#g" "$file"
            sed -i -e "s#__APPLY_ROLE__#${{ secrets.TERRAFORM_PLAN_ROLE }}#g" "$file"

@jose.amengual it seems my suggestion would not work

let me think about this

that is why I added the cache

in my fork, this works just fine

I got the problem, I’ll come back with the workaround tomorrow. It is a night in my time now and I’m too tied to find the right solution. But caching still looks not good for me as it’s breaks gitops conception

the caching is optional in my code for that reason, but let me know tomorrow

to explain the problem and scenarios: I will fully explains the use cases: