#atmos (2024-09)

@Andriy Knysh (Cloud Posse) @Igor Rodionov

@Miguel Zablah could you pls send me full github actions log?

@Igor Rodionov I will send it via DM

could you try to run locally

 atmos terraform output iam --stack data-gbl-dev --skip-init -- -json

Is there anything suspicious output

?

1 sec

nop it looks good

is it correct json ?

yes it looks as correct json I will run jq to see if it fails there but it looks good

I see the error I think

jq type fails

it’s bc that cmd will still echo the switch context

and then it will give the json

for example:

Switched to workspace "data-gbl-dev-iam".
{
...
}

Hm.. what version of atmos are you using?

1.88.0

let me update to latest

Yea. Try pls

I have meeting now

Will be back in 1 hour

no worries I will put the update here in a bit

@Miguel Zablah in your atmos.yaml, update the logs section to this

logs:
  # Can also be set using 'ATMOS_LOGS_FILE' ENV var, or '--logs-file' command-line argument
  # File or standard file descriptor to write logs to
  # Logs can be written to any file or any standard file descriptor, including `/dev/stdout`, `/dev/stderr` and `/dev/null`
  file: "/dev/stderr"
  # Supported log levels: Trace, Debug, Info, Warning, Off
  # Can also be set using 'ATMOS_LOGS_LEVEL' ENV var, or '--logs-level' command-line argument
  level: Info

/dev/stderr tells Atmos to log messages to /dev/stderr instead of /dev/stdout (which pollutes the output from terraform)

same issue with the latest version of atmos

and I have /dev/stderr

it’s not related to Atmos version, please check the logs section

I see okay let me change this

if yiu have already stderr, then it’s not related. This output is from TF itself

Switched to workspace "data-gbl-dev-iam"

(not controlled by Atmos)

I see so what about doing a sed '1d' before saving it to file maybe that will fix it

@Miguel Zablah I suppose 1d is color special char that is part of “switch ” output

@Miguel Zablah can you try full command

atmos terraform output iam --stack data-gbl-dev --skip-init -- -json 1> output_values.json

1d is just means delete first line haha

what do you have in json file?

yeah that will saved all the output including the first line: Switched to workspace "data-gbl-dev-iam".

I think we can maybe validate it instead

so you have this: https://github.com/cloudposse/github-action-atmos-terraform-apply/blob/main/action.yml#L332

atmos terraform output ${{ inputs.component }} --stack ${{ inputs.stack }} --skip-init -- -json 1> output_values.json    

we can add some steps here like this:

atmos terraform output ${{ inputs.component }} --stack ${{ inputs.stack }} --skip-init -- -json 1> tmb_output_values.json

cat tmb_output_values.json | (head -n 1 | grep -q '^[{[]' && cat tmb_output_values.json || tail -n +2 tmb_output_values.json) 1> output_values.json

this will ensure that if the value dose not start with { or [ it will delete the first line

the case is that we do not have the problem in our tests

are your test running the latest atmos version?

so I want to find the reason instead quick fixes

1.81.0

let me downgrade to that version to see if this is new

https://github.com/cloudposse/github-action-atmos-terraform-apply/actions/runs/10460642342/job/28967242146

and yeah I think your right lets find that out

this is our tests

same issue hmm

let me check your test

@Igor Rodionov this test is using openTofu right?

we have both

oh let me check the tf I think that one is opentofu

terraform and opentofu

ah I see them sorry

haha

okay I’m using a different tf version, your using: 1.5.2

the strange thing is that when you do the plan on this test you have the: Switched to workspace "plat-ue2-sandbox". but on apply I don’t see that

that’s fine

@Igor Rodionov @Andriy Knysh (Cloud Posse) I have copy the action to my repo and added this 2 lines and this makes it work:

atmos terraform output ${{ inputs.component }} --stack ${{ inputs.stack }} --skip-init -- -json 1> tmb_output_values.json

cat tmb_output_values.json | (head -n 1 | grep -q '^[{[]' && cat tmb_output_values.json || tail -n +2 tmb_output_values.json) 1> output_values.json

can we maybe add this fix to the action? idk why for me it’s giving the msg Switched to workspace "plat-ue2-sandbox". have you guys configure the /dev/stderr to write to a file? bc by default it will aslo write to terminal or maybe if your running it docker it treats this differently?

@Igor Rodionov bumping this up

https://github.com/cloudposse/atmos/issues/170

As you may know, we predominantly use geodesic. Anyone open to working with us to implement it? I can create the asdf-atmos repo

yep, I even use it myself. I use geodesic as a base image of my custom Docker image. AFAIK, there are some packages installed already in it, like OpenTofu. the others can be installed using cloudposse/packages repository - that’s probably the easiest way. however, that’s somehow limited to the set of packages that this repository offers. I started to explore the idea of installing only mise (tool like asdf) and then, copying .mise.toml config file from my repository into it. that way mise can install all the dependencies via plugins. Renovate supports mise, too, which makes upgrades collected within .mise.toml file smooth and very transparent.

unless I’m not aware of something and trying to reinvent the wheel here

Understood, it can definitely be layered on. @Michal Tomaszek do you have experience writing plugins for asdf?

I do! I can help with this if Michael doesnt have the time

sadly, I have no experience with plugins for asdf. however, I’m happy to contribute under someone’s guidance.

@RB i created this from their template, and modified it

https://github.com/cloudposse/asdf-atmos

But I can’t seem to get it working.

cc @Michal Tomaszek

nevermind, it works

just had to run asdf global atmos latest after installing it

Cool! I’ll check this out tomorrow. Thanks Erik

https://github.com/asdf-vm/asdf-plugins/pull/1042

Or even a GitHub Action that runs atmos validate stacks before other actions run?

@Andriy Knysh (Cloud Posse)

There’s the setup atmos action, and after that you can just run any atmos command like validate stacks

The precommits would be smart, but I don’t believe we have any documentation on that

I think we should have a page on setting up precommiti hooks here https://atmos.tools/core-concepts/projects/

@Andriy Knysh (Cloud Posse) do we need a task for that?

Feel free to vendor this action we created for implementing this functionality if you think it’ll benefit the community:

name: "Atmos Validate Stacks"

on:
  pull_request:
    paths:
      - "stacks/**"
      - "components/**"
    branches:
      - main

env:
  ATMOS_CLI_CONFIG_PATH: ./rootfs/usr/local/etc/atmos

jobs:
  validate_stacks:
    runs-on: [Ubuntu]
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Extract ATMOS_VERSION from Dockerfile
        run: |
          version=$(grep 'ARG ATMOS_VERSION=' Dockerfile | cut -d'=' -f2)
          echo "atmos_version=$version" >> "$GITHUB_ENV"

      - name: Setup Atmos
        uses: cloudposse/github-action-setup-atmos@v2
        with:
          atmos-version: ${{ env.atmos_version }}
          install-wrapper: false

      - name: Validate stacks
        run: atmos validate stacks

thanks @Michael Rosenfeld

I believe so

cc @Igor Rodionov

@Matt Calhoun

Yes, as without that you won’t be able to do an exact “apply” of what was planned.

@Andrew Ochsner are you not seeing this behavior?

well i assumed it didn’t because our pipeline team built a pipeline that saves the plan then saves the hcl lockfile separately (via save plan… so they override the path)… any chance you can point me to the code that does it? or i can go back and validate w/ the team

So… it appears we leveraged mostly from github-action-atmos-terraform-plan which saves the lock files as a separate step (implying the storePlan doesn’t by default, just the plan file that you give it). So i’m back to my connundrum of the getPlan not overwriting the lock file if it exists (because it’s committed to my repo)https://github.com/cloudposse/github-action-atmos-terraform-plan/blob/main/action.yml#L276-L299

@Matt Calhoun is OoO and won’t have a chance to look at it until next week

ok

@Igor Rodionov

@Matt Calhoun

The reason we are checking that is that we don’t expect that a plan file with the same name as the one we are retrieving to exist on disk at that point, so if it does, we’re considering that an error condition. We could consider adding flag for overwrite, but could you provide the use case for this so we can think it through a little bit more? Thanks!

sure.. it’s a little bit related to the question above…. we have a savePlan that saves the plan and then another savePlan that saves the lockfile. I didn’t write it so id ont’ know if the first savePlan actually saves the lock file.. if so this is all moot.

it was the 2nd get plan for the lock file that we run into issues… and i definitely want it to overwrite whatever lock file is there and use the one that was generated or existed when the plan was built

so i think i don’t care about this as much so long as the save plan saves the lock file then we can get rid of the extra step somoene put in there

(@Andrew Ochsner as a side note, just wanted to make sure you knew that the github-action-terraform-plan-storage action was written to support multiple clouds, with AWS and Azure presently implemented. It’s a pluggable implementation, so if you’re doing this for GCP, it would be ideal to contribute it back to this action)

The azureblob, and s3 implementations are here https://github.com/cloudposse/github-action-terraform-plan-storage/tree/main/src/lib

oh 100% would… we’re leveraging azure plans but storing in s3 buckets lol

Example error -

╷
│ Error: invalid stack manifest 'mgmt/demo/root.yaml'
│ yaml: unmarshal errors:
│   line 83: mapping key "s3_bucket_name" already defined at line 77
│
│   with module.iam_roles.module.account_map.data.utils_component_config.config,
│   on .terraform\modules\iam_roles.account_map\modules\remote-state\main.tf line 1, in data "utils_component_config" "config":
│    1: data "utils_component_config" "config" {
│
╵
exit status 1

VSCodes even pointing at the line and im like its not there lol.

Ok I got it back to functioning again. Essentially I had many old template files in where team members had put some double variables in the template. In the past it didn’t seem to care, but today it was finding everywhere one of those was and forcing me to correct. Not a huge issue other than I wish I know why it decided to do that now.

@Ryan can you please confirm you don’t need any assistance here?

I am good now, apologies. Really strange come back from a few weeks OOO. Thanks for checking.

You’d need to go into the base component directory switch to the correct workspace Dump the state create a new workspace switch to the new workspace push the state

You can also override the workspace name in the stack configurations to maintain the original workspace

Great info, thanks!

I recently found myself in the same position, and I advocated to start with a fresh org/set of accounts. There is documentation in Atmos on Brownfield but if there is an option to rebuild/migrate, it might be easier to go down this road.

Take a look at the following prs. This is how I’ve personally gotten this to work.

• https://github.com/cloudposse/terraform-aws-components/pull/945 - account map • https://github.com/cloudposse/terraform-aws-components/pull/943 - account These prs weren’t merged in based on Nurus comment. I haven’t had the time to address the needs in a separate data source component. On my eventual plate. But if you feel ambitious, please feel free to address it so we can have a maintained upstream component for this.

For the account component, i used the yaml to define my accounts which gets stored in its outputs. Usually this component will create the accounts too. (The original reason why i didn’t want to create a new component is because the logic of this component is fairly complex and i didn’t want to copy that logic and have it break between the existing and new flavor of the account components). This is a root level component meaning only super user can apply it.

the account-map is also a root level component and it reads the output of the account component and has its own set of outputs. These are read by all the downstream components and can be read by any role.

These flavors above are as-is and unofficial from me so if you decide to use, please use with caution

please review this doc https://atmos.tools/core-concepts/components/terraform/remote-state#atmos-configuration

when using remote-state, your atmos.yaml will not work if places in the root of the repo (since Terraform executes the providers from the component directory)

you need to place atmos.yaml into one of the known paths

noted & thanks @Andriy Knysh (Cloud Posse) Let me check that.

Hi @Andriy Knysh (Cloud Posse) I fixed this issue, thank you!

@Andriy Knysh (Cloud Posse)

@Dan Miller (Cloud Posse) can you please take a look at this ?

Yes that shouldve been included on that page, but it’s missing. Here’s a link to our Cloud Posse reference architecture that has the same workflows

https://docs.cloudposse.com/layers/gitops/example-workflows/#atmos-terraform-plan-matrix-reusable

ah thanks! but I think this is outdated the atmos-terraform-drift-detection

dose not work for me

I have solve the issue with this actions:

cloudposse/github-action-atmos-terraform-select-components

that is the same that @Igor Rodionov and @Yonatan Koren help solve for plan and apply here: https://sweetops.slack.com/archives/C031919U8A0/p1724845173352589

then the matrix for the plan that that job outputs is not valid when doing this plan

The latest version of all workflows are on that same page I linked above. This page is automatically updated with the latest version, whereas atmos.tools is manually updated with a point in time version. I recommend using the docs.cloudposse version for now, and I will update atmos.tools

https://docs.cloudposse.com/layers/gitops/example-workflows/

yeah I think this needs some update this workflow is not working

hi @Dan Miller (Cloud Posse) Do we have any document for Gitlab CICD?

@Dan Miller (Cloud Posse) with the docs on the workflow that you provided I was able to make the drift detection work, thanks!

hi @Dan Miller (Cloud Posse) Do we have any document for Gitlab CICD? unfortunately no, we only use GitHub at this time

Thank you!

Some components do not have the variables defined. Not by design, but just we hadn’t needed them before. Feel free to open a PR against the components repo to add the missing variables and pass them through to the underlying module(s)

@Erik Osterman (Cloud Posse) You mean I had to customize the component to suit my needs. Right?

You’d need to modify the component to expose the inputs of the module with copied and pasted variables.

E.g. ec2 instance module has a variable for ami so the ami variable needs to be set in 2 additional places, one at the module level within the component and it would need a variable definition

In component main.tf

# in component
module "ec2" {
  # ...
  ami = var.ami
  # ...
}

And in component variables.tf

variable "ami" {
  default = null
}

Now the input can be exposed to the Atmos yaml interface

components:
  terraform:
    ec2-instance/my-fav-singleton:
      # ...
      vars:
        ami: ami-0123456789

For one of the components, rds, i recall copying all the variables from the upstream module directly into the component (with convention {component}-[variables.tf](http://variables.tf)) and exposing each input in the module definition within the component which worked fairly well

https://github.com/cloudposse/terraform-aws-rds/blob/main/variables.tf

https://github.com/cloudposse/terraform-aws-components/blob/main/modules/rds/rds-variables.tf

You mean I had to customize the component to suit my needs. Right? Yes, although if it’s just adding some variables I think it’s suitable to upstream for everyone

I think the method @RB is also a good technique, similar to monkeypatching or swizzling.

https://developer.hashicorp.com/terraform/language/files/override

See https://atmos.tools/best-practices/terraform/#use-terraform-overrides-to-extend-monkey-patch-vendored-components

Thanks @RB, @Erik Osterman (Cloud Posse) If I modify config in components like [main.tf](http://main.tf) and [variables.tf](http://variables.tf) , It works for me but If I run the command atmos vendor pull in the future, It will overwrite my change.

Yes that’s true. You could modify the component locally, make sure it works, if it does, upstream it

It works for me but If I run the command atmos vendor pull in the future, It will overwrite my change.
Unless you use terraform overrides pattern

That is pretty cool how the override will just merge the definitions together. So basically you can do something like this, is that right?

# components/terraform/ec2-instance/main_override.tf
variable "root_volume_type" {}

module "ec2_instance" {
  root_volume_type = var.root_volume_type
}

Strange that that page mentions almost every block except module

I think it works with module blocks

But even ChatGPT says I’m wrong, so

Let’s assume it does work, for convenience, would you folks be open to PRs for all components to follow a convention like [variables.tf](upstream-module>-<http://variables.tf) ?

Not following. From my POV, the whole point of overrides is you have full control and we don’t need a strict convention persay.

Or in otherwords, the point is not to upstream override files

So let’s use the component ec2-instance as an example. We know that root_volume_type is missing from the component variables.tf file but exists in the upstream module terraform-aws-ec2-instance.

In the rds component example, we use [rds-variables.tf](http://rds-variables.tf) which is a copy of its upstream module’s variables.tf file. After implementing that change, the rds components variables.tf file only contains the region and other component-specific variables.

We can do the same for ec2-instance and other components too. This way the override method could be used as a stop-gap until all components are updated.

One more benefit of this approach is that cloudposse can also have a github action that monitors the upstream module so when its inputs are modified, it can auto copy those inputs directly into the component itself.

Difficulties here

• adding the input directly to the module definition, however, this can also be remediated using something like hcledit.

• If multiple upstream modules are used in a component, such as eks/cluster, then a copied file would also need to have each input var prefixed with the module name to avoid collisions

friendly ping ^

Not sure we will invest in automating those kind of updates. That said we plan to support JIT vendoring of modules, and using the existing support for provider and backend generation, you will be able to provision modules directly as components with state.

Not exactly the same thing, but means fewer custom components are required

Oh wow so a lot of the components can be deleted if all they do is wrap an upstream module.

But there are some components that wrap a single module but also have some component-logic under the hood. I imagine those components would still be retained unless their logic gets some how moved into its upstream module

I think there might still need to be a stop gap in place for components that use multiple modules or for components that don’t have logic that can be easily moved into the modules it wraps, no?

We will continue to mange/develop components, but adopt a hybrid model

For the modules that still need to be developed, would y’all consider the {upstream-module}-[variables.tf](http://variables.tf)?

I still don’t follow why such a convention is necessary. Can you elaborate? What problem is it addressing

Ah sorry, let me think of what i can add to https://sweetops.slack.com/archives/C031919U8A0/p1726067246212129?thread_ts=1725969302.837589&cid=C031919U8A0

It’s just all about scalability. If you have anything thats not DRY so in this case copying vars from module to component, would require a system to keep these vars in line. So if we maintain separate files, as opposed to a single variables.tf, between component vars and module vars then we can simplify upgrading to a newly exposed var by downloading the modules vars into the spec {module}-[variables.tf](http://variables.tf)

So instead of one component variables.tf file that contains

# e.g. component vars
variable region {}

# e.g. module vars
variable "engine_version" {}

So for example, we put var region in [variables.tf](http://variables.tf) and put var engine_version in [rds-variables.tf](http://rds-variables.tf)

Aha i see what you mean now. Let me think on it

@Igor M do you mean the the terminal UI when you execute the atmos command?

That’s right

we def should not show the abstract components in the UI (since we don’t provision them). I’ll check it, the fix will be in the next release

another minor observation for the UI - the up/down/left/right letter keys don’t work when within filter (/) mode

In Atmos, when you break your infrastructure into components, you’re essentially working with one component at a time. The behavior you’re describing would, therefore, work out of the box.

When you run Atmos commands like atmos terraform apply $component_name --stack foo, you only target that specific component. This is somewhat analogous to the Terragrunt --include-dir behavior, but with Atmos, you’re naturally working with one component at a time. Atmos does not support a DAG at this time for automatic dependency order applies. For that we use workflows instead.

Thank you so much for your reply! What do you mean by workflows?

https://atmos.tools/core-concepts/workflows/

Please see some related threads in this channel

https://sweetops.slack.com/archives/C031919U8A0/p1723482923252549?thread_ts=1722872544.379589

We are working on improvements for this, using YAML explicit types

Alright. I think this is closer

The key part is toRawJson

Almost, going to grab some food. I’m getting closer

@Andriy Knysh (Cloud Posse)

hmm Do I have a setting wrong, Does output need to do json? I put it on trace and saw Executing ‘terraform output vpc -s lab’

interestingt the tfvars look correct in the vpc, yet in the component not well

{
   "private_subnets": "[\"subnet-00ec468ff0c78bf7c\",\"subnet-029a6827865e98783\"]",
   "stage": "lab",
   "vpc_id": "vpc-09ead1bd292122e33"
}

bug I guess

@Dennis DeMarco this is a limitation that we have in Atmos using YAML and Go templates in the same files.

this is an invalid YAML (because { ... } is a map in YAML (same as in JSON), but double {{ breacks the YAML parser

a: {{ ... }}

so we have to quote it

a: "{{ ... }}"

but when we qoute it, it’s a string in YAML. And the string is sent to the TF component input

I personally have not found a good way to deal with that. There is a thread with the same issue, and one person said they found a solution (I can try to find it)

I tried it, but did not get good results today

another way to deal with that is to change the TF input type from list(string) to string and then do jsondecode on it (but that will require changing the TF components)

having said that, we are working on a new way to define atmos.Component function - not as a Go template, but as YAML custom type

a: !terraform.output <component> <stack>

this should be done in the next week or so, we’ll let you know

Ah alrighty. Thank you. I did a work around by just using data aws_subnets with the vpc id that came over.

But I was like a dog with a bone trying to make it work today hah

yes, and as I mentioned, you can change the var type to string and then do jsondecode on it

Thank you very much

TF will get this as input

"[\"subnet-00ec468ff0c78bf7c\",\"subnet-029a6827865e98783\"]"

which is a string, and after jsondecode you will get list(string)

but now we need to move this core stuff to another stack file

in Atmos, stack and stack file (stack manifest) are diff things

you can use any file names, in any folder, and move the components b/w them

stack files are for people to organize the stack hierarchy (folder structure)

stack is a logical construct, and it can be defined in any stack manifest (or in more than one)

yes, it can be pepe.yaml, but in this case we need to move it to another stack ( not file) to run the workflows at different times

so oyu want to move some components into another Atmos stack (like from plat-ue2-dev to core-ue2-dev)?

correct, because at some point they might be owned by someone else

I remember you told me to avoid overriding the variables using to name stacks since it is a bad idea

1. Import the components into the new stack manifest file
2. (remove them from the old manifest file)
3. Override terraform workspace for the components

I remember you told me to avoid overriding the variables using to name stacks since it is a bad idea

it’s a bad pattern, but can be used in cases like that (and documented)

documented, can you point me to the doc?

when you say import you are talking about moving it to the new .yaml file , not using import: on the yaml file, right?

Also related https://sweetops.slack.com/archives/C031919U8A0/p1725628118394529?thread_ts=1725566301.529239&channel=C031919U8A0&message_ts=1725628118.394529

https://atmos.tools/tutorials/atmos-component-migrations-in-yaml/#overriding-the-workspace-name

thanks guys