#atmos (2022-12)

2022-12-06

Susie avatar

What is the difference between command and job when writing workflows?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Introduction to Atmos | atmos

Atmos is a workflow automation tool to manage complex configurations with ease. It’s compatible with Terraform and many other tools.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

only command is supported

Susie avatar

oh, i must have seen job somewhere else

Susie avatar

thank you for the link

Susie avatar

is there an eta for when the documentation will be finished

Susie avatar

How do i configure the planfile?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

configure the planfile?

Susie avatar

i want the workflow to save a .tfplan file when i run plan. sifting through the code online, i see that there is a “planFile” var that is populated. i see the file get generated but then it is automatically deleted. i’d like this file to persist so that i can use it to run terraform destory --auto-approve tfplan

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it gets automatically deleted after a successful apply only

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
atmos terraform apply and atmos terraform deploy commands support --from-plan flag. If the flag is specified, the commands will use the previously generated planfile instead of generating a new varfile
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Introduction to Atmos | atmos

Atmos is a workflow automation tool to manage complex configurations with ease. It’s compatible with Terraform and many other tools.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) is highlighting that all terraform arguments are passed through.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

THis means that configure the planfile, you would do it just as normal with terraform, by passing the -out argument. https://developer.hashicorp.com/terraform/cli/commands/plan#out-filename

Command: plan | Terraform | HashiCorp Developerattachment image

The terraform plan command creates an execution plan with a preview of the changes that Terraform will make to your infrastructure.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. atmos terraform plan $component --stack $stack -out=planfile

Susie avatar

i see the problem. i was placing any terraform flag options directly after the terraform command. atmos was saying it “didn’t recognize component -out=tfplan”. I didn’t realize that the component needed to immediately follow terraform commands and then flags.

EX: this works: terraform plan $component --out=tfplan this does not: terraform plan -out=tfplan $component

fiesta_parrot2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Great! I’m glad you got it working

2022-12-07

2022-12-08

mike avatar

Hey all. Trying to learn Atmos. I think the tuorial here: https://docs.cloudposse.com/tutorials/atmos-getting-started/ is broken. I cloned the latest from <https://github.com/cloudposse/tutorials.git> and can follow through part of the tutorial (all the way through step 4). However, step 5 seems broken. When I try to run atmos workflow deploy-all -s example I get this error:

 ✗ . [none] (HOST) 02-atmos ⨠ atmos workflow deploy-all -s example
Error: required flag(s) "file" not set
Usage:
  atmos workflow [flags]

Flags:
      --dry-run        atmos workflow <name> -f <file> --dry-run
  -f, --file string    atmos workflow <name> -f <file>
  -h, --help           help for workflow
  -s, --stack string   atmos workflow <name> -f <file> -s <stack>

required flag(s) "file" not set

If I try to set the file I get this:

 ✗ . [none] (HOST) 02-atmos ⨠ atmos workflow deploy-all -s example -f example.yaml
file 'stacks/workflows/example.yaml' does not exist

It looks like atmos is expecting the workflow to be defined in a separate file (not in example.yaml). Is anyone else seeing this? I am doing something wrong?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

the tutorial is out of date and has issues. You need to configure the base path for the workflows in atmos.yaml, see https://atmos.tools/cli/configuration#workflows

Introduction to Atmos | atmos

Atmos is a workflow automation tool to manage complex configurations with ease. It’s compatible with Terraform and many other tools.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Introduction to Atmos | atmos

Atmos is a workflow automation tool to manage complex configurations with ease. It’s compatible with Terraform and many other tools.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

atmos will join workflows.base_path with the file name provided on the command line in --file flag

mike avatar

Ok, thank you

mike avatar

Is there some sort of reference for the Atmos yaml files and what they can contain?

mike avatar

The stack files that is (I think, sorry I am new to this).

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Introduction to Atmos | atmos

Atmos is a workflow automation tool to manage complex configurations with ease. It’s compatible with Terraform and many other tools.

mike avatar

Thank you so much!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Dan Miller (Cloud Posse)

2022-12-09

2022-12-15

Release notes from atmos avatar
Release notes from atmos
07:14:37 PM

v1.17.0 what Add local search to atmos docs Add atmos describe affected CLI command Update docs website why

Quickly search through documentation

The command atmos describe affected produces a list of the affected Atmos components and stacks given two Git commits. The command compares the final component sections (after all imports and deep-merging) for all Atmos components in all stacks, and produces a list of affected (changed) components in the stacks. The command also checks the changed files…

Release v1.17.0 · cloudposse/atmosattachment image

what

Add local search to atmos docs Add atmos describe affected CLI command Update docs website

why

Quickly search through documentation

The command atmos describe affected produces a list of…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is a very exciting announcement for atmos. This makes Atmos much easier to use in a CI/CD context. Using this command, you can easily compute all affected stacks so that they can be planned.

Release v1.17.0 · cloudposse/atmosattachment image

what

Add local search to atmos docs Add atmos describe affected CLI command Update docs website

why

Quickly search through documentation

The command atmos describe affected produces a list of…

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@jose.amengual you wanted to try that for Atlantis projects (let us know how it goes and if we need to improve anything)

jose.amengual avatar
jose.amengual

yes!!! and I did look at it

jose.amengual avatar
jose.amengual

I wonder if filters can be used?

jose.amengual avatar
jose.amengual

so that a specific stack or component can be listed

jose.amengual avatar
jose.amengual

in the atlantis world when using -p for project I could do : atlantis plan -p pepe-auto-ue2-vpc

jose.amengual avatar
jose.amengual

at this point I know the component name ( or I can parsed) and the stack name base on my naming convention so I could be smart and only describe affected base on a specific stack

jose.amengual avatar
jose.amengual

and the other useful thing will be that the output shows the atlantis stack name so you can match it with the -p flag in the atlantis command

jose.amengual avatar
jose.amengual

like

`{
      "stack": "tenant2-ue2-staging",
      "component_type": "terraform",
      "component": "test/test-component-override-3",
      "affected": "stack.vars"
      "atlantis_project: "tenant2-ue2-staging-test-component-ovverride-3"
   },
jose.amengual avatar
jose.amengual

if the atlantis integration is enabled

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, so ideal (futuristic/non-existant) implementation, atlantis would support running an external command to determine which projects are affected. So instead of relying exclusively on affected files (which is primitive), it could also exec a command which return a list of affected projects.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If that existing, then this would be extremely powerful as you would get all the power of atlantis with all the power of atmos.

jose.amengual avatar
jose.amengual

do not call my Atlantis immature Erik!!!!! LOL

jose.amengual avatar
jose.amengual

when you say atlantis would support running an external command to determine which projects are affected. this is already possible if I understand correctly

jose.amengual avatar
jose.amengual

we might be thinking the same here, I wanted atlantis to avoid having to PLAN an all projects and only plan on the projects that changed base on the PR trigger

jose.amengual avatar
jose.amengual

If using the github action with atmos like I’m doing now you could do that preprocess before hits atlantis and then create an atlantis.yaml file that only contains those affected projects

jose.amengual avatar
jose.amengual

IF using atmos inside atlantis, you can run a pre_workflow_hook to run atmos to generate the atlantis.yaml file before is parsed, again, with only the the projects that changed

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@jose.amengual could you explain in a few sentences what would you like to add the atmos describe affected to help with atlantis?

jose.amengual avatar
jose.amengual

sure

jose.amengual avatar
jose.amengual

• support filtering per stack and component =

atmos describe affected -s prod-ue2
atmos describe affected vpc -s prod-ue2
atmos describe affected vpc

• when atlantis integration is enabled extend output to contain the project name for the component changed i.e:

`{
      "stack": "tenant2-ue2-staging",
      "component_type": "terraform",
      "component": "test/test-component-override-3",
      "affected": "stack.vars"
      "atlantis_project: "tenant2-ue2-staging-test-component-ovverride-3"
   },
2
Release notes from atmos avatar
Release notes from atmos
07:34:39 PM

v1.17.0 what Add local search to atmos docs Add atmos describe affected CLI command Update docs website why

Quickly search through documentation

The command atmos describe affected produces a list of the affected Atmos components and stacks given two Git commits. The command compares the final component sections (after all imports and deep-merging) for all Atmos components in all stacks, and produces a list of affected (changed) components in the stacks. The command also checks the changed files…

Release notes from atmos avatar
Release notes from atmos
09:34:36 PM

v1.18.0 what Update workflow commands of type shell why Workflow commands of type shell could be any complex shell commands (or scripts) in the workflow YAML definition file

Release v1.18.0 · cloudposse/atmosattachment image

what

Update workflow commands of type shell

why

Workflow commands of type shell could be any complex shell commands (or scripts) in the workflow YAML definition file

Release notes from atmos avatar
Release notes from atmos
09:54:38 PM

v1.18.0 what Update workflow commands of type shell why Workflow commands of type shell could be any complex shell commands (or scripts) in the workflow YAML definition file

2022-12-16

2022-12-19

jose.amengual avatar
jose.amengual

@Andriy Knysh (Cloud Posse) having some issues with atmos describe affected

jose.amengual avatar
jose.amengual
atmos describe affected
ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
jose.amengual avatar
jose.amengual

my git works fine but I do have instead-of settings

jose.amengual avatar
jose.amengual

I wonder if there is a problem with that

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

according to https://github.com/src-d/go-git/issues/550, go-git should use the correct transport depending on the URL. Look at the responses, some other Git settings might be wrong. Other than that, go-git does not have anything special for SSH (well, PlainClone function allows specifying auth credentials, but we don’t want to use it anyway)

#550 can you give me a sample about clone a private repo using ssh key?

can you give me a sample about clone a private repo using ssh key? thanks

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

i did not test it with SSH so can’t say anything else than what the link describes

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Comment on #550 can you give me a sample about clone a private repo using ssh key?

If it helps, I did something like this to create the auth object needed to clone an ssh repo:

s := fmt.Sprintf("%s/.ssh/id_rsa", os.Getenv("HOME"))
sshKey, err = ioutil.ReadFile(s)
signer, err := ssh.ParsePrivateKey([]byte(sshKey))
auth = &gitssh.PublicKeys{User: "git", Signer: signer}
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

something like those we’ll to test

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@jose.amengual i’ll will test this when I get a moment and will prob add an ENV var and a CLI flag to the describe affected command to tell it to find and use the SSH credentials

1

2022-12-20

jose.amengual avatar
jose.amengual

how do I deploy a multiregion component in atmos?

jose.amengual avatar
jose.amengual

in this case, a regional cluster that needs to be created in the main region an it’s id is then used for all the other regions

jose.amengual avatar
jose.amengual

I guess I could do a data lookup to the resources I needs to find using the same module…..

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@jose.amengual you use “Component Remote State”

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

here is what we added to a PR, not deployed to the main site yet

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Component Remote State | atmos

Component Remote State is used when we need to get the outputs of an Atmos component,

jose.amengual avatar
jose.amengual

I’m trying not to use that module

OliverS avatar
OliverS

What is the reason, @jose.amengual?

jose.amengual avatar
jose.amengual

the remore-state module requires some special atmos configs and I think data lookups keep the state more in line with the actual infrastructure

jose.amengual avatar
jose.amengual

potentially you can have a bunch of atmos config yaml changes that the remote-state module will considere as actual changes but in reality were never deployed

jose.amengual avatar
jose.amengual

that is why I prefer the data lookups

OliverS avatar
OliverS

by data lookups do you mean using data terraform_remote_state ...?

jose.amengual avatar
jose.amengual

no

OliverS avatar
OliverS

or straight data aws_whatever

jose.amengual avatar
jose.amengual

dat.aws_vpc etc…

jose.amengual avatar
jose.amengual

yes

2022-12-21

2022-12-27

Viacheslav avatar
Viacheslav

Hi there! I have a terraform project with complex tfvars variable of type “map of objects”, which also contains nested maps of objects

{
 "complexvar":
	{
	  "toplevel": {
	    "objectA" : {
	      "bottomlevel1" : {
		    "attribute1": true
		    "attribute2": true
	      },
	      "bottomlevel2" : {
		    "attribute1": true
		    "attribute2": true
	      }
	    },
	    "objectB" : {
	      "bottomlevel3" : {
		    "attribute1": "abc"
		    "attribute2": "abc"
	      }
	    }
	  }
	}
}

and I am trying to update attributes of “*bottomlevel**” objects with “vars” property of the stack:

vars:
  complexvar:
    objectA:
      bottomlevel1: 
        attribute1: false

When I do following, atmos generates a new tfvars.json only with bottomlevel1 object and removes bottomlevel2 and objectB instead of deep merge. Does atmos support deep merge of terraform.tfvars.json and stack vars?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It sounds like you might be mixing your own tfvar files with the ones we generate. Atmos only deep merges what it manages, which are stack configurations. Translate all your tfvars into stack configs and it will definitely handle this use case

Viacheslav avatar
Viacheslav

@Erik Osterman (Cloud Posse) Thank you! It is an acceptable solution in my case

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The key thing here is atmos really knows very little about terraform. It is a general purpose workflow automation tool. One of the built in features is it can generate varfiles.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We designed atmos this way so it can automate all sorts of tools not just terraform

Release notes from atmos avatar
Release notes from atmos
05:24:37 PM

v1.19.0 what Add Sources of Component Variables to atmos describe component command Update docs why The atmos describe component command outputs the final deep-merged component configuration in YAML format. The output contains the following sections: atmos_component - Atmos component name atmos_stack - Atmos stack name backend - Terraform backend configuration backend_type - Terraform backend type command - the binary to execute when provisioning the component (e.g. terraform, terraform-1, helmfile)…

Release v1.19.0 · cloudposse/atmosattachment image

what

Add Sources of Component Variables to atmos describe component command Update docs

why The atmos describe component command outputs the final deep-merged component configuration in YAML form…

2022-12-28

Patrice Lachance avatar
Patrice Lachance

Hi all, since backend.tf.json files need to be generated when ‘vendoring’ components from terraform-aws-components, why would atmos users leave auto_generate_backend_file: false and generate backend files manually?

jose.amengual avatar
jose.amengual

if you want atmos to generate those backend.tf files you need that setting set to true

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yes

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and it’s not related to vendoring, you can use vendoring or not, and you can auto generate the backend files or not

Patrice Lachance avatar
Patrice Lachance

yes, I understood that, I was just asking why would users prefer to generate the backend files manually instead of having atmos to it automatically

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if you have a single backend for the entire infrastructure, then it’s your choice to have the backend files committed or auto generated - the result will be the same

2
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

but atmos supports multiple backends. eg per account, per tenant, per region. in this case it will be generated automatically based on the information in the configuration

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and in this case it can’t be one static backend file

2022-12-29

    keyboard_arrow_up