#atmos (2023-04)

you can use workflows https://atmos.tools/core-concepts/workflows/

you can create reusable workflows and then provide atmos stack on the command line

perfect, thank you, this looks exactly like what we are looking for. apologies for the newbie question but we are all picking up atmos for the first time and enjoying the ride!

thanks for using it, let us know if you need any help

awesome. thank you Andriy

watch out with this guy @Andriy Knysh (Cloud Posse) he is nice but not as nice as me

(we work together)

We’re also using workflows for this. So we define all the steps to bring up a layer of your stack.

(or destroy)

and they are sequential, they do not run in parallel right?

not yet, we have a task to add parallel steps

I prefer sequential

@Andriy Knysh (Cloud Posse)

as a side note, Go templates could help with basic ARN generation as well.

if we set context or something we could potentially provide all necessary values and Atmos gen that ARN so we don’t have to do that in TF

a bit more context: https://sweetops.slack.com/archives/G014YEKDH4K/p1680713411920479

cc @Jeremy G (Cloud Posse) (this is an off-shoot of that thought/discussion)

I don’t like the idea of Atmos generating an ARN rather than using Terraform to do it. Also, Atmos is not Terraform and does not directly have access to remote state.

You might be able to refine our current remote state Terraform implementation with a more generic remote state “mixin”, but unless you want to run through a separate Terraform apply to read the values, I don’t think Atmos can wire the outputs to the inputs for you. You might come up with some kind of “convention over configuration” where an input has a specific naming format and the mixin is configured to place a value with that name as a key in an output map and then end up with something like

locals {
  var_name = try(modules.remote[var.component]["var_name"], var.var_name)
}

I would need to see some fully worked-out examples to see if that is really simplifying anything. I don’t think it would be possible to satisfy the constraints of a mixin, which is that the component should work without any code modification whether or not the mixin is present.

right. definitely a complex scenario. if it shows up as a need more often, i might take a rough stab at a POC

One important design decision is atmos isn’t bound to AWS. We want to keep it platform agnostic. Our recommendation is to make the components support references and have it dereference those inputs. That’s how we currently do everything

If atmos starts reading terraform remote state we’re treading down a slippery slope. There’s all the configuration of backends to accesss remote state and a dozen types of remote ends

I could imagine atmos running a command that output some settings and those settings are accessible

yes. Also, Atmos does not know anything about roles, credentials etc. Underlaying tools (e.g. terraform) do. So Atmos can’t access AWS or any other clouds/platforms on its own

That’s a solid point @Erik Osterman (Cloud Posse).

@Andriy Knysh (Cloud Posse) that’s why I was suggesting a TF component that does it, but I get how other providers may be problematic for that.

What about reading things like name, etc from other stacks to reference them?

That’s all YAML and agnostic

That’s another chicken/egg problem for sure so definitely not an easy thing.

Just random thoughts.

this is done using the remote-state module

https://atmos.tools/core-concepts/components/remote-state

Yeah, just thinking through ways not to define a bunch of those and doing something more dynamic

what happens if you remove the /

https://atmos.tools/core-concepts/components/vendoring/#schema-componentyaml

@Andriy Knysh (Cloud Posse) interesting usage of vendor @Michael Dizon is using it to vendor modules and not components

Since we support generating the backend in atmos, really any module we have can be a component

so when i remove the / I get this error:

error downloading '<https://github.com/cloudposse/terraform-aws-ec2-instance.git?ref=0.47.1>': /usr/bin/git exited with 128: fatal: not a git repository (or any of the parent directories): .git

ah interesting. so presently, the intent is for this to point to the terraform-aws-components repo?

no, it can point to any endpoint supported by go-getter

As implemented, we usually use it for components in our monorepo

what about

uri: github.com/cloudposse/terraform-aws-ec2-instance.git//?ref={{.Version}}
version: 0.47.1

https://github.com/hashicorp/go-getter#url-format

formatted it like this and got the same result

github.com/cloudposse/terraform-aws-ec2-instance?ref={{.Version}}

per: https://github.com/hashicorp/go-getter#protocol-specific-options

I can confirm this appears to be a bug.

erik@Eriks-MacBook-Pro /tmp % export ATMOS_LOGS_LEVEL=Trace    
erik@Eriks-MacBook-Pro /tmp % atmos vendor pull --component bar
Pulling sources for the component 'bar' from 'github.com/cloudposse/terraform-aws-ec2-instance.git//?ref=0.47.1' and writing to 'components/terraform/bar'

error downloading '<https://github.com/cloudposse/terraform-aws-ec2-instance.git?ref=0.47.1>': /usr/bin/git exited with 128: fatal: not a git repository (or any of the parent directories): .git

ah

@Andriy Knysh (Cloud Posse) https://github.com/hashicorp/go-getter/issues/114

interesting.. this worked:

uri: github.com/cloudposse/terraform-aws-ec2-instance.git///?ref={{.Version}}

oh, interesting

I was just going to say, I was able to do //test to vendor the tests, so thought it was a problem with root only

but that makes sense that /// works

If you get this working, could I trouble you to open a doc here on how to vendor modules?

https://github.com/cloudposse/atmos/blob/master/website/docs/core-concepts/components/component-vendoring.md

sure no problem!

tomorrow

@Erik Osterman (Cloud Posse) https://github.com/cloudposse/atmos/pull/364

re-reviewed

ok, in a meeting right now. will update after i come out

@Erik Osterman (Cloud Posse) committed your suggestions. learned something new today, didn’t know github could do that!

Thanks for the contributions!

dude thanks for the amazing tools and modules!

https://atmos.tools/core-concepts/components/vendoring#vendoring-modules-as-components

atmos may make this possible but it’d require a lot of additional logic, two steps makes more sense

because the account ID won’t be known, and you need to instantiate a new provider to work on that second account

so there will necessarily be multiple calls to “terraform” which I don’t think atmos is capable of today

unless it has a terragrunt style “apply-all” command that i don’t know about

I see that atmos has workflows which allow sequential commands, but I don’t see anything that would enforce the “wait” or dependency ordering. I think remote outputs and some dynamic stuff in the providers can handle finding the account id. I’m mainly just wondering on the orchestration part.

The two-steps approach makes sense, but how is this typically done in a CI/CD environment? I basically want to add a new account to some config, submit a PR, then have it created and a baseline config applied

I’ve never fully automated that. Usually just have a common “account-baseline” module that we add to the new account after creation

could be done in CI/CD in two steps still

Yeah it does look like there is a status check API on the create account operation: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/organizations/describe-create-account-status.html

Probably not worth optimizing with full automation. Using the CLI to poll the status and then run the baseline tf apply as a second step seems reasonable

well the terraform resource for adding an account to an org shouldn’t be “created” until this status check would return true anyway, i don’t think you’ll need that CLI bit

That’s what I was wondering, but haven’t found confirmation of that

yeah. in my experience, AWS account creation happens really fast, like maybe a few seconsd

that’s what makes me think it is not waiting for the full completion

as i’ve seen in most guides it can take several minutes

And the API docs seem to indicate it’s a background process that could take minutes: https://docs.aws.amazon.com/organizations/latest/APIReference/API_CreateAccount.html

Oh, that’s probably why I’ve perceived it as only being seconds.

Actually looking at the TF provider code, it does look like it is calling and waiting on that status endpoint. So maybe it’s just fast the majority of the time, but could take a while

Yes, we do this

https://atmos.tools/core-concepts/workflows/

In our commercial reference architecture (for sale), we leverage workflows to bring everything up

Workflows can call other workflows

This is our account factory https://docs.cloudposse.com/components/catalog/aws/account/

Thanks, @Erik Osterman (Cloud Posse). Sounds like I’m on the right track. Do you all ever use nested OUs?Unless i’m missing something, that account factory only supports one level deep. I’m curious if that is an intentional best practice

yes & no. We don’t recommend them b/c of resource naming and disambiguation.

architecturally I like it, but in practice it’s problematic if you subscribe to our opinionated naming conventions.

ah fair, that makes sense

in our convention, org = namespace ou = tenant account = stage region = environment resource = name

so imagine an org called acme with an ou called plat (For platform), with an account called prod with resources in us-east-2 (we use use2), and a cluster (eks)

the resource becomes acme-plat-prod-use2-eks and then all the sub-resources assocaited with the cluster

now introduce sub-OUs, and you and 4-5 characters, and now you’re definitely hitting limits on resource names for many systems.

That’s helpful, thanks!

Hi, to jump on this, one option I’ve considered is to use a convention along the lines of

namespace = org tenant = “smallest” ou stage = account environment = region name = resource

taking your example: an org called acme with an ou called plat with child ou’s called foo and bar with an account called prod in each ou with resources in us-east-2 and a resource (eks)

resource in plat-prod: acme-plat-prod-use2-eks resource in (plat-)foo-prod: acme-foo-prod-use2-eks resource in (plat-)bar-prod: acme-bar-prod-use2-eks

There is a loss of visibility inherent here (which can only be partially mitigated by adding a “ou-path” tag/attribute/etc to each null-label), and some tweaking of the account module required (add support for the parent_id property in the YAML for an optional second layer of aws_organizations_organizational_unit resources).

However, I don’t think this would actually break anything in account-map or elsewhere, tenant still refers to an ou, just with an additional constraint.

Have you ever considered something like this? Or is “tenant=ou” by convention or technical limitation more immutable than I have considered here?

I suppose you would be in trouble if you reused an ou name across different branches of the your aws org hierarchy, but should be fairly easy to guard against

Yeah, we would have that issue is we have multiple OUs with child OUs based on environment, like dev and prod

(disregard my most recent messages, deleted)

I am jumping on a call - will get back

Agreed Austin, that wouldn’t work in the model I have described here. However, the refarch’s current OU heirarchy model is entirely flat with by necessity, large, low-resolution OUs, so env-based child OUs wouldn’t be viable in that model either.

There’s a tension here between the refarch’s flat structure (which I otherwise love), and a desire for nested OUs — admittedly in my attempt in an approach that adds a irritating uniqueness constraint on the name of “child OUs”.

The other option: acme-plat_foo-prod-use2-eks , is my least favourite, as not only does it have the length issues Erik mentioned, but it feels like an abuse of the tenant keyword.

In my hand-waved model the level-1 OU would be a broad project category, the level-2 OU would be a specific project, and each level-2 OU would have a dev, stag and prod account trio.

This would let you add governance controls and shared atmos-yaml configuration with slightly more granularity than in the refarch, with a downside that “projects” (or whatever an equivalent keyword is in your organisation’s internal parlance) would need to by either intentionally uniquely named, or assigned a random id-phrase.

in reflection, I think you can mostly get way with the hierarchical OUs, provided you a) use the most specific OU as the tenant, and ensure that the most-specific OU is always unique across all parent OUs

Maybe there’s someway to create a rego policy to enforce this. Since atmos supports rego, this would then be maintainable.

So with the standard model, to apply different SCPs between dev and prod, you would have to do that on each explicit account in the Atmos stack, correct? To me that’s the only real benefit to the OU nesting is being apply to take advantage of the policy inheritance

Keep in mind it’s all IAC and with atmos, you get imports

So there’s really no benefit to the nesting.

With imports it’s all still DRY.

Awesome to hear you’re building out those pipelines.

We did build atmos describe affected for this purpose, but have not implemented the apply-affected command you describe.

Mostly b/c gitops best practices for terraform involve reviewing the plan before apply, and that woudl require tighter integration with your build process

you could implement this probably quite easily with some jq foo and xargs

Then add your own subcommand for it https://atmos.tools/core-concepts/subcommands/

Ahh, i see. Thank you so much for the quick response

Fwiw, we’re implementing GitHub Actions support

Awesome

you could implement this probably quite easily with some jq

run atmos describe affected, use jq to select the components and stacks, then in a loop you can call atmos terraform apply

and you can add all of that into a custom command

sounds good and i am working on that now. Thanks so much

by custom command we mean two possible things:

1. atmos custom command https://atmos.tools/core-concepts/subcommands/

1. You can add that script as a workflow step using the type shell https://atmos.tools/core-concepts/workflows/

The first one sounds cool to use

@Andriy Knysh (Cloud Posse) do not trust this guy, he works with that other guy that is not as nice as me

yea, and note that you can create a custom atmos command using any complex script (including other atmos commands including other custom commands), and then you can use that custom command in a workflow step of type atmos

how does the atmos Github action does this today?

in a shell loop

atmos custom commands can be as simple as calling a shell script

# Custom CLI commands
commands:
  - name: aws
    description: Execute AWS commands
    commands:
      - name: assume-role
        description: Execute 'aws assume-role' command
        steps:
          - set-aws-assume-role-credentials

or as complex as this example

  - name: manage-s3-assets
    description: Pushes or syncs assets to the given s3 bucket
    verbose: true
    flags:
    - name: environment
      shorthand: e
      description: Environment
      required: true
    - name: stage
      shorthand: s
      description: Stage
      required: true
    - name: mode
    - name: from
      description: From Path
      required: true
    - name: datasource
      description: Target
      required: true
    - name: datasource_path
      description: Target path
      required: true
    component_config:
      component: "deploy"
      stack: "xxx-{{ .Flags.environment }}-{{ .Flags.stage }}"
    env:
    - key: AWS_REGION
      value: '{{ .ComponentConfig.settings.standard_env.AWS_REGION }}'
    - key: S3_CMD
      value: '{{ index .ComponentConfig.settings.aws_commands.s3_cmd .Flags.mode}}'
    - key: BUCKET_NAME
      value: "xx-{{ .Flags.environment }}-{{ .Flags.stage }}-{{ .Flags.datasource }}"
    - key: AWS_PROFILE
      value: "xx-gbl-{{ .Flags.stage }}-terraform"
    steps:
    - aws s3 ${S3_CMD} {{ .Flags.from }} s3://${BUCKET_NAME}/${DATA_SOURCE_PATH}

we are going to create a custom command so people can run this in any pipeline and once we.move to github actions we will definitely use your actions.

we are stuck with bitbucket internally but for external repos we can use github (after approval )

I do not use the remote-state module cloudposse uses but I do data lookups so in my remote state I will have something like :

data "aws_vpc" "main" {
  tags = {
    Name = format("%s-%s-%s", var.namespace, var.environment, var.vpc_name)
  }
}

because as you said the [context.tf](http://context.tf) maintains the naming convention is easy to find any resource

now if is a resource you did not create you can still craft a data lookup to find the existing vpc base on tags and such

Thanks @jose.amengual for your response. I’m talking about scenario that we have a VPC that was created outside of cloudposse context and you are require to use an existing VPC or any other existing resource.

CloudPosse recommends to use their own aws components or writing your own stacks as long as you follow their conventions and then we can also use how many data sources that we need as you mentioned and develop our own building blocks.

There are some differences between startup companies that would much appreciate your intervention in all of their aws account management and so on.

Within enterprise companies is the exact opposite, they want you to follow their rules, their policies, they would like to provide you part of the infrastructure and we should use it according to their guidelines as long as it comply with product requirements that you deploy on their tenant.

I was more curious to know how cloudposse team face in such scenarios and keep using their aws terraform components.

Nothing about atmos requires using context or remote state. We use those as our best practices for terraform, but atmos doesn’t enforce it.

Atmos is used by massive enterprises

With conventions far different from what we do in our reference architecture, which is why everything is configurable via the CLI config

We recommend companies use what serves them best. We provide components we use, but users of atmos, enterprises and startups alike are not required to use our components and the tool works with any vanilla terraform or the components we provide. You can use it with any cloud, GCP, azure, AWS, etc.

JFYI, the components that we open source are designed to be internally consistent, and won’t work as well when trying to leverage them in existing environments that have different assumptions.

Got it, Thanks for your clarification Erik, We actually already did a huge project with Atmos for provisioning environments on Azure for enterprise companies , it works pretty well and we are happy with our outcome.

We align it to our needs by developing some super module which wire all the connections between modules, we were influenced by Azure Cloud Adoption Framework module but since we had a special requirements we decided to develop our own super module and plug it with Atmos.

The thing is that now we should do the same for aws infrastructre provisioning and we know that cloudposse has their own aws terraform components. Therefore, we thought about give a try and use it instead of reinvent the wheel with another intermediate layer between Atmos and modules for our needs but looks like we can not avoid it.

atmos does the following: 1) allows you to organize component/stack configurations in a hierarchical and DRY way; 2) generates TF varfiles and backend configs from the stack config

you can use any module/component w/o atmos, you just need to cd into the component folder and provide all the required variables. Then use plain TF commands

@Andriy Knysh (Cloud Posse) I think what he wants to do is akin to our atmos terraform shell

https://atmos.tools/cli/commands/terraform/shell

If you run atmos terraform shell, it will set up your environment so you can run a “plain terraform plan command”

yes, terraform shell can be used

but only if you have atmos.yaml defined

the error above says that atmos.yaml does not exist (https://atmos.tools/quick-start/configure-cli)

so if you want Atmos to generate all the varfiles for the component, you need: 1) create atmos.yaml (https://atmos.tools/quick-start/configure-cli) ; 2) configure Atmos components and stacks (https://atmos.tools/quick-start/create-components, https://atmos.tools/quick-start/create-atmos-stacks)

on the other hand, if you want to use a TF module/component w/o Atmos (just TF code), then you cd into the folder and provide all the variables and backend config, then use plain TF commands

this is a working example (which we are using for testing) https://github.com/cloudposse/atmos/tree/master/examples/complete with everything defined (atmos.yaml, components, stacks, etc.)

Okay. Will explore this more. Our organisation is using terragrunt so wanted to explore how to use cloudposse modules with it. Thank you.

if you need help with using the components, let us know. Also, we can help with integrating Atmos if you want to explore it

Aha, so in that case, wouldn’t it be more about overriding the terraform command? We haven’t tried it, but others have mentioned they are using terragrunt.

Aha, so when you use the Cloud Posse developed components, they will frequently leverage the stack configurations. Under the hood they use our other open source child modules.

Our Cloud Posse components frequently use a terraform data provider which reads remote state and looks up configuration from the stacks. That’s why you’re getting this error.

To be clear, it doesn’t require atmos, but it requires stack configurations. If that’s not what you want, then you’ll need to fork it.

Oh Okay. Thanks a lot. Will try that approach

does it have ‘.git’ folder? In the command atmos describe affected --file affected.json --verbose=true --repo-path $BITBUCKET_CLONE_DIR

it will not work if it’s not a valid git folder

yes i verified it in local and reinitialized using git init as-well

and it is correctly pointing to right head aswell

we never tested it with Bitbucket, might be an issue with that. We are using go-git Go lib to do all things with Git

or might be not related to Bitbucket

if you just copy the whole repo into a temp dir (including the .git folder), then use the command atmos describe affected --file affected.json --verbose=true --repo-path xxxx ?

manually copy, on your computer

Ahhh, i will do that and see, how it goes. Thank you

yes, just clone the entire folder into a temp folder on your computer

then use it in --repo-path argument

at least you’ll know that this part works

sounds good and i will post the outcome, once it’s done

it is giving me the same kind of error but the directory has .git folder

the target remote repo is not a Git repository. Check that it was initialized and has '.git' folder: repository does not exist

for some reason, https://github.com/go-git/go-git does not like that .git folder (this part of the error repository does not exist is from go-git)

https://stackoverflow.com/questions/8927070/git-error-conq-repository-does-not-exist (Bitbucket users complain)

please review it

(this issues are difficult to answer since Atmos does not do anything with Git and it’s folders, go-git does. Whatever the lib does, we don’t control it in Atmos code, and go-git is far from perfect)

Ahhh, Got it. Thanks so much for the quick response

So we fixed the repository does not exist error

now is giving us authentication required

the two commands before atmos describe affected are git status and git config -l and the both work right before the atmos command

so it is estrange that atmos does not seem to use the credential in the same shell/run/pipeline

atmos uses go-git

whatever it does

if we run this locally against the same repo it works so there is definitely something missing in the pipeline

there are a lot of discussions about that for go-git https://github.com/go-git/go-git/issues/116

that def works in GitHub actions (we are using atmos describe affected in the actions)

in github we have 0 issues

but in a container, it does not for a private repo

that’s go-git issues, which can prob be solved by using

repo, err := git.PlainClone(pathToRepo, false, &git.CloneOptions{
			Auth: &http.BasicAuth{
				Username: "yourUsername",
				Password: personalAccessToken},
			URL:      "<https://github.com/go-git/go-git>",
			Progress: os.Stdout,
		})

but it does not look like a good solution if you need to specify your username and a PAT (as the command’s arguments?)

yes, I think there is a setting that we might not have access to allow the runners to authenticate using ssh

this worked atmos describe affected --ref refs/heads/main --ssh-key ${BITBUCKET_SSH_KEY_FILE} --verbose=true