#atmos (2022-08)

this breaks it

the stack YAML file is for prod

you should not override the context vars per component since atmos uses the context vars to find the component in the stack

and here

terraform:
    # Temporary dev bucket in prod account
    template-bucket-dev:
      metadata:
        component: s3-bucket
        inherits:
          - template-bucket
      vars:
        stage: dev

ah yes

Still getting used to that, thanks!

the stage is already dev, so the command atmos terraform plan template-bucket-dev --stack prod can’t find it anymore

everything for stage dev should go to stacks/org/dev/us-east-1.yaml in your case

ok I see

yeah, this is a weird case because we’re migrating from a single account to multi-account but we need to apply this change in the single account - so it’s basically a hack I’m trying to do, within our new stack structure

i.e. we’re mid-migration

the YAML file names don’t matter, the folder sructure does not matter, you can name the files anything and use any level of subfolders

what matters is the context vars (tenant, environment, stage) inside the files - atmos uses it to find the component in the stack

I have changed it to:

components:
  terraform:
    # Temporary dev bucket in prod account
    template-bucket-dev:
      metadata:
        component: s3-bucket
        inherits:
          - template-bucket
      vars:
        attributes:
          - dev
        privileged_principal_arns:
          - arn:aws:iam::123456789123:role/eg-dev-backend-task:
              - ""

and that should get the job done for now

ok I see - thanks for the clarification

I think it will stick in my understanding finally

Appreciate your help @Andriy Knysh (Cloud Posse)!

this is the key https://sweetops.slack.com/archives/C031919U8A0/p1660010904513129?thread_ts=1660009575.038859&cid=C031919U8A0

the bonus is, you an rename the files (file names are for humans), move the files to any subfolder, and it should continue working

That does seems like it will result in overall more stability, which is great

btw, here

included_paths:
    - "org/**/*"
  name_pattern: "{stage}"
  excluded_paths:
    - "catalog/**/*"
    - "**/_defaults.yaml"

you don’t need - "catalog/**/*"

b/c the included_paths only include "org/**/*"

and you exclude "**/_defaults.yaml" which are any _defaults files in orgs/…/…

ok, I think I picked that up from an example somewhere.. i assumed it meant it wouldn’t process any of those files when scanning for config unless they are included by others..

Is that what excluded_paths does?

included_paths include folders to scan

excluded_paths exclude some folders/files from the included

"catalog/**/*" will never be scanned since it’s not included in "org/**/*"

ah I see

the latest atmos example uses the latest stack structure here https://github.com/cloudposse/atmos/tree/master/examples/complete/stacks

Thank you - i’ll review that again

Is the actual purpose of excluded_paths to reduce scanning effort or to make sure the catalog entries aren’t actually ‘executed’ ?

both

1) _defaults.yaml file are not the actual stacks, they are just imported (like files from catalog), so we don’t need to scan all of them to find the component in the stack

2) we need to give atmos only the top-level stack files to scan, otherwise it will scan some YAML files that are just meant for import

makes sense, thanks

btw, these commands will help you with the stack configs and finding misconfigurations

atmos validate stacks
atmos describe stacks
atmos describe component xxx -s yyy

https://atmos.tools/cli

oh nice - i’ve never seen this site!

it’s not done yet, but the list of the CLI commands is complete

also see these https://github.com/cloudposse/atmos/pull/146

https://github.com/cloudposse/atmos/pull/133

I just tried validate stacks - no output means valid?

thanks for all that. We’re definitely at a productive level with atmos. Just some minor things like above, due to my knowledge gaps.

also, you can define custom atmos commands (they will be shown in atmos help) https://github.com/cloudposse/atmos/blob/master/atmos.yaml#L66

https://github.com/cloudposse/atmos/pull/168

The resource is deleted as part of our compliance component

I would be open to adding a feature flag to our vpc component

Ah interesting. I have a team member in progress on building a small component around default_vpc_deletion right now that we were going to upstream.

I could see adding it to an existing component, but for right now I think it’s not a bad stand along action so it can be dealt with on a per-region basis.

That works too

Feel free to upstream

Actually, what about adding it to the existing account settings component?

Do you use that one?

We use that and that was the other component I thought of adding it to, BUT account-settings is a global component and deleting the default VPC would be a per-region action.

It’s likely possible to create a submodule that we could iterate over regions to delete default VPCs, but dunno if it’d be the best fit.

Ohhhhh snap!

Ok, good point

Nah, proceed then with your current approach

The aws provider has native support for it so i don’t think the awsutils provider is needed for default vpc deletion anymore

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/default_vpc

@RB I just had that same conversation with a teammate — The issue I don’t like with deleting the default VPC via AWS provider + default_vpc resource is that it requires two applies: One to gain ownership over the VPC resource and then one to destroy the VPC resource.

My teammate was saying he wanted to go the official provider route because it was official, but really I think it’s just more work for no gain. So IMO the awsutils way is superior.

Ah that is very interesting. I have not used the native resource and was unaware of this limitation.

It’s probably worth a ticket with the aws provider. You’re right, the awsutils provider seems to be superior here

Ya, official route is “consistent” with terraform patterns, but a PIA and not gitops friendly.

Hence, our resource still wins.

Hey everyone, @Matt Gowie’s teammate was me. I was less concerned about it being official than idiomatic. To my knowledge, tf will destroy resources on apply but only if they are child resources of a resource that tf is managing. My understanding is that when AWS implemented a complete lifecycle for default VPC they did it as an independent resource, so you have to run an apply to pick it up, and then you can do a terraform destroy to get rid of it (but only if a flag is set in the resource). Hashi’s implementation is consistent with CRUD principles given what they are provided from amazon, imo.

I actually like the account-settings idea, because it feels like this should be a flag at account creation that AWS doesn’t provide. This would work like the official default GKE provider where there’s a “remove_default_node_pool” flag and terraform reconciles that after it creates the cluster.

It’s definitely more work to delete it in every region though.

the account-settings is deployed in gbl tho with a primary region so you’d only be able to affect your primary region and not your secondary regions

it would be easier to create a ~net net~ new regional component vpc-delete-default and deploy that imho

not sure I follow on “net net regional component”, but I am new to atmos so not fully up-to-speed on its idiom. I just mean that the desired behavior is probably account-level, definitely in terms of security. If you don’t want unsecured VPCs in one region, you won’t want them in any region.

corrected. net net => new

terraform providers have a region input and providers cannot be dynamically added so a regional component would be vpc-delete-default component imported into say ue2-automation and uw2-automation to delete the default vpc in both ue2 and uw2

Got it I think this is the direction we are headed in.

I wish this were a flag at account creation on the AWS side though, that feels like the missing piece to me. ty for the discussion everyone!

Ya, in an ideal world, it should be a feature flag on account creation.

cc: @Matt Gowie @Erik Osterman (Cloud Posse)

cc: @Andriy Knysh (Cloud Posse)

workspaces work well TFC, Atlantis an many other tools, I think this people are stuck on early version of TF but I tried to stay away from workspaces as much as possible because I do not want to add more stuff to my TF workflow but not because they do not work

atmos uses workspaces too, no?

Yes, atmos uses workspaces. The workspaces atmos uses are the name of the <root-stack>-<optional derived component name> .

It would be nice to write up an article using atmos, in defense of terraform workspaces using tfvars. 100% — Would love to see it. The copy pasta of directories method seems archaic. Even with root modules only being child modules it still feels deeply wrong.

There will be a Masterpoint blog started in the coming month or two and I’ll add this to my list of potential post ideas if none of ya’ll haven’t jumped on it by then.

Strongly agree

the main issue they describe in the article is actually not related to TF workspaces. They are talking about diff versions of the same TF component (e.g. some TF components using diff database)

If you force your development and production environments to use the same exact code, how do you test new versions of your infrastructure? Let's say you want to change your application to be backed by a different type of datastore, or add in some new AWS resource that was just released. How do you do that in your development environment without doing it in production at the same time?

it’s the issue of TF components versioning

which is a big issue if you have a lot of diff variations, but easily solved if you have just a few diff versions

components
   terraform
     component1
     component1_v2

yes, but this article is used to throw out the idea of workspaces and so this article can easily be used to toss out atmos

i speak from debate experience of trying to push atmos btw.

in any case, TF workspaces have nothing to do with TF component versioning. You’ll encounter the same issue whether you are using TF workspaces or not

Ya this is a worthy topic. There is so much FUD about terraform workspaces that is consistently spread by influential persons and companies using terraform.

I would love to dispel it

is there a cloudposse blog ?

1. In atmos, we are using workspaces, but can “easily” switch it off

1. “easily” from the code point of view. Not “easily” if you consider S3 backend where we have workspace_key_prefix as the bucket folder and workspace as subfolder. Backend needs to be considered here as well; otherwise everything in S3 will be in just one big folder which is not easy to maintain

cloudposse.com/blog

but we don’t post regularly

(other than our office hours updates, which some what dilute the blog)

the second issue they mentioned is forgetting to switch to the correct namespace. With plain TF, this is a big issue and we can agree with them on that

this is difficult w/o automation

atmos does the switch automatically on every command execution, and prints the message to the console about what workspace is being used

exactly, this is what i want from the blog post because atmos makes working with workspaces very very very easy

in fact, when first working with atmos, i had no idea it was using workspaces in the background

and as far as ive worked with atmos, it’s never failed to choose the appropriate workspace for me

@RB you can quote that in your blog post

wait, what about feedback on the design?

isn’t that great too?

Hrmmm good feedback

Have you checked out the tutorials?

Those are simpler albeit a little out of date

Yeah though googling lead me to : https://atmos.tools/core-concepts/fundamentals

https://github.com/cloudposse/tutorials

If you get a chance to run through it and have any problems please post here

There was a tiny bit of irony in finding a tool that abstracts out the complication of layering and composing complex infrastructure - only to have the example in the git repo being the full complete complex infrastructure and none of the intermediate layering of complexity

Then let’s open a PR to fix it

Haha it’s true

Is it a case that the project is new/moving so quickly the docs site is a bit bare right now?

So it’s one of the challenges we have. First as the creators it’s “easy” for us. But we know it’s a lot to take in. How to build up to that? We want to find a good way to do that. Even in OO programming it’s hard, and we took the concepts from there

http://atmos.tools needs more content!

We have a lot more internal documentation for customers but it’s taking a while for us to upstream it

Think if you’re doing a tutorial series for AWS for example, maybe showing intermediate stages starting off with a single component/layer of a VPC, then having two layer components of a public/private subnet setup (think IGW/NAT GW) or an isolated subnet setup (think just raw subnets and VPC endpoints for aws services - lambda maybe?) and then sitting something on top of those two options, to show the power of a stack where you have multiple middle options and can drop EKS on those depending on some layered config might have some “real world” value

I’ve come to find atmos because I am finding it increasingly difficult to provide our developers a composable stack where we can have multiple VPC layouts, because the compute platform of EKS is not really bothered by those complexities down in the subnets, but it’s critical the vpc/subnets exist first - and resolving that dependency web is proving painful

Yes that’s the problem we had as consultants going into new situations. We always had to solve the problem of different architecture and couldn’t develop the terraform for every architecture. We didn’t want to do code generation either because that’s hard to test.

Yeah the code generation route is pain too, ideally the atmos style of providing a recipe of components with some overrides in the vars seems exactly the right solution…

So we built atmos so we could write our modules once and compose them in any way expressing it as yaml. Smaller states. Decoupled lifecycles of various kinds of resources. The mistake we see all the time is someone has a module with the vpc and the eks cluster in it. But the lifecycles are totally different.

They should have 2 different states.

yup, exactly the pain we’re feeling

plus the risk of accidentally nuking the VPC…

So that’s comforting that atmos is attempting to solve this mess

Feel free to DM me and I can show you behind the scenes how we organize it

Thanks for the offer, before I embark on the tutorials - noticed as you suggested they’re 17 months old - do they still work with the current version of atmos or has API changed?

(basically asking do I need to go fish out a much older binary for atmos to play along)

nevermind, looks like you’ve bundled a docker image (TIL about geodesic)

let’s see if it it all works

geodesic tutorial worked bar one thing: https://docs.cloudposse.com/tutorials/geodesic-getting-started/#geodesic-usage-patterns the second install option just plain didn’t do anything

the atmos example blows up because the fetch-weather url 404s now

This might be helpful too. This is the normal layout for an atmos mono repo. The only difference is that we usually dont use the “infra” directory.

https://github.com/cloudposse/atmos/tree/master/examples/complete

Generally there are 2 main directories, components/terraform/<root terraform modules> and stacks/

# stacks/uw2-dev.yaml
# or stacks/orgs/acme/dev/us-west-2.yaml
vars:
  # company prefix
  namespace: acme
  # our word for short region code 
  environment: uw2
  # our word for account
  stage: dev

components:
  terraform:
    vpc/defaults:
      metadata:
        type: abstract
        # corresponds to components/terraform/vpc
        component: vpc
     vars:
       enabled: true

    vpc/example:
      metadata:
        # real is implicit and optional
        type: real
        component: vpc
        inherits:
        - vpc/defaults
      vars:
        name: example

Then

atmos terraform plan vpc/example --stack uw2-dev

This creates a vpc component called example in the dev account in us-west-2

does atmos have a simple solution to creating a different terraform state file per component?

Yes, atmos will generate the backend based on the yaml inputs and use a workspace key prefix

Atmos uses workspaces behind the scenes and the workspace is equivalent to the <root-stack>-<component name> if using a derived component (via metadata) like in the above fashion or simply <root-stack> without using a derived component.

This ensures a unique workspace per component instantiation per root stack

is there the concept of stacks with shared components? say there’s two stacks, dev-a and dev-b, both of them are just eks setups (two control planes needed) but they can both share subnets and the vpc - possible?

I appreciate this would cause pain when doing terraform destroy, not knowing the other stack exists - so maybe it’s just a bad idea

yeah just ignore that, it’s a stupid idea to cross the streams, it’s delicate enough in terraform as it is

Yes, the components are shared. See this.

# catalog/vpc/example.yaml

components:
  terraform:
    vpc/defaults:
      metadata:
        type: abstract
        # corresponds to components/terraform/vpc
        component: vpc
     vars:
       enabled: true

    vpc/example:
      metadata:
        # real is implicit and optional
        type: real
        component: vpc
        inherits:
        - vpc/defaults
      vars:
        name: example

# stacks/uw2-dev.yaml
import:
- catalog/vpc/example

# stacks/uw2-prod.yaml
import:
- catalog/vpc/example

thats a shared component reused across 2 accounts

oh damn.

that is powerful

ya… it is and we really need to blog on it more lol

now, if the documentation was exhaustive…

blogs are cool, lots of examples of where terraform falls short and how to solve that problem on the docs site will be great

and we all know terraform falls short in some catastrophic ways…

say all the terraform resources are prefixed in their name with a string, and all I want to do is provision an entire stack with a dynamic prefix - and the state be named accordingly:

say in terraform land it’d be TF_VAR_PREFIX=abc terraform plan and TF_VAR_PREFIX=abc terraform apply to bring up stacks where the states live in s3:\\examplebucket\states\${AWS_REGION}\${PREFIX}\${COMPONENT}.state so the backend key is dynamic based on component, and prefix (and optionally region)

@Daniel Loader check this out — https://github.com/cloudposse/atmos/blob/master/examples/complete/stacks/catalog/terraform/test-component-override.yaml#L11-L18

In that example, the component has a specific name that will end up in the workspace name (test/test-component-override) and it references that its code comes from test/test-component. I believe that is what you’re looking for.

AFAIK, you can’t accomplish that from the command line, but in the IaC / infrastructure as data world, you don’t really want to as that would go against best practice.

that’s fair feedback, was looking for a way to provision dynamic sandboxes for devs, that last 1-8hrs at the most

Yeah you can do that via a pipeline that adds some YAML to your stacks and then runs a fresh apply of that new stack. Then have a job come back around and clean it up later.

And we do that with atmos. The difference is we commit that file. What’s rad is then you have a record of it it, versus if you just do it on the command line without state, it’s inconsistent

Atmos supports wildcard includes, so we create a folder called previews/ and programmatically commit a file to that folder which then gets automatically depoyed

Delete the file and it automatically gets destroyed