#gcp (2024-03)

did you correctly configure GCP backend in Atmos manifests? something like this:

  terraform:
    # Backend
    backend_type: gcs
    backend:
      gcs:
        bucket: "xxxxxxx-bucket-tfstate"
        prefix: "terraform/tfstate" 

In a _defaults.yaml in a parent stack folder that in imported we have:

terraform:
  backend_type: gcs
  backend:
    gcs:
      bucket: poc-tfstate-bucket

and seeing in S3 atmos creating a folder for component then a sub-folder for stack,

can you give more details on this? (I don’t understand “atmos creating a folder for component then a sub-folder for stack” - the folders for components and stacks must already exist in your repo)

In AWS the bucket has these folders, that are components.

Followed by these stacks folders

ok, so you are saying that the S3 backend creates those folders in the S3 bucket, but you are not seeing the GCP backend creating those folders in the GCP bucket?

yes

it depends on TF backend and workspaces. In AWS S3 backend, for example, the workspace_key_prefix (which you can specify in Atmos config for each component; if not specified, then Atmos component name is used) is the main folder name. The TF workspaces for diff stacks are subfolders. That’s how the S3 backend in TF works

(I’m not sure if GCP backend works the same way, or it just creates a flat folder structure - you need to confirm that in terrform docs for GCP)

Thanks

you need to confirm if GCP backend has a similar concept like workspace_key_prefix in the AWS S3 backend (which creates those top-level folders in the S3 state bucket)

Thank You, will look into that feature.

https://developer.hashicorp.com/terraform/language/settings/backends/gcs#prefix

I think it’s the prefix (which is the same as orkspace_key_prefix in the S3 backend)

you can configure prefix for each component in Atmos manifests

components:
  terraform:
     <my-component>:
        backend:
           gcp:
              prefix: <my-component>

Atmos does not generate prefix for GCP automatically, it only does it for S3 backend workspace_key_prefix (we will take a look at this to add it to Atmos to generate prefix automatically using Atmos components names if prefix is not configured for a component)

for now, try the above config on your components, it should generate the folders inside the state bucket (Named states for workspaces are stored in an object called <prefix>/<name>.tfstate.)

Thanks, that is working.

super

thank you for testing and pointing it out - we’ll add the auto-generation of prefix for GCP backend to Atmos (similar to what it does for workspace_key_prefix for S3 backend)

auto-generation of prefix for gcs backend was added to Atmos 1.65.0

https://github.com/cloudposse/atmos/releases/tag/v1.65.0

This one has some examples for GCP

https://github.com/slalombuild/terraform-atmos-accelerator

thanks @Erik Osterman (Cloud Posse)

That makes sense. Projects might be like accounts in AWS. So you have project level isolation.

yes !

example : https://medium.com/google-cloud/multiple-gcp-providers-in-terraform-64d20673519e

@Monish Devendran please review these docs (they are for AWS, but you can update them for GCP)

https://atmos.tools/design-patterns/organizational-structure-configuration

https://atmos.tools/cli/configuration#stacks

the pattern is, you define your stack config per Org, project, account, etc. (configure the context variables in each). Then define or import the Atmos components into each top-level stack.

you can define defaults for each component in catalog, then import the defaults into the stacks, and override the component config per stack if needed

(you can DM me your setup, I’ll review)

@Andriy Knysh (Cloud Posse), Thanks . setup and will send

https://github.com/monish-devendran/platform-orchestrator

my repo

very basic setup, im trying to read above links and trying to see. also can you send me a link to configure terraform cloud

i’ll review your repo, thanks for sharing

configure terraform cloud

you mean https://developer.hashicorp.com/terraform/tutorials/cloud-get-started

yes , my statefiles i want to run in tf cloud

Or more specifically use terraform cloud strict as a state backend? Or also use the deployment capabilities as well of terraform cloud

@Monish Devendran I’ve sent you a zip file with the infra configuration for GCP with a few stages (dev, prod, staging) and a few GCP projects with multiple Atmos components in each (specific component type like Cloud Functions and GCS buckets per GCP project). We can sync anytime to review it

Hi @Erik Osterman (Cloud Posse), So my use case is to use terraform cloud to run terraform apply and store them in tf cloud

In that case you will need to commit the varfiles and configure terraform cloud to use the appropriate varfile with each workspace

To do that see this comment, https://github.com/hashicorp/terraform/issues/15966#issuecomment-1868823558

Heads up, @RB I saw you commented on this thread and this appears to be solution.