#gcp (2024-03)

Google Cloud Platform

2024-03-01

Christopher McGill avatar
Christopher McGill

Question. In atmos.yaml we are using “auto_generate_backend_file: true” and seeing in S3 atmos creating a folder for component then a sub-folder for stack, which it them places the terraform.tf into. When we run the same layout of components/stacks/config against GCP GCS we are seeing only a state file be created, no folders, example core-usc1-auto.tfstate which it is renaming the terraform.tf to which is the name of the stage. Has anyone seen this behaviour or can advise? Thanks

Google Cloud Platform

Google Cloud Platform lets you build, deploy, and scale applications, websites, and services on the same infrastructure as Google.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

did you correctly configure GCP backend in Atmos manifests? something like this:

  terraform:
    # Backend
    backend_type: gcs
    backend:
      gcs:
        bucket: "xxxxxxx-bucket-tfstate"
        prefix: "terraform/tfstate" 
Google Cloud Platform

Google Cloud Platform lets you build, deploy, and scale applications, websites, and services on the same infrastructure as Google.

Christopher McGill avatar
Christopher McGill

In a _defaults.yaml in a parent stack folder that in imported we have:

terraform:
  backend_type: gcs
  backend:
    gcs:
      bucket: poc-tfstate-bucket
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)


and seeing in S3 atmos creating a folder for component then a sub-folder for stack,

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

can you give more details on this? (I don’t understand “atmos creating a folder for component then a sub-folder for stack” - the folders for components and stacks must already exist in your repo)

Christopher McGill avatar
Christopher McGill

In AWS the bucket has these folders, that are components.

Christopher McGill avatar
Christopher McGill

Followed by these stacks folders

Christopher McGill avatar
Christopher McGill
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

ok, so you are saying that the S3 backend creates those folders in the S3 bucket, but you are not seeing the GCP backend creating those folders in the GCP bucket?

Christopher McGill avatar
Christopher McGill

yes

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it depends on TF backend and workspaces. In AWS S3 backend, for example, the workspace_key_prefix (which you can specify in Atmos config for each component; if not specified, then Atmos component name is used) is the main folder name. The TF workspaces for diff stacks are subfolders. That’s how the S3 backend in TF works

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(I’m not sure if GCP backend works the same way, or it just creates a flat folder structure - you need to confirm that in terrform docs for GCP)

Christopher McGill avatar
Christopher McGill

Thanks

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you need to confirm if GCP backend has a similar concept like workspace_key_prefix in the AWS S3 backend (which creates those top-level folders in the S3 state bucket)

Christopher McGill avatar
Christopher McGill

Thank You, will look into that feature.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Backend Type: gcs | Terraform | HashiCorp Developerattachment image

Terraform can store the state remotely, making it easier to version and work with in a team.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

I think it’s the prefix (which is the same as orkspace_key_prefix in the S3 backend)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you can configure prefix for each component in Atmos manifests

components:
  terraform:
     <my-component>:
        backend:
           gcp:
              prefix: <my-component>
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Atmos does not generate prefix for GCP automatically, it only does it for S3 backend workspace_key_prefix (we will take a look at this to add it to Atmos to generate prefix automatically using Atmos components names if prefix is not configured for a component)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

for now, try the above config on your components, it should generate the folders inside the state bucket (Named states for workspaces are stored in an object called <prefix>/<name>.tfstate.)

Christopher McGill avatar
Christopher McGill

Thanks, that is working.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

super

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

thank you for testing and pointing it out - we’ll add the auto-generation of prefix for GCP backend to Atmos (similar to what it does for workspace_key_prefix for S3 backend)

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

auto-generation of prefix for gcs backend was added to Atmos 1.65.0

https://github.com/cloudposse/atmos/releases/tag/v1.65.0

2024-03-23

Monish Devendran avatar
Monish Devendran

Hello Team, Im planning to start a GCP setup via atmos. Can some one provide me with basic structure ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This one has some examples for GCP

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
slalombuild/terraform-atmos-accelerator

An opinionated, multi-cloud, multi-region, best-practice accelerator for Terraform.

1
Monish Devendran avatar
Monish Devendran

thanks @Erik Osterman (Cloud Posse)

Monish Devendran avatar
Monish Devendran

I want to target multiple services in different projects, example : cloudfunction should go to project1 and all cloudrun should be project2 and storage in project3. So how can i design such a way each stack can go to different project

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That makes sense. Projects might be like accounts in AWS. So you have project level isolation.

Monish Devendran avatar
Monish Devendran

yes !

Monish Devendran avatar
Monish Devendran
How to use multiple GCP providers in Terraform!!attachment image

In this article, you will not just learn about terraform providers but will also learn how to deploy cloud resources across multiple…

Monish Devendran avatar
Monish Devendran

so i might have different service account for each project

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Monish Devendran please review these docs (they are for AWS, but you can update them for GCP)

https://atmos.tools/design-patterns/organizational-structure-configuration

Organizational Structure Configuration Atmos Design Pattern | atmos

Organizational Structure Configuration Atmos Design Pattern

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
CLI Configuration | atmos

Use the atmos.yaml configuration file to control the behavior of the atmos CLI.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

the pattern is, you define your stack config per Org, project, account, etc. (configure the context variables in each). Then define or import the Atmos components into each top-level stack.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you can define defaults for each component in catalog, then import the defaults into the stacks, and override the component config per stack if needed

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(you can DM me your setup, I’ll review)

Monish Devendran avatar
Monish Devendran

@Andriy Knysh (Cloud Posse), Thanks . setup and will send

Monish Devendran avatar
Monish Devendran
monish-devendran/platform-orchestrator

extended from atmos to leverage orchestration in terraform

1
Monish Devendran avatar
Monish Devendran

very basic setup, im trying to read above links and trying to see. also can you send me a link to configure terraform cloud

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

i’ll review your repo, thanks for sharing

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)


configure terraform cloud

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Terraform Cloud | Terraform | HashiCorp Developerattachment image

Collaborate on version-controlled configuration using Terraform Cloud. Follow this track to build, change, and destroy infrastructure using remote runs and state.

Monish Devendran avatar
Monish Devendran

yes , my statefiles i want to run in tf cloud

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Or more specifically use terraform cloud strict as a state backend? Or also use the deployment capabilities as well of terraform cloud

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Monish Devendran I’ve sent you a zip file with the infra configuration for GCP with a few stages (dev, prod, staging) and a few GCP projects with multiple Atmos components in each (specific component type like Cloud Functions and GCS buckets per GCP project). We can sync anytime to review it

1
Monish Devendran avatar
Monish Devendran

Hi @Erik Osterman (Cloud Posse), So my use case is to use terraform cloud to run terraform apply and store them in tf cloud

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In that case you will need to commit the varfiles and configure terraform cloud to use the appropriate varfile with each workspace

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Comment on #15966 Feature: Conditionally load tfvars/tf file based on Workspace

I was able to do this with Terraform Cloud by adding an environment variable to the workspace:
Key: TF_CLI_ARGS
Value: -var-file "dev.tfvars"

Now when I run terraform apply it turns into terraform apply -var-file dev.tfvars. It will do this for all terraform commands. (See here: https://developer.hashicorp.com/terraform/cli/config/environment-variables#tf_cli_args-and-tf_cli_args_name)

It would be better if we could apply this in the code itself though.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Heads up, @RB I saw you commented on this thread and this appears to be solution.

2024-03-24

2024-03-28

Monish Devendran avatar
Monish Devendran

Can someone help me,

Im trying to pass a secret which is stored in akeyless,

data "akeyless_secret" "secret" { path = "/GCP/Secrets/cf-triggers/tf-cf-triggers" }

provider "google" { project = "cf-triggers" credentials = data.akeyless_secret.secret }

resource "google_pubsub_topic" "example" { name = "akeyless_topic" message_retention_duration = "86600s" }

❯ terraform apply
data.akeyless_secret.secret: Reading...
data.akeyless_secret.secret: Read complete after 1s [id=/GCP/Secrets/cf-triggers/tf-cf-triggers]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Incorrect attribute value type
│ 
│   on main.tf line 38, in provider "google":
│   38:   credentials = data.akeyless_secret.secret
│     ├────────────────
│     │ data.akeyless_secret.secret is object with 4 attributes
│ 
│ Inappropriate value for attribute "credentials": string required.
Monish Devendran avatar
Monish Devendran

Has anyone faced this issue, not able to pass secret from akeyless

    keyboard_arrow_up