#github-actions (2024-07)

Aha, looks like the action needs to install terraform. Good catch

@Igor Rodionov

Cc @Gabriela Campana (Cloud Posse) @Andriy Knysh (Cloud Posse)

It actually was installing terraform before, but was removed on v3 release

Maybe that was so you can choose between installing terraform and tofu ?

https://github.com/cloudposse/github-action-atmos-affected-stacks/releases/tag/3.0.0
The notable changes in v3 are:

v3 works only with atmos
= 1.63.0
v3 drops install-terraform input because terraform is not required for affected stacks call
v3 drops atmos-gitops-config-path input and the ./.github/config/atmos-gitops.yaml config file. Now you have to use GitHub Actions environment variables to specify the location of the atmos.yaml.

v3 drops install-terraform input because terraform is not required for affected stacks call

This was true, until we added atmos.Component function, which I believe executes terraform outputs

@Andriy Knysh (Cloud Posse) , can you confirm? cc @Igor Rodionov

if you are using atmos.Component in templates, then yes, terraform must be installed

@Gabriela Campana (Cloud Posse) let’s create a task to restore this functionality, with a feature flag.

For now, @Marat Bakeev you should be able to work around this by adding the step to install terraform (or open tofu) in your GHA workflow

Unfortunately, it will still fail, even with terraform installed. It also wants node. For now, we just hardcoded our values and are not using lookups, but it would be great to be able to use them.

what wants Node? it’s probably in some of your templates

I’m using reference arch from you guys, I don’t know. I enabled templates to get the lookups working, and it started asking for node

it’s probably related to the GH actions. Nothing in the templates needs Node (unless it’s some function in the templates)

@Matt Calhoun @Dan Miller (Cloud Posse) any ideas? This is based on our latest refarch from last week

@Marat Bakeev by any chance is the terraform component using node under the hood? e.g. with a local exec?

DEV-2397 : Restore functionality in github-action-atmos-affected-stacksto install Terraform with a feature flag

@Erik Osterman (Cloud Posse) honestly, I’m not sure. We didn’t add anything!

https://github.com/cloudposse/github-action-atmos-affected-stacks/pull/42

Here is the PR to rollback terraform installation. Waiting for a review to merge

@Marat Bakeev The PR was released at v3.1.0

Could you pls rerun your workflows to see if the problem solved

@Alexander Matveev see this?

@Marat Bakeev Got it. Thanks!

@Igor Rodionov it seems to have fixed it, thanks!

They’re using exactly the same workflow file:

jobs:
  remediate-drift:
    if: github.event.action == 'labeled' && contains(github.event.issue.labels.*.name, 'apply')
    name: Remediate Drift
    runs-on: ["self-hosted", "terraform"]
    steps:
      - name: Remediate Drift
        uses: cloudposse/github-action-atmos-terraform-drift-remediation@v2
        with:
          issue-number: ${{ github.event.issue.number }}
          action: remediate
          atmos-version: ${{ vars.ATMOS_VERSION }}
          atmos-config-path: ${{ vars.ATMOS_CONFIG_PATH }}

Last week it was working fine, now it can’t find aws -_-

Are you able to downgrade in the meantime?

the lookups would stop working %) and I’m not sure it would help, but I can try

It didn’t help, it still complains about missing aws executable -_- Last week’s runs weren’t touching kubernetes, so I guess that’s why it worked

The root cause of the issue is that .github/workflows/atmos-terraform-drift-remediation.yaml misses - uses: unfor19/install-aws-cli-action@v1 which exists both in atmos-terraform-plan-matrix.yaml and atmos-terraform-apply-matrix.yaml

@Igor Rodionov

@Marat Bakeev @Alexander Matveev is right. You need to install aws cli like we had in plan and apply workflows. The case is that the Kubernetes component uses AWS CLI in provider,tf to auth on the K8s API. This is an edge case. In theory, there is a wide range of third-party APIs that need different CLIs to auth a service. For k8s, I will do the fix in our workflows; that’s ridiculous. We fixed it in the plan and apply, but missed it in drift remediation. Thanks for pointing out.

@Marat Bakeev @Erik Osterman (Cloud Posse) Here is the PR to fix our refarch

https://github.com/cloudposse/refarch-scaffold/pull/678

Aha, yes, in the older EKS component this is true. The latest release does not. @dan I think this is a painless upgrade. Cc @Jeremy G (Cloud Posse)

Not exactly painless upgrade, as it requires multiple manual steps, but not terrible, either. Takes about 20 minutes per cluster.

@Matt Calhoun

@Gabriela Campana (Cloud Posse) please help follow up

Re, node I have to think it’s based on the self hosted runner images not containing some packages, like node. Also the node 16 is very old.

@Andriy Knysh (Cloud Posse)

@Marat Bakeev can you send me whole *atmos-terraform-drift-detection.yaml* so I will see where you added the steps. Thanks

Re: atmos.Component not working in the action, did you set vars.ATMOS_VERSION to the latest version?

atmos-version: ${{ vars.ATMOS_VERSION }}

<no value> is a result from Go templates if it was not evaluated. Something is not configured in the GH action: atmos.yaml , Atmos version, etc. Since it’s working locally (both components iam-role/azure-agent and ec2-instance/agent1), then someting is missing in the action config (that can be Atmos version, or path to atmos.yaml, or config inside atmos.yaml)

We just discussed this internally. @Jeremy G (Cloud Posse) has additional context

I am concerned about azure appearing in strings here. I have no experience with Azure and I don’t know how much, if at all, our stuff has been tested with Azure. While we would like to be cloud agnostic, our stuff has been focused on AWS.

I’m not exactly sure what you have. While I check on that, you should be getting Terraform installed for you. That gets configured in rootfs/usr/local/etc/atmos/atmos.yaml like this:

integrations:
  github:
    gitops:
      terraform-version: 1.5.7 

As for Node, we have gone through some iterations about which image to use for our self-hosted runners. Most people want to preserve the option to use containers in their workflows, so although if you don’t need it, it is more efficient to not have it, we have decided to generally recommend enabling Docker-in-Docker for runners.

Additionally, while it is general Cloud Posse policy to use explicit version pinning on resources like Docker images, for runners, it is counter productive, because the first thing the runner does when it starts up is check to see if an update is available, and if one its, it installs it and then restarts. So having an explicit pin does not achieve the goal of pinning which version is actually run, and adds extra workload by forcing auto updates.

Putting the above together, we now recommend your runner use the following configuration snippet:

dind_enabled: true
image: ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner-dind:ubuntu-20.04

I believe we sent you a configuration that uses the default image, which is now based on Ubuntu 22.04, and I think that explains the issues you were having with Node.

If you are sure you never need Docker in your runners, you can use

dind_enabled: false
image: ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:ubuntu-20.04

Also, I recommend upgrading eks/actions-runner-controller to version 1.470.1

@Marat Bakeev are you unblocked?

That’s a lot of minutes used in a month.