#github-actions (2024-07)

Discussions related to GitHub Actions

2024-07-02

Marat Bakeev avatar
Marat Bakeev

I think there’s an issue with github-action-atmos-affected-stacks - if we start using atmos lookups, then the action fails, because there’s no terraform in the action issues/41

cloudposse/github-action-atmos-affected-stacks
2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Aha, looks like the action needs to install terraform. Good catch

cloudposse/github-action-atmos-affected-stacks
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Cc @Gabriela Campana (Cloud Posse) @Andriy Knysh (Cloud Posse)

Marat Bakeev avatar
Marat Bakeev

It actually was installing terraform before, but was removed on v3 release

RB avatar

Maybe that was so you can choose between installing terraform and tofu ?

RB avatar

https://github.com/cloudposse/github-action-atmos-affected-stacks/releases/tag/3.0.0
The notable changes in v3 are:

v3 works only with atmos
= 1.63.0
v3 drops install-terraform input because terraform is not required for affected stacks call
v3 drops atmos-gitops-config-path input and the ./.github/config/atmos-gitops.yaml config file. Now you have to use GitHub Actions environment variables to specify the location of the atmos.yaml.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)



v3 drops install-terraform input because terraform is not required for affected stacks call

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This was true, until we added atmos.Component function, which I believe executes terraform outputs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) , can you confirm? cc @Igor Rodionov

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if you are using atmos.Component in templates, then yes, terraform must be installed

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Gabriela Campana (Cloud Posse) let’s create a task to restore this functionality, with a feature flag.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

For now, @Marat Bakeev you should be able to work around this by adding the step to install terraform (or open tofu) in your GHA workflow

Marat Bakeev avatar
Marat Bakeev

Unfortunately, it will still fail, even with terraform installed. It also wants node. For now, we just hardcoded our values and are not using lookups, but it would be great to be able to use them.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

what wants Node? it’s probably in some of your templates

Marat Bakeev avatar
Marat Bakeev

I’m using reference arch from you guys, I don’t know. I enabled templates to get the lookups working, and it started asking for node

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it’s probably related to the GH actions. Nothing in the templates needs Node (unless it’s some function in the templates)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Matt Calhoun @Dan Miller (Cloud Posse) any ideas? This is based on our latest refarch from last week

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Marat Bakeev by any chance is the terraform component using node under the hood? e.g. with a local exec?

Marat Bakeev avatar
Marat Bakeev

@Erik Osterman (Cloud Posse) honestly, I’m not sure. We didn’t add anything!

Igor Rodionov avatar
Igor Rodionov

Here is the PR to rollback terraform installation. Waiting for a review to merge

Igor Rodionov avatar
Igor Rodionov

@Marat Bakeev The PR was released at v3.1.0

Igor Rodionov avatar
Igor Rodionov

Could you pls rerun your workflows to see if the problem solved

Marat Bakeev avatar
Marat Bakeev

@Alexander Matveev see this?

Alexander Matveev avatar
Alexander Matveev

@Marat Bakeev Got it. Thanks!

Marat Bakeev avatar
Marat Bakeev

@Igor Rodionov it seems to have fixed it, thanks!

2

2024-07-03

2024-07-04

2024-07-07

2024-07-08

actions Archives - The GitHub Blog avatar
actions Archives - The GitHub Blog
08:25:35 PM

GitHub Actions: GPU hosted runners are now generally available GitHub Actions: GPU hosted runners are now generally available The post GitHub Actions: GPU hosted runners are now generally available appeared first on The GitHub Blog.

GitHub Actions: GPU hosted runners are now generally available

GitHub Actions: GPU hosted runners are now generally available

The GitHub Blog - Updates, ideas, and inspiration from GitHub to help developers build and design software.

Updates, ideas, and inspiration from GitHub to help developers build and design software.

2024-07-09

Marat Bakeev avatar
Marat Bakeev

I now suddenly have issues with Atmos Terraform Drift Remediation - it fails on authentication to Kubernetes - getting credentials: exec: executable aws not found. We changed atmos version from 1.80 to 1.83, though. Is it not supported?

Marat Bakeev avatar
Marat Bakeev

They’re using exactly the same workflow file:

jobs:
  remediate-drift:
    if: github.event.action == 'labeled' && contains(github.event.issue.labels.*.name, 'apply')
    name: Remediate Drift
    runs-on: ["self-hosted", "terraform"]
    steps:
      - name: Remediate Drift
        uses: cloudposse/github-action-atmos-terraform-drift-remediation@v2
        with:
          issue-number: ${{ github.event.issue.number }}
          action: remediate
          atmos-version: ${{ vars.ATMOS_VERSION }}
          atmos-config-path: ${{ vars.ATMOS_CONFIG_PATH }}
Marat Bakeev avatar
Marat Bakeev

Last week it was working fine, now it can’t find aws -_-

Marat Bakeev avatar
Marat Bakeev
2024-07-09T04:57:58.8486056Z kubernetes_manifest.ec2_node_class["default"]: Refreshing state...
2024-07-09T04:57:58.8486559Z 
2024-07-09T04:57:58.8487036Z Planning failed. Terraform encountered an error while generating this plan.
2024-07-09T04:57:58.8487606Z 
2024-07-09T04:57:58.8487614Z 
2024-07-09T04:57:58.8487996Z Error: Failed to determine GroupVersionResource for manifest
2024-07-09T04:57:58.8488459Z 
2024-07-09T04:57:58.8488741Z   with kubernetes_manifest.ec2_node_class["default"],
2024-07-09T04:57:58.8489707Z   on ec2-node-class.tf line 26, in resource "kubernetes_manifest" "ec2_node_class":
2024-07-09T04:57:58.8490581Z   26: resource "kubernetes_manifest" "ec2_node_class" {
2024-07-09T04:57:58.8490982Z 
2024-07-09T04:57:58.8491110Z Get
2024-07-09T04:57:58.8492081Z "<https://CA639ED0C99EB1C5F6EF28CFA19B151B.gr7.ap-southeast-2.eks.amazonaws.com/api?timeout=32s>":
2024-07-09T04:57:58.8493149Z getting credentials: exec: executable aws not found
2024-07-09T04:57:58.8493554Z 
2024-07-09T04:57:58.8494120Z It looks like you are trying to use a client-go credential plugin that is not
2024-07-09T04:57:58.8494812Z installed.
2024-07-09T04:57:58.8495016Z 
2024-07-09T04:57:58.8495477Z To learn more about this feature, consult the documentation available at:
2024-07-09T04:57:58.8496772Z       <https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins>
2024-07-09T04:57:58.8497665Z exit status 1
2024-07-09T04:57:58.8497877Z 
2024-07-09T04:57:58.8498069Z Terraform apply failed
2024-07-09T04:57:58.8510018Z ##[error]Process completed with exit code 1.
Marat Bakeev avatar
Marat Bakeev
2024-07-09T04:56:36.7059673Z Current runner version: '2.317.0'
2024-07-09T04:56:36.7065783Z Runner name: 'platform-runner-f7gcw-4z6vs'
2024-07-09T04:56:36.7106854Z Runner group name: 'Default'
2024-07-09T04:56:36.7107832Z Machine name: 'platform-runner-f7gcw-4z6vs'
2024-07-09T04:56:36.7110563Z ##[group]GITHUB_TOKEN Permissions
2024-07-09T04:56:36.7112679Z Contents: read
2024-07-09T04:56:36.7113113Z Issues: write
2024-07-09T04:56:36.7113600Z Metadata: read
2024-07-09T04:56:36.7114039Z ##[endgroup]
2024-07-09T04:56:36.7116415Z Secret source: Actions
2024-07-09T04:56:36.7117113Z Prepare workflow directory
2024-07-09T04:56:36.7904054Z Prepare all required actions
2024-07-09T04:56:36.8080857Z Getting action download info
2024-07-09T04:56:37.1516171Z Download action repository 'cloudposse/github-action-atmos-terraform-drift-remediation@v2' (SHA:65aabe2c0189b45b603be815dae1d839d23bd90a)
2024-07-09T04:56:38.0288473Z Getting action download info
2024-07-09T04:56:38.3326806Z Download action repository 'actions/github-script@v6' (SHA:d7906e4ad0b1822421a7e6a35d5ca353c962f410)
2024-07-09T04:56:39.0027454Z Download action repository 'cloudposse/github-action-atmos-terraform-apply@v2' (SHA:3ec18a48ec4acec3c2760036343603171f17938b)
2024-07-09T04:56:39.7414471Z Getting action download info
2024-07-09T04:56:40.1422584Z Download action repository 'actions/checkout@v4' (SHA:692973e3d937129bcbf40652eb9f2f61becf3332)
2024-07-09T04:56:40.8160936Z Download action repository 'cloudposse/github-action-setup-atmos@v2' (SHA:7137afeb830d41684b464f9677a13566d4bcb550)
2024-07-09T04:56:41.4994630Z Download action repository 'hashicorp/setup-terraform@v3' (SHA:651471c36a6092792c552e8b1bef71e592b462d8)
2024-07-09T04:56:42.1969633Z Download action repository 'actions/setup-node@v4' (SHA:60edb5dd545a775178f52524783378180af0d1f8)
2024-07-09T04:56:42.9492869Z Download action repository 'cloudposse/github-action-atmos-get-setting@v1' (SHA:644b6ec2d51b24455905f7158804a41962c6d6b2)
2024-07-09T04:56:43.6624238Z Download action repository 'jaxxstorm/[email protected]' (SHA:71d17cb091aa850acb2a1a4cf87258d183eb941b)
2024-07-09T04:56:47.1942062Z Download action repository 'aws-actions/[email protected]' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502)
2024-07-09T04:56:47.8832345Z Download action repository 'cloudposse/github-action-terraform-plan-storage@v1' (SHA:0b13c5198170a1636557f008e1737c662ed2eb86)
2024-07-09T04:56:48.6733651Z Download action repository 'infracost/actions@v3' (SHA:d5dd739fb5eae81f31bf09fea5c290186a91964a)
2024-07-09T04:56:49.5715885Z Complete job name: Remediate Drift
2024-07-09T04:56:49.6204174Z A job started hook has been configured by the self-hosted runner administrator
2024-07-09T04:56:49.6356034Z ##[group]Run '/etc/arc/hooks/job-started.sh'
2024-07-09T04:56:49.6366017Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-09T04:56:49.6366579Z ##[endgroup]
2024-07-09T04:56:49.6646354Z [0;37m2024-07-09 04:56:49.656  DEBUG --- Running ARC Job Started Hooks[0m
2024-07-09T04:56:49.6648104Z [0;37m2024-07-09 04:56:49.657  DEBUG --- Running hook: /etc/arc/hooks/job-started.d/update-status[0m
2024-07-09T04:56:49.7212378Z ##[group]Run cloudposse/github-action-atmos-terraform-drift-remediation@v2
2024-07-09T04:56:49.7213119Z with:
2024-07-09T04:56:49.7213397Z   issue-number: 128
2024-07-09T04:56:49.7213714Z   action: remediate
2024-07-09T04:56:49.7214023Z   atmos-version: 1.83.1
2024-07-09T04:56:49.7214440Z   atmos-config-path: ./rootfs/usr/local/etc/atmos/
2024-07-09T04:56:49.7214922Z   debug: false
2024-07-09T04:56:49.7215447Z   token: ***
2024-07-09T04:56:49.7215734Z ##[endgroup]
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Are you able to downgrade in the meantime?

Marat Bakeev avatar
Marat Bakeev

the lookups would stop working %) and I’m not sure it would help, but I can try

Marat Bakeev avatar
Marat Bakeev

It didn’t help, it still complains about missing aws executable -_- Last week’s runs weren’t touching kubernetes, so I guess that’s why it worked

Alexander Matveev avatar
Alexander Matveev

The root cause of the issue is that .github/workflows/atmos-terraform-drift-remediation.yaml misses - uses: unfor19/install-aws-cli-action@v1 which exists both in atmos-terraform-plan-matrix.yaml and atmos-terraform-apply-matrix.yaml

1
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov

Igor Rodionov avatar
Igor Rodionov

@Marat Bakeev @Alexander Matveev is right. You need to install aws cli like we had in plan and apply workflows. The case is that the Kubernetes component uses AWS CLI in provider,tf to auth on the K8s API. This is an edge case. In theory, there is a wide range of third-party APIs that need different CLIs to auth a service. For k8s, I will do the fix in our workflows; that’s ridiculous. We fixed it in the plan and apply, but missed it in drift remediation. Thanks for pointing out.

Igor Rodionov avatar
Igor Rodionov

@Marat Bakeev @Erik Osterman (Cloud Posse) Here is the PR to fix our refarch

https://github.com/cloudposse/refarch-scaffold/pull/678

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Aha, yes, in the older EKS component this is true. The latest release does not. @dan I think this is a painless upgrade. Cc @Jeremy G (Cloud Posse)

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

Not exactly painless upgrade, as it requires multiple manual steps, but not terrible, either. Takes about 20 minutes per cluster.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Matt Calhoun

2024-07-10

2024-07-11

Marat Bakeev avatar
Marat Bakeev

Hey @Igor Rodionov @Erik Osterman (Cloud Posse) - we have more issues with github actions. Atmos Terraform Drift Detection fails to run if we have lookups enabled in atmos. Again, it requires terraform and nodejs to be installed. It can be fixed by adding that as steps to atmos-terraform-drift-detection.yaml

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.5.7
      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: 16

Another issue is - even with all of this added, lookups do not work from github actions. The lookup returns nothing.

Error: reading IAM Instance Profile (<no value>): operation error IAM: GetInstanceProfile, https response error StatusCode: 400, RequestID: d0ab23da-fb09-4a7e-a5b8-3a34b24bdc77, api error ValidationError: 1 validation error detected: Value '<no value>' at 'instanceProfileName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\w+=,.@-]+

The lookup looks like this:

instance_profile: '{{ (atmos.Component "iam-role/azure-agent" .stack).outputs.role.instance_profile }}'

This same code works fine locally in geodesic shell!

if I run atmos terraform output iam-role/azure-agent --stack core-apse2-auto I can see the output I’m looking up is there:

Terraform has been successfully initialized!
role = {
  "arn" = "arn:aws:iam::BOGUS:role/nsp-core-apse2-auto-azure-agent"
  "id" = "AROABOGUS6SEEH3A63"
  "instance_profile" = "nsp-core-apse2-auto-azure-agent"
  "name" = "nsp-core-apse2-auto-azure-agent"
  "policy" = ""
}

and local atmos picks it up atmos terraform plan ec2-instance/agent1 --stack core-apse2-auto

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Gabriela Campana (Cloud Posse) please help follow up

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Re, node I have to think it’s based on the self hosted runner images not containing some packages, like node. Also the node 16 is very old.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse)

Igor Rodionov avatar
Igor Rodionov

@Marat Bakeev can you send me whole *atmos-terraform-drift-detection.yaml* so I will see where you added the steps. Thanks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Re: atmos.Component not working in the action, did you set vars.ATMOS_VERSION to the latest version?

atmos-version: ${{ vars.ATMOS_VERSION }}

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

<no value> is a result from Go templates if it was not evaluated. Something is not configured in the GH action: atmos.yaml , Atmos version, etc. Since it’s working locally (both components iam-role/azure-agent and ec2-instance/agent1), then someting is missing in the action config (that can be Atmos version, or path to atmos.yaml, or config inside atmos.yaml)

Dan Miller (Cloud Posse) avatar
Dan Miller (Cloud Posse)

We just discussed this internally. @Jeremy G (Cloud Posse) has additional context

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

I am concerned about azure appearing in strings here. I have no experience with Azure and I don’t know how much, if at all, our stuff has been tested with Azure. While we would like to be cloud agnostic, our stuff has been focused on AWS.

I’m not exactly sure what you have. While I check on that, you should be getting Terraform installed for you. That gets configured in rootfs/usr/local/etc/atmos/atmos.yaml like this:

integrations:
  github:
    gitops:
      terraform-version: 1.5.7 

As for Node, we have gone through some iterations about which image to use for our self-hosted runners. Most people want to preserve the option to use containers in their workflows, so although if you don’t need it, it is more efficient to not have it, we have decided to generally recommend enabling Docker-in-Docker for runners.

Additionally, while it is general Cloud Posse policy to use explicit version pinning on resources like Docker images, for runners, it is counter productive, because the first thing the runner does when it starts up is check to see if an update is available, and if one its, it installs it and then restarts. So having an explicit pin does not achieve the goal of pinning which version is actually run, and adds extra workload by forcing auto updates.

Putting the above together, we now recommend your runner use the following configuration snippet:

dind_enabled: true
image: ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner-dind:ubuntu-20.04

I believe we sent you a configuration that uses the default image, which is now based on Ubuntu 22.04, and I think that explains the issues you were having with Node.

If you are sure you never need Docker in your runners, you can use

dind_enabled: false
image: ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:ubuntu-20.04

Also, I recommend upgrading eks/actions-runner-controller to version 1.470.1

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Marat Bakeev are you unblocked?

1

2024-07-12

2024-07-15

2024-07-16

    keyboard_arrow_up