#github-actions (2023-08)

@Igor Rodionov

context: https://github.com/kayac/ecspresso/issues/598

Didn’t you mean to do exit 1? with exit code 0 it’s a success and the workflow continues. That however fails the run and you may not want that. At the price of duplication if there’s more steps to follow, you can include the condition to every step

if: steps.changed-files.outputs.any_changed == 'true'

I guess this is the very thing you wanted to avoid

you could also split jobs so job1 only runs changed-files action and exposed an output and then job2 runs based on the if condition - no need to duplicate to all steps

My main issue is I want the test job to be required for PR’s. So if files in a certain folder has changed run the tests, if not changed in that folder the test will just return success to make the required test pass in the PR to allow it to be merged into main.

I see, I can’t think of any GHA native way to achieve it

@Dan Miller (Cloud Posse)

I’m not familiar with tj-actions/changed-files , but we do something similiar with Atmos and the github-action-atmos-affected-stacks action

It doesn’t matter how you detect the affected stacks or changed files. You get a bool on the output and the question is what you can do with it to end the pipeline cleanly based on it

Yes, I tried this with no success the exit isn’t working

are you looking at the comment that I linked? it doesn’t use exit …

accepted answers can be misleading

Yes

  gh run cancel ${{ github.run_id }}
  gh run watch ${{ github.run_id }}

This is actually working to stop the workflow, unfortunately I need the test to return success to make the PR checks pass.

Just a random guess, host machine is out of disk space

doesn’t seem to be the case

@Jeremy G (Cloud Posse) does this ring a bell? I saw you commenting back in 2020 on this issue: https://github.com/awslabs/amazon-ecr-credential-helper/issues/102

@Igor Rodionov

the weird thing is we’ve made no changes to the set up. this is all the same as it was before and it just fails

of course it is right before the very moment we need it lol

@johncblandii know this case

sec

try this

add that to the user data?

yea

we have this in there:

mkdir -p /root/.docker
echo '{ "credsStore": "ecr-login" }' >/root/.docker/config.json
mkdir -p /home/ec2-user/.docker
echo '{ "credsStore": "ecr-login" }' >/home/ec2-user/.docker/config.json

should the path match?

no

ok, gotcha

my path is from our module

that docker-config.json should be like that

ok, have to step away, but i’ll run this in a few

For GHA, did you give the job write permission?

permissions:
  contents: write

IIRC, in a GitHub Action using official Docker actions, you do not need to have any credential helpers or .docker/config.json, just use the docker/login-action action (which is done for you by cloudposse/github-action-docker-build-push).

yes, it has that

and yes, that’s the action we’re using. it just started failing out of nowhere

There was an outage. Is it still failing?

yes. it started last week

was still failing today

is failing now cuz of the tee above, but i’m pushing a fix

I don’t know, then. I haven’t seen new problems crop up recently.

yeah, reverted and it is still failing

Confirmed this was in place

permissions:
  contents: write
  id-token: write

cloudposse/[email protected]

confirmed adding credHelpers didn’t help.

I created a reduced test case and docker/login-action continues to fail no matter what I do with the same error:

Error saving credentials: error storing credentials - err: exit status 1, out: not implemented I’ll see what else I can find out. Definitely the worst timing to hit this bug

just adding here for context as I go.

testing on ubuntu-latest works perfect fine on GHA.

testing on a linux2 bastion and I could easily replicate this problem

I changed this section and everything worked on the bastion. Testing this on the runners as well. So far it looks promising

mkdir -p /root/.docker
echo '{}' >/root/.docker/config.json
mkdir -p /home/ec2-user/.docker
echo '{}' >/home/ec2-user/.docker/config.json

confirmed we’re back green

So what changed?

there is a new ami being used with the filter that came out this month

so could be something in that ami

1 sec…

amzn2-ami-hvm-2.0.20230808.0-x86_64-ebs

that date coincides with when i saw some errors

i believe this one was fine amzn2-ami-hvm-2.0.20230727.0-x86_64-ebs

or maybe the one before that. i didn’t track those as i didn’t consider it to be an AMI change

have you tried OIDC?

OIDC tokens have expiry too

@Max Lobur (Cloud Posse)

@Dan Miller (Cloud Posse)

yes absolutely. Check out this action: https://github.com/docker/login-action#aws-elastic-container-registry-ecr

although we typically use an AWS role that has access to ECR and assume the role with Github OIDC

similarly: https://github.com/aws-actions/amazon-ecr-login

Thanks @Dan Miller (Cloud Posse)

Btw I found out that if you rerun a job it automatically runs jobs that depend on it (ie have a needs property that points to it). So it might work to re-run the password getter job, this would auto run the job that needs that password.