SweetOps #github-actions for February, 2024

Discussions related to GitHub Actions

2024-02-05

wbrown43

We are encountering an issue where pre-commit https://github.com/cloudposse/github-action-pre-commit is failing ERROR: terraform-docs is required by terraform_docs pre-commit hook but is not installed or in the system's PATH. There is a terraform install step that completes prior to the pre-commit step. Here is a screen shot of the run:

cloudposse/github-action-pre-commit

A GitHub action to run pre-commit and allow overriding the git config user name and email

Erik Osterman (Cloud Posse)

06:14:12 PM

@Jeremy G (Cloud Posse)

cloudposse/github-action-pre-commit

A GitHub action to run pre-commit and allow overriding the git config user name and email

Jeremy G (Cloud Posse)

07:10:42 PM

@wbrown43 Sorry about that. Please read https://sweetops.slack.com/archives/CRG4CCS03/p1707089830893669

Most Cloud Posse projects install build-harness under the project’s repository root to manage things like building Docker images or generating README.md from README.yaml. In turn, build-harness installs some tools it needs when it needs them. We had been using INSTALL_PATH to determine where build-harness installs its tools, intending that they be installed under the build-harness tree, but this is in conflict with Geodesic, which uses INSTALL_PATH to determine where to install the Geodesic wrapper/command. This was causing build-harness to install gomplate in /usr/local/bin , which was not the intention.

So we changed build-harness to install its tools into PACKAGES_INSTALL_PATH instead. Unfortunately, this broke some workflows, particularly pre-commit workflows, which use INSTALL_PATH to direct where build-harness installs tools needed for the workflow. The quick fix is to replace INSTALL_PATH with PACKAGES_INSTALL_PATH in the workflow where the intent is to have build-harness install tools in a particular directory.

wbrown43

07:20:25 PM

Thank you @Jeremy G (Cloud Posse) that worked

wbrown43

05:25:41 PM

@johncblandii

2024-02-13

actions Archives - The GitHub Blog

09:45:36 PM

Deprecation notice: v1 and v2 of the artifact actions Deprecation notice: v1 and v2 of the artifact actions The post Deprecation notice: v1 and v2 of the artifact actions appeared first on The GitHub Blog.

Deprecation notice: v1 and v2 of the artifact actions

The GitHub Blog - Updates, ideas, and inspiration from GitHub to help developers build and design software.

Updates, ideas, and inspiration from GitHub to help developers build and design software.

2024-02-14

Saichovsky

08:58:25 PM

Hello,

I have a job that looks something like this:

jobs:
  pop-values-from-list:
    runs-on: my-runner
    env:
      MY_LIST: '[1,2,3,4,5,6,7,8,9,0]' # list has 10 items
    steps:
      - name: Pop 10% of list
        run: |
          pop_my_list() {
            percentage=$1
            list=$2
            MY_LIST=$(process_and_pop $percentage $list) >>"$GITHUB_OUTPUT"
          }
          pop_my_list 10% $MY_LIST # MY_LIST now has 9 items
      - run: WaitForAsyncProcess
      - name: Pop 20% of list
        run: |
          pop_my_list() {
            percentage=$1
            list=$2
            MY_LIST=$(process_and_pop $percentage $list) >>"$GITHUB_OUTPUT"
          }
          pop_my_list 20% $MY_LIST # let's assume that MY_LIST now has 7 items
      - run: WaitForAsyncProcess
...
      - name: Pop everything that remains in list
        ...

This is of course a simplified version of what I am trying to achieve. The problem here is that I have to keep repeating the function definition for pop_my_list()

Is there a way I can have the function defined once and invoke it across steps without repetition?

Saichovsky

09:41:16 PM

Figured it out.

What I did is that I copied the function definition to a file (escaping the $ symbols) in the first step, then sourced the file in subsequent steps before calling the function

Igor Rodionov

02:15:54 PM

@Saichovsky, nice workaround.

managedkaos

04:12:15 AM

You could also consider putting the function in an action (either Dockerfile or Javascript based) and then call it as a true action in your workflow. then could also be used in other repos.

2024-02-15

2024-02-20

Tony

10:20:03 AM

Hello all - we are trying to update our workflows and actions to node20 and are using your excellent github-action-matrix-outputs-write and github-action-matrix-outputs-read actions. github-action-matrix-outputs-read unfortunately is still using an older version of actions/download-artifact which uses node16 and I see an automatic PR in your repo to update it is failing :( Are there any plans to update this to use v4 of actions/download to address this? The v4 actions have breaking changes and now uploaded artifacts are immutable so it may be that github-action-matrix-outputs-write needs some changes at the same time so that they continue to work as expected?

Erik Osterman (Cloud Posse)

02:46:01 PM

@Igor Rodionov can you help update this

Erik Osterman (Cloud Posse)

02:46:45 PM

@Gabriela Campana (Cloud Posse)

Igor Rodionov

03:03:59 PM

Sure

Tony

03:15:50 PM

That would be great, thanks guys! I tried simply updating the version of actions/download-artifact (ans omse other older version) in a local copy but it didn’t work - the download find no artifacts, so I think your write action needs an update of some kind too, though it is not obvious to me what that update should be..

Igor Rodionov

04:40:49 PM

The case is that actions/download-artifact v3 is not compatible with v4 . Github changed download API so it needs additional work to update.

Erik Osterman (Cloud Posse)

06:07:08 PM

Can you elaborate @Igor Rodionov? You mean the official GHA doesn’t support GitHub’s own API?

Igor Rodionov

06:08:07 PM

sure

Igor Rodionov

06:10:53 PM

@Erik Osterman (Cloud Posse)

download-artifact@v4+ is not currently supported on GHES yet. If you are on GHES, you must use v3.
Downloading artifacts that were created from action/upload-artifact@v3 and below are not supported.
In v3, Artifacts are mutable so it’s possible to write workflow scenarios where multiple jobs upload to the same Artifact. In v4, Artifacts are immutable (unless deleted). So you must change each of the uploaded Artifacts to have a different name and filter the downloads by name to achieve the same effect. This was key feature we used. Now we need to change write/read action to suppor that

Igor Rodionov

06:11:04 PM

links

Igor Rodionov

06:11:05 PM

https://github.com/actions/download-artifact/blob/main/docs/MIGRATION.md

Migration

• Migration • Multiple uploads to the same named Artifact • Overwriting an Artifact • Merging multiple artifacts

Several behavioral differences exist between Artifact actions v3 and below vs v4. This document outlines common scenarios in v3, and how they would be handled in v4.

Multiple uploads to the same named Artifact

In v3, Artifacts are mutable so it’s possible to write workflow scenarios where multiple jobs upload to the same Artifact like so:

jobs:
  upload:
    strategy:
      matrix:
        runs-on: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.runs-on }}
    steps:
      - name: Create a File
        run: echo "hello from ${{ matrix.runs-on }}" > file-${{ matrix.runs-on }}.txt
      - name: Upload Artifact
        uses: actions/upload-artifact@v3
        with:
          name: my-artifact # NOTE: same artifact name
          path: file-${{ matrix.runs-on }}.txt
  download:
    needs: upload
    runs-on: ubuntu-latest
    steps:
      - name: Download All Artifacts
        uses: actions/download-artifact@v3
        with:
          name: my-artifact
          path: my-artifact
      - run: ls -R my-artifact

This results in a directory like so:

my-artifact/
  file-macos-latest.txt
  file-ubuntu-latest.txt
  file-windows-latest.txt

In v4, Artifacts are immutable (unless deleted). So you must change each of the uploaded Artifacts to have a different name and filter the downloads by name to achieve the same effect:

jobs:
  upload:
    strategy:
      matrix:
        runs-on: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.runs-on }}
    steps:
    - name: Create a File
      run: echo "hello from ${{ matrix.runs-on }}" > file-${{ matrix.runs-on }}.txt
    - name: Upload Artifact
-     uses: actions/upload-artifact@v3
+     uses: actions/upload-artifact@v4
      with:
-       name: my-artifact
+       name: my-artifact-${{ matrix.runs-on }}
        path: file-${{ matrix.runs-on }}.txt
  download:
    needs: upload
    runs-on: ubuntu-latest
    steps:
    - name: Download All Artifacts
-     uses: actions/download-artifact@v3
+     uses: actions/download-artifact@v4
      with:
-       name: my-artifact
        path: my-artifact
+       pattern: my-artifact-*
+       merge-multiple: true
    - run: ls -R my-artifact

In v4, the new pattern: input will filter the downloaded Artifacts to match the name specified. The new merge-multiple: input will support downloading multiple Artifacts to the same directory. If the files within the Artifacts have the same name, the last writer wins.

Overwriting an Artifact

In v3, the contents of an Artifact were mutable so something like the following was possible:

jobs:
  upload:
    runs-on: ubuntu-latest
    steps:
      - name: Create a file
        run: echo "hello world" > my-file.txt
      - name: Upload Artifact
        uses: actions/upload-artifact@v3
        with:
          name: my-artifact # NOTE: same artifact name
          path: my-file.txt
  upload-again:
    needs: upload
    runs-on: ubuntu-latest
    steps:
      - name: Create a different file
        run: echo "goodbye world" > my-file.txt
      - name: Upload Artifact
        uses: actions/upload-artifact@v3
        with:
          name: my-artifact # NOTE: same artifact name
          path: my-file.txt

The resulting my-file.txt in my-artifact will have “goodbye world” as the content.

In v4, Artifacts are immutable unless deleted. To achieve this same behavior, you can use overwrite: true to delete the Artifact before a new one is created:

jobs:
  upload:
    runs-on: ubuntu-latest
    steps:
      - name: Create a file
        run: echo "hello world" > my-file.txt
      - name: Upload Artifact
-       uses: actions/upload-artifact@v3
+       uses: actions/upload-artifact@v4
        with:
          name: my-artifact # NOTE: same artifact name
          path: my-file.txt
  upload-again:
    needs: upload
    runs-on: ubuntu-latest
    steps:
      - name: Create a different file
        run: echo "goodbye world" > my-file.txt
      - name: Upload Artifact
-       uses: actions/upload-artifact@v3
+       uses: actions/upload-artifact@v4
        with:
          name: my-artifact # NOTE: same artifact name
          path: my-file.txt
+         overwrite: true

Note that this will create an entirely new Artifact, with a different ID from the previous.

Merging multiple artifacts

In v3, multiple uploads from multiple jobs could be done to the same Artifact. This would result in a single archive, which could be useful for sending to upstream systems outside of Actions via API or UI downloads.

jobs:
  upload:
    strategy:
      matrix:
        runs-on: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.runs-on }}
    steps:
      - name: Create a File
        run: echo "hello from ${{ matrix.runs-on }}" > file-${{ matrix.runs-on }}.txt
      - name: Upload Artifact
        uses: actions/upload-artifact@v3
        with:
          name: all-my-files # NOTE: same artifact name
          path: file-${{ matrix.runs-on }}.txt

The single all-my-files artifact would contain the following:

.
  ∟ file-ubuntu-latest.txt
  ∟ file-macos-latest.txt
  ∟ file-windows-latest.txt

To achieve the same in v4 you can change it like so:

jobs:
  upload:
    strategy:
      matrix:
        runs-on: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.runs-on }}
    steps:
      - name: Create a File
        run: echo "hello from ${{ matrix.runs-on }}" > file-${{ matrix.runs-on }}.txt
      - name: Upload Artifact
        uses: actions/upload-artifact@v3
        with:
-         name: all-my-files
+         name: my-artifact-${{ matrix.runs-on }}
          path: file-${{ matrix.runs-on }}.txt
+ merge:
+   runs-on: ubuntu-latest
+   needs: upload
+   steps:
+     - name: Merge Artifacts
+       uses: actions/upload-artifact/merge@v4
+       with:
+         name: all-my-files
+         pattern: my-artifact-*

Note that this will download all artifacts to a temporary directory and reupload them as a single artifact. For more information on inputs and other use cases for actions/upload-artifact/merge@v4, see the action documentation.

Igor Rodionov

06:11:24 PM

https://github.com/actions/download-artifact?tab=readme-ov-file#v4---whats-new

Igor Rodionov

06:11:40 PM

so we can not just pin new version

Tony

07:16:32 AM

Perhaps you could simply prefix them all with the matrix-step-namein the write action and then in the read action, pass the pattern argument as <matrix-step-name>* and merge-multiple: true to actions/download-artifact. I think that would essentially get you back to what you have today right?

Erik Osterman (Cloud Posse)

02:31:36 PM

Yes, I that sounds like what should happen. Merge on download and use jq to aggregate.

Erik Osterman (Cloud Posse)

02:32:40 PM

Interestingly I faced a related issue (on an unrelated workflow) over three weekend. I put it on hold but think I will do exactly this.

Tony

09:52:01 AM

@Igor Rodionov @Erik Osterman (Cloud Posse) have you guys given any thought to an ETA for a fix for the github actions? Thanks

Igor Rodionov

09:52:42 AM

@Tony I am working on it now. I will delivery the fix this week

Tony

09:52:59 AM

Awesome @Igor Rodionov - thanks!

Igor Rodionov

11:46:07 AM

@Tony I just released new versions. Could you pls try it and test if it works smoothly and as expected.

https://github.com/cloudposse/github-action-matrix-outputs-write/releases/tag/1.0.0

https://github.com/cloudposse/github-action-matrix-outputs-read/releases/tag/1.0.0

Tony

12:51:45 PM

Hi @Igor Rodionov Great! I will hopefully be able to look at this tomorrow or Friday latest and report back to you.

Tony

02:32:58 PM

@Igor Rodionov I have tested today on my pipeline which merges output from two parallel jobs - the output with the old version and the new version of your actions looks the same so it does appear to be working as expected.

Tony

02:33:06 PM

Thanks again for actioning this so quickly

#github-actions (2024-02)

2024-02-05

2024-02-13

2024-02-14

2024-02-15

2024-02-20

2024-02-21

2024-02-28

2024-02-29