#github-actions (2023-04)

Discussions related to GitHub Actions

2023-04-03

samuel avatar

Hi, good morning I am new to github Action, I have this worflow that working ok name: Deploy on: [push] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@master - name: Copiar el contenido del repositorio con scp uses: appleboy/scp-action@master env: HOST: ${{ secrets.HOST }} USERNAME: ${{ secrets.USERNAME }} PORT: ${{ secrets.PORT }} KEY: ${{ secrets.SSHKEY }} with: source: “pagweb/, app/” target: “/var/www/html, /var/www” strip_components: 1

samuel avatar

when the files are copied the content of the two sources is copied, I want that webpage is copied to /var/www/html and the app is copied to another destination example var/www

samuel avatar

what could be doing wrong

Darren Cunningham avatar
Darren Cunningham

please wrap in a code block (three backticks) so the formatting isn’t jacked

Darren Cunningham avatar
Darren Cunningham

initial guess is that it doesn’t look like scp-action supports directories

target - target path on the server
source - scp file list
samuel avatar

this is working ok, but what I do to seeting two target webpage is copied to /var/www/html and the app is copied to another destination example var/www

Darren Cunningham avatar
Darren Cunningham

you can either comb through the action code to see if it supports that or you can just run the action twice, not the most efficient but it should work

samuel avatar

ok when i solve it i’ll let you know

samuel avatar

the issue was solve

Darren Cunningham avatar
Darren Cunningham

nice, are you able to share the solution?

samuel avatar

yes, the code is here

samuel avatar

name: Deploy

on: [push]

jobs: deploy:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@master
- name: Copiar el contenido del repositorio con scp
  uses: appleboy/scp-action@master
  env:
    HOST: ${{ secrets.HOST }}
    USERNAME: ${{ secrets.USERNAME }}
    PORT: ${{ secrets.PORT }}
    KEY: ${{ secrets.SSHKEY }}
  with:
    source: "pagweb/*"
    target: "/var/www/html"
    -uses: actions/checkout@master
- name: Copiar el contenido de folder app con scp
  uses: appleboy/scp-action@master
  env:
    HOST: ${{ secrets.HOST }}
    USERNAME: ${{ secrets.USERNAME }}
    PORT: ${{ secrets.PORT }}
    KEY: ${{ secrets.SSHKEY }}
  with:
    source: "app/*"
    target: "/home/ubuntu/app"
    strip_components: 1
Darren Cunningham avatar
Darren Cunningham

gotta get used to using code blocks

Darren Cunningham avatar
Darren Cunningham

wrap :point_up: in three backticks `

1
Darren Cunningham avatar
Darren Cunningham

/snippet is a good option too as that has type formatting

Darren Cunningham avatar
Darren Cunningham
on: [push]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master
    - name: Copiar el contenido del repositorio con scp
      uses: appleboy/scp-action@master
      env:
        HOST: ${{ secrets.HOST }}
        USERNAME: ${{ secrets.USERNAME }}
        PORT: ${{ secrets.PORT }}
        KEY: ${{ secrets.SSHKEY }}
      with:
        source: "pagweb/*"
        target: "/var/www/html"
        -uses: actions/checkout@master
    - name: Copiar el contenido de folder app con scp
      uses: appleboy/scp-action@master
      env:
        HOST: ${{ secrets.HOST }}
        USERNAME: ${{ secrets.USERNAME }}
        PORT: ${{ secrets.PORT }}
        KEY: ${{ secrets.SSHKEY }}
      with:
        source: "app/*"
        target: "/home/ubuntu/app"
        strip_components: 1
Darren Cunningham avatar
Darren Cunningham

it’s just a little easier to read

Darren Cunningham avatar
Darren Cunningham
Darren Cunningham avatar
Darren Cunningham

snippets are annoying IMO because I don’t like the collapse/expand but I can see how they’d be helpful with really long posts

1
Darren Cunningham avatar
Darren Cunningham

also be cautious about using @master when using public GH actions — I suggest using the explicit sha for a commit to better protect yourself. while unlikely, it’s an attack vector to just blindly always run the latest version of a GH action. meaning, if somebody was able to add malicious code they could access the host, username & password you’re using in this case.

1
this1

2023-04-04

2023-04-05

2023-04-06

Pawel Rein avatar
Pawel Rein

What do people use for monitoring / metrics collection from github actions runs? I found meercode.io mentioned in older conversations. Is there a way to use Grafana?

Darren Cunningham avatar
Darren Cunningham

currently using Datadog, previously had a home grown solution that used a AWS Lambda (Golang) that was the handler of GitHub Webhooks that would record metrics to CloudWatch. It worked just fine but didn’t have any bells or whistles and was yet another thing that we “owned” so using a managed product is easier so worth the capex.

Darren Cunningham avatar
Darren Cunningham

IMO meercode’s pricing looks silly. I essentially compare pricing against GitHub, meaning that your product should provide as much value for the same pricing. GitHub’s market price is $4/user and $21/user, so does meercode add 7x and ~3x the value of GitHub for their respective tiers

1
Darren Cunningham avatar
Darren Cunningham

even if you go with GitHub’s original team pricing at $9 we’re still talking 3x

Darren Cunningham avatar
Darren Cunningham

Datadog is $8/month per committer

Darren Cunningham avatar
Darren Cunningham

and no it doesn’t provide 2x the value of GitHub, but it’s still worth it to us

Pawel Rein avatar
Pawel Rein

Thanks @Darren Cunningham, very useful input. I am aware of the Datadog integration. Datadog collector is opensource so I thought maybe I can find inspiration there to build something to ship to prometheus. Then I found out there’s a prometheus exporter on github but it only ships one metric. Going with either would still mean owning it. We already pay for a Datadog alternative that doesn’t have this integration, so moving to Datadog is not an option.

johncblandii avatar
johncblandii

Adding this to our pre-commit GHA helps keep our stacks formatted consistently.

.pre-commit-config.yaml

repos:
  - repo: <https://github.com/google/yamlfmt>
    rev: v0.9.0
    hooks:
      - id: yamlfmt
  - repo: <https://github.com/antonbabenko/pre-commit-terraform>
    rev: v1.77.1
    hooks:
      - id: terraform_fmt
      - id: terraform_docs
        args: ["--args=--lockfile=false"]

.yamlfmt

formatter:
  type: basic
  retain_line_breaks: true
  indentless_arrays: false
  drop_merge_tag: true

GHA in thread

fiesta_parrot1
johncblandii avatar
johncblandii
      - uses: actions/setup-go@v4
        with:
          go-version: "1.18.0"

      - run: go install github.com/google/yamlfmt/cmd/yamlfmt@latest

      - uses: actions/setup-python@v3
        with:
          python-version: 2.x
          token: ${{ secrets.GHA_SERVICE_ACCOUNT_TOKEN }}

      # pre-commit checks: fmt + terraform-docs
      # We skip tf_validate as it requires an init
      # of all root modules, which is to be avoided.
      - uses: cloudposse/[email protected]
        env:
          SKIP: tf_validate
        with:
          token: ${{ secrets.GHA_SERVICE_ACCOUNT_TOKEN }}
          git_user_name: ${{ secrets.GHA_SERVICE_ACCOUNT_USER_NAME }}
          git_user_email: ${{ secrets.GHA_SERVICE_ACCOUNT_USER_EMAIL }}
          git_commit_message: "chore(pre-commit): linting"
          extra_args: --all-files
johncblandii avatar
johncblandii

so basically the same as the CP GHA example, but the first 2 steps are new (setup-go and go install ....)

2023-04-10

actions Archives | The GitHub Blog avatar
actions Archives | The GitHub Blog
07:35:39 PM

Manage caches in your Actions workflows from Web Interface Caching dependencies and other commonly reused files enables developers to speed up their GitHub Actions workflows and make them more efficient. We have now enabled Cache Management from the web interface to enable developers to get more transparency and control over their cache usage within their GitHub repositories. Actions users who use actions/cache can now: […]

Manage caches in your Actions workflows from Web Interface | GitHub Changelogattachment image

Caching dependencies and other commonly reused files enables developers to speed up their GitHub Actions workflows and make them more efficient. We have now enabled Cache Management from the web interface to enable developers to get more transparency and control over their cache usage within their GitHub repositories. Actions users who use actions/cache can now: […]

2023-04-11

hatz avatar

Hey all, I am attempting to use the github-action-matrix-outputs-write github action and facing an unexpected error that I’m hoping I can get some assistance with. Here is the usage of the action:

    - uses: cloudposse/[email protected]
      if: ${{ inputs.monorepo == 'true' }}
      id: out
      with:
        matrix-step-name: ${{ github.job }}
        matrix-key: ${{ env.APPLICATION_NAME }}
        outputs: |-
          comment: ${{ env.URL }}

This is the error I’m getting

Error: bad file '-': yaml: mapping values are not allowed in this context

Here’s some more output:

Copying to tool cache
Run JSON=$(echo "comment: URL for deploy: <https://frontend-docs-259.website.dev>" | yq '. + {}' -o json | jq . -c -M -e)
  JSON=$(echo "comment: URL for deploy: <https://frontend-docs-259.website.dev>" | yq '. + {}' -o json | jq . -c -M -e)
  echo "result=${JSON}" >> $GITHUB_OUTPUT
hatz avatar

@Igor Rodionov ^ it looks like you’re a primary contributor on this action

hatz avatar

After some more testing it kinda seems like the - character in my url is what’s causing the issue. If I just have a comment of "Hello World" it seems to work, so maybe this is just needing to figure out some syntax

2023-04-12

Igor Rodionov avatar
Igor Rodionov

@hatz Hello. Thanks for your question. Let me try to reproduce the issue

hatz avatar

Awesome! I did end up doing a whole lot of testing and I do have it all working now, but there are some odd things. The first big fix I found wasx just throwing quotes around things, like this:

        outputs: |
          url: "${{ env.URL }}"
hatz avatar

which seems to fix the - issue because that character in yaml is technically for an array

hatz avatar

Another odd thing I found is that an individual key doesn’t like to be given multiline yaml. For example if I have this input in my github action

  paths:
    description: 'A multiline yaml string of key/value pairs for each Name and Path to include in the comment'
    required: false
    default: |
      Application: application
      Dashboard: /dashboard

And then try to directly use it to the matrix write action like

          paths: ${{ inputs.paths }}

It ends up breaking because of the newlines and a similar map issue. I got around this currently by turning my multiline yaml into a singleline yaml with newline characters, passing that into the write, and then turning it back into a multiline yaml on the other side. for example

    - name: Process paths for output
      shell: bash
      run: |
        echo "paths=$(echo '${{ inputs.paths }}' | awk '{printf "%s\n", $0}')" >> $GITHUB_ENV

    - uses: cloudposse/[email protected]
      id: out
      with:
        matrix-step-name: ${{ github.job }}
        matrix-key: ${{ env.APPLICATION_NAME }}
        outputs: |
          url: "${{ env.URL }}"
          paths: '${{ env.paths }}'

And then on the read side doing a echo -e so that the newlines get expanded.

Happy to share more details with you if you’d like! Passing around the multiline yaml stuff gets annoying complicated in github actions I’ve found so this might be some edge cases

Igor Rodionov avatar
Igor Rodionov

@hatz I create a PR with new test for your case

Igor Rodionov avatar
Igor Rodionov
#17 Fix special chars

what

• Fix special chars issue

Hey all, I am attempting to use the github-action-matrix-outputs-write github action and facing an unexpected error that I’m hoping I can get some assistance with. Here is the usage of the action:

    - uses: cloudposse/[email protected]
      if: ${{ inputs.monorepo == 'true' }}
      id: out
      with:
        matrix-step-name: ${{ github.job }}
        matrix-key: ${{ env.APPLICATION_NAME }}
        outputs: |-
          comment: ${{ env.URL }}

This is the error I’m getting

Error: bad file '-': yaml: mapping values are not allowed in this context

Here’s some more output:

Copying to tool cache
Run JSON=$(echo "comment: URL for deploy: <https://frontend-docs-259.website.dev/>" | yq '. + {}' -o json | jq . -c -M -e)
  JSON=$(echo "comment: URL for deploy: <https://frontend-docs-259.website.dev/>" | yq '. + {}' -o json | jq . -c -M -e)
  echo "result=${JSON}" >> $GITHUB_OUTPUT

references

https://sweetops.slack.com/archives/CQA2BH8AG/p1681239551909169

Igor Rodionov avatar
Igor Rodionov

Could you check that I get the issue right ?

hatz avatar

Also I’m not sure how much it matters but I am currently using the yaml : | instead of the : |- - not sure if the newline at the end breaks anything or not but thought I’d mention it! example:

        outputs: |
          url: '${{ env.URL }}'
          paths: '${{ env.paths }}'
Igor Rodionov avatar
Igor Rodionov

@hatz I’m rewriting the action to nodejs

Igor Rodionov avatar
Igor Rodionov

otherwise it would be always edge cases that breaks everything

hatz avatar

That is awesome! I’m currently building some internal composite actions for our engineers to do ephemeral deployments, and doing stuff straight in bash has been kind of a headache because of newlines and special characters and all that.

hatz avatar

Shell gets brittle really fast when trying to do multiline stuff I’ve found. And github actions seems to behave slightly different than my own shell so it’s been a lot of experimenting

hatz avatar

If you want any help or need someone to do any kind of testing let me know! I’ve got my side working (with some tweaks to pre/post-process) it with 0.3.1 and definitely happy to test new stuff!

Igor Rodionov avatar
Igor Rodionov

I cut the new release

Igor Rodionov avatar
Igor Rodionov

Could you pls test your special chars issues?

Igor Rodionov avatar
Igor Rodionov

for multiline unfortunately you have to use such workaround

Igor Rodionov avatar
Igor Rodionov
        outputs: |
          url: '${{ env.URL }}'
          paths: ${{ toJson(env.paths) }}
Igor Rodionov avatar
Igor Rodionov

this is the limitation of GitHub actions substitution.

hatz avatar

Hey! I haven’t had a chance to test this out yet, been OOO for a few days. I will probably get this tomorrow or friday!

1
hatz avatar

Hey @Igor Rodionov - My first initial test led me to believe this didn’t work but I’m trying a couple other things. It downloaded all 9 of the files that I expected to see, but the ‘output’ only had one entry for each object. Possible I might have something on my side so I’ll let you know!

hatz avatar

Alright one error I’m finding so far is what it considers to be valid yaml.

paths: 'Application:
'
fd_name: 'frontend-asset-manager'
---------------------
YAMLParseError: Missing closing 'quote at line 2, column 21:

paths: 'Application:
                    ^

This seems to be related to the multi-line yaml that I have where the value is an empty string

here is my input

        outputs: |
          url: ${{ env.URL }}
          paths: '${{ inputs.paths }}'
          fd_name: ${{ env.FD_NAME }}
hatz avatar

I have the closing quote on the second line but it doesn’t seem to see that

hatz avatar

The new error output though is much better

hatz avatar

Here’s the error I get if I don’t throw single quotes around it

Error: Outputs should be valid YAML 
---------------------
url: <https://frontend-usdf-382.figureapps.dev>
paths: Application:

fd_name: frontend-usdf
---------------------
YAMLParseError: Nested mappings are not allowed in compact mappings at line 2, column 8:

paths: Application:
       ^
hatz avatar

I’ve tried with both outputs: | and outputs: |- too

Igor Rodionov avatar
Igor Rodionov
paths: '${{ toJson(inputs.paths) }}'
Igor Rodionov avatar
Igor Rodionov

try

1
hatz avatar

Alright this worked on the write step now:

paths: '${{ toJson(inputs.paths) }}'

it ends up looking like this:

  with:
    matrix-step-name: ephemeral-deploy
    matrix-key: frontend-auth-382
    outputs: url: <https://frontend-auth-382.url.com>
      paths: '"Application:\n"'
      fd_name: frontend-auth

However, on the read side I’m still only actually seeing one entry.

No artifact name specified, downloading all artifacts
Creating an extra directory for each artifact that is being downloaded
starting download of artifact 1ad31b158455284757e5320e8e4ca65ed4c909fce6554d4713a764e6480d9f11 : 1/9
Total number of files that will be downloaded: 1
starting download of artifact 1de3311f3a5fb2f26beb247ba1ee2e3866eac7f2cd014ca2b940117f87d46d22 : 2/9
Total number of files that will be downloaded: 1
starting download of artifact 4fc8f69a7cd20390d89bc18ea6400ec381bfea1ff6fe77c64d0b0439ab3de809 : 3/9
Total number of files that will be downloaded: 1
starting download of artifact 4fea876fd1027053d0ddf38e74b805fbe8a4e1944613e2c7aecb501d79f1d512 : 4/9
Total number of files that will be downloaded: 1
starting download of artifact 6c4186c01f171e8d190e6b8688645a0591030d0cced58c86b353d565a0211bda : 5/9
Total number of files that will be downloaded: 1
starting download of artifact 86464d497618037bd50d8e9925b69000e8e433dae9c05a2a99e356fd93147d90 : 6/9
Total number of files that will be downloaded: 1
starting download of artifact a23bd2a5d63a624994e3258b3dc50af5a26bdb6fe7ec483b51a89546f99ba47b : 7/9
Total number of files that will be downloaded: 1
starting download of artifact bf4c9fa23d050fb0912454d8a2a9d6d40f16c348289320fde9e4d9e59f21d635 : 8/9
Total number of files that will be downloaded: 1
starting download of artifact d7a846ded11eb21020889310d301d57d579499d0034000632dd9dfe1f27571c8 : 9/9
Total number of files that will be downloaded: 1
There were 9 artifacts downloaded

Everything gets downloaded. but then the json seems to not be merging them together

        json=$(echo '${{ steps.read-matrix.outputs.result }}')
        echo $json
{"url":{"matrix_key":"<https://frontend.url.com>"},"paths":{"matrix_key":"\"Application:\\n\""},"fd_name":{"matrix_key":"frontend"}}
hatz avatar

also worth noting that no single quotes had a slightly different write input

paths: ${{ toJson(inputs.paths) }}

became

  paths: "Application:\n"

but the read was the same issue

hatz avatar

Manually inspecting all 9 of the artifacts that got uploaded during the job and I see what I expect for each one, so it seems the merging on the read somehow isn’t quite working

example artifact uploads (all 9 are different values but same keys)

{"matrix_key":{"url":"<https://frontend-auth.url.com>","paths":"Application:\n","fd_name":"frontend-auth"}}
{"matrix_key":{"url":"<https://frontend-group.url.com>","paths":"Application:\n","fd_name":"frontend-group"}}
hatz avatar

Oh I see the issue. It seems like that json structure changed. I think the current version of the -write action is now incompatible with the -read action. Here’s an example json from 0.3.1 release:

{"frontend-auth":{"url":"<https://frontend-auth.url.com>","paths":"Application:\\n\\n","fd_name":"frontend-auth"}}
hatz avatar

I think the bug is that it is using the matrix_key as a string literal instead of a value ref

hatz avatar

@Igor Rodionov Alright I submitted a PR for that specific issue here: https://github.com/cloudposse/github-action-matrix-outputs-write/pull/20

hatz avatar

I’m now running into a new issue where the newline character in the json is not being evaluated properly (because in github actions I think it wants \\n instead of \n, but at least the first issue should be fixed)

hatz avatar

Alright I fixed my issue around the newline character, in my post-processing of the read action I had some weird stuff to handle the \\n and I forgot to remove it in my testing, so it’s all working well with the above PR fix I made + the toJson() you recommended!

hatz avatar

@Igor Rodionov can you check that PR out today? I’d love to get it merged/released so we can start using it internally!

hatz avatar

Also can you elaborate on why you do an assertion for empty fields? I have a case where passing empty fields is valid, seems like a bit of an over-reaching assertion on the input data to not allow empty string. Thoughts?

hatz avatar

@Igor Rodionov

  with:
    matrix-step-name: ephemeral-deploy
    outputs: url: ''
  paths: 'Application:\n\n'
  fd_name: ''
  errors: '\n\n

\n:x: argocd_token is null. Please reach out to #devops so we can help you onboard.\n

'

e.g. if an error happens I’m passing errors through, but that means other fields (like my url) never got created so that becomes ''

Error: Expected '' to not equal ''

2023-04-13

samuel avatar

Hi good afternoon everybody, I tray aneble githubaction in a github Enterprise, to done this I need setting a s3 bucket, but I am have issue, this could be that I am doing something wrong

carlos.clemente avatar
carlos.clemente
1
samuel avatar
My first problem is that I don't know what it should be aws service Url
samuel avatar
samuel avatar

I Created the bucket s3 I only see arn ejemplo: arnawss3:::test-github-action02 but the url service no

samuel avatar

The question is I should do a procedure to create it

samuel avatar

if anyone has a step by step, I would appreciate it

samuel avatar

I found this documentation

samuel avatar

this command runer in console of my github Enterprise server ?

samuel avatar

yes now I get it the finger

samuel avatar
openssl s_client -connect github.pro.edenor:443 < /dev/null 2>/dev/nullopenssl x509 -fingerprint -noout -sha1 -in /dev/stdin SHA1 Fingerprint=211766891169

2023-04-14

2023-04-19

2023-04-20

actions Archives | The GitHub Blog avatar
actions Archives | The GitHub Blog
04:15:39 PM

GitHub Actions: Create and share your own deployment protection rules for safe and controlled deployments GitHub Actions: Create and share your own deployment protection rules for safe and controlled deployments

GitHub Actions: Create and share your own deployment protection rules for safe and controlled deployments | GitHub Changelogattachment image

GitHub Actions: Create and share your own deployment protection rules for safe and controlled deployments

2023-04-24

actions Archives | The GitHub Blog avatar
actions Archives | The GitHub Blog
05:55:35 PM

GitHub Actions: macOS 13 is now available GitHub Actions: macOS 13 is now available

GitHub Actions: macOS 13 is now available | GitHub Changelogattachment image

GitHub Actions: macOS 13 is now available

actions Archives | The GitHub Blog avatar
actions Archives | The GitHub Blog
05:55:35 PM

GitHub Actions: Faster macOS runners are now available in open public beta! GitHub Actions: Faster macOS runners are now available in open public beta!

GitHub Actions: Faster macOS runners are now available in open public beta! | GitHub Changelogattachment image

GitHub Actions: Faster macOS runners are now available in open public beta!

actions Archives | The GitHub Blog avatar
actions Archives | The GitHub Blog
06:55:34 PM

GitHub Actions Importer Feature Flags GitHub Actions Importer Feature Flags

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
#1339 Option to Not Set as Latest Release

I’m dealing with release/X.Y branches, each time we have the tool run it increments properly by the filter by commitish feature, however on older versions lets say 1.x vs 2.x, 1.x releases will still get set as “latest”. It would be nice to be able to disable this either by branch or by perhaps only allowing latest to be set on tags that are tied back to the primary branch.

2023-04-26

hatz avatar

Is it possible for someone to check out PR I have open on one of the cloudposse github actions? https://github.com/cloudposse/github-action-matrix-outputs-write/pull/20

#20 Fix matrix_key as a string literal instead of a value reference

The json creation was using "matrix_key" as a string literal instead of referencing the matrix_key variable that gets set from the input. This PR fixes the issue.

Before this action on v0.4.0 would write the following json:

{"matrix_key":{"url":"<https://frontend-auth.url.com/>","paths":"Application:\n","fd_name":"frontend-auth"}}

With this PR it writes the following, which matches the behavior of v0.3.1:

{"frontend-auth":{"url":"<https://frontend-auth.url.com/>","paths":"Application:\n","fd_name":"frontend-auth"}}

This was tested on a private repository so I can’t share the full build, but hopefully the above is enough information.

Related slack message in SweetOps Slack: https://sweetops.slack.com/archives/CQA2BH8AG/p1682360941807629?thread_ts=1681319531.187979&cid=CQA2BH8AG


We should probably update the release notes of v0.4.0 to indicate that it is a broken version and is not compatible with the github-action-matrix-outputs-read action.

cc: @goruha

Igor Rodionov avatar
Igor Rodionov

@hatz let me check

Igor Rodionov avatar
Igor Rodionov

@hatz I approved and merged the PR

Igor Rodionov avatar
Igor Rodionov

thanks for you contribution. this is the nice catch

2023-04-28

Soren Jensen avatar
Soren Jensen

Hi All, I got some issues with our setup of GitHub and tests before merging into main. We are working with a single mono repo with the following structure.

bootstrap/
├── .github/workflows
│   ├── aws_core.yml
│   ├── client.yml
│   ├── data_api.yml
│   └── ingestion_api.yml
├── aws_core/
├── client/
├── data_api/
├── ingestion_api/
├── shared_lib/

Each module has its own folder with a nice mixture between terraform, python and nodejs files. and it’s own pipeline defined in .github We got a few share libs with code used in multiple modules which triggers multiple pipelines. E.g. a chance in the shared_lib will trigger the pipelines for all our api’s.

Pipeline for aws_core got the following triggers:

name: aws_core

on:
  push:
    paths:
      - .github/workflows/aws_core.yml
      - scripts/load_env.py
      - aws_core/**

Pipeline for the data_api

name: data_api

on:
  push:
    paths:
      - .github/workflows/data_api.yml
      - scripts/load_env.py
      - scripts/requirements.txt
      - data_api/**
      - api_lambda_infra/**
      - shared_lib/**

Pipeline for shared_lib

name: shared_lib

on:
  push:
    paths:
      - .github/workflows/shared_lib.yml
      - scripts/requirements.txt
      - shared_lib/**

Each pipeline got it’s own test job, the first problem is when opening up a PR to merge a branch into main if you select Auto-Merge it merges before all the pipelines has completed, so some tests might fail after the merge is complete breaking prod.

The second problem is as you can see in the pipelines above they are only triggered on certain changes as we only want the relevant pipelines to run when working on a branch. It’s common we modify code in multiple modules on the same branch, e.g. a change there require a new endpoint in the api and a change in the client. Committing this step by step will trigger one pipeline and in next commit another. Before merging into main it would be nice to force all pipelines to run.

Any suggestions on how to get this working? How do others working with a mono repo solve these problems?

    keyboard_arrow_up