#codefresh

Archive: https://archive.sweetops.com/codefresh/

2019-10-19

deftunix

hi all, do you have any example of usage of codefresh to deploy on vm (not using packer, I have seen the example package-gcp) and/or aws beanstalk?

deftunix

just another quick one: do you have any example of integration with monitoring system to measure quality, productivity and velocity?

Codefresh Status - Incident History
03:04:40 PM

https://status.codefresh.io/incidents/xx6mqg203nj6\|Docker has reported an incident Oct 19, 14:52 UTC Resolved - This incident has been resolvedOct 18, 23:06 UTC Identified - Docker has reported an incident. It’s affecting the following components:

  • Docker Package Repositories

More information here:

Docker has reported an incident

Codefresh’s Status Page - Docker has reported an incident.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

2019-10-18

Our team has decided to bite the bullet and start using helm 3. In looking at the source code for the official helm step, it appears as though the helm version can be passed as an argument. Has anyone successfully done this or will I need to fork the step

Harrison Heck

Any way to exclude a file in the modified files expression in the triggers?

Kristofer Svardstal

@Harrison Heck Glob expressions aren’t great at excluding, but you could try !(filenamehere). Or, if you have conditions in your steps, you could run a check in one step to manually extract it and go from there.

Harrison Heck

I’ve tried that exclusion but it didn’t work. Specifically it was /!(docker/pulumi/*). I’m not sure if it’s supposed to work in CF or if they don’t support negation.

Harrison Heck

The whole point of these trigger rules though are to bypass starting a job for this. That’s a LOT of overhead for no reason.

Kristofer Svardstal

Understandable. Let me see what I can come up with.

Codefresh Status - Incident History
11:24:39 PM

https://status.codefresh.io/incidents/xx6mqg203nj6\|Docker has reported an incident Oct 18, 23:06 UTC Identified - Docker has reported an incident. It’s affecting the following components:

  • Docker Package Repositories

More information here:

Docker has reported an incident

Codefresh’s Status Page - Docker has reported an incident.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

2019-10-17

Erik Osterman

Not likely…

Alex Siegman

I’m confused. Why would you not be able to do that? Take the “passed in” stuff, have a step that does some logic and does a cf_export on the desired build args, then in the build step use the ${{}} to snag those exported environment variables. Doesn’t cf_export make the variables available to the pipeline and all future steps? Granted, I may not be understanding the initial problem.

Erik Osterman

Docker Build args have a special section

Erik Osterman

If there’s a deterministic number of args, each one could be defined and mapped to an env

Erik Osterman

But if the list I’d build args vary uncontrollably not sure how to do it

Alex Siegman

Yeah, if you want a different number of build args, that might not be possible with the build step’s build args, but if you want the same build args with different values or whatever, then I think that’s doable.

Codefresh Status - Incident History
05:09:38 PM

https://status.codefresh.io/incidents/gd72cldlm28q\|Docker Hub has reported an ongoing incident Oct 17, 16:55 UTC Resolved - Docker System Status Current system status information:

All Systems OperationalOct 16, 15:41 UTC Investigating - This could affect your Codefresh builds.

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721

Docker Hub has reported an ongoing incident

Codefresh’s Status Page - Docker Hub has reported an ongoing incident.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

@Alex Siegman different number of build args and different arg names

I just have some very similar applications which i’ve wanted to share pipelines for because they build & deploy the same way

its just one builds w/ a couple build args and the others do not

i could add a conditional step (on the repo name) for that project, but at that point i think id prefer to just use a seperate pipeline

2019-10-16

Codefresh Status - Incident History
03:59:39 PM
Docker Hub has reported an ongoing incident

Codefresh’s Status Page - Docker Hub has reported an ongoing incident.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

codefresh users, do you use a single github CI user token to add triggers on all pipelines? have you found a more secure way of doing this that is still maintainable

Erik Osterman

One thing to consider is introducing a gitops style workflow for adding pipelines.

Erik Osterman

Basically, make everyone a ‘user’ in codefresh with no specific permissions

1
Erik Osterman

cerate the github CI integration with trigger admin permissions

Erik Osterman

then use PR review/approval process to rollout the pipelines using the codefresh cli

Alex Siegman

Anyone set up Codefresh with GSuite SAML custom app for SSO? I did so a couple days ago and can’t get it to work, and it’s been 2 days with no response from support =/ . I followed their docs, and when I try to sync team, I get a BAD REQUEST response code. Just want to make sure I lined up the right properties in the right places.

@Alex Siegman can you send me the ticket number please ?

looking for help in escaping yaml… need to run ~this in Codefresh pipeline:

commands:
      - kubectl run --env ...

and the env variable is: LOGIN='{"randomDelay": 120000, "pauseAfter": 90000}'

how do I escape that to be valid…?

Alex Siegman

Use a block scalar with >- perhaps? https://yaml-multiline.info/

YAML Multiline Strings

Find the right syntax for your YAML multiline strings.

1
Alex Siegman

something like:

commands:
  - >-
    kubectl run --env LOGIN='{"randomDelay": 120000, "pauseAfter": 90000}'
    more command that will be folded in to a single line above
  - next new command

No clue if that’ll work, but I’m pretty sure you can use : in block scalars since they are terminated by indentation

yeah this totally worked, small note on our specific case, had to do: ... --env "LOGIN={\"randomDelay\": 120000, \"pauseAfter\": 90000}" - more escaping

thanks, I think that may work!

Erik Osterman
codefresh-io/codefresh-pipeline-runner

GitHub action with ability to run codefresh pipeline - codefresh-io/codefresh-pipeline-runner

has anyone tried to dynamically pass in build args in the build step (in a shared pipeline setting where you could have multiple triggers from different services)

Erik Osterman

…as in Docker build args?

yeah

prob not possible using their builtin build step huh

2019-10-15

Codefresh Status - Incident History
05:34:46 PM

https://status.codefresh.io/incidents/z63nfcjcpt08\|Docker Hub Incident Oct 15, 17:24 UTC Update - More information here:

Docker Hub Incident

Codefresh’s Status Page - Docker Hub Incident.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Codefresh Status - Incident History
06:19:52 PM

https://status.codefresh.io/incidents/z63nfcjcpt08\|Docker Hub Incident Oct 15, 18:07 UTC Resolved - Docker Hub has reported that the affected services are operational again.Oct 15, 17:24 UTC Update - More information here:

Harrison Heck

Anyone know how to have cf_export not actually echo the value? Not sure why it does this.

Kostis (Codefresh)

@Harrison Heck you can write directly to the underlying file https://codefresh.io/docs/docs/codefresh-yaml/variables/#directly-writing-to-the-file

Variables

Codefresh is a Docker-native CI/CD platform. Instantly build , test and deploy Docker images.

Harrison Heck

That’s it though huh? I’m not sure why cf_export echos the value by default anyways, it’s a security risk.

Harrison Heck

Having to echo to the file is not the best experience.

Codefresh Status - Incident History
07:54:42 PM

https://status.codefresh.io/incidents/rd7b4ldr73w9\|Docker Hub has reported an ongoing incident Oct 15, 19:43 UTC Investigating - Docker Hub has reported an ongoing incident

This could affect your Codefresh builds.

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721

Docker Hub has reported an ongoing incident

Codefresh’s Status Page - Docker Hub has reported an ongoing incident.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Codefresh Status - Incident History
09:14:40 PM

https://status.codefresh.io/incidents/rd7b4ldr73w9\|Docker Hub has reported an ongoing incident Oct 15, 20:59 UTC Monitoring - From DokerHub: [Monitoring] Docker hub web login is working again. We will continue to monitor the situation.

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721Oct 15, 19:43 UTC Investigating - Docker Hub has reported an ongoing incident

This could affect your Codefresh builds.

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721

Docker Hub has reported an ongoing incident

Codefresh’s Status Page - Docker Hub has reported an ongoing incident.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Codefresh Status - Incident History
10:49:38 PM

https://status.codefresh.io/incidents/rd7b4ldr73w9\|Docker Hub has reported an ongoing incident Oct 15, 22:32 UTC Resolved - From DockerHub:

All Systems OperationalUpdated a few seconds ago Our system status page is a real-time view of the performance and uptime of Docker products and services.

More information here: https://status.docker.comOct 15, 20:59 UTC Monitoring - From DokerHub: [Monitoring] Docker hub web login is working again. We will continue to monitor the situation.

More information here:…

Docker Hub has reported an ongoing incident

Codefresh’s Status Page - Docker Hub has reported an ongoing incident.

2019-10-11

can I set this from within codefresh.yml?

dustinvb

Yes, under the spec section of the pipeline.

spec:
  terminationPolicy:
    - type: branch
      event: onCreate
      ignoreBranch: true
  concurrency: 1
  triggerConcurrency: 1
  externalResources: []

Looks like our spec examples need to be updated. Will put in that request now.

what happens to the other builds if I set concurrency: 1? do they get queued or do they get terminated?

dustinvb

They are queued.

Then you set a termination policy that enforces the behavior selected when they get scheduled to run.

Erik Osterman

this is great!

Erik Osterman

@Igor Rodionov @aknysh @Jeremy Grodberg

Erik Osterman

@Alex Siegman

Erik Osterman

we should drop the check for concurrency and just use this.

aknysh

yea that’s good

Jeremy Grodberg

But @Erik Osterman aren’t we generally testing for concurrency of a build in a different pipeline?

Erik Osterman

no

Erik Osterman

same pipeline

Jeremy Grodberg

I’m thinking we check to make sure all PR build/deploys are finished before running destroy. That’s the only concurrency check that comes to mind.

aknysh

we check if the same pipeline is running and wait if it is

2019-10-10

Codefresh still does not support limiting pipeline concurrency?

If I wanted to only ever have 1 instance of pipeline running, and all others to queue - this is still only done by a step with codefresh cli script?

Erik Osterman

I think there’s an option to cancel previous build

Erik Osterman

but not limit concurrency

Erik Osterman

@dustinvb did this change?

dustinvb

This is found under the pipeline SETTINGS -> POLICIES tab.

2019-10-09

Erik Osterman

https://community.codefresh.io/ launched! that’s nice

Codefresh

The official Codefresh Community Forums

2019-10-08

Harrison Heck

Has anyone integration CodeFresh with DataDog? This way we can see all of the events there and create monitors from it? For example, we were using this to create an alert that got sent to OpsGenie whenever our infrastructure repo failed.

dustinvb

You can try this. Not certain it is what you’re looking for but it’s what I have on hand.

 SendDatadogEvent:
    title: Sending Datadog Event
    image: concourse/datadog-event-resource:latest
    commands:
      - >-
        echo '
        {
          "source": {
            "application_key": "${{DATADOG_APPLICATION_KEY}}",
            "api_key": "${{DATADOG_API_KEY}}",
            "filter": ""
           },
           "params": {
             "title": "${{CF_REPO_NAME}}",
             "text": "${{CF_BUILD_URL}}",
             "priority": "low",
             "alert_type": "success",
             "host": "<https://g.codefresh.io>",
             "aggregation_key": "${{CF_BUILD_ID}}",
             "source_type": "codefresh",
             "tags": [
               "codefresh"
             ]
           }
        }' \| /opt/resource/out ./event.json
1
Harrison Heck

Perfect, I can adapt that as needed. Hopefully one day there is native integration.

2019-10-07

Michael Kolb

@ Thanks for the sharing the information

Michael Kolb

Anyone else having issues with support response times from [email protected]?

Alex Siegman

I regularly have multiple days go by on seemingly simple tickets. I’ve only put a handful in. Luckily nothing has been an outage incident, but it took something like 2 weeks to get a “it doesn’t work that way here’s a link to relevant documentation” which is not service I could brag about or recommend. I’m hoping my first few tickets have been a fluke though.

Oleg Sucharevich

@Michael Kolb what is the issue?

Michael Kolb

@Oleg Sucharevich I have a technical issue with image push to ECR that is outstanding now for over 2 days with multiple emails threads

Michael Kolb

From the logging that i get the pipeline is not finding the image in http://r.cfcr.io/navican/ from the candidate that is built. The sha256 references are not matching up

Oleg Sucharevich

Can you please send me the build id?

Oleg Sucharevich

Will escalate it

Oleg Sucharevich

Thanks!

Michael Kolb

@Oleg Sucharevich Thanks for the help

Oleg Sucharevich

Sure

2019-10-04

@Michael Kolb … I just had the same issue with variables in manual pipelines. It happens because the CodeFresh environment variables are left undefined. The key is to override the variables in the ‘options’ passed to the run

 if (servicePipelineMap.has(key)) {
          const { pipeline, trigger } = servicePipelineMap.get(key)
          const data = {
            trigger,
            sha,
            branch: targetBranch,
            variables: {
              CF_PULL_REQUEST_ID: id,
              CF_PULL_REQUEST_NUMBER: number,
              CF_BRANCH: targetBranch,
              CF_BASE_BRANCH: prBranch,
              CF_PULL_REQUEST_TARGET: targetBranch,
              CF_COMMIT_AUTHOR: user.login,
            },
          }
          app.log(data.variables)
          const workflowId = await sdk.pipelines.run({ name: pipeline }, data)
          app.log(
            `${pipeline} triggered a pipeline with id ${workflowId} on PR ${number} to ${ref}`
          )
        } else {
          app.log(`There is no pipeline associated with this service`)
        }
Erik Osterman

@Michael Kolb this is with the codefresh js sdk?

Erik Osterman
codefresh-io/js-sdk

Contribute to codefresh-io/js-sdk development by creating an account on GitHub.

They need to do a sprint or two on properly documenting their API/SDK. The swagger doc is not very helpful as it lacks example responses. The CLI is much more reliable and is entirely based off the SDK. The SDK is powerful but lacks documentation. The only way I was able to not be blocked was by forking the CLI and reading the source code…

codefresh-io/js-sdk

Contribute to codefresh-io/js-sdk development by creating an account on GitHub.

Yes @Erik Osterman Being as I need to manually trigger pipelines outside of CodeFresh I found the JS-SDK to be the easiest to work with. They should make a point of featuring it.

dustinvb

@Kostis (Codefresh) Can you add this to our documentation backlog to document our JS and GO SDK?

I found it only by going through their github org

2019-10-02

Michael Kolb

yes, i have been running automatically and manually. The manual iterations appear to be the iterations that fail. I have had to run the pipeline manually at times, since I have been experiencing intermittent Push step failures to AWS ECR. It is unfortunate that this attribute of the variable is not documented. https://codefresh.io/docs/docs/codefresh-yaml/variables/

Variables

Codefresh is a Docker-native CI/CD platform. Instantly build , test and deploy Docker images.

Erik Osterman

@here public #office-hours starting now! join us to talk shop https://zoom.us/j/508587304

2019-10-01

Michael Kolb

Anyone else out there having issues with pushing images to AWS ECR? Codefresh support has gone dark and all my pipelines now fail to push to AWS ECR.

Alex Siegman

As if there’s an outage, I don’t think so, i’ve successfully pushed a few to ECRs in us-east-1 in the last hour or so

Michael Kolb

thanks… my pipelines push to us-west-2. All of them are failing including ones that worked before

Michael Kolb

are you using the integrated registries or

provider: ecr
Michael Kolb

Has anyone seen issues with codefresh variables interpolating with the build step?

stage: build-images
type: build
title: Build Api Image
description: build api image
working_directory: '${{CF_VOLUME_PATH}}/${{CF_REPO_NAME}}'
dockerfile: '${{dockerfile}}'
image_name: '${{image_name}}'
tag: '${{vtag}}-${{CF_PULL_REQUEST_NUMBER}}'

I am getting a scenario where the build step fails on subsequent commits to the feature branch and will not interpolate the ${{CF_PULL_REQUEST_NUMBER}} at run time.

I get the following at build time:

Building image: my_image_name:0.1.0-${{cf_pull_request_number}} using unique hash: 6903d14a13db6d2cef7a7134925ea0f68468bd00
Erik Osterman

That variable is only populated when pipeline is triggered by a Pr webhook

Erik Osterman

Are you running it manually by any Chance?

2019-09-24

Codefresh Status - Incident History
12:54:42 PM

https://status.codefresh.io/incidents/ggtjcd7jfxcn\|GitHub has reported an incident Sep 24, 12:44 UTC Investigating - GitHub has reported an incident This might impact your builds.

More information here: https://www.githubstatus.com/incidents/b54zv9nz8m01

GitHub has reported an incident

Codefresh’s Status Page - GitHub has reported an incident.

Incident on 2019-09-24 11:30 UTC

GitHub’s Status Page - Incident on 2019-09-24 11:30 UTC.

Codefresh Status - Incident History
02:39:44 PM

https://status.codefresh.io/incidents/ggtjcd7jfxcn\|GitHub has reported an incident Sep 24, 14:28 UTC Resolved - This incident has been resolved.Sep 24, 12:44 UTC Investigating - GitHub has reported an incident This might impact your builds.

More information here: https://www.githubstatus.com/incidents/b54zv9nz8m01

GitHub has reported an incident

Codefresh’s Status Page - GitHub has reported an incident.

2019-09-20

Codefresh Status - Incident History
05:24:39 PM

http://status.codefresh.io/incidents/3bs50kjfk2vs\|Codefresh Incident - Builds are stuck in pending Sep 20, 17:16 UTC Update - We are continuing to investigate this issue.Sep 20, 17:15 UTC Investigating - Currently we’re investigating an issue that’s affecting the execution of builds in the system.

Codefresh Incident - Builds are stuck in pending

Codefresh’s Status Page - Codefresh Incident - Builds are stuck in pending.

Codefresh Status - Incident History
06:29:40 PM

https://status.codefresh.io/incidents/3bs50kjfk2vs\|Codefresh Incident - Builds are stuck in pending Sep 20, 18:21 UTC Identified - Issue identified: We are experiencing a connectivity issue with a third-party vendor, which affects the scheduling flow for our builds. The ETA to resolve this issue is ~1 hour.Sep 20, 17:16 UTC Update - We are continuing to investigate this issue.Sep 20, 17:15 UTC Investigating - Currently we’re investigating an issue that’s affecting the execution of builds in the system.

Codefresh Incident - Builds are stuck in pending

Codefresh’s Status Page - Codefresh Incident - Builds are stuck in pending.

whats up with codefresh today?

aknysh

down

Codefresh Status - Incident History
07:14:38 PM

https://status.codefresh.io/incidents/3bs50kjfk2vs\|Codefresh Incident - Builds are stuck in pending Sep 20, 18:21 UTC Identified - Issue identified: We are experiencing a connectivity issue with GCP Cloud SQL (PostgreSQL) which affects the scheduling flow for our builds. The ETA to resolve this issue is ~1 hour.Sep 20, 17:16 UTC Update - We are continuing to investigate this issue.Sep 20, 17:15 UTC Investigating - Currently we’re investigating an issue that’s affecting the execution of builds in the system.

Alex Siegman

Yeah, UI is broken too - teammate is getting the new user experience~

Oleg Sucharevich

We are on it, will update once we back

3

my coworkers first day using the app as well

Codefresh Status - Incident History
08:49:40 PM

https://status.codefresh.io/incidents/3bs50kjfk2vs\|Codefresh Incident - Builds are stuck in pending Sep 20, 20:45 UTC Update - We are continuing to monitor for any further issues.Sep 20, 20:34 UTC Monitoring - Builds are currently able to be executed. We’re still monitoring the system.Sep 20, 18:21 UTC Identified - Issue identified: We are experiencing a connectivity issue with GCP Cloud SQL (PostgreSQL) which affects the scheduling flow for our builds. The ETA to resolve this issue is ~1 hour.Sep 20, 17:16 UTC Update - We are continuing to investigate this issue.Sep 20, 17:15 UTC…

Codefresh Incident - Builds are stuck in pending

Codefresh’s Status Page - Codefresh Incident - Builds are stuck in pending.

Oleg Sucharevich

We are back

1
Oleg Sucharevich

the status page will be updated now as well

Codefresh Status - Incident History
09:19:38 PM

https://status.codefresh.io/incidents/3bs50kjfk2vs\|Codefresh Incident - Builds are stuck in pending Sep 20, 21:04 UTC Update - Builds are able to progress now. We’re still investigating an issue with our Helm-repository integration and audit-log data availability.Sep 20, 20:45 UTC Update - We are continuing to monitor for any further issues.Sep 20, 20:34 UTC Monitoring - Builds are currently able to be executed. We’re still monitoring the system.Sep 20, 18:21 UTC Identified - Issue identified: We are experiencing a connectivity issue with GCP Cloud SQL (PostgreSQL) which affects the…

Codefresh Status - Incident History
09:54:39 PM

https://status.codefresh.io/incidents/3bs50kjfk2vs\|Codefresh Incident - Builds are stuck in pending Sep 20, 21:36 UTC Update - We are continuing to monitor for any further issues.Sep 20, 21:36 UTC Update - Helm-repository integration is working now. Old audit-log data will be available soon.Sep 20, 21:04 UTC Update - Builds are able to progress now. We’re still investigating an issue with our Helm-repository integration and audit-log data availability.Sep 20, 20:45 UTC Update - We are continuing to monitor for any further issues.Sep 20, 20:34 UTC Monitoring - Builds are currently able to be…

Codefresh Incident - Builds are stuck in pending

Codefresh’s Status Page - Codefresh Incident - Builds are stuck in pending.

Codefresh Status - Incident History
10:39:51 PM

https://status.codefresh.io/incidents/3bs50kjfk2vs\|Codefresh Incident - Builds are stuck in pending Sep 20, 22:23 UTC Resolved - Issue related to stuck builds is completely resolvedSep 20, 21:36 UTC Update - We are continuing to monitor for any further issues.Sep 20, 21:36 UTC Update - Helm-repository integration is working now. Old audit-log data will be available soon.Sep 20, 21:04 UTC Update - Builds are able to progress now. We’re still investigating an issue with our Helm-repository integration and audit-log data availability.Sep 20, 20:45 UTC Update - We are continuing to monitor for…

Codefresh Status - Incident History
10:39:51 PM

https://status.codefresh.io/incidents/r8lzcjzbrv3j\|Old Audit-log data is not available Sep 20, 22:29 UTC Identified - We’re currently working on restoring the old audit log data. It wil be available soon

Old Audit-log data is not available

Codefresh’s Status Page - Old Audit-log data is not available.

2019-09-18

Erik Osterman

public #office-hours starting now! join us to talk shop https://zoom.us/j/508587304

2019-09-16

2019-09-14

creslinux

Afternoon. Im rebuilding my k8s and codefresh - my first, built two weeks back, got tangled. When i try to delete or edit a service in kube-system i hit replicasets.extensions is forbidden: User "system<i class="em em-serviceaccount"></i>kube-system:codefresh-user" cannot list resource "replicasets" in API group "extensions" in the namespace "kube-system".

creslinux

This has been applied without error - which i thought would give control to the SA codefresh user over kube-system

dustinvb

Can you try editing the apiGroups to: - apiGroups: ["extensions", ""]

creslinux

Ignore above - the problem here was the nodes did not fully come up on the az, fixed my terraform and now working

creslinux

Im hitting an RBAC problem,

Message             services "dev-mm-user-fe-default-srv" is forbidden: User                                                                                
                     "system<i class="em em-serviceaccount"></i>kube-system:codefresh-user" cannot get resource "services" in API group ""                                       
                     in the namespace "eu-dev-public-ns"     

Ive applying the following, that i hoped would allow DF to deploy to this ns/svc - but didnt

apiVersion: <http://rbac.authorization.k8s.io/v1>
kind: ClusterRoleBinding
metadata:
  name: codefresh-user
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io>
  kind: ClusterRole
  name: codefresh-role
subjects:
- kind: ServiceAccount
  name: codefresh-user
  namespace: eu-dev-public-ns
creslinux

any hints would be greatly received

Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Erik Osterman

We use Helmfile and deploy the Codefresh account like this

Erik Osterman

Not sure if it is helpful…

creslinux

Hi @Erik Osterman thanks for the link - it might be useful. I’m very much learning as i go, do i need to add this as a helm repo ?

creslinux

will read on helmfile.d

Erik Osterman

Actually Helmfile is a way to manage helm releases

Erik Osterman

Helmfile will add the repos for you

Erik Osterman

We also have a #helmfile channel

2
Erik Osterman
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Erik Osterman

Read more about it here

2
creslinux

same troubles

services "dev-mm-user-fe-default-srv" is forbidden: User "system<i class="em em-serviceaccount"></i>kube-system:codefresh-service-account" cannot update resource "services" in API group "" in the namespace "eu-dev-public-ns"

2019-09-13

2019-09-12

creslinux

Morning guys

creslinux

Struggling here, a pipe was working now failing – big change could be is my EKS was bumped from 1.13 to 1.14

Executing command: /cf-deploy-kubernetes /codefresh/volume/cf-generated/deployment.yml
Server minor version: 14
Warning: No variable was found in /codefresh/volume/cf-generated/deployment.yml, syntax is {{VAR}}
---> Kubernetes objects to deploy in /codefresh/volume/cf-generated/deployment.yml :

Nothing obvious to me - I’ve put YAML in the pipe from prior working runs

Erik Osterman

Can you share the pipeline yaml?

Alex Siegman

So, I’m having an issue. I’m trying to conditionally do a pipeline step based off both a condition and an approval step match, but I can’t find the right YAML voodoo to make it work, and there are ZERO examples that I can find in the docs as well.

example yaml step:

  deploy_helmfile_sqitch:
    title: Deploy with helmfile
    stage: Deploy
    image: "${{TOOL_DOCKER_REPO_PREFIX}}/${{AWS_ENV_NAME}}:latest"
    working_directory: /conf/
    environment:
      - 'KUBECONFIG=${{CF_KUBECONFIG_PATH}}'
      - 'REPO_ROOT=${{CF_VOLUME_PATH}}/${{CF_REPO_NAME}}'
    shell: bash
    commands:
      # Announce the release version
      - echo "* Preparing to deploy ${RELEASE_NAME} with image ${IMAGE_NAME}:${IMAGE_TAG}"
      - echo "*     with flavor \"$FLAVOR\" into namespace ${RELEASE_NAMESPACE}"
      - kubectl config use-context "${KUBE_CONTEXT}"
      - cd $(conf-directory helmfiles) && echo Working directory is now $PWD
      - helmfile --namespace="${RELEASE_NAMESPACE}" --environment="${FLAVOR}"
        -f "${REPO_ROOT}/deploy/releases/sqitch.yaml" sync
    when:
      steps:
      - name: ask_for_permission
        on:
        - approved
      conditions:
        any:
          comment_is_not_airflow: '"${{COMMENT}}" != "airflow"'
Alex Siegman

This returns Line: 113 \| deploy_helmfile_sqitch: "conditions" is not allowed. Current value: [object Object]

Igor Rodionov

@Alex Siegman have you tried to play with field when for approval step ?

Igor Rodionov
Approval

How to pause pipelines for manual approval

Igor Rodionov
Approval

How to pause pipelines for manual approval

Alex Siegman

yeah, i can get the when step to work with either a steps section or a conditions section, but not both

Alex Siegman

unless I need to do something like:

when:
  all:
    steps:
      #steps go here
    conditions:
      # conditions go here

I didn’t think to try that

Erik Osterman

@dustinvb

dustinvb

@Kristofer Svardstal can you take a look? Still no power at my home.

Erik Osterman

Wow! That’s horrible

Kristofer Svardstal

Hey hey!

Kristofer Svardstal

I believe this should be possible, so just bear with me while I get an example together.

Kristofer Svardstal

@Alex Siegman Hey, so, under when, conditions should be condition.

Alex Siegman

Sometimes it’s the small things in life. Will try that, thank you

Erik Osterman

@Igor Rodionov

Erik Osterman

Any tips

2019-09-10

Kristofer Svardstal

You can also ping me, @Kristofer Svardstal, and I will try to assist!

1

2019-09-09

oscar

Spinnaker vs Codefresh? Thoughts

Alex Siegman

I’d love to have an opinion on this. I want to like Spinnaker but never had a chance to demo it due to a high cost to start. Lots of operations and dev time involved in running your own CICD system. CodeFresh on the other hand is flipping expensive in and of itself, but probably really cost effective until maybe 10 or so concurrency.

oscar

Thanks for the input.

New firm very keen to use Spinnaker (invest the time) instead of investing the constant money into Codefresh each month (we’d want enterprise onpremise so we don’t have to expose internal infrastructure to WWW… @Erik Osterman any opinions on that exposure & a solution?)

Erik Osterman

look into concourse ci, argo, and spinnaker.

dustinvb

Codefresh guy sharing his thoughts…

I often beat Spinnaker in side-by-sides just b/c of the high cost of implementation (often it is not even running by the end of the Codefresh evaluation)

There are other costs often not taken into account for running any services on-premise.

  1. Installation
  2. Maintenance
  3. Availability

These all translate to someone’s time which translates to money. This includes running Codefresh on-premise.

With our Hybrid offering we assume these costs in our product pricing and assume the responsibilities above which would normally be left to engineering.

We offer a unique Hybrid solution where we deploy a runner agent as a pod to a behind-the-firewall Kubernetes cluster requiring only egress out to Codefresh API to operate. The operations will take place on customer’s the Kubernetes cluster the pod itself will upgrade itself and scale out jobs to the capacity of the cluster.

https://codefresh.io/docs/docs/enterprise/behind-the-firewall/

If you’re looking at running Codefresh on premise there is a Helm chart available to install and upgrade but there is a cost in engineering effort behind this as well as any other on-premise service.

Also, I noticed you no longer are representing your moose hat here in Slack…

Codefresh behind the firewall

How to run Codefresh pipelines in your own secure infrastructure

dustinvb

Sorry, I haven’t been that active in this channel. Been a busy summer. If anyone ever has a question relating to Codefresh just ping me on here using @dustinvb and I’ll get back to you ASAP

2
2

2019-09-06

2019-09-05

anyone running codefresh pipeline invoked by another pipeline successfully with variables being passed in? (the official step is called codefresh-run) the pipeline being invoked doesn’t seem to be getting the variables for me

Alex Siegman

i think the cloudposse example app might have that… i have a “release” pipeline that triggers off a github release, and it calls deploy pipelines conditionally based on the release tag format

Alex Siegman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Alex Siegman

We have a disabled trigger on the target pipeline, but I can’t quite remember what that solved at the moment

yeah im using their official codefresh-run step instead of invoking the cli, i might just end up going back to the cli command

jafow


i might just end up going back to the cli command
FWIW this is what we did for similar reasons.

I have a step using the codefresh cli and I have an identical step using their official one and using the official one doesn’t work :/

dustinvb

I am sorry this example is not working can you please send me the URL to verify and fix?

Here is a working example of a matrix pipeline using the codefresh-run to call the dvt pipeline.

https://github.com/dustinvanbuskirk/example-voting-app/blob/master/.codefresh/codefresh-cd.yml#L164-L180

dustinvanbuskirk/example-voting-app

Docker’s Example Voting App. Contribute to dustinvanbuskirk/example-voting-app development by creating an account on GitHub.

dustinvb

I also have a Knapsack example that uses codefresh-run in a parallel step type.

version: "1.0"
stages:
  - "clone"
  - "build"
  - "test-prep"
  - "pipeline-tests"
steps:
  main_clone:
    type: "git-clone"
    description: "Cloning main repository..."
    repo: "dustinvanbuskirk/rails-app-with-knapsack"
    revision: "${{CF_BRANCH}}"
    stage: "clone"
  BuildTestDockerImage:
    title: Building Test Docker image
    type: build
    arguments:
      image_name: 'dustinvanbuskirk/rails-app-with-knapsack-test'
      tag: '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}'
      dockerfile: Test.Dockerfile
    stage: "build"
  CleanAllureFolder:
    title: Cleaning Allure Folder
    stage: "test-prep"
    image: alpine
    commands:
      - rm -rf ${{CF_VOLUME_PATH}}/allure-results
  ImportKnapsackReport:
    title: "Importing Knapsack Report"
    image: ${{BuildTestDockerImage}}
    stage: "test-prep"
    environment:
      - KNAPSACK_GENERATE_REPORT=true
    commands:
      - cf_export KNAPSACK_REPORT=$(cat knapsack_rspec_report.json \| jq -c \| base64 \| tr -d '\n')
  RunKnapsackTestsPipelines:
    title: "Running Knapsack Tests Pipelines"
    type: parallel
    stage: "pipeline-tests"
    steps:
      RunKnapsackTestsPipeline1:
        title: "Running Knapsack Test Pipeline 1"
        type: codefresh-run
        fail_fast: true
        arguments:
          PIPELINE_ID: "knapsack-demo/knapsack-demo-rspec-runner"
          TRIGGER_ID: "dustinvanbuskirk/rails-app-with-knapsack"
          BRANCH: ${{CF_BRANCH}}
          SHA: ${{CF_REVISION}}
          VARIABLE:
            - CI_NODE_TOTAL=3
            - CI_NODE_INDEX=0
            - KNAPSACK_REPORT='${{KNAPSACK_REPORT}}'
            - PARENT_BUILD_ID=${{CF_BUILD_ID}}
          NO_CACHE: false
          NO_CF_CACHE: false
          ENABLE_NOTIFICATIONS: false
          RESET_VOLUME: false
          DETACH: false
      RunKnapsackTestsPipeline2:
        title: "Running Knapsack Test Pipeline 2"
        type: codefresh-run
        fail_fast: true
        arguments:
          PIPELINE_ID: "knapsack-demo/knapsack-demo-rspec-runner"
          TRIGGER_ID: "dustinvanbuskirk/rails-app-with-knapsack"
          BRANCH: ${{CF_BRANCH}}
          SHA: ${{CF_REVISION}}
          VARIABLE:
            - CI_NODE_TOTAL=3
            - CI_NODE_INDEX=1
            - KNAPSACK_REPORT=${{KNAPSACK_REPORT}}
            - PARENT_BUILD_ID=${{CF_BUILD_ID}}
          NO_CACHE: false
          NO_CF_CACHE: false
          ENABLE_NOTIFICATIONS: false
          RESET_VOLUME: false
          DETACH: false
      RunKnapsackTestsPipeline3:
        title: "Running Knapsack Test Pipeline 3"
        type: codefresh-run
        fail_fast: true
        arguments:
          PIPELINE_ID: "knapsack-demo/knapsack-demo-rspec-runner"
          TRIGGER_ID: "dustinvanbuskirk/rails-app-with-knapsack"
          BRANCH: ${{CF_BRANCH}}
          SHA: ${{CF_REVISION}}
          VARIABLE:
            - CI_NODE_TOTAL=3
            - CI_NODE_INDEX=2
            - KNAPSACK_REPORT=${{KNAPSACK_REPORT}}
            - PARENT_BUILD_ID=${{CF_BUILD_ID}}
          NO_CACHE: false
          NO_CF_CACHE: false
          ENABLE_NOTIFICATIONS: false
          RESET_VOLUME: false
          DETACH: false
  CopyAllureResultsFromS3:
    title: "Copying Allure Results From S3"
    stage: "pipeline-tests"
    image: dustinvanbuskirk/cfstep-knapsack:alpha
    commands:
      - aws s3 cp --recursive s3://${{BUCKET_NAME}}/${{CF_BUILD_ID}}/ ${{CF_VOLUME_PATH}}/allure-results/
  UploadTestReports:
    title: "Uploading Test Reports"
    stage: "pipeline-tests"
    image: codefresh/cf-docker-test-reporting
    environment:
      - ALLURE_DIR=${{CF_VOLUME_PATH}}/allure-results

2019-09-04

David Holsgrove

Hey, general question - say I’ve built a docker image in a codefresh pipeline (tagged in the CF registry and also pushed out to ECR) and want to use that as the image in a different pipelines freestyle step - how do I auth to the private registry to pull the image? Dont mind if CF is the easiest, but can’t seem to find a way to use my ECR credentials which let me push images to also pull images?

Erik Osterman

#office-hours starting now! ask questions, get answers. free for everyone. https://zoom.us/j/508587304

Alex Siegman

So - I’ve got a new pipeline stuck in “pending” state - is there any way to get info on what that means / why that is? zero other builds are running

2019-09-03

creslinux

Anybody have any hints for deploying an EKS cluster via Codefesh, the example in their repo fails as described in the CF repo issue https://github.com/codefresh-io/eks-installer/issues/5

Failed to run freestyle step: Save Terraform statefile to Codefresh context; caused by NonZeroExitCodeError: Container for step title: Save Terraform statefile to Codefresh context, step type: freestyle, operation: Freestyle step failed with exit code: 2 · Issue #5 · codefresh-io/eks-installer

Running freestyle step: Save Terraform statefile to Codefresh context Pulling image codefresh/cli:0.8.54 Digest: sha256:ff73c256681cbf6e6ef8304748d50e6f54903096b7dad9116554694b5fedfbb6 Status: Imag…

dustinvb

Let me escalate this to engineering.

dustinvb

I do not have another example of automated creation and addition in a pipeline looks like this needs a fix to the API call responsible for adding the EKS cluster to our Kubernetes cluster integration.

creslinux

Awesome, 10pm here - that sounds like my home/bed time as im a total newbie and could not help other that test works after a change

creslinux

thanks @dustinvb

2019-08-31

Codefresh Status - Incident History
02:49:51 PM

http://status.codefresh.io/incidents/k86gxg1bt83g\|Codefresh build clusters networking issues Aug 31, 14:39 UTC Investigating - Amazon has reported connectivity issues affecting some instances. They are currently working on investigating the issues More information can be found on https://status.aws.amazon.com/

Codefresh build clusters networking issues

Codefresh’s Status Page - Codefresh build clusters networking issues.

Codefresh Status - Incident History
09:04:44 PM

http://status.codefresh.io/incidents/k86gxg1bt83g\|Codefresh build clusters networking issues Aug 31, 20:47 UTC Resolved - AWS Incident is no longer affecting CF.Aug 31, 14:39 UTC Investigating - Amazon has reported connectivity issues affecting some instances. They are currently working on investigating the issues More information can be found on https://status.aws.amazon.com/

Codefresh build clusters networking issues

Codefresh’s Status Page - Codefresh build clusters networking issues.

2019-08-29

for those using direnv in codefresh, do you have a seperate file that you rename to .envrc when the pipeline runs? How do you handle committing that to the repo for the pipeline and having a local copy

Erik Osterman
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

Erik Osterman

I don’t follow your question

Erik Osterman

But this is how we did it

2019-08-28

Codefresh Status - Incident History
08:54:39 AM

http://status.codefresh.io/incidents/18b61rq0kdcb\|Bitbucket has reported an incident on their service Aug 28, 08:47 UTC Resolved - The incident has been resolved and all systems are operational.Aug 27, 17:23 UTC Investigating - Incident details: Bitbucket commits API high response time and timeouts This incident affects: API, SSH, Git via HTTPS, Mercurial via HTTPS, and Pipelines.

More info: https://bitbucket.status.atlassian.com/incidents/d1nyzp56331y

Bitbucket has reported an incident on their service

Codefresh’s Status Page - Bitbucket has reported an incident on their service.

Bitbucket commits API high response time and timeouts

Atlassian Bitbucket’s Status Page - Bitbucket commits API high response time and timeouts.

2019-08-27

Codefresh Status - Incident History
05:29:43 PM

http://status.codefresh.io/incidents/18b61rq0kdcb\|Bitbucket has reported an incident on their service Aug 27, 17:23 UTC Investigating - Incident details: Bitbucket commits API high response time and timeouts This incident affects: API, SSH, Git via HTTPS, Mercurial via HTTPS, and Pipelines.

More info: https://bitbucket.status.atlassian.com/incidents/d1nyzp56331y

Bitbucket has reported an incident on their service

Codefresh’s Status Page - Bitbucket has reported an incident on their service.

Bitbucket commits API high response time and timeouts

Atlassian Bitbucket’s Status Page - Bitbucket commits API high response time and timeouts.

2019-08-22

Codefresh Status - Incident History
10:29:38 AM

http://status.codefresh.io/incidents/lwr3123l1smt\|Github services are partially degraded Aug 22, 10:14 UTC Investigating - GitHub informs: Issues, PRs, Dashboard, Projects, Notifications are partially degraded

Github services are partially degraded

Codefresh’s Status Page - Github services are partially degraded.

Codefresh Status - Incident History
11:19:39 AM

http://status.codefresh.io/incidents/lwr3123l1smt\|Github services are partially degraded Aug 22, 11:11 UTC Update - GitHub reports a major outage on Issues, PRs, Dashboard, Projects, Notifications servicesAug 22, 10:14 UTC Investigating - GitHub informs: Issues, PRs, Dashboard, Projects, Notifications are partially degraded

Codefresh Status - Incident History
12:34:43 PM

http://status.codefresh.io/incidents/lwr3123l1smt\|Github services are partially degraded Aug 22, 12:23 UTC Resolved - This incident has been resolved.Aug 22, 11:11 UTC Update - GitHub reports a major outage on Issues, PRs, Dashboard, Projects, Notifications servicesAug 22, 10:14 UTC Investigating - GitHub informs: Issues, PRs, Dashboard, Projects, Notifications are partially degraded

Github services are partially degraded

Codefresh’s Status Page - Github services are partially degraded.

2019-08-21

is it possible to have a pipeline that listens on a github org? allowing it to create new projects/pipelines based on new repo creation?

woops forgot @Erik Osterman already asked this above ^^

Erik Osterman

haha

Erik Osterman

Erik Osterman

yea, not yet.

Erik Osterman

however, using repo templates might not be a bad start

Erik Osterman

also, in the repo template, include a pipeline-spec

Erik Osterman

and a makefile

Erik Osterman
version: '1.0'
kind: pipeline
metadata:
  name: ourcompany/example-app/destroy-pull-request
  isPublic: false
  project: etan-tolling/example-app
  deprecate:
    repoPipeline: true
    implicitGitCloneService:
      context: github
      scm:
        provider: github
        owner:
          name: ourcompany
        name: example-app
spec:
  triggers:
    - name: ourcompany/destroy-pr
      type: git
      repo: ourcompany/example-app
      events:
        - pullrequest.closed
        - pullrequest.unlabeled
      pullRequestAllowForkEvents: false
      commentRegex: /.*/gi
      branchRegex: /.*/gi
      branchRegexInput: regex
      provider: github
      disabled: false
      options:
        noCache: false
        noCfCache: false
        resetVolume: false
      context: CF-default
      contexts: []
      variables: []
  steps: {}
  stages: []
  variables: []
  contexts:
    - Slack
    - GitHub
  specTemplate:
    location: git
    repo: ourcompany/example-app
    path: ./codefresh/destroy-pull-request.yml
    context: github
Erik Osterman

that’s an example spec

Erik Osterman
codefresh create pipeline -f destroy-pull-request.yml
aknysh

A Codefresh pipeline for creating pipelines

aknysh
Programmatic Creation of Codefresh Pipelines - Part 2 - Codefresh

At Codefresh, we know that any CI/CD solution must be attractive to both developers and operators (SREs). One of the major advantages of Codefresh is the graphical user interface that includes dashboards for Kubernetes and Helm deployments. These graphical dashboards are very useful to developers who are just getting started with deployments and pipelines. We … Continued

^ that requires a monorepo

2019-08-12

Alex Siegman

Seems that pending-approval steps don’t parse variables in the title field. I’d love to be able to customize that text for better UX on reused pipelines. Anyone done something similar? Trying to do this:

  ask_for_permission:
    type: pending-approval
    title: Deploy release of ${{APP_NAME}}:${{IMAGE_TAG}} to ${{FLAVOR}}?
    stage: Prepare

But I get the text seen in title with no substitution for the variables. Is this a feature request for the codefresh folks?

Erik Osterman

@Alex Siegman totally agree

Erik Osterman

We’ve submitted a similar feature request

Erik Osterman

I think the more that do it, the better

Erik Osterman

Go to http://support.codefresh.io to add your vote for this

Erik Osterman

I know @Jeremy Grodberg requested it too

Jeremy Grodberg
06:43:15 PM

@Jeremy Grodberg has joined the channel

Alex Siegman

Feature request sent

for codefresh, sometimes my pipeline will pull a previously built image to use as the cache and other times it will say the following:

Previous built image: <http<i class="em em-//r.cfcr.io/org/[email protected]"></i>hash> exists on the daemon and will be us
ed as cache for the build 

so im wondering if you have many devs working concurrently on the same repo (in different branches) then does that mean that it’s more likely your pipeline will need to pull the image (adds ~1.5-2min for a 2.5GB image) as opposed to already having the image existing on the daemon

Erik Osterman

yes, I think it depends on if the build hits a worker (and thus different daemon) with the image cached.

Erik Osterman

@Igor Rodionov spent a lot of time looking into this recently

when you say worker

you mean k8s worker i assume that codefresh is running on?

Erik Osterman

so behind the scenes they run Docker-in-Docker (for security reasons).

Erik Osterman

it spawns a dind on demand

Erik Osterman

i thin their SaaS product works slightly different than their enterprise product though

Erik Osterman

best thing is to go to <http://support.codefresh.io> and open issue

Erik Osterman

they’ll explain better than me

Erik Osterman

@dustinvb is on vacay i think

is dustin the only cf guy on sweetops?

i was under the impression each pipeline had a persistent volume that would store many docker image layers

Erik Osterman

this is just an informal community driven channel. @Dan Garfield and @ are here too, but may not be SME.

Erik Osterman

And @ for enterprises sales questions

no worries, i wasnt planning on reaching out to anyone directly but if they read this channel and have comments, im not opposed

i guess im wondering how I can maximize the image existing on the daemon so the build doesnt spend an extra 1.5~2min pulling the image first

Erik Osterman

yea, we’ve been bit by this as well. for on prem, we reduced the number of build nodes and increased the size of the nodes.

Erik Osterman

not sure what to do for the hosted version though to improve cache hits (or if its even the problem we had)

that makes sense

for on prem

2019-08-05

dustinvb

@Erik Osterman Do you have an example step where you utilize Helmfile in a Codefresh pipeline?

dustinvb

Have someone who would like to see how this is used in a Codefresh pipelin

Erik Osterman

yes, but not maintained/current.

Erik Osterman
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

dustinvb

There is nothing your team uses currently as part of Codefresh deploy with a helmfile in a pipeline? I thought this was something you were doing regularly for a majority of your Helm deploys.

Erik Osterman

oh that

Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman

but the ctl command there is optional. we have an inline approach we use with our latest pipelines.

Erik Osterman

instead of that deploy_helmfile, we use a step more like

Erik Osterman
deploy_helmfile:
    title: Deploy with helmfile
    stage: Deploy
    image: "${{TOOL_DOCKER_REPO_PREFIX}}/${{AWS_ENV_NAME}}:latest"
    working_directory: /conf/
    environment:
      - 'KUBECONFIG=${{CF_KUBECONFIG_PATH}}'
      - 'REPO_ROOT=${{CF_VOLUME_PATH}}/${{CF_REPO_NAME}}'
      - 'APP_HOST=${{APP_HOST_DOMAIN}}'
    shell: bash
    commands:
      # Announce the release version
      - echo "* Preparing to deploy ${RELEASE_NAME} with image ${IMAGE_NAME}:${IMAGE_TAG}"
      - echo "*     with flavor \"$FLAVOR\" into namespace ${RELEASE_NAMESPACE}"
      - kubectl config use-context "${KUBE_CONTEXT}"
      - cd $(conf-directory helmfiles) && echo Working directory is now $PWD
      - direnv exec . helmfile --namespace="${RELEASE_NAMESPACE}" --environment="${FLAVOR}"
        -f "${REPO_ROOT}/deploy/helmfile.yaml" sync
    when:
      steps:
      - name: ask_for_permission
        on:
        - approved

2019-07-27

2019-07-26

Erik Osterman

@dustinvb have you seen any “auto discovery” feature for codefresh?

Erik Osterman

e.g. a repo could just define something like this:

Erik Osterman
version: "1.0"
kind: "pipeline"
metadata:
  name:  foobar/tests
  isPublic: false
  labels:
    tags: []
  deprecate:
    applicationPort: '8080'
    repoPipeline: true
  project: foobar
spec:
  triggers:
    - branchRegex: /.*/gi
      branchRegexInput: regex
      commentRegex: /.*/gi
      context: github-webhooks
      events:
        - push
      provider: github
      pullRequestAllowForkEvents: false
      repo: foobar
      type: git
  contexts:
    - terraform modules
  variables: []
  specTemplate:
    location: url
    url: >-
      <https://raw.githubusercontent.com/cloudposse/foobar/master/codefresh/terraform.yml>
  stages: []
  steps: {}
Erik Osterman

it would automatically see a new foobar repo was created, detect the config above, and execute it.

Erik Osterman

this would tie in to org webhooks (not repo webhooks)

dustinvb

No I haven’t seen anything like this in use. I had a request just today to have a codefresh terraform module to perform something similar. Can you ping our shared channel and double check with Amir and Kostis there?

Erik Osterman

ok

thatd be awesome

this would be great with github templates

Erik Osterman

Wow, interesting idea

Erik Osterman

I hadn’t considered that

2019-07-25

Codefresh Status - Incident History
03:14:46 AM

http://status.codefresh.io/incidents/7xxcvrbmgx6m\|Platform Latency Jul 26, 03:10 UTC Update - We are continuing to investigate this issue.Jul 26, 03:09 UTC Investigating - Engineering is investigating reports of latency in platform.

Platform Latency

Codefresh’s Status Page - Platform Latency.

Codefresh Status - Incident History
04:49:42 AM

http://status.codefresh.io/incidents/7xxcvrbmgx6m\|Platform Latency Jul 26, 04:45 UTC Resolved - We resolved the issue and the platform performance is back to normalJul 26, 03:10 UTC Update - We are continuing to investigate this issue.Jul 26, 03:09 UTC Investigating - Engineering is investigating reports of latency in platform.

Platform Latency

Codefresh’s Status Page - Platform Latency.

2019-07-24

thanks Erik you guys always have an example for everything I need to start looking there first

1
Erik Osterman

they might not always be the “latest” (since usually it percolates from customer engagements)

2019-07-23

Codefresh Status - Incident History
12:44:42 PM

http://status.codefresh.io/incidents/c1g630llmx51\|Codefresh UI outage Jul 23, 12:32 UTC Resolved - This incident has been resolved.Jul 23, 12:30 UTC Investigating - We are currently investigating this issue.

Codefresh UI outage

Codefresh’s Status Page - Codefresh UI outage.

anyone use pipelines to create new projects/pipelines w/codefresh cli

Erik Osterman
cloudposse/codefresh

Codefresh repos and pipelines in code. Contribute to cloudposse/codefresh development by creating an account on GitHub.

2019-07-22

Codefresh Status - Incident History
04:14:45 PM

http://status.codefresh.io/incidents/qbqypwr1wy5x\|GitHub has reported an incident Jul 22, 15:58 UTC Investigating - GitHub has reported an incident This might impact your builds.

More information here: https://www.githubstatus.com/incidents/tq7pm7mn9b1k

GitHub has reported an incident

Codefresh’s Status Page - GitHub has reported an incident.

Incident on 2019-07-22 15:46 UTC

GitHub’s Status Page - Incident on 2019-07-22 15:46 UTC.

Codefresh Status - Incident History
11:09:43 PM

http://status.codefresh.io/incidents/qbqypwr1wy5x\|GitHub has reported an incident Jul 22, 22:54 UTC Resolved - GH has reported this incident is now resolved.Jul 22, 15:58 UTC Investigating - GitHub has reported an incident This might impact your builds.

More information here: https://www.githubstatus.com/incidents/tq7pm7mn9b1k

GitHub has reported an incident

Codefresh’s Status Page - GitHub has reported an incident.

Incident on 2019-07-22 15:46 UTC

GitHub’s Status Page - Incident on 2019-07-22 15:46 UTC.

Codefresh Status - Incident History
03:24:40 AM

http://status.codefresh.io/incidents/24snchj6zx5v\|Internal Incident Jul 23, 03:20 UTC Investigating - We’re experiencing an intermittent issue with our API. We’re still trying to identify the root cause.

Internal Incident

Codefresh’s Status Page - Internal Incident.

Codefresh Status - Incident History
06:04:41 AM

http://status.codefresh.io/incidents/24snchj6zx5v\|Internal Incident Jul 23, 06:00 UTC Resolved - We have resolved the issue which caused intermediate network issues in CodefreshJul 23, 03:20 UTC Investigating - We’re experiencing an intermittent issue with our API. We’re still trying to identify the root cause.

Internal Incident

Codefresh’s Status Page - Internal Incident.

2019-07-18

Codefresh Status - Incident History
05:44:44 PM

http://status.codefresh.io/incidents/qlt15vbwtvj0\|Docker Hub has reported an incident Jul 18, 17:37 UTC Investigating - Docker Hub has reported an incident affecting their Docker Hub Registry and Docker Hub Web components. This could affect Codefresh builds.

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5d30ac0eca873b5c9dfe7768

Docker Hub has reported an incident

Codefresh’s Status Page - Docker Hub has reported an incident.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Codefresh Status - Incident History
05:59:40 PM

http://status.codefresh.io/incidents/qlt15vbwtvj0\|Docker Hub has reported an incident Jul 18, 17:47 UTC Monitoring - Docker Hub has reported the affected components as operational.

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5d30ac0eca873b5c9dfe7768Jul 18, 17:37 UTC Investigating - Docker Hub has reported an incident affecting their Docker Hub Registry and Docker Hub Web components. This could affect Codefresh builds.

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5d30ac0eca873b5c9dfe7768

Codefresh Status - Incident History
06:14:44 PM

http://status.codefresh.io/incidents/pdmzg65xr6ll\|GitHub has reported an incident Jul 18, 18:07 UTC Investigating - GitHub has reported an incident affecting Web and API requests.

More information here: https://www.githubstatus.com/ / https://www.githubstatus.com/incidents/jzcjznsv759m

GitHub has reported an incident

Codefresh’s Status Page - GitHub has reported an incident.

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

Incident on 2019-07-18 18:00 UTC

GitHub’s Status Page - Incident on 2019-07-18 18:00 UTC.

Codefresh Status - Incident History
06:14:44 PM

http://status.codefresh.io/incidents/qlt15vbwtvj0\|Docker Hub has reported an incident Jul 18, 18:11 UTC Resolved - Incident reported as fully Resolved.

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5d30ac0eca873b5c9dfe7768Jul 18, 17:47 UTC Monitoring - Docker Hub has reported the affected components as operational.

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5d30ac0eca873b5c9dfe7768Jul 18, 17:37 UTC Investigating - Docker Hub has reported an incident affecting their Docker Hub Registry…

Codefresh Status - Incident History
07:04:45 PM

http://status.codefresh.io/incidents/pdmzg65xr6ll\|GitHub has reported an incident Jul 18, 18:54 UTC Monitoring - GitHub has reported that Web and API error rates are back to normal levels.Jul 18, 18:07 UTC Investigating - GitHub has reported an incident affecting Web and API requests.

More information here: https://www.githubstatus.com/ / https://www.githubstatus.com/incidents/jzcjznsv759m

Codefresh Status - Incident History
07:34:45 PM

http://status.codefresh.io/incidents/pdmzg65xr6ll\|GitHub has reported an incident Jul 18, 19:26 UTC Resolved - GitHub has reported this incident as ResolvedJul 18, 18:54 UTC Monitoring - GitHub has reported that Web and API error rates are back to normal levels.Jul 18, 18:07 UTC Investigating - GitHub has reported an incident affecting Web and API requests.

More information here: https://www.githubstatus.com/ / https://www.githubstatus.com/incidents/jzcjznsv759m

GitHub has reported an incident

Codefresh’s Status Page - GitHub has reported an incident.

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

Incident on 2019-07-18 18:00 UTC

GitHub’s Status Page - Incident on 2019-07-18 18:00 UTC.

Codefresh Status - Incident History
10:14:42 PM

http://status.codefresh.io/incidents/v0vc702nlwzl\|Codefresh issue affecting r.cfcr.io registry Jul 18, 22:03 UTC Investigating - We are investigating an issue affecting our http://r.cfcr.io registry component

Codefresh issue affecting r.cfcr.io registry

Codefresh’s Status Page - Codefresh issue affecting http://r.cfcr.io registry.

Codefresh Status - Incident History
10:44:41 PM

http://status.codefresh.io/incidents/v0vc702nlwzl\|Codefresh issue affecting r.cfcr.io registry Jul 18, 22:29 UTC Monitoring - There was a temporary network issue. The http://r.cfcr.io registry is operational again. We’ll keep monitoring it.Jul 18, 22:03 UTC Investigating - We are investigating an issue affecting our http://r.cfcr.io registry component

Codefresh Status - Incident History
11:34:44 PM

http://status.codefresh.io/incidents/v0vc702nlwzl\|Codefresh issue affecting r.cfcr.io registry Jul 18, 23:25 UTC Resolved - This incident has been resolved.Jul 18, 22:29 UTC Monitoring - There was a temporary network issue. The http://r.cfcr.io registry is operational again. We’ll keep monitoring it.Jul 18, 22:03 UTC Investigating - We are investigating an issue affecting our http://r.cfcr.io registry component

Codefresh issue affecting r.cfcr.io registry

Codefresh’s Status Page - Codefresh issue affecting http://r.cfcr.io registry.

2019-07-17

2019-07-11

ah enterprise feature

2019-07-10

Erik Osterman

Public #office-hours starting now! Join me here: https://zoom.us/meeting/register/dd2072a53834b30a7c24e00bf0acd2b8

Have any questions? This is your chance to ask us anything.

dustinvb

@Erik Osterman Do you have details on use of the iam auth in a Codefresh pipeline? I am in a pairing call discussing the use and wondered if you have steps somewhere as an example of this in a pipeline. I am looking through your GitHub now.

Erik Osterman

I don’t have an example of using “STS” tokens

Erik Osterman

I can show you how it works, but not a codefresh pipeline

dustinvb

Mind helping me out on a call real fast with customer and talk me through how it would look in a pipeline and I’ll put the steps together post call?

Erik Osterman

sure

dustinvb

I believe it will be a freestyle step with the auth

Erik Osterman

is this for terraform by any chance?

Erik Osterman
cloudposse/terraform-aws-elasticsearch

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash. - cloudposse/terraform-aws-elasticsearch

Erik Osterman

(here’s how we’re running terratest)

Erik Osterman

we have 40+ repos updated so far

dustinvb

No this is using aws authenticator to pass in the ARN for use during the annotation creation for the ingress around the Amazon ALB.

dustinvb

Maybe I am not understanding this but the ARN is needed in the Kube config for the role to be able to use alb annotations.

Erik Osterman

oh, we haven’t tried using aws-iam-authenticator with codefresh.

dustinvb

Okay that’s cool. Then never mind the request. I think I’ll just need to write the steps.

Erik Osterman

yea, we’ve taken a different approach for deploying the ALB ingress.

Erik Osterman

we provision backing services with terraform under atlantis

Erik Osterman
cloudposse/terraform-aws-kops-aws-alb-ingress

Terraform module to provision an IAM role for aws-alb-ingress-controller running in a Kops cluster, and attach an IAM policy to the role with permissions to manage Application Load Balancers. - clo…

Erik Osterman

then we write the arn to SSM

Erik Osterman

when we deploy the ALB ingress chart, then we pull the ARN out of SSM using chamber.

dustinvb

Thanks think they’re going with a different approach will gather more information but right now per example they have alb annotations defined in the values of a Helm chart that they pass to the ingress of the service to update alb.

Erik Osterman

and we deploy the alb ingress chart with helmfile

Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Erik Osterman

that said, I’m not keen on the ALB ingress.

Erik Osterman

it’s lame. it creates a new ALB for every ingress.

Erik Osterman

provisioning ALBs is slow (and they are expensive) relatively speaking.

Erik Osterman

you’ll quickly run out of ALBs (per the limits) and need to request a limit increase

per https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html isn’t the limit the same for ELBs? In regards to pricing, ELBs and ALBs are about the same price

AWS Service Limits - Amazon Web Services

See the default limits for AWS services for an AWS account.

You could create a single ingress for all services running in a single namespace

Erik Osterman

In our case, we setup environments for every pull request. Every pull request is in a new namespace.

Erik Osterman

It’s so nice how nginx-ingress can just reuse the load balancer

Erik Osterman

yes/no, it means that you need to rearchitect how the ingresses are defined. I think it’s nicer to define the ingress rules along side the service than globally in the namespace.

yeah I agree. Decoupling the services from the ingress is just an added PITA for whoever is configuring the service

are you able to limit which pipelines/projects have access to which k8s clusters?

i.e. only prod pipelines have access to the prod cluster or only pipelines in a specific project can have access to a project specific cluster

Erik Osterman
Access control

How to restrict resources in a company environment

2019-07-03

Erik Osterman

Public Office Hours starting now! Join me here: https://zoom.us/meeting/register/dd2072a53834b30a7c24e00bf0acd2b8

Have any questions? This is your chance to ask us anything.

2019-07-02

Codefresh Status - Incident History
02:14:46 PM

http://status.codefresh.io/incidents/8j9lk4jt96yf\|Dockerhub registry issue - Impact on your Codefresh builds Jul 2, 14:09 UTC Investigating - We are currently investigating an issue with pulling images from dockerhub (502 errors). This seems like an overall issue in Dockerhub and may affect your builds in Codefresh even if your own images aren’t hosted in Dockerhub. We’ll provide more updates.

Dockerhub registry issue - Impact on your Codefresh builds

Codefresh’s Status Page - Dockerhub registry issue - Impact on your Codefresh builds.

Codefresh Status - Incident History
02:49:37 PM

http://status.codefresh.io/incidents/8j9lk4jt96yf\|Dockerhub registry issue - Impact on your Codefresh builds Jul 2, 14:42 UTC Monitoring - Issue seems to be resolved, it was related to an issue on cloudflare that impacted services like dockerhub, npm and others. We’re currently monitoring the system to make sure everything is up.Jul 2, 14:09 UTC Investigating - We are currently investigating an issue with pulling images from dockerhub (502 errors). This seems like an overall issue in Dockerhub and may affect your builds in Codefresh even if your own images aren’t hosted in Dockerhub. We’ll provide…

Codefresh Status - Incident History
07:34:39 PM

http://status.codefresh.io/incidents/8j9lk4jt96yf\|Dockerhub registry issue - Impact on your Codefresh builds Jul 2, 19:20 UTC Resolved - According to Cloudflare’s Status Page, the issue is now resolved. Thus, all indirectly impacted services (DockerHub, NPM, among others) should be functional as well.

More info here: https://www.cloudflarestatus.com/incidents/9wjyx63y2xsyJul 2, 14:42 UTC Monitoring - Issue seems to be resolved, it was related to an issue on cloudflare that impacted services like dockerhub, npm and others. We’re currently monitoring the system to make sure everything is up.Jul 2,…

Dockerhub registry issue - Impact on your Codefresh builds

Codefresh’s Status Page - Dockerhub registry issue - Impact on your Codefresh builds.

Cloudflare Status

Welcome to Cloudflare’s home for real-time and historical data on system performance.

2019-06-27

do you guys have multiple pipelines for deploying to dev, qa, prod for a specific service. or one single long running pipeline?

Erik Osterman

So what works for us is to reuse the same deploy.yml (manifest) but define the pipeline multiple times for each environment or cluster

Erik Osterman

e.g. deploy-testing would have environment variables setup for deploying perhaps to the staging cluster in the testing namespace.

Erik Osterman

then all of those pipelines can be triggered at once by calling the codefresh cli from inside of a deploy pipeline

Erik Osterman

and those can be concurrent or serial

sweet

thats exactly what i was going to do

less so

the triggering them all at once part. it was going to be more serial for us

merge into develop -> deploy dev merge into master -> deploy qa create git release off mater -> deploy prod

Erik Osterman

yea, makes sense

Erik Osterman

in fact, you can create one pipeline now that calls each one of those pipelines serially with an approval step in between

Erik Osterman

on an unrelated note, new codefresh pricing is online

Erik Osterman
Pricing for Basic, Pro, and Enterprise Plans for CI/CD | Codefresh

Start free for both public and private repositories, no credit card required. Unlimited builds, unlimited private repos, built-in Docker registry, built-in Helm repository

i saw

Erik Osterman

it was updated a couple weeks ago

they reached out a week or two ago

its nice

Erik Osterman
Erik Osterman

much more palatable pricing

oh yeah

2019-06-26

oscarsullivan_old

I’ve really bungled my pipelines. I’m deploying containers using Geodesic but I realised that master != prod, as silly as that sounds…

  deploy:
    title: Deploying Docker Image with Geodesic
    stage: "deploy"
    image: <http<i class="em em-//r.cfcr.io/he/${{CF_BRANCH_TAG_NORMALIZED}}.he.co.uk"></i>master>
    volumes:
      - ./deployment:/deployment
    commands:
      - ansible --version

oscarsullivan_old

So when branch is develop, the develop Geodesic module is used. Staging:staging… But… hold on… master branch uses which geodesic module?

oscarsullivan_old

Obviously there are some expensive changes I can make like renaming my prod geodesic module to master etc but I don’t like that

oscarsullivan_old

how can I create a sort of dictionary or something

oscarsullivan_old

Now I had tried this…

oscarsullivan_old

nvm think I may have solved that pickle

Erik Osterman

Lmk if you still want/need any pointers

but I realised that `master` != `prod

That statement may have just changed my mind on how I usually name my master aws accounts. Maybe root is the better name for it.

oscarsullivan_old

Because codefresh isnt a language I couldn’t use dictionaries of elif statements so I just have a conditional that sets STAGE variable to prod if branch is master and if branch isnt master I set STAGE to CF BRANCH NAME which works just great!

Erik Osterman

yep, that’s a good approach

Erik Osterman

basically have a step that executes some business logic (E.g. in bash) and calls cf_export

oscarsullivan_old

I had a real panic when I went to soft launch a production machine first time with all these tools

1
Erik Osterman

It’s a little bit of a mindwarp the first time!

oscarsullivan_old

For ref and @

oscarsullivan_old
    revision: master
  ask_for_permission:
    type: pending-approval
    stage: "deploy"
    title: Deploy release?
    when:
      branch:
        only: [ master ]
  check_geodesic_master:
    title: Set Geodesic module - production
    stage: "deploy"
    description: Set Geodesic module
    image: alpine:latest
    commands: 
      - cf_export STAGE=prod
    when:
      branch:
        only: [ master ]
  check_geodesic_other:
    title: Set Geodesic module
    stage: "deploy"
    description: Set Geodesic module
    image: alpine:latest
    commands: 
      - cf_export STAGE=${{CF_BRANCH_TAG_NORMALIZED}}
    when:
      branch:
        ignore: [ master ]
  deploy:
    title: Deploying Docker Image with Geodesic
    stage: "deploy"
    image: <http<i class="em em-//r.cfcr.io/he/${{STAGE}}.he.co.uk"></i>master>
    volumes:
      - ./deployment:/deployment
    commands:
      - ansible --version
      - ansible-playbook ${{PLAYBOOK}} -i inventory/${{STAGE}}.yml -u he --private-key=~/ssh/id_rsa -e "tag=${{CF_BRANCH_TAG_NORMALIZED}}"
    when:

oscarsullivan_old

Naming conventions:

Branches: develop Staging Master

Geodesic modules: develop Staging Prod

Ansible inventory names: Develop Staging Prod

Container tags: Develop Staging Master

oscarsullivan_old

So as you can see a dictionary would have been beaut

oscarsullivan_old

No caps. Just on mobile

2019-06-20

Can you use integration secrets in a pipeline? Can’t find anything in the docs. Might be completely off-base here, but I’m looking for a way to easily query ECR within a step.

Erik Osterman

sorry, so many terms and they can be overloaded

Erik Osterman

by integration secrets, do you mean what codefresh calls in their UI “Shared Configurations” (under “Account Settings”)

Erik Osterman

These can have secrets

Erik Osterman

then you can “import configuration” to a pipeline.

Erik Osterman

(though “import” is a semi-misnomer. they are not copied, but linked to)

@Erik Osterman Thanks for replying. Sorry about the lack of context. No, I meant to refer to the secrets that are part of the integrations, in the Integrations section. When you add an ECR registry, you add your AWS credentials. Was wondering if there was a way to access/use said creds in a pipeline.

Erik Osterman

Ahhhhaaa

Erik Osterman

Now I see what you mean. Yes, that would be nice.

Erik Osterman

sec

Erik Osterman

you can try something like this:

Figured it wasn’t possible haha. Should be an enhancement though. If codefresh were to add secrets management that would be beyond great.

Erik Osterman
codefresh get context github --decrypt -o yaml \| yq -y .spec.data.auth.password
1
Erik Osterman

Now this example is for the github integration

Erik Osterman

I’ve never tried it for the docker registry integrations

2019-06-19

Codefresh Status - Incident History
03:24:41 PM

http://status.codefresh.io/incidents/mspyvvq8yr3d\|Docker Hub Incident: Jun 19, 15:23 UTC Investigating - Docker Hub has reported an incident: Components: Docker Hub Registry, Docker Hub Web More info here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5d0a4fd6b684ad142660aa9d

Docker Hub Incident:

Codefresh’s Status Page - Docker Hub Incident:.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Erik Osterman

Codefresh Case Study with Cloud Posse

Erik Osterman

#office-hours starting now! https://zoom.us/j/684901853

Have a demo of using Codefresh for ETL

Erik Osterman

These are the ETL jobs: https://github.com/singer-io

Singer

Simple, Composable Open Source ETL. Singer has 30 repositories available. Follow their code on GitHub.

Codefresh Status - Incident History
06:39:44 PM

http://status.codefresh.io/incidents/mspyvvq8yr3d\|Docker Hub Incident: Jun 19, 18:30 UTC Identified - The issue has been identified and a fix is being implemented.Jun 19, 15:23 UTC Investigating - Docker Hub has reported an incident: Components: Docker Hub Registry, Docker Hub Web More info here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5d0a4fd6b684ad142660aa9d

Docker Hub Incident:

Codefresh’s Status Page - Docker Hub Incident:.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Erik Osterman

How we run ETL jobs on codefresh: https://youtu.be/dSJzEOo37MA

Erik Osterman

How to bring up environments when GitHub labels are added or removed:

Codefresh Status - Incident History
09:19:36 PM

http://status.codefresh.io/incidents/mspyvvq8yr3d\|Docker Hub Incident: Jun 19, 21:07 UTC Resolved - Docker has reported the incident has been solved.Jun 19, 18:30 UTC Identified - The issue has been identified and a fix is being implemented.Jun 19, 15:23 UTC Investigating - Docker Hub has reported an incident: Components: Docker Hub Registry, Docker Hub Web More info here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5d0a4fd6b684ad142660aa9d

Docker Hub Incident:

Codefresh’s Status Page - Docker Hub Incident:.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

2019-06-13

jafow

hello I have question about running pipelines locally via codefresh CLI

jafow

first of all thanks for the codefresh CLI. it’s nice and easy to use

jafow

I’d like to confirm whether I’m doing something wrong here that’s causing my local builds to count toward my builds quota

jafow

or, OTOH if that’s intended & expected.

jafow


Note that the engine has transparent network access to all the other settings in your Codefresh account and therefore will work exactly the same way as if it was run on Codefresh infrastructure (e.g. use the connected Docker registries you have setup in the UI)

jafow

but if I’m running a pipeline with --local and --local-volume opts, then I’d expect I’m like, fully local

2019-06-12

Erik Osterman

Public #office-hours starting now! Join us on Zoom if you have any questions. https://zoom.us/j/684901853

is there a way to exec into the codefresh container in your pipeline?

im trying to get the python kube client to run on the container and codfresh might be setting the kube-context up a little different. i wanted to avoid passing in the kube api token if possible

Erik Osterman

I have used tmate for this

2019-06-10

Hey ya’ll. Running into a weird issue. I’m trying to run the following step in a freestyle project:

steps:
  install_deps:
    title: Install deps
    image: 'python:3'
    working_directory: ${{main_clone}}
    commands:
      - python3 -m venv venv
      - . venv/bin/activate
      - pip install -r requirements.txt
      - ./scripts/test.sh

And it falls on its face. It seems to be getting confused about the order in which the commands are supposed to be run? This runs perfectly fine on CircleCI.

How does CodeFresh run commands, does it use the shell of the image I specify, or something else?

Erik Osterman

Based on what you’ve shared, there’s no main_clone step defined. I know that it used to be implicit, but I think that codefresh has been moving towards making that explicit.

Erik Osterman

Also, it helps if you share the literal error, otherwise we don’t know what’s broken.

Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman

Here’s an example of the main_clone step

Sure thing, this is the error:

Traceback (most recent call last):                                                                                                       
  File "<string>", line 1, in <module>                                                                                                   
AttributeError: module 'nltk' has no attribute '__version__'                                                                             
Traceback (most recent call last):                                                                                                       
  File "<string>", line 1, in <module>                                                                                                   
AttributeError: module 'nltk' has no attribute 'download'                                                                                
Reading environment variable exporting file contents.                                                                                    
[SYSTEM]                                                                                                                                 
 Message             Failed to run freestyle step: Install deps                                                                          
 Caused by           Container for step title: Install deps, step type: freestyle, operation: Freestyle step failed                      
                     with exit code: 1                                                                                                   
 Documentation Link  <https://codefresh.io/docs/docs/codefresh-yaml/steps/freestyle/>                                                      
 Action Items        Fix command : ./scripts/test.sh                                                                            
 Exit code           1                                                                                                                   
 Name                NonZeroExitCodeError   
Erik Osterman

Can you share your corresponding circle config?

Managed to get around it by moving the steps inside a Dockerfile instead of the codefresh.yml.

Still new to this platform so will post any other issues I encounter on here.

2019-06-02

Codefresh Status - Incident History
07:09:37 PM

http://status.codefresh.io/incidents/p0503g8v7xv0\|Codefresh availability Jun 2, 19:05 UTC Investigating - We are currently having an availability issue, from initial investigation this is happening due to Google Cloud Platform cross region issue. We are working to resolve the issue

Codefresh availability

Codefresh’s Status Page - Codefresh availability.

Codefresh Status - Incident History
07:39:51 PM

http://status.codefresh.io/incidents/p0503g8v7xv0\|Codefresh availability Jun 2, 19:23 UTC Monitoring - Issue have been resolved and system is fully operational, we’re still monitoring the status Jun 2, 19:05 UTC Investigating - We are currently having an availability issue, from initial investigation this is happening due to Google Cloud Platform cross region issue. We are working to resolve the issue

Codefresh Status - Incident History
08:44:40 PM

http://status.codefresh.io/incidents/p0503g8v7xv0\|Codefresh availability Jun 2, 20:32 UTC Update - We are still having issues due to a cross region incident over at GCP: https://status.cloud.google.com/ Jun 2, 19:23 UTC Monitoring - Issue have been resolved and system is fully operational, we’re still monitoring the status Jun 2, 19:05 UTC Investigating - We are currently having an availability issue, from initial investigation this is happening due to Google Cloud Platform cross region issue. We are working to resolve the issue

Codefresh availability

Codefresh’s Status Page - Codefresh availability.

Codefresh Status - Incident History
12:59:53 AM

http://status.codefresh.io/incidents/p0503g8v7xv0\|Codefresh availability Jun 3, 00:45 UTC Resolved - GCP has updated its Status Page and has confirmed that the incident is resolved. We validated that Codefresh is working as expected now. The incident is resolved.

More information about GCP’s incident here: https://status.cloud.google.com/incident/compute/19003 Jun 2, 20:32 UTC Update - We are still having issues due to a cross region incident over at GCP: https://status.cloud.google.com/ Jun 2, 19:23 UTC Monitoring - Issue have been resolved and system is…

Codefresh availability

Codefresh’s Status Page - Codefresh availability.

2019-05-31

is there a way to delete/update helm repos in the codefresh ui?

2019-05-30

what is happening in “Initializing Process”

just started using codefresh and im getting “TypeError: Cannot read property ‘branch’ of undefined”

Mat Geist

adding a tag key to the build step fixed that error for me

Erik Osterman

can you share your yaml as a snippet?

version: '1.0'
stages:
  - prepare
  - build
steps:
  main_clone:
    title: Cloning repo
    type: git-clone
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: '${{CF_REVISION}}'
    stage: prepare
    git: github
  build_docker_image:
    title: Building Docker Image
    type: build
    stage: build
    image_name: bug-tracker
    working_directory: ./
    dockerfile: Dockerfile

haha

adding tag to build_docker_image

Erik Osterman

aha! yes

Erik Osterman
we require the tag here else the pipeline will fail to initialize by dustinvanbuskirk · Pull Request #76 · cloudposse/testing.cloudposse.co

@osterman Ran into an issue where pipeline would fail to initialize due to the tag argument not being present in the build step. Can you please review? I used the SHORT SHA for the tag.

do we get docker layer caching automatically w/codefresh

Erik Osterman

yes

the ui is so nice

Erik Osterman
Erik Osterman

example of layer caching working

i see a borken image

Erik Osterman

in slack?

Erik Osterman

wierd

Mat Geist

yeah slack does that sometimes.. refreshing or clicking will show it

aknysh

if you click on it, it shows

i see it now

do you guys use the codefresh image registry too?

Erik Osterman

yep

Erik Osterman

we use that and ECR

Erik Osterman

for simpler setups without the requirements of ECR use the codefresh registry

Erik Osterman

ECR is trickier plus max 1000 tags per repo

Erik Osterman

(which means you cannot use same repo for prod and ci)

i see

bout to test out this layer caching

circle charges for it and its super fickle (barely ever works). but i believe codefresh has a persistent volume per account

Erik Osterman

PV per pipeline

Erik Osterman

i believe

Erik Osterman

@dustinvb

amazing

Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman

here are our pipelines to get you started

how come your build_image doesnt require tag

cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman

i think this is a recent thing

Erik Osterman

none of our pipelines have it, but they started breaking

ah

dustinvb

Yes, very recent. Already contacted my team about this to update in our docs.

dustinvb

And Yes PV per pipeline. When successful volume stored for next pipeline execution./

are we supposed to setup the webhook in gh ourselves or does codefresh do that automatically for us?

i see that, but no webhook was created in my repo

i just created it myself (not sure if the above is a bug or im understanding it incorrectly)

dustinvb

This should be happening automatically. Can you send me the time you setup the GIT Trigger, Trigger Name and the Pipeline you created this trigger on? I am also assuming you’re using GH SaaS?

what do you mean GH Saas?

yeah

So i had a git push that didnt trigger at this time Thu May 30 13<i class="em em-07"></i>03 2019 -0700

thats PST btw

dustinvb

You said the webhook was not under the repository settings?

dustinvb

You had to add manually?

dming you privately

is there a way to resync your github repositories on codefresh? I created a new repo for shared codefresh configs (build, deploy, etc) and I’m trying to select that repo in the dropdown but it doesn’t show up

dustinvb

This should be syncing automatically but we could be seeing some cache at work here. Can you gather your OS and Browser and send that over to our support team using the Intercom Chat Bubble widget?

2019-05-29

Erik Osterman

public #office-hours starting now! join us here: https://zoom.us/j/684901853

2019-05-28

2019-05-23

2019-05-22

Codefresh Status - Incident History
05:34:54 PM

http://status.codefresh.io/incidents/pwzcgr2qszcz\|GitHub Incident: delay in notification deliveries May 22, 17:18 UTC Identified - GitHub is experiencing an incident related to their Notifications. This impacts email and webhook deliveries.

More information here: https://www.githubstatus.com/incidents/nf4gcjtcsqdb

GitHub Incident: delay in notification deliveries

Codefresh’s Status Page - GitHub Incident: delay in notification deliveries.

Incident on 2019-05-22 16:56 UTC

GitHub’s Status Page - Incident on 2019-05-22 16:56 UTC.

Erik Osterman

Public/Free Office Hours with Cloud Posse starting now!!

https://zoom.us/j/684901853

Codefresh Status - Incident History
12:00:05 AM

http://status.codefresh.io/incidents/pwzcgr2qszcz\|GitHub Incident: delay in notification deliveries May 22, 23:46 UTC Resolved - This incident has been resolved. May 22, 17:18 UTC Identified - GitHub is experiencing an incident related to their Notifications. This impacts email and webhook deliveries.

More information here: https://www.githubstatus.com/incidents/nf4gcjtcsqdb

GitHub Incident: delay in notification deliveries

Codefresh’s Status Page - GitHub Incident: delay in notification deliveries.

Incident on 2019-05-22 16:56 UTC

GitHub’s Status Page - Incident on 2019-05-22 16:56 UTC.

2019-05-21

2019-05-20

Alejandro Rivera

Hi, I’m trying to get a list of files that changed from current branch off a different branch. e.g. git diff $BRANCH –name-only on a pipeline. Seems that that other branches don’t get fetched so I ran a git fetch $BRANCH, but git asks for credentials. Got ssh key in place as suggested on the documentation but it’s not getting picked up by git. Any ideas on what I’m missing or a workaround for this?

Olivier Vanden Bos

client_ui: title: Building Game UI image: ${{client_image}} stage: build_client working_directory: ${{main_clone}} commands: - mkdir -p ~/.ssh - ssh-keyscan -H http://github.com >> ~/.ssh/known_hosts - echo “${SSH_KEY}” | tr '”${SPLIT_CHAR}”' ‘\n’ > ~/.ssh/id_rsa - chmod 600 ~/.ssh/id_rsa - ‘/bin/bash docker-build-ui.sh’

Olivier Vanden Bos

@Alejandro Rivera ^^

Alejandro Rivera

thanks Olivier!!!

Olivier Vanden Bos

Cache wiped out on a yml change. is it expected ?

Olivier Vanden Bos

Looking a 2 successive build - 27 min with cache and next one end up to be 2 hours

Olivier Vanden Bos

@dustinvb is it expected ?

Olivier Vanden Bos

literally the only changes on the depot were the yml file of codefresh..

@Olivier Vanden Bos will check and get back to you as soon as possible. This is the same issue you described in the email you sent me right?

Olivier Vanden Bos

They all seem related. It is almost if the codefrsh/volume was shared accross pipelines and concurrent build could collide somehow

Olivier Vanden Bos

I managed to repro locally though - with simulating the volume persistent accross pipelines. I am still digging.

Olivier Vanden Bos

Found the issue, was on our side

2019-05-17

Mat Geist
05:27:08 PM

trying to use the codefresh build-harness to setup pipelines ( https://github.com/cloudposse/build-harness/tree/master/modules/codefresh ), running into an issue Error: Default git context is not specified. according to the spec, context is optional ( https://codefresh-io.github.io/cli/pipelines/spec/ ), and i cant find a reference to what the “auto select branch” option should be. looking at the yaml from an existing pipeline shows only the same 3 options im trying to provide in the spec template (location, repo, path)

Erik Osterman
cloudposse/codefresh

Codefresh repos and pipelines in code. Contribute to cloudposse/codefresh development by creating an account on GitHub.

Olivier Vanden Bos

Running out of resources @dustinvb

Olivier Vanden Bos

here are my current real blockers

Olivier Vanden Bos

Error response from daemon: Error processing tar file(exit status 1): write /GameClient/Assets/Scenes/CoralReefRewardScene/R&D/Scenes/RS-LayerTest.unity: no spac e left on device

Olivier Vanden Bos

@ FYI

dustinvb

Okay this is why I put you on enterprise to begin with. I knew you were going to be building in a way that you’d need extra space. I believe these pipelines are around 40GB. Do you have an estimate on how big these Docker images might get?

dustinvb

@ Could you assist here and just check this build out? We’re building using unity and the images are going to be very large. I am not sure but I think we’re pressing the storage limit defined in the runtime.

Olivier Vanden Bos

40GB seems a lot

Olivier Vanden Bos

Lemme check what’s our real size on Jenkins

Olivier Vanden Bos

Docker source image with UNity is 7GB

Olivier Vanden Bos

my full local repo that build 3 targets is around 8GB

Olivier Vanden Bos

now are you including the git files themselves ? .git ?

Olivier Vanden Bos

my .git looks like 20Gb

dustinvb

Hmm well that should have fit. Let me check with my support team on this.

Olivier Vanden Bos

General questions about best strategy to generate artifacts ala Jenkins

Olivier Vanden Bos

we used to zip and upload to S3. I did add the S3 integration.

Olivier Vanden Bos

Do you have any example on how to zip and upload ?

Olivier Vanden Bos

We deploy as well as a static website - still on S3 - but not zipped this time. any useful plugin ?

Olivier Vanden Bos

@dustinvb I am planning to use directly the aws s3-cli ? Is there anything simpler ? I do not see in the doc how to use the s3 artifact integration ? should I deal with the credentials in the script itself ?

dustinvb
codefresh-contrib/s3copy

A plugin to work with s3 buckets. Contribute to codefresh-contrib/s3copy development by creating an account on GitHub.

dustinvb

I am looking for a step to aid with archive now.

dustinvb

For that s3 copy you just need to provide a service account in S3 with bucket access using these variables in the pipeline variables.

AWS_ACCESS_KEY_ID=<id> AWS_SECRET_ACCESS_KEY=<key> AWS_DEFAULT_REGION=<region>

If you plan on using this user over and over you may want to put these variables into a shared config you can import for other builds.

https://codefresh.io/docs/docs/configure-ci-cd-pipeline/shared-configuration/#creating-shared-configuration

Shared configuration

How to keep your pipelines DRY

dustinvb

Sorry haven’t forgot still looking. Steps that often require this have this build in. Trying to find an a-la-carte example.

dustinvb
  ArchiveFiles:
    image: alpine:latest
    commands:
      - apk add --update zip
      - tar -czvf ./archive.tgz ./folder/or/file.txt

I do not see that we have a step for this.

Please try the step below using an alpine image. I am checking with team about OOTB step.

Olivier Vanden Bos

perfect thanks !

Olivier Vanden Bos

@jafow FYI ^^

Mat Geist

ive traced my issue down to codefresh create -f failing no matter what i give it, giving me Error: "{\"status\"<i class="em em-500,\"code\""></i>\"1001\",\"name\"<i class="em em-\"INTERNAL_SERVER_ERROR\",\"message\""></i>\"Internal server error\",\"context\":{}}" not really sure where to go from here

dustinvb

You go right here… Sorry about the delay. I am traveling and often offline on weekends.

Please send me your YAML file for review. It is possible you’re missing a value or somethign.

I have some recent examples I updated here for comparison.

https://github.com/codefresh-contrib/example-voting-app/tree/master/.codefresh/specs

codefresh-contrib/example-voting-app

Docker’s Example Voting App. Contribute to codefresh-contrib/example-voting-app development by creating an account on GitHub.

Mat Geist

thanks for the reply! https://gist.github.com/mgeist/163026ccfb89626808ec24ca647dc589 using https://codefresh-io.github.io/cli/pipelines/spec/ as a reference, specifically “Pipeline with a remote spec template brought from a git repository”

dustinvb

Can you please try adding a name to your git context and let me know if that corrects the behavior? https://github.com/codefresh-contrib/example-voting-app/blob/master/.codefresh/specs/cd.yaml#L13

Name can be anything to identify the trigger.

I will make sure the team updates our spec examples if this resolves the issue.

codefresh-contrib/example-voting-app

Docker’s Example Voting App. Contribute to codefresh-contrib/example-voting-app development by creating an account on GitHub.

Mat Geist

same thing, code 1001, 500 internal server error

dustinvb

Can you update your example yaml with the latest?

dustinvb

Do you have a GIT context configured? Can you add these two fields?

https://github.com/codefresh-contrib/example-voting-app/blob/master/.codefresh/specs/dvts.yaml#L20-L21

context being the name of your context. If not, at a minimum please try to add the default revision.

codefresh-contrib/example-voting-app

Docker’s Example Voting App. Contribute to codefresh-contrib/example-voting-app development by creating an account on GitHub.

Mat Geist
# codefresh get contexts
NAME            TYPE
CF_HELM_DEFAULT helm-repository
github          git.github

and updated script: https://gist.github.com/mgeist/163026ccfb89626808ec24ca647dc589 no dice

Mat Geist

finally got to the bottom of this, turns out pipeline names will break if they contain /. thanks for the assistance and FYI @dustinvb

dustinvb

Didn’t expect that to be the case. I will discuss with the team exposing this as an error on create.

jafow

It seems like a step in my codefresh.yml ends and proceeds to the next step before the final command exits

jafow

here’s what I’m trying to do:

jafow

pull down terraform and unzip the archive

jafow

here’s the steps in my yaml file:

steps:
  clone_repo:
    stage: prepare
    type: git-clone
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: '${{CF_REVISION}}'
    git: github
  wtf:
    stage: prepare
    image: alpine:latest
    commands:
      - 'apk add curl wget unzip file'
      - 'ls -l'
      - 'curl <https://releases.hashicorp.com/terraform/${{TERRAFORM_VERSION}}/terraform_${{TERRAFORM_VERSION}}_linux_amd64.zip> -o /tmp/terraform.zip'
  next_part:
    stage: prepare
    title: 'hello world'
    image: alpine:latest
    commands:
      - 'ls -lh'
      - 'unzip /tmp/terraform.zip'

jafow

here’s some things I’ve tried:

  • fixing some nice typos
  • changing the step where curl runs
  • running curl with -s ?
jafow

a common error that I get that makes me think the step is ending before the curl exits is this:

End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.
jafow

thrown by unzip

jafow

hey at least it’s friday tho ¯_(ツ)_/¯

jafow

my work around for this was just to put these into a dockerfile and build it. that works really well

dustinvb

I am glad you found a work around. Please note that while we do participate here our Intercom “chat” bubble is staffed 7am-6pm PDT M-F whenever these issues come up please check with the support staff there. We also have hours outside of the ones mentioned.

jafow

thanks @dustinvb I’ll use the chat bubble next time

2019-05-16

Did you know https://codecov.io/ works with codefresh: https://docs.codecov.io/docs/supported-ci-providers?

I did not realize, since we did code coverage in composition and env variables did not get passed, but after passing these into the composition candidate made it “just work”:

- CODECOV_SLUG=${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}
- CF_BUILD_URL=${{CF_BUILD_URL}}
- CF_BUILD_ID=${{CF_BUILD_ID}}
- CF_BRANCH=${{CF_BRANCH}}
- CF_REVISION=${{CF_REVISION}}

(on top of CODECOV_TOKEN)

Code Coverage Done Right | Codecov

Develop healthier code using Codecov’s leading, dedicated code coverage solution. Try it free

^^ we had codecov working on codefresh already, but that just adds extra metadata on the codecov side - like link to build etc.

1
Erik Osterman

Thanks for sharing

Mat Geist

playing with codefresh, ran into an interesting issue where not including the tag key for the build step, despite it being marked as having a default value in the docs, was throwing this error TypeError: Cannot read property 'branch' of undefined during the initialization process. couldnt find anything about this anywhere so hopefully this helps someone

2019-05-15

dustinvb

@Olivier Vanden Bos please join me here. We run this community Slack in partnership with Cloud Posse.

Olivier Vanden Bos
06:08:48 PM

@Olivier Vanden Bos has joined the channel

Erik Osterman

@Olivier Vanden Bos welcome!

1
Erik Osterman

doing public office hours now: https://zoom.us/j/684901853

James D. Bohrman

I wish I’d seen this

Erik Osterman

Every Wednesday at 11:30 am pst

Erik Osterman

Join is next week!

James D. Bohrman

In my calender weekly!

Erik Osterman

if anyone has questions on codefresh…

Olivier Vanden Bos

Can I invite some of my developers to this slack channel ?

Erik Osterman

please do - it’s open for everyone

Erik Osterman
Olivier Vanden Bos

Question About pushing images to dockerhub

Olivier Vanden Bos

I usually do that locally to push : docker tag game-client imbellus/game-client-build:1 docker push imbellus/game-client-build:1

Olivier Vanden Bos

Error in code fresh is : Running push step: Pushing to Docker Registry Pulling image ovandenbos-imb/game-client18865fc9fba8ac10129b14677adfbac370858d320c646fa7697ad46791b9f90a Digest: sha256:18865fc9fba8ac10129b14677adfbac370858d320c646fa7697ad46791b9f90a Status: Image is up to date for <http18865fc9fba8ac10129b14677adfbac370858d320c646fa7697ad46791b9f90a> Finished pulling <http18865fc9fba8ac10129b14677adfbac370858d320c646fa7697ad46791b9f90a> from internal registry Tagging image as: <http1> The push refers to repository [http://docker.io/library/game-client] denied: requested access to the resource is denied [SYSTEM] Error: Failed to push image <http1;> caused by NonZeroExitCodeError: Container for operation: push failed with exit code: 12

Olivier Vanden Bos

for the yaml: client_image: title: Building Docker Image type: build stage: build image_name: game-client working_directory: ./ tag: ‘${{CF_BRANCH_TAG_NORMALIZED}}’ dockerfile: Dockerfile

push_to_registry: title: Pushing to Docker Registry type: push stage: postbuild candidate: ‘${{client_image}}’ tag: ‘1’ registry: dockerhub

Olivier Vanden Bos

any idea of what I am doing wrong ?

Olivier Vanden Bos

‘dockerhub ‘ registry is setup and ‘Test’ successful

Olivier Vanden Bos

the obvious step that I am missing is the ‘rename’ part in tagging : docker tag game-client imbellus/game-client-build

Erik Osterman

The image_name needs to be relative to DockerHub

Erik Osterman

e.g. for us, it would be cloudposse/image-client since cloudposse is our org

Erik Osterman

Here’s how we do it

Erik Osterman
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

Erik Osterman

this works because our CF_REPO_OWNER (cloudposse) is the same as our dockerhub

Olivier Vanden Bos

Cool. lemme try.

Olivier Vanden Bos

It worked thank

Olivier Vanden Bos

Freestyle npm step stuck

Olivier Vanden Bos

my log looks like that :

Olivier Vanden Bos

the page is actually freezing and I cannot copy to the clipboard, but that’s a different issue

Olivier Vanden Bos

this works locally for me

Olivier Vanden Bos

maybe because we are using a private repo on git ?

Olivier Vanden Bos

“dependencies”: { “chart.js”: “^2.8.0”, “chartjs-plugin-datalabels”: “^0.5.0”, “detect-browser”: “^4.4.0”, “gameui-design-system”: “<git+ssh://[email protected]/ImbellusBetta/gameui-design-system.git#v3.7.3>”, “lodash.debounce”: “^4.0.8”, “math3d”: “^0.2.2”, “object-path”: “^0.11.4”,

@dustinvb can you please give this one a look when you get a chance?

dustinvb

@Olivier Vanden Bos Can you send me the Build’s URL from the logs page you pulled the excerpt. I’d like to examine the commands in more detail there is a possibility that we need pass credentials to the step to allow access to private repository.

dustinvb

I see this is using the ssh method to clone the private repository. I believe this is a Docker build process. I am checking now to see how we can get the context you’ve created and convert that to be able to specify a SSH key.

dustinvb

I think we will need to incorporate something like the following into the Dockerfile itself using a combination of build arg to import the key to be used. Do you have a service account user that you can create an ssh key for that has access to the npm repository?

Olivier Vanden Bos

Sorry @dustinvb this one I fixed. Found the doc and that fixed it. In deed, the ‘Freeze’ was really annoying and lack of logs (but could get better logs with the download panel)

dustinvb

If you see this freeze again can you please send me the build’s url and the step and spot where the freeze occurred. I’d like to have support look. If you have the original I wouldn’t mind looking at that.

Olivier Vanden Bos

I can

Olivier Vanden Bos

this one is the latest one

Olivier Vanden Bos

froze for 1h 10m

Olivier Vanden Bos

I might have cancelled it

dustinvb

Can I get your OS + version and Browser + version?

Olivier Vanden Bos

Windows 10 Chrome Version 74.0.3729.157

2019-05-12

Codefresh Status - Incident History
04:14:53 PM

http://status.codefresh.io/incidents/9qzp713fcw4r\|Codefresh UI doesn't load May 12, 16:02 UTC Investigating - We are currently investigating an issue where Codefresh UI doesn’t load. Build automatic triggers are not affected.

Codefresh UI doesn't load

Codefresh’s Status Page - Codefresh UI doesn’t load.

Codefresh Status - Incident History
04:34:47 PM

http://status.codefresh.io/incidents/9qzp713fcw4r\|Codefresh UI doesn't load May 12, 16:20 UTC Resolved - The incident has been resolved May 12, 16:02 UTC Investigating - We are currently investigating an issue where Codefresh UI doesn’t load. Build automatic triggers are not affected.

2019-05-10

oscarsullivan_old

How do I go about launching two containers (a la docker-compose) in a pipeline? I run GOSS in my pipeline but it depends on access to a database which I’d like to be inside the pipeline instead of a real one

Nikola Velkovski

You will need to run docker compose in daemon mode.

Nikola Velkovski

so that execution can continue and you can do testing.

Nikola Velkovski

the database should be inside the docker-compose

Nikola Velkovski

and you will need a waiting script.

Nikola Velkovski

what I do is stick this in the docker-compose for the app

    command:
      - /bin/sh
      - -c
      - \|
        while ! nc -z mysql-test 3306; do
          echo "Database is not ready yet."
          sleep 1
        done
Nikola Velkovski

mysql-test is the name of the service in docker-compose

Nikola Velkovski

does this help ?

Nikola Velkovski

also -exit-code-from is helpful https://docs.docker.com/compose/reference/up/

docker-compose up

Usage: up [options] [–scale SERVICE=NUM…] [SERVICE…] Options: -d, –detach Detached mode: Run containers in the background, print new container names. Incompatible with –abort-on-container-exit. –no-color Produce monochrome output. –quiet-pull Pull without…

2019-05-07

Codefresh Status - Incident History
10:55:16 AM

http://status.codefresh.io/incidents/7w08fg1r2xdc\|Build initialization issue May 7, 10:50 UTC Resolved - We had experienced a 30 minutes issue which affected initializing new builds. 2:51 am - 3:19 am PDT

Build initialization issue

Codefresh’s Status Page - Build initialization issue.

2019-05-06

Codefresh Status - Incident History
04:05:13 PM

http://status.codefresh.io/incidents/0sylk4ggy89g\|codefresh.io website is down May 6, 16:00 UTC Monitoring - Consumer website is back up. We are monitoring. May 6, 15:59 UTC Update - We are continuing to work on a fix for this issue. May 6, 15:56 UTC Identified - Our consumer facing website is down but you can still log into your services at https://g.codefresh.io

Our hosting provider for our consumer facing site is looking into server issues.

codefresh.io website is down

Codefresh’s Status Page - http://codefresh.io website is down.

Codefresh Status - Incident History
04:55:21 PM

http://status.codefresh.io/incidents/0sylk4ggy89g\|codefresh.io website is down May 6, 16:36 UTC Resolved - This incident has been resolved. May 6, 16:00 UTC Monitoring - Consumer website is back up. We are monitoring. May 6, 15:59 UTC Update - We are continuing to work on a fix for this issue. May 6, 15:56 UTC Identified - Our consumer facing website is down but you can still log into your services at https://g.codefresh.io

Our hosting provider for our consumer facing site is looking into server issues.

2019-05-04

Codefresh Status - Incident History
03:20:12 PM

http://status.codefresh.io/incidents/fkplnsqzcq70\|Docker Hub Planned Maintenance THIS IS A SCHEDULED EVENT May 4, 16:00 UTC  -  May 5, 02:15 UTC May 4, 15:04 UTC Scheduled - Docker Hub will be performing scheduled maintenance this Saturday May 4, 2019 from approximately 9:00AM to 7:15PM, US Pacific Daylight Time (UTC-7). During this window, Docker Hub will be operating in a read-only mode. Registry logins and image pulls will continue to work for a majority of this time frame. Pushes however will generally be unavailable. Maintenance activities, approximate timelines…

Docker Hub Planned Maintenance

Codefresh’s Status Page - Docker Hub Planned Maintenance.

Codefresh Status - Incident History
04:09:43 PM

http://status.codefresh.io/incidents/fkplnsqzcq70\|Docker Hub Planned Maintenance May 4, 16:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May 4, 15:04 UTC Scheduled - Docker Hub will be performing scheduled maintenance this Saturday May 4, 2019 from approximately 9:00AM to 7:15PM, US Pacific Daylight Time (UTC-7). During this window, Docker Hub will be operating in a read-only mode. Registry logins and image pulls will continue to work for a majority of this time frame. Pushes however will generally be…

Codefresh Status - Incident History
02:29:58 AM

http://status.codefresh.io/incidents/fkplnsqzcq70\|Docker Hub Planned Maintenance May 5, 02:16 UTC Completed - The scheduled maintenance has been completed. May 4, 16:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May 4, 15:04 UTC Scheduled - Docker Hub will be performing scheduled maintenance this Saturday May 4, 2019 from approximately 9:00AM to 7:15PM, US Pacific Daylight Time (UTC-7). During this window, Docker Hub will be operating in a read-only mode. Registry logins and image pulls will continue to work…

Docker Hub Planned Maintenance

Codefresh’s Status Page - Docker Hub Planned Maintenance.

2019-05-02

When are you guys going to roll out the new pricing schema @?

dustinvb

Mark, I am opening this on your Codefresh Slack channel.

Erik Osterman

hahah! love this image

I made this a slack emoji a while back, some of my developers REALLY hate it, others love it

Michael Kolb

How do you turn off the new projects feature…. It is breaking my ability to work with and launch pipelines

dustinvb

@Michael Kolb Can you send this information to [email protected] along with some additional details around what items are no longer working after the Project Features implementation? Thank you.

2019-05-01

whats the new unlimited builds pricing model for codefresh going to be like?

Erik Osterman

@ can help

02:37:34 AM

@ has joined the channel

Erik Osterman

for additional context see #gitops

@btai we’re looking to have the model which will be per concurrent steps, so no matter how many build you’re running overall it’s all about the concurrency of steps. Are you looking for specific pricing? Or just general idea of how it will work?

Curious what the specific pricing will be

2019-04-30

Michael Kolb

Does Codefresh have API that accepts commands to start an existing pipeline? I have an integration test that will take hours and would like the option to start another pipeline once the integration tests complete.

dustinvb
codefresh-contrib/example-voting-app

Docker’s Example Voting App. Contribute to codefresh-contrib/example-voting-app development by creating an account on GitHub.

2019-04-25

Codefresh Status - Incident History
05:40:00 PM

http://status.codefresh.io/incidents/435r6pktn4bz\|Docker incident: Docker Hub elevated errors Apr 25, 17:33 UTC Investigating - Docker has reported an incident affecting Docker Hub Web and Docker Hub Automated Builds components. It shouldn’t affect normal pulling and pushing operations. We will keep monitoring this.

More information here: https://status.docker.com/

Docker incident: Docker Hub elevated errors

Codefresh’s Status Page - Docker incident: Docker Hub elevated errors.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Codefresh Status - Incident History
06:29:59 PM

http://status.codefresh.io/incidents/435r6pktn4bz\|Docker incident: Docker Hub elevated errors Apr 25, 18:10 UTC Resolved - Docker has reported this incident as Resolved. “[Resolved] The issue has been resolved. The DB is backup again.”

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5cc1ed78790d0e1ca1c8fcd4 Apr 25, 17:33 UTC Investigating - Docker has reported an incident affecting Docker Hub Web and Docker Hub Automated Builds components. It shouldn’t affect normal pulling and pushing operations. We will keep monitoring this.

More…

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

2019-04-24

oscarsullivan_old

Hi.. anyone got experience using submodules and codefresh?

oscarsullivan_old
version: '1.0'
stages:
 - build
 - push
 - prepare
 - deploy

steps:
  get_git_token:
    title: Reading Github token
    image: codefresh/cli
    commands:
      - cf_export GITHUB_TOKEN_EXPORT=$(codefresh get context github --decrypt -o yaml \| yq -y .spec.data.auth.password)
  updateSubmodules:
    image: codefresh/cfstep-gitsubmodules
    environment:
      - GITHUB_TOKEN=${{GITHUB_TOKEN_EXPORT}}
      - CF_SUBMODULE_SYNC=true
      - CF_SUBMODULE_UPDATE_RECURSIVE=false
  debug:
    title: Debug Submodules
    image: codefresh/cli
    commands:
      - codefresh get contexts
      - ls -lah /codefresh/volume/my_app/models
      - ls -lah /codefresh/volume/my_app/lib/library
  BuildingDockerImage:
    title: Building Docker Image
    stage: "build"
    type: build
    image_name: ${{IMAGE_NAME}}
    working_directory: ./
    tag: '${{CF_BRANCH_TAG_NORMALIZED}}'
    dockerfile: pm2.Dockerfile
    build_arguments:
      - APP_ENV=.env-${{CF_PULL_REQUEST_TARGET}}
      - PORT=${{PORT}}
oscarsullivan_old

So it gets my git token, updates the submodules in /codefresh/volume/my_app

oscarsullivan_old

then does an ls pf those dirs to show me it has cloned (it has)

oscarsullivan_old

and then in the docker build step I have: COPY models/* models/

oscarsullivan_old

but the models/ dir is empty in the container..

oscarsullivan_old

So I can only assume that /codefresh/volume/my_app is not the path that my_app is being cloned into during init and therefore not the context of Docker when building

dustinvb

I would recommend trying to set the working_directory: ${{main_clone}} for the step to ensure you’re in the clone repository directory.

2019-04-23

Michael Kolb

the docker hub release

Michael Kolb

i found the command that i need kubectl config use-context <<cluster name>>

Michael Kolb

the container had all the tools installed. Unfortunately, there was not documentation on the docker hub site.

2019-04-22

Michael Kolb

Has anyone been using cf_export stating the name of an existing environment variable? I have tried to use it, but the variable output is getting altered. FreeStye Bash Code

 - FEATURE=$(echo ${{CF_BRANCH}} \| cut -c 9- )
      - echo $FEATURE
      - cf_export $FEATURE

Output

TI-595                                                                                                                                                  
Exporting TI-595=-595

I am seeing the cf_export command parse the static environment variable value vs. storing the defined value with the assignment cf_export $FEATURE.

aknysh

@Michael Kolb here’s a pipeline step example to setup ENV vars

aknysh
  env:
    title: Setup Environment
    stage: Init
    fail_fast: true
    image: ${{build_image}}
    working_directory: &cwd ${{main_clone}}/${{PROJECT}}
    commands:
      - cf_export BUILD_HARNESS_VERSION=0.18.0

      # Github Commenter
      - cf_export GITHUB_OWNER=${{CF_REPO_OWNER}}
      - cf_export GITHUB_REPO=${{CF_REPO_NAME}}
      - cf_export GITHUB_COMMENT_TYPE=pr
      - cf_export GITHUB_PR_ISSUE_NUMBER=${{CF_PULL_REQUEST_NUMBER}}
      - cf_export GITHUB_COMMENT_FORMAT_FILE=${{CF_VOLUME_PATH}}/${{CF_REPO_NAME}}/codefresh/terraform/comment.txt.gotmpl
aknysh

it exports Codefresh env vars and also those defined in Codefresh UI (Global config section and ENV vars section in each pipeline)

aknysh

you mean the example does not use the ${{...}} syntax?

Michael Kolb

So, in the documentation it says that you can state an existing variable state the name of an existing environment variable (like EXISTING_VAR) https://codefresh.io/docs/docs/codefresh-yaml/variables/#using-cf_export-command

Variables

Codefresh is a Docker-native CI/CD platform. Instantly build , test and deploy Docker images.

Michael Kolb

However, this hasn’t stored that variable assignment

Michael Kolb

and the displayed use case does not show assignment in the code

Michael Kolb
ersion: '1.0'
steps:
  freestyle-step-1:
    description: Freestyle step..
    title: Free styling
    image: alpine:latest
    commands:
      - cf_export VAR1=VALUE1 VAR2=VALUE2 EXISTING_VAR

  freestyle-step-2:
    description: Freestyle step..
    title: Free styling 2
    image: ${{VAR1}}
    commands:
      - echo $VAR2
      - curl http://$EXISTING_VAR/index.php
Michael Kolb

is the documentation referencing a pipeline variable defined in the pipeline vs. in the pipeline yaml?

dustinvb

Yes pipeline variables are in the pipeline SPEC file. If you need variables for the pipeline to be set during pipeline execution then you’d use cf_export to have them pop out to pipeline.

dustinvb

The existing variable section is mentioning step-2 in the YAML above.

dustinvb

You do not need to specify a variable name before the cf_export

Michael Kolb

thanks, the documentation page could use more examples of cf_export variable assignment like the example that you provided - cf_export GITHUB_OWNER=${{CF_REPO_OWNER}}

dustinvb

@Kostis (Codefresh) Can you add this to your backlog to add in some more use cases to this?

dustinvb

For example command export?

dustinvb
codefresh-contrib/example-voting-app

Docker’s Example Voting App. Contribute to codefresh-contrib/example-voting-app development by creating an account on GitHub.

Kostis (Codefresh)

@dustinvb will do

Michael Kolb

they display assignment as static references

Michael Kolb

and the reference EXISTING_VAR is misleading if this is pulling from the pipeline variable definition vs. an existing variable

Michael Kolb

documentation clarification would help, since I have had to run several builds trying to figure out how the cf_export process handles environment variables

Michael Kolb

is there any documentation on the codefresh/kube-helm image that provides connection to Kubernetes & Tiller?

dustinvb
codefresh-contrib/images

useful docker images. Contribute to codefresh-contrib/images development by creating an account on GitHub.

codefresh-contrib/cfstep-helm

Docker image for Codefresh Helm step. Contribute to codefresh-contrib/cfstep-helm development by creating an account on GitHub.

2019-04-19

How are you guys running stuff like integration tests and system tests? CF can’t directly access internal services that are deployed, so are you guys using some kind of proxy container in the cluster to run these kinds of tests?

Erik Osterman
codefresh-io/venona

Codefresh runtime-environment agent. Contribute to codefresh-io/venona development by creating an account on GitHub.

Erik Osterman

This allows you to run it all in your k8s cluster

Erik Osterman

we’ve used this for things like connecting to artifactory or using consul

dustinvb

Yup this runner allows you to talk to the internal pods of the Kubernetes cluster using the container ips if needed.

this will just show up as a different run time environment on your pipeline?

dustinvb

Add the runner to the cluster and then select the runner for the pipeline and everything happens on your cluster.

Erik Osterman

related to this, I have a question

Erik Osterman

we are about to make our kops cluster private. I know we can use venona, but how do the k8s integrations work then?

dustinvb

Yes

dustinvb
Codefresh behind the firewall

How to run Codefresh pipelines in your own secure infrastructure

Is there a way to split up what runs on CF runtime env and what runs on our infra? I assume that venona still reports everything up to CF UI, right?

dustinvb

Yes all comes back to UI

dustinvb

Each pipeline has a runtime setting. You can use our CLI to call another pipeline from your original as a step in the original and have that child pipeline run on your infra.

1

I think this should cover our needs, thanks!

1

2019-04-18

dustinvb

Is there a how-to on helmfile in Codefresh like an example codefresh.yml with an explanation on the steps?

Erik Osterman

Not that well documented

Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman

The “ctl” command can be ignored. It just shows how to call Helmfile

Erik Osterman

In this example, we support both blue/green and rolling

Erik Osterman

Also, this example shows how we use the monochart

Erik Osterman

I’d be happy to jump on a call and walk you through it

2019-04-15

dustinvb

@Erik Osterman Do you happen to have an example of running https://github.com/cloudposse/github-commenter as a step in Codefresh pipeline using a Docker image? I’ve seen in other project snippets of codefresh.yml formatted examples but nothing in this project.

cloudposse/github-commenter

Command line utility for creating GitHub comments on Commits, Pull Request Reviews or Issues - cloudposse/github-commenter

dustinvb

Came up with this really quickly.

  GitHubCommenter:
    title: Add GitHub Comment
    image: cloudposse/github-commenter:latest
    environment:
      - GITHUB_TOKEN=${{GITHUB_TOKEN}} #Must be created see link for info below.
      - GITHUB_OWNER=${{CF_REPO_OWNER}}
      - GITHUB_REPO=${{CF_REPO_NAME}}
      - GITHUB_COMMENT_TYPE=pr
      - GITHUB_PR_ISSUE_NUMBER=${{CF_PULL_REQUEST_NUMBER}}
      - GITHUB_COMMENT="" #Your custom comment goes here
aknysh
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

dustinvb

Thanks!

aknysh
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

aknysh
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

2019-04-11

Codefresh Status - Incident History
11:35:20 AM

http://status.codefresh.io/incidents/yg8v2wq51f93\|Bitbucket access issue Apr 11, 11:15 UTC Investigating - We are currently investigating a problem affecting bitbucket integrations

Bitbucket access issue

Codefresh’s Status Page - Bitbucket access issue.

Codefresh Status - Incident History
01:24:51 PM

http://status.codefresh.io/incidents/gf0xpyqp3rp5\|Build services disruption Apr 11, 13:11 UTC Investigating - We are currently investigating this issue

Build services disruption

Codefresh’s Status Page - Build services disruption.

Codefresh Status - Incident History
03:00:05 PM

http://status.codefresh.io/incidents/yg8v2wq51f93\|Bitbucket access issue Apr 11, 14:43 UTC Resolved - The incident has been resolved Apr 11, 11:15 UTC Investigating - We are currently investigating a problem affecting bitbucket integrations

Bitbucket access issue

Codefresh’s Status Page - Bitbucket access issue.

Codefresh Status - Incident History
03:00:06 PM

http://status.codefresh.io/incidents/gf0xpyqp3rp5\|Build services disruption Apr 11, 14:43 UTC Resolved - The incident has been resolved Apr 11, 13:11 UTC Investigating - We are currently investigating this issue

Build services disruption

Codefresh’s Status Page - Build services disruption.

Erik Osterman
GitOps with Terraform on Codefresh (Webinar)

Infrastructure as code, pipelines as code, and now we even have code as code! =P In this talk, we show you how we build and deploy applications with Terraform using GitOps with Codefresh. Cloud Posse is a power user of Terraform and have written over 140 Terraform modules. We’ll share how we handl

Erik Osterman

Here are the slides from the webinar today.

Erik Osterman

Video will be posted as soon as it’s available.

oscarsullivan_old

Thank you

Erik Osterman
Create new S3 Bucket by osterman · Pull Request #75 · cloudposse/testing.cloudposse.co

what Demo of adding a new user bucket why GitOps rocks! =)

Erik Osterman
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

Codefresh Status - Incident History
07:07:23 PM

http://status.codefresh.io/incidents/c56tcftbjvs3\|Helm repositories issue Apr 11, 19:04 UTC Resolved - A fix has been deployed to production and confirmed to have fixed the issue. Helm repositories are fully operational. Apr 11, 18:49 UTC Monitoring - The issue has been identified, and the corresponding fix was applied. Helm repositories are accessible again, We’re closely monitoring the platform to ensure everything is working as expected. Apr 11, 18:47 UTC Identified - Currently Helm repositories provided by Codefresh are having issues. This issue is under…

Helm repositories issue

Codefresh’s Status Page - Helm repositories issue.

2019-04-10

dustinvb

Awesome! I made this little example app a little more deployable… Still working out the kinks. https://github.com/codefresh-contrib/example-voting-app

codefresh-contrib/example-voting-app

Docker’s Example Voting App. Contribute to codefresh-contrib/example-voting-app development by creating an account on GitHub.

dustinvb

Requires a few IPs right now and only works with Cloud LB not sure how to make this more portable using Istio or NGINX as both would require a DNS integration using the LB as is while requiring 2 IPs per running Helm Release is a bit much but this is also meant to demo a few things then toss after playing.

2019-04-09

Erik Osterman

We’ve added support for DocumentDB (MongoDB) to our Codefresh Enterprise terraform module: https://github.com/cloudposse/terraform-aws-codefresh-backing-services

cloudposse/terraform-aws-codefresh-backing-services

Terraform module to provision AWS backing services necessary to run Codefresh Enterprise - cloudposse/terraform-aws-codefresh-backing-services

Erik Osterman

This makes everything that is absolutely essential to running Codefresh Enterprise (onprem) a fully managed service by AWS.

2019-04-05

oscarsullivan_old
Status: Image is up to date for cloudposse/build-harness:0.18.0                                                            
make: *** No rule to make target 'codefresh/notify/slack/deploy'.  Stop.                                                   
[SYSTEM] Error: Failed to run freestyle step: Send notification to Slack channel; caused by NonZeroExitCodeError: Container
 for step title: Send notification to Slack channel, step type: freestyle, operation: Freestyle step failed with exit code:
 2    
  SendSlackDeployNotification:
    title: Send notification to Slack channel
    stage: "deploy"
    image: cloudposse/build-harness:${{BUILD_HARNESS_VERSION}}
    commands:
      - make codefresh/notify/slack/deploy

Anyone aware of why this would be happening? LGTM according to build-harness readme and example here: https://github.com/cloudposse/example-app/blob/29f91a718522e4a702d77d172a41ed1f779d42fe/codefresh/pull-request.yaml#L107

cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

oscarsullivan_old
oscarsullivan_old

likewise make init fails

Erik Osterman

Need more details.

oscarsullivan_old

Not too sure what else to share other than BUILD_HARNESS_VERSION == 0.18.0

Erik Osterman

@Igor Rodionov

oscarsullivan_old

So have tried the Codefresh commands of:

    ...
    commands:
      - make init
oscarsullivan_old

and as above

Igor Rodionov
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Igor Rodionov

?

Igor Rodionov

@oscarsullivan_old ^

Igor Rodionov

check this env vars

Igor Rodionov
cloudposse/build-harness

Collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more - cloudposse/build-harness

Igor Rodionov
cloudposse/build-harness

Collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more - cloudposse/build-harness

Igor Rodionov

To debug it would be useful to see env vars and codefresh.yaml

casey

@oscarsullivan_old I just setup a pipeline using the codefresh/notify/slack/deploy/webapp target using BUILD_HARNESS_VERSION=0.18.0

casey

I would check your environment variables and make sure they are set correctly, I was getting similar errors, and it was because of env variables not being set

oscarsullivan_old

Thanks, I’ll double check both these points (version and undeclared variables)

oscarsullivan_old

it is most likely the declaration of env vars

oscarsullivan_old

I did no special setup to use the build-harness tmeplates

casey

does anyone have suggestions for the strategy I should use for deploying to production from codefresh? I feel unsafe giving the ClusterRole codefresh-role permission to deploy to the whole cluster

casey

Or is that the only way possible?

oscarsullivan_old

I think I use this

oscarsullivan_old
module "codefresh_user" {
  source    = "git:<i class="em em-<https"></i>//github.com/cloudposse/terraform-aws-iam-system-user.git?ref=tags/0.4.1>"
  namespace = "${var.namespace}"
  stage     = "${var.stage}"
  name      = "codefresh"
}

resource "aws_iam_user_policy_attachment" "default" {
  user       = "${module.codefresh_user.user_name}"
  policy_arn = "arn<img src="/assets/images/custom_emojis/aws.png" class="em em-aws">iam:policy/AmazonEC2ContainerRegistryPowerUser"
}
oscarsullivan_old

but not using K8s rn

2019-04-04

oscarsullivan_old

what format do you use for your Secrets Manager in #aws to connect to the Codefresh private repo? Can anyone please share an example

2019-04-03

oscarsullivan_old

Oh wow

oscarsullivan_old

yeh you’re right

oscarsullivan_old

that’s perfect actually. thanks

Codefresh Status - Incident History
09:39:51 AM

http://status.codefresh.io/incidents/22c56qdq2wrb\|Webhooks issue Apr 3, 09:27 UTC Resolved - The incident been resolved Apr 3, 09:24 UTC Investigating - We are currently investigating an issue where git webhooks don’t trigger pipelines. Running builds manually works as expected

Webhooks issue

Codefresh’s Status Page - Webhooks issue.

Codefresh Status - Incident History
12:45:20 PM

http://status.codefresh.io/incidents/d70jkhs9pp9z\|Codefresh availability issue Apr 3, 12:29 UTC Investigating - We are currently investigating a problem affecting Codefresh availability

Codefresh availability issue

Codefresh’s Status Page - Codefresh availability issue.

Codefresh Status - Incident History
01:00:17 PM

http://status.codefresh.io/incidents/d70jkhs9pp9z\|Codefresh availability issue Apr 3, 12:46 UTC Monitoring - Codefresh site is up, we’re still monitoring the system Apr 3, 12:29 UTC Investigating - We are currently investigating a problem affecting Codefresh availability

Codefresh Status - Incident History
01:44:53 PM

http://status.codefresh.io/incidents/d70jkhs9pp9z\|Codefresh availability issue Apr 3, 13:34 UTC Update - Google just updated us on an ongoing incident, we’re monitoring the issue with google team Apr 3, 12:46 UTC Monitoring - Codefresh site is up, we’re still monitoring the system Apr 3, 12:29 UTC Investigating - We are currently investigating a problem affecting Codefresh availability

Codefresh Status - Incident History
02:34:48 PM

http://status.codefresh.io/incidents/d70jkhs9pp9z\|Codefresh availability issue Apr 3, 14:23 UTC Resolved - This incident has been resolved. Apr 3, 14:22 UTC Update - We are continuing to monitor for any further issues. Apr 3, 13:34 UTC Update - Google just updated us on an ongoing incident, we’re monitoring the issue with google team Apr 3, 12:46 UTC Monitoring - Codefresh site is up, we’re still monitoring the system Apr 3, 12:29 UTC Investigating - We are currently investigating a problem affecting Codefresh availability

Codefresh availability issue

Codefresh’s Status Page - Codefresh availability issue.

2019-04-02

oscarsullivan_old

How would you go about checking if .env is a directory or file inside a docker image inside a pipeline?

oscarsullivan_old

You could have a freestyle step running a bash IF -f or -d; you could have some sort of condition in Dockerfile maybe?

oscarsullivan_old

And also to add geodesic to CI, good to just use https://github.com/cloudposse/prod.cloudposse.co/blob/master/codefresh.yml ?

cloudposse/prod.cloudposse.co

Example Terraform/Kubernetes Reference Infrastructure for Cloud Posse Production Organization in AWS - cloudposse/prod.cloudposse.co

oscarsullivan_old

Seems to be only pushing to codefresh registry which works for me. Unsure why it defines an old build harness?

oscarsullivan_old

Erik, where’s an example of you executing a command in a geodesic module from a freestyle step in codefresh?

Erik Osterman
CI/CD Demonstration of Terraform (GitOps) by osterman · Pull Request #22 · cloudposse/testing.cloudposse.co

what Demonstrate how to do CI/CD of Terraform with Codefresh why Larger goal is to apply this to all reference architectures and terraform modules

Erik Osterman

We should move that “use_codefresh” direnv function into geodesic

oscarsullivan_old

thanks

oscarsullivan_old

I get this funny bash script actually

oscarsullivan_old
  DeployingDockerImage:
    title: Deploying Docker Image with Ansible
    image: <http<i class="em em-//r.cfcr.io/user/acme/${{MODULE}}.acme.co.uk"></i>master>
    command:
      - ansible --version 
oscarsullivan_old
02:30:13 PM
Erik Osterman

@oscarsullivan_old that’s the default behavior of geodesic

Erik Osterman

that’s why you can run

Erik Osterman

docker run myco/myinfra \| bash

Erik Osterman

to install geodesic

Erik Osterman

to avoid that, do this:

oscarsullivan_old

what did I miss there Erik?

Erik Osterman
CI/CD Demonstration of Terraform (GitOps) by osterman · Pull Request #22 · cloudposse/testing.cloudposse.co

what Demonstrate how to do CI/CD of Terraform with Codefresh why Larger goal is to apply this to all reference architectures and terraform modules

oscarsullivan_old

because command not cmd?

Erik Osterman
cmd:
      - "-l"
      - "-c"
      - "./tests/run.sh"
Erik Osterman

i am not sure about command vs cmd

Erik Osterman
Freestyle

Run commands inside a Docker container

Erik Osterman

looks like cmd is canonical

oscarsullivan_old

ah I went from this

CollectAllMyDeps:
  title: Install dependencies
  image: python:3.6.4-alpine3.6
  commands:
    - pip install .
oscarsullivan_old

Ok will try to make it closer looking to what you’ve got

Erik Osterman

oh, so yea, cmd is the arg passe to the entrypoint

oscarsullivan_old

could be the difference

Erik Osterman

vs commands is run after the entrypoint

oscarsullivan_old

ahhh

Erik Osterman

you can try this:

oscarsullivan_old

that makes sense

oscarsullivan_old
step_name:
  title: Step Title
  description: Step description
  image: image/id
  working_directory: ${{step_id}}
  commands: 
    - bash-command1
    - bash-command2
  cmd:
    - arg1
    - arg2
oscarsullivan_old

although that confuses me again

Erik Osterman
cmd: ["-l", "-c", "true"]
commands:
      - "my command"
oscarsullivan_old
cmd: ["--version"]
commands:
      - "ansible"
Erik Osterman

no

Erik Osterman

hrm… maybe

Erik Osterman

oscarsullivan_old

oscarsullivan_old

seems dull I can’t just have commands: ansible --version

oscarsullivan_old

and have to split it up

Erik Osterman

nono

oscarsullivan_old

I’ll give these new combos a go

Erik Osterman

in that example, you are passing --version to the entrypoint which is bash

Erik Osterman

so you should get the version of bash back

oscarsullivan_old

ooh

Erik Osterman

you could alternatively change the entrypoint

Erik Osterman

study up on ENTRYPOINT vs CMD in docker

Erik Osterman

then see what we’re doing in the Dockerfile for geodesic

oscarsullivan_old


study […]
Thanks

oscarsullivan_old

Must admit I don’t really get the differences

oscarsullivan_old

Seen both in action but no further

Erik Osterman

yea, the subtle nuances are often misunderstood by Dockerfile authors

Erik Osterman

and you see them misused and abused

Erik Osterman

I liken it to the system call

int execve(const char *filename, char *const argv[],
                  char *const envp[]);
Erik Osterman

ENTRYPOINT ~ filename

Erik Osterman

CMD ~ argv

Erik Osterman

(this is my mental model, not the docker explanation)

oscarsullivan_old

lol Erik it was because I had command not commands

oscarsullivan_old
  DeployingDockerImage:
    title: Deploying Docker Image with Ansible
    image: <http<i class="em em-//r.cfcr.io/xxx/xxx/${{MODULE}}.xxxx.co.uk"></i>master>
    commands:
      - ansible --version 
      # ansible-playbook pod.yml -i inventory/${{inventory}}.yml

Output:

Status: Downloaded newer image for <http<i class="em em-//r.cfcr.io/xxx/xxx/sandbox.xxx.co.uk"></i>master>                               
ansible 2.7.9                                                                                                              
  config file = None                                                                                                       
  configured module search path = [u'/conf/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']               
  ansible python module location = /usr/lib/python2.7/site-packages/ansible                                                
  executable location = /usr/bin/ansible                                                                                   
  python version = 2.7.15 (default, Jan 24 2019, 16<i class="em em-32"></i>39) [GCC 8.2.0]                                                     
Reading environment variable exporting file contents.                                                                      
Successfully ran freestyle step: Deploying Docker Image with Ansible                              
oscarsullivan_old

Now I gotta get my private Role + Playbook accessible

oscarsullivan_old

Thinking either have them in conf, have some kind of wget situation and publicise them (not ideal), try and do an import like with terraform modules?????

Erik Osterman

Our strategy for this is to write the key to chamber

Erik Osterman

However, I think for your purposes, it might be sufficient to write the key to a codefresh secret

Erik Osterman

You can easily add an SSH key to the agent using an environment variable

Erik Osterman
source <(ssh-agent -s)
ssh-add - <<<${ANSIBLE_SSH_PRIVATE_KEY}
Erik Osterman
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

oscarsullivan_old

Or my geodesic module has a git clone.. but then where do I store the deploy key hmmmm

oscarsullivan_old

Have come up with this solution

oscarsullivan_old
04:21:05 PM
Erik Osterman

You could also clone as a step in the codefresh pipeline

Erik Osterman

that would be accessible from inside of geodesic

Erik Osterman

the benefit of doing it that way is that you can leverage the git integrations already available in codefresh without putting a secret (e.g. /conf/ssh/deployment-repo-key into git)

2019-04-01

oscarsullivan_old

So it turned out I was using ‘global variables’ and build variables wrong

oscarsullivan_old
oscarsullivan_old

I thought this would mean when I run the build pipeline my Dockerfile’s ARG PORT would get a value of 3060

oscarsullivan_old

Turns out it doesn’t

oscarsullivan_old
Build an Image with Build Arguments

Codefresh is a Docker-native CI/CD platform. Instantly build , test and deploy Docker images.

oscarsullivan_old

and that’s fine.. happy to define it in a codefresh.yml file

oscarsullivan_old

but then I realised how do I do multi-branch / stage CI… surely I’ll need multiple codefresh.yml because one example argument is APP_ENV

Erik Osterman

Did you try to use interpolations with:

Erik Osterman
build_arguments:
    - key=value
Erik Osterman

e.g.

Erik Osterman
build_arguments:
    - APP_ENV=${{APP_ENV}}
Erik Osterman

Also, what we do sometimes is create a reusable pipeline like build.yaml that we call from something like some-app.yaml; the some-app.yaml can trigger the build pipeline and pass settings to it.

oscarsullivan_old

Ahhh I did not try interpolating. I went from not declaring args in .Yaml to declaring hardcoded and removing the vars from the console

oscarsullivan_old

I was going to have a pipeline for each stage inside the codefresh UI which runs off the same codefresh.yml and simply has the build args in the UI

oscarsullivan_old

Does any of this sound like I’ve done / misunderstood it incorrectly, as having multiple codefresh.yml doesn’t feel DRY.. especially to only change a few build args. Thanks

oscarsullivan_old

Ah.. unless it should be ENV instead of ARG idk

how many variables are you setting at build time? I usually just use environment variables that are set at run time instead of build time. I add the environment variables through the pod definition. This allows my Dockerfile to be pretty generic

oscarsullivan_old

So far 2. Vant see it breaching 5

I see your screenshot that your are setting aws secrets and the port for your container. I don’t think the port matters on your Docker container, so I am assuming its more about your aws secrets. I never was a fan of keeping keys lying around, and I liked having everything controlled through IAM. May I suggest taking a look at https://github.com/uswitch/kiam ? That’s what we use to make pods have specific IAM permissions, and it works great.

uswitch/kiam

Integrate AWS IAM with Kubernetes. Contribute to uswitch/kiam development by creating an account on GitHub.

oscarsullivan_old

Thanks Mark. I’ve actually removed it now. It was when I was trying put ECS

oscarsullivan_old

The two args I have are: application port and application env file.

oscarsullivan_old

Because I then do: COPY ${APP_ENV} .env

2019-03-28

oscarsullivan_old

I’m a bit stuck. I setup my EKS cluster with the CP module. I’ve connected it to Codefresh. I’ve created a Codefresh service to launch my image. I get an error saying no nodes available. I check EC2 console and I have two worker instances (that’s a synonym of node, right).

Does anyone have any insight on what I may have missed from just reading that and seeing the screenshots below? thanks

oscarsullivan_old
oscarsullivan_old

EKS cluster

oscarsullivan_old
aknysh

if you run kubectl get pods --all-namespaces, what do you see?

oscarsullivan_old
~ ⨠ kubectl --kubeconfig="/conf/kube/eks" get pods --all-namespaces
error: the server doesn't have a resource type "pods"
oscarsullivan_old

hmm

oscarsullivan_old

oh wait sorry my token expired

oscarsullivan_old
kubectl --kubeconfig="/conf/kube/eks" get pods --all-namespaces
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
default       pod-68b9fd45b4-w88jb       0/1     Pending   0          40m
kube-public   pod-78c67897fc-tvg28       0/1     Pending   0          45m
kube-system   coredns-854797898c-jhkfd   0/1     Pending   0          3h
kube-system   coredns-854797898c-wp4vz   0/1     Pending   0          3h
oscarsullivan_old

what’s weird is that I deployed pod onto default and kube-public a wihle ago

oscarsullivan_old

it is since removed form kube-public in codefresh though

oscarsullivan_old

and kube-dns appears in codefresh portal with replicas 0/2

Erik Osterman

@aknysh others have had trouble with getting nodes registered

Erik Osterman

What does get nodes show?

aknysh

kubectl get nodes

Erik Osterman

(On phone)

oscarsullivan_old
kubectl --kubeconfig="/conf/kube/eks" get nodes --all-namespaces
No resources found.
aknysh
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

oscarsullivan_old

Hmm no it’s actually a CP project Erik pointed me to last week

oscarsullivan_old

ah wait

oscarsullivan_old

yeh that example

oscarsullivan_old

that’s it

aknysh

for worker nodes to join the cluster, you need to apply that config map

oscarsullivan_old

Right that adds up with just what I was reading here https://learn.hashicorp.com/terraform/aws/eks-intro#configuring-kubectl-for-eks

AWS EKS Introduction | Terraform - HashiCorp Learn

A Terraform configuration based introduction to EKS.

aknysh

either using the example code, or manually

aknysh

let me know if that still is not working

aknysh

it was tested about 2 months ago, was ok

oscarsullivan_old

thanks will follow this:

###########################################################################################################################################
#
# NOTE: To automatically apply the Kubernetes configuration to the cluster (which allows the worker nodes to join the cluster),
# the requirements outlined here must be met:
# <https://www.terraform.io/docs/providers/aws/guides/eks-getting-started.html#preparation>
# <https://www.terraform.io/docs/providers/aws/guides/eks-getting-started.html#configuring-kubectl-for-eks>
# <https://www.terraform.io/docs/providers/aws/guides/eks-getting-started.html#required-kubernetes-configuration-to-join-worker-nodes>
#
# If you want to automatically apply the Kubernetes configuration, set `var.apply_config_map_aws_auth` to "true"
#
###########################################################################################################################################
aknysh

but recently a few people reported issues with workers joining the cluster

Erik Osterman
node not joining cluster · Issue #13 · cloudposse/terraform-aws-eks-cluster

Hi, during the deploymnet everithing went well, but when i tryed to query the cluster i get kubectl get no No resources found. this are the logs from the node. Mar 19 1401 ip-172-18-14-119 kube…

aknysh

so if you have the same issue, I’ll have to look into that

Erik Osterman

@aknysh

aknysh

i saw that

aknysh

as i mentioned, we did not see issues like that before

aknysh

@oscarsullivan_old try the example and let us know

oscarsullivan_old

oscarsullivan_old

where do you store the equivalent of ~/.kube/config for geodesic so you dont have to specify it?

oscarsullivan_old

that would technically be /conf/.kube/config correct?

oscarsullivan_old


don’t have to
e.g kubectl ~–kubeconfig=”/conf/kube/eks”~ get nodes –all-namespaces

aknysh

i think geodesic has an ENV var for that, @Erik Osterman do you remember which one?

aknysh

(or maybe it’s just for kops)

Erik Osterman

It’s a kubectl env

Erik Osterman

Yes

Erik Osterman

That one

oscarsullivan_old

so if I set ENV KUBECONFIG=/conf/kube/eks should be ok?

Erik Osterman

@aknysh what’s the path setting g for the module

Erik Osterman

That is what should change to dev shm

aknysh
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

oscarsullivan_old

groovy that worked

oscarsullivan_old


so if I set ENV KUBECONFIG=/conf/kube/eks should be ok?

oscarsullivan_old

ok let me try the example proj now

oscarsullivan_old
 ⧉  sandbox (aws:default)
 ✓   (-sandbox-admin) ~ ⨠ kubectl get nodes --all-namespaces
No resources found.
 ⧉  sandbox (aws:default)
 ✓   (-sandbox-admin) ~ ⨠ kubectl get pods --all-namespaces
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
default       pod-68b9fd45b4-w88jb       0/1     Pending   0          54m
kube-public   pod-78c67897fc-tvg28       0/1     Pending   0          59m
kube-system   coredns-854797898c-jhkfd   0/1     Pending   0          3h
kube-system   coredns-854797898c-wp4vz   0/1     Pending   0          3h
aknysh

with var.apply_config_map_aws_auth` set to “true”

oscarsullivan_old

ok let me try

Erik Osterman

@aknysh Where does the module write the config?

aknysh

you specify the path

Erik Osterman

Can we specify then dev shm?

aknysh

kubeconfig_filename = "${path.module}/kubeconfig${var.delimiter}${module.eks_cluster.eks_cluster_id}.yaml"

aknysh

need to change that to a var

oscarsullivan_old
 ✓   (healthera-sandbox-admin) eks ⨠ kubectl get nodes --all-namespaces
NAME                                          STATUS     ROLES    AGE   VERSION
ip-10-14-108-66.eu-west-2.compute.internal    NotReady   <none>   10s   v1.12.7
ip-10-14-152-132.eu-west-2.compute.internal   NotReady   <none>   10s   v1.12.7
oscarsullivan_old

@aknysh getting closer

oscarsullivan_old

cool so kube-dns has deployed now

oscarsullivan_old
aknysh

do you see the workers?

Erik Osterman

See above

Erik Osterman

@aknysh can you update the issue with this new information ?

oscarsullivan_old
 ✓   (healthera-sandbox-admin) eks ⨠ kubectl get nodes --all-namespaces
NAME                                          STATUS   ROLES    AGE   VERSION
ip-10-14-108-66.eu-west-2.compute.internal    Ready    <none>   47s   v1.12.7
ip-10-14-152-132.eu-west-2.compute.internal   Ready    <none>   47s   v1.12.7
 ⧉  sandbox (aws:default)
 ✓   (-sandbox-admin) eks ⨠ kubectl get nodes --all-namespaces
NAME                                          STATUS   ROLES    AGE   VERSION
ip-10-14-108-66.eu-west-2.compute.internal    Ready    <none>   1m    v1.12.7
ip-10-14-152-132.eu-west-2.compute.internal   Ready    <none>   1m    v1.12.7
 ⧉  sandbox (aws:default)
 ✓   (-sandbox-admin) eks ⨠ kubectl get pods --all-namespaces
NAMESPACE     NAME                       READY   STATUS             RESTARTS   AGE
default       pod-68b9fd45b4-w88jb       0/1     ImagePullBackOff   0          1h
kube-public   pod-78c67897fc-tvg28       0/1     ImagePullBackOff   0          1h
kube-system   aws-node-mzj4z             1/1     Running            0          1m
kube-system   aws-node-vzj9j             1/1     Running            0          1m
kube-system   coredns-854797898c-jhkfd   1/1     Running            0          4h
kube-system   coredns-854797898c-wp4vz   1/1     Running            0          4h
kube-system   kube-proxy-4b7r7           1/1     Running            0          1m
kube-system   kube-proxy-hxkng           1/1     Running            0          1m
aknysh

i can update the issue, yes

aknysh

but i pointed out a few times there to do what @oscarsullivan_old just did

aknysh

so either that did not work for them, or something else was wrong

oscarsullivan_old

yeh how come that isn’t true by default

oscarsullivan_old

who wouldn’t want their workesr auto connecting???

oscarsullivan_old
apply_config_map_aws_auth="true"

I just added this to my terraform.auto.tfvars

oscarsullivan_old

ok I’ve got somethingI can try

aknysh

as i remember there were some thoughts around that, but don’t remember exactly the details

oscarsullivan_old

workinggg

oscarsullivan_old

codefresh docs a bit dated

oscarsullivan_old

the ui said just input the [image]:[tag]

oscarsullivan_old

but hte docs said the full path

oscarsullivan_old

oh wait no lol still failed

aknysh

is it your image?

oscarsullivan_old
oscarsullivan_old

I felt I had correctly linked codefresh, codefresh repo, codefresh pipeline pushing to repo, and EKR to codefresh correctly

aknysh

i think you need to deploy Docker pull secrets to k8s https://github.com/cloudposse/charts/tree/master/incubator/dockercfg

cloudposse/charts

The “Cloud Posse” Distribution of Kubernetes Applications - cloudposse/charts

aknysh

1 sec

oscarsullivan_old

It doesn’t have any secrets

oscarsullivan_old

(if that’s a valid answer)

aknysh
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

oscarsullivan_old

That’s even when it is codefresh’s registry?

aknysh

yes

oscarsullivan_old

Alright thanks

oscarsullivan_old

will look at htose last 3 links and report back

oscarsullivan_old

heh I’m using the GUI

oscarsullivan_old

ok let’s try that

oscarsullivan_old

super easy with GUI

oscarsullivan_old

darn still a failure

aknysh

when we deploy an app to the cluster, we usually create a helmfile in the same repo

aknysh
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

aknysh

we call it Helm Cartridge

aknysh

the helmfile deploys the pull-secret chart and then the app chart

oscarsullivan_old

Right.. had this fundamental misunderstanding:

Kubernetes deployments are based on a “pull” approach. When you deploy your application to a Kubernetes cluster you don’t upload the application itself (which usually happens with traditional deployments). Instead, Kubernetes will pull the Docker images to its nodes on its own.

https://codefresh.io/docs/docs/deploy-to-kubernetes/access-docker-registry-from-kubernetes/

Accessing a Docker registry from your Kubernetes cluster

Allowing Kubernetes to pull Docker images from your registry

oscarsullivan_old

My understanding was that since I’ve linked Codefresh and EKS and Codefresh + Codefresh registry that was enough??

I went through this a little while ago, I created a secret in the namespace I was deploying to

kubectl create secret docker-registry ${name_of_secret} --docker-server=<http://r.cfcr.io> --docker-username=${codefresh_username} --docker-password=${codefresh_cr_generated_password} -n ${namespace}

Then when you deploy you have to set the imagePullSecret in the Deployment spec

aknysh

k8s itself needs permissions to pull images

aknysh

(that’s for @oscarsullivan_old ^)

aknysh

when pulling the image during deployment

oscarsullivan_old

These docs imply this gives permissions

yeah thats the pull secret that you upload as a secret to kubernetes

aknysh

it has nothing to do with Codefresh, which just sends k8s resource definitions to the cluster, the cluster itself pulls the images

oscarsullivan_old

I went through this a little while ago, I created a secret in the namespace I was deploying to

kubectl create secret docker-registry ${name_of_secret} --docker-server=<http://r.cfcr.io> --docker-username=${codefresh_username} --docker-password=${codefresh_cr_generated_password} -n ${namespace}

Then when you deploy you have to set the imagePullSecret in the Deployment spec

oscarsullivan_old

It’s just From now on, this cluster on this namespace will be able to deploy Docker images from the selected Registry. sort of says what you’re saying @aknysh

oscarsullivan_old


the cluster itself pulls the images

aknysh

yes, it does the same as the chart

aknysh

(too many moving parts )

oscarsullivan_old

so if that does the same as the chart

oscarsullivan_old

need I do Mark’s step?

oscarsullivan_old

and need I do what you’re referencing needs doing?

aknysh

For a quick test, do what Mark showed

aknysh

We use the chart and the helmfile to automate all of that

oscarsullivan_old


–docker-password=${codefresh_cr_generated_password}
Is this the token you can generate or my actual codefresh password???

thats the token

oscarsullivan_old

and the username is the name of the token

oscarsullivan_old

or my username

its whatever you set here

right now it would be your username, unless you have a service account

oscarsullivan_old

fab

in the future I think they are going to allow you to generate users for CFCR that are not tied to an individual user

oscarsullivan_old

yeh makes sense

oscarsullivan_old

once you added that to the cluster’s namespace with kubectl, did you need to referene the secret?

yup, I referenced it in the helm chart

in the Deployment spec

oscarsullivan_old

Ah damn I don’t have any helm bits setup

oscarsullivan_old

Should I get familiar with that via http://kubernetes.io ‘s tutorial?

oscarsullivan_old


There is nothing specific to Codefresh regarding the usage of Docker registry secrets, and therefore following the official Kubernetes documentation is the recommended approach.
Yep ok

if you are not using helm you can just set it in the deployment spec itself https://kubernetes.io/docs/concepts/containers/images/

Images

If you need access to multiple registries, you can create one secret for each registry. Kubelet will merge any imagePullSecrets into a single virtual .docker/config.json

cat <<EOF > pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: foo
  namespace: awesomeapps
spec:
  containers:
    - name: foo
      image: janedoe/awesomeapp:v1
  imagePullSecrets:
    - name: myregistrykey
EOF

here is an example for a pod from their documentation

myregistrykey would be the name of the secret you created in the namespace

oscarsullivan_old

Can I execute an ansible playbook in a code fresh pipeline?

oscarsullivan_old

Instead of K8s deployment it pulls the latest container with ansible. K8s doesn’t feel in scope for the nest 3 months after having tried it for a day it is clear it is a 6-9 month project

Erik Osterman

Yes you can execute anything that fits in a container

Erik Osterman

I think geodesic has ansible too :-)

Erik Osterman

So just run geodesic as part of your Codefresh pipeline

oscarsullivan_old

Ooo wow ok that saves me a lot!

oscarsullivan_old

Hadn’t considered using my module lol

oscarsullivan_old

Will need to start actually pushing that up now / placing it in a CI pipeline

Erik Osterman

That’s the beauty of it! It’s a container. Run it anywhere containers are run… locally or in CICD. Same workflow.

oscarsullivan_old
Freestyle

Run commands inside a Docker container

2019-03-27

casey

has anyone used chamber in a codefresh pipeline to inject secrets into helm chart? Wondering whats that best approach for this?

casey

never mind found a link in the docs, here is reference for anyone who cares

2019-03-25

dustinvb

Codefresh tackles the move from VMs to micro-services supporting build, test, delivery and deployment of micro-services using micro-service based pipelines built upon a scalable platform meant to scale as micro-services emerge which require 10x-1000x the amount of builds in a CI/CD platform as your typical Legacy build VM based applications. Each micro-service can bring in it’s own version of a programming language (best for the job) leaving it up to a VM to build and test offers little flexibility to remain as agile as today’s containers allow. Using a pipeline built out of micro-services gives your CI/CD the same flexibility as the applications you’re building. Around all of this Codefresh has build a UI that integrates with your Version Control System and Kubernetes. Giving you a soup to nuts view of your CI/CD around containers and Kubernetes. We also provide private Docker registry to aid in your archive/deployment of Docker containers. Please see https://steps.codefresh.io/ for a listing of some of the provides micro-services you can include as steps in your pipeline and of course to this document about Codefresh pipelines: https://codefresh.io/docs/docs/configure-ci-cd-pipeline/introduction-to-codefresh-pipelines/

Codefresh: FreshSteps

One small step for your YAML, one giant leap for your pipeline.

Introduction to Codefresh pipelines

Understand how Codefresh pipelines work

2019-03-23

Avineshwar Pratap Singh

Noob question: Hoping to understand (read if there is a resource already) the motivation behind creating *codefresh*. Just to be clear, this is not an ask what is *codefresh* but more around the lines of what kind of issues it tackles better.

@Erik Osterman

Erik Osterman

@Avineshwar Pratap Singh can you ping me on Monday? I am AFK.

Avineshwar Pratap Singh

@Erik Osterman looking forward!

2019-03-21

dustinvb

I already booked the time to attend! This is a do not miss in my book.

1
1

2019-03-20

oscarsullivan_old

https://codefresh.io/docs/docs/getting-started/create-a-basic-pipeline/#running-unit-tests-automatically Is this still valid? Can’t seem to find the ‘Workflow’–>‘Unit Test’ screen.

Getting Started - Create a Basic Pipeline

Continuous Integration with Codefresh

oscarsullivan_old

Ah, I had ‘YAML’ mode on

oscarsullivan_old

I am noticing that I’m unable to ‘Launch’ when using YAML mode? I get a warning confirming this isn’t supported.

oscarsullivan_old

Alright switched to BASIC mode. It defaulted to YAML, but they seem to do the same things! .. do prefer to have it source vontrolled…. feel like I’ve missed a step

Erik Osterman

you can definitely have it source controlled

Erik Osterman

3 ways to define it:

Erik Osterman
  • 1) inline using UI (great for debugging, but not as a workflow)
Erik Osterman
  • 2) from a remote URL
Erik Osterman
  • 3) from source control (how we typically do it)
Erik Osterman

you can also even define your pipeline creation process as code as we do here:

Erik Osterman
cloudposse/codefresh

Codefresh repos and pipelines in code. Contribute to cloudposse/codefresh development by creating an account on GitHub.

Erik Osterman
03:04:05 AM
1
1

2019-03-19

Codefresh Status - Incident History
03:54:49 PM

http://status.codefresh.io/incidents/rgm481pf7gh1\|Codefresh Incident Mar 19, 15:53 UTC Investigating - We are investigating an issue on our platform that is affecting build functionality. We will go to provide more information here as investigation progresses.

Codefresh Incident

Codefresh’s Status Page - Codefresh Incident.

oscarsullivan_old

For Codefresh, builds are effectively PRs aren’t they? As well as daily/weekly builds?

oscarsullivan_old

Surely 220/m would go v quickly

oscarsullivan_old

just trying to work that out

Codefresh Status - Incident History
04:29:54 PM

http://status.codefresh.io/incidents/rgm481pf7gh1\|Codefresh Incident Mar 19, 16:15 UTC Monitoring - There was an issue during a DB upgrade process. The issue was quickly identified and now is resolved. But we’ll keep monitoring the platform to ensure it is working as expected Mar 19, 15:53 UTC Investigating - We are investigating an issue on our platform that is affecting build functionality. We will go to provide more information here as investigation progresses.

Erik Osterman
dustinvanbuskirk/github-commit-statistics

Small script to gather github commit statistics. Contribute to dustinvanbuskirk/github-commit-statistics development by creating an account on GitHub.

dustinvb

This should give you a semi reasonable estimate based on commits to your org repos and the total across all of them.

oscarsullivan_old

Brill thanks. Just had a discussion about moving away from jenkins

oscarsullivan_old

Just weighing up the costs (setup/migration + long term operating costs) of other platforms.

oscarsullivan_old

Jenkins currently costing us $84/m (t2.large + EBS) for 2 concurrent builds… However it is a ball ache to do properly.

oscarsullivan_old

The maintenance cost alone is huge

dustinvb

Yes, our SaaS model gets you out of maintenance and gives you a team of support engineers on our side to assist you if you do run into anything.

dustinvb

I encourage you to reach out to [email protected] to ask any questions about pricing.

oscarsullivan_old

Just had a count of historic builds on Jenkins

oscarsullivan_old

we only just breached 60 builds last month

oscarsullivan_old

and only on 35 builds 19 days into march

dustinvb
Jenkins Plugins

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software.

oscarsullivan_old

turns out we should be fine on the Basic plan

dustinvb

I haven’t had much success with it.

oscarsullivan_old

no just manually counted lol

dustinvb

OH well… If you can manually count then probably wouldn’t even be a metric to concern yourself with. Instead look at concurrency and the runtime sizes memory/compute for each pipeline.

oscarsullivan_old

Having come into my org and not having setup the Jenkins and knowing it isn’t properly backed up, I wouldnt even dare install that plugin Dustin!

oscarsullivan_old

That’s why I made a point here at work that we should move away

oscarsullivan_old

I shouldn’t feel scared of infrastructure!

dustinvb

Jenkins is a bit brittle.

oscarsullivan_old

If I’m honest, I COULD quite easily have a Dockerised Jenkins with a mounted EBS volume that gets backed up……. but I just don’t think I want to

oscarsullivan_old

I would rather we used a SaaS CI tool like CircleCI/Travis/Cloud Fresh

oscarsullivan_old

But at the same time it would be quite hard to have this dockerised jenkins scale to use slave workers etc

oscarsullivan_old

and then restore the config when updating the EC2

dustinvb

I’ve done it. Terraform + Chef and then ran all the builds from Docker. It was better but I still had to deal with things like the Maven plugin…

oscarsullivan_old

if I commit to Codefresh I have to commit to dockerising all our products hmmmm

dustinvb

Not true

oscarsullivan_old

It would feel like an anti-pattern to dockerise all our applications, have CI run on the dockerised version, but not deploy it as a container

dustinvb

Easy to roll in anything you’re doing on the Linux Slave with Jenkins over to an Ubuntu or CentOS Docker image and incorporate that same scripting and push binaries. That’s supported.

oscarsullivan_old

Oh? That does sound doable

oscarsullivan_old

Yes.. interesting

dustinvb

Depends, we have customers that push jars for example that are used by their customers that are not focused on the Docker artifact. I think it depends on what your plan is for the artifact. If it makes sense to run in a container and it is possible that’s an optin.

oscarsullivan_old

I literally could use our same pipeline but in codefresh

dustinvb

Yup, and if you’re like me you already have the scripting in version control in something like a make file.

oscarsullivan_old

I do not, I have been here for just over a month and desperately want it out of a platform

oscarsullivan_old

Practically unreadable in Jenkins (not using Blueocean or Jenkinsfile)

dustinvb

Just leaving the item of building the slave Docker image by running apt-get and other package solutions to prepare the image for the build. Then you can use that in a multistage later on.

oscarsullivan_old

Does Codefresh have a CF vs other page?

oscarsullivan_old

I..e vs Travis and Circle

dustinvb

Yes we have several

dustinvb
Continuous Delivery Blog - Codefresh

Learn how to achieve continuous delivery to Kubernetes with Helm, Istio, Selenium testings, security testing, performance testing, and more.

dustinvb

Check the blog here.

oscarsullivan_old

Thanks, will search in that

dustinvb

@Kostis (Codefresh) Is our technical writer and does the comparisons posted on the blog if you have any questions.

oscarsullivan_old

Haha no luck using the search bar for “Travis” and “Circle”

dustinvb
Migrate from TravisCI to Codefresh - The New Stack

TravisCI was recently bought by Idera. And it seems that this acquisition, with reported layoffs, is  encouraging people to look for alternatives. Here’sa tutorial on how to use Coefresh as an alternative.

oscarsullivan_old

Does it count as a build if I create a PR, build triggered, and I then update the PR?

oscarsullivan_old

Is that 1 or 2 builds?

oscarsullivan_old

Say first build fails

oscarsullivan_old

and then I fix it

dustinvb

2 builds, but could restrict this to not create on specific events.

oscarsullivan_old

Is the notion of a build applicable when I’m just testing? I figure yes, as in order to test it must build in advance

dustinvb

Say if you have filter on commit and PR open, the commit would have already built unless you wanted to do more in regards to a commit pipeline vs. pr pipeline. Then you’d add pull request synchronized event to build on your commits to existing PR.

oscarsullivan_old

Alright thanks.. quite a niche scenario from what I’ve seen.. Devs generally test in advance of PR

oscarsullivan_old

but just wanted to know whether it is easy to burn through build count

dustinvb

Oscar, I’d suggest emailing Michael about the pricing. He can work with you in regards to concerns around these soft build limits in our plans and find you a plan that fits your budget.

oscarsullivan_old

Thanks Dustin. I did mean to reply actually but got caught up elsewhere.

dustinvb
05:04:14 PM
oscarsullivan_old

I swear I’m not a busy-bod

oscarsullivan_old

I do really do work

dustinvb
05:06:08 PM
dustinvb

Not you ?

oscarsullivan_old

No.. more like

oscarsullivan_old
05:06:44 PM
dustinvb

Oscar, Michael has confirmed there is alternative plans he can discuss with you. Just FYI.

oscarsullivan_old

Thanks, have replied to the existing email chain

Codefresh Status - Incident History
07:25:28 PM

http://status.codefresh.io/incidents/rgm481pf7gh1\|Codefresh Incident Mar 19, 19:11 UTC Update - We’re still working on resolving some performance issues. We’re constantly monitoring the system. Once all minor issues are fully resolved, this incident will be closed Mar 19, 16:15 UTC Monitoring - There was an issue during a DB upgrade process. The issue was quickly identified and now is resolved. But we’ll keep monitoring the platform to ensure it is working as expected Mar 19, 15:53 UTC Investigating - We are investigating an issue on our platform that is…

Codefresh Incident

Codefresh’s Status Page - Codefresh Incident.

Codefresh Status - Incident History
07:55:13 PM

http://status.codefresh.io/incidents/rgm481pf7gh1\|Codefresh Incident Mar 19, 19:46 UTC Resolved - The issue is now resolved. Performance is now at expected levels across all the platform. We’re going to keep monitoring the system to ensure a consistent behavior in terms of performance Mar 19, 19:37 UTC Update - UPDATE: we’re currently performing a DB-upgrade operation as part of the actions to resolve the current performance issues Mar 19, 19:11 UTC Update - We’re still working on resolving some performance issues. We’re constantly monitoring the system. Once…

2019-03-15

dustinvb

Simple Step for Aqua scanning in a Codefresh pipeline. https://github.com/codefresh-contrib/cfstep-aqua

codefresh-contrib/cfstep-aqua

Codefresh Step for Aqua Docker Image Scans. Contribute to codefresh-contrib/cfstep-aqua development by creating an account on GitHub.

dustinvb
04:19:17 PM

This is the results passed back to Codefresh registry and there is a link to the detailed report in Aqua

2
Erik Osterman

@dustinvb did something change in the last 2-3 weeks in codefresh that would lead to this to start failing?

chsh -s /bin/sh nobody
Erik Osterman

got it

Erik Osterman

something in alpine:3.7 has changed

Erik Osterman

so that it prompts for a password

Erik Osterman

but doesn’t happen in 3.8

Erik Osterman

i’ll figure this out.

dustinvb

I normally see this error related to npm install using global flag. Not sure why it’s going to fail here. I have to add the unsafe arg to npmconfig to get around it.

oscarsullivan_old

+

oscarsullivan_old

When I use -g for npm it expects sudo permissions. This is when using node-alpine

Erik Osterman
chsh always asking a password , and get `PAM: Authentication failure`

Today I tried to switch to another shell. First I tried fish, and used chsh -s fish to change fish to default. After some time I found it cannot use ~/.bashrc (&& needs to be replaced by a…

Erik Osterman

this worked for me

Erik Osterman

echo "auth sufficient pam_shells.so" > /etc/pam.d/chsh in my alpine container

Erik Osterman

@dustinvb can i unexport an env? (E.g. cf_unexport)

dustinvb

Sorry no, not a feature at this time.

2019-03-14

Erik Osterman

Just incase anyone is planning on rolling out Codefresh Enterprise (on prem), in AWS, here’s our terraform module for it: https://github.com/cloudposse/terraform-aws-codefresh-backing-services

cloudposse/terraform-aws-codefresh-backing-services

Terraform module to provision AWS backing services necessary to run Codefresh Enterprise - cloudposse/terraform-aws-codefresh-backing-services

1
Erik Osterman

this is to leverage as many fully-managed AWS services as possible that are required to run Codefresh

2019-03-13

Codefresh Status - Incident History
09:00:31 AM

http://status.codefresh.io/incidents/qnqv9rlw0jm6\|GitHub Incident - Degraded performance across Web, Pages and Notifications Mar 13, 08:51 UTC Resolved - This incident has been resolved. Mar 12, 15:18 UTC Investigating - GitHub has reported an incident related to degraded performance across Web, Pages and Notifications. More information here: https://www.githubstatus.com/

GitHub Incident - Degraded performance across Web, Pages and Notifications

Codefresh’s Status Page - GitHub Incident - Degraded performance across Web, Pages and Notifications.

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

dustinvb

Sorry been traveling see you’re already here @oscarsullivan_old

oscarsullivan_old

Indeed! Looking forward to spending tomorrow using Codefresh. Thanks for the demo-call Dustin!

2

2019-03-12

Codefresh Status - Incident History
03:20:25 PM

http://status.codefresh.io/incidents/qnqv9rlw0jm6\|GitHub Incident - Degraded performance across Web, Pages and Notifications Mar 12, 15:18 UTC Investigating - GitHub has reported an incident related to degraded performance across Web, Pages and Notifications. More information here: https://www.githubstatus.com/

GitHub Incident - Degraded performance across Web, Pages and Notifications

Codefresh’s Status Page - GitHub Incident - Degraded performance across Web, Pages and Notifications.

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

oscarsullivan_old

How come the github status comes to this channel @Erik Osterman?

Erik Osterman

The Codefresh status page includes statuses of everything they depend on

oscarsullivan_old

Ah ok, cool

oscarsullivan_old

I just subscribed to github status rss for my firm’s slack.. awesome idea.

Erik Osterman

Yea it’s worked wonderfully! We have a channel at CloudPosse with feeds for all the status pages (including AWS) that we depend on

2019-03-08

dustinvb
Git providers

Codefresh is a Docker-native CI/CD platform. Instantly build , test and deploy Docker images.

2

2019-03-06

dustinvb
Access control

How to restrict resources in a company environment

2

2019-03-04

has anyone ever set up deploy conditionals to only deploy when a branch is tagged?

dustinvb

Yes, Mark I have.

I normally evaluate that the variable ${{CF_BASE_BRANCH}} is set to master

This will have a value on tags.

Real world example.

when:
      condition:
        all:
          releaseTag: 'match("${{CF_BRANCH}}", "^([0-9]+)([.][0-9]+){1,2}.*", true) == true'
          masterBranch: '"${{CF_BASE_BRANCH}}" == "master"'
dustinvb

This will look at branch for semantic version as tag and also make sure the branch used was master not some feature.

you rock, dustin

It looks like CF_BASE_BRANCH does not exist on Bitbucket? Do you know any other way to pull the name of the branch, not the tag? even when I do

    when:
      condition:
        all:
          releaseTag: 'match("${{CF_BRANCH}}", "^([0-9]+)([.][0-9]+){1,2}.*", true) == true'
      branch:
        only:
          - master

the branch still gets pulled as the tag eg 1.0.0

when I do a build with echo "CF_BASE_BRANCH=${{CF_BASE_BRANCH}}", it says there is bad variable substitution

dustinvb

During the event we pick up tag and use for CF_BRANCH based on the event. I am not sure why BASE_BRANCH is not available. Please send this through Codefresh’s Intercom to be tracked by support. BTW new events for Bitbucket have been released providing more filtering options around PRs.

2019-03-01

Erik Osterman

@dustinvb is there are way to update shared configurations using cli?

Erik Osterman

thx!!

Erik Osterman

feels very much like kubectl

Erik Osterman

too bad it doesn’t support -f - for stdin

Erik Osterman
Error: Failed to read file; caused by Error: File extension is not recognized

Erik Osterman

@dustinvb got what I wanted to implement (assume roles) for codefresh

2019-02-28

dustinvb

@Erik Osterman Do you have examples of pulling terraform modules from git repos into Codefresh pipeline for use with a step to run terraform commands?

Erik Osterman

yes, but easier for me to share over a zoom

Erik Osterman

how soon do you need it?

dustinvb

I have a customer that’s working on it right now but I have a meeting coming up.

dustinvb

Would you mind me connecting the two of you to go over how it works?

Erik Osterman

sure thing!

dustinvb

Cool let me see if he’s available

dustinvb

Do you have a codefresh.yaml that we’d be able to look at? Looks like your calendar has you pretty buried.

Michael Kolb

Hello Erik, an example codefresh.yaml would be helpful to understand how you manage the Terraform module ssh github downloads. I am working with Terraform’s hashicorp/terraform:light image for the Terraform cli invocation. Dustin provided the reference document, https://codefresh.io/docs/docs/codefresh-yaml/steps/git-clone/#reuse-a-git-token-from-codefresh-integrations, for using an SSH key with GIt and Codefresh.

Architecturally, we are working on separating our Terraform infrastructure code from our k8s container application code and creating a terraform directory with terraform code in the container application Github repository so that the applications AWS environment dependencies, sns, sqs, etc, are built and maintained with application deployment. We are also exploring POC of using an environment specific terraform subdirectories, i.e, terraform/dev, to store encrypted tfvars files for ease of code promotion across the dev, test, staging, and production environments. Any Codefresh Terraform insights that you could provide would be helpful.

Git-Clone

Customizing the git checkout process

Erik Osterman

Hey @Michael Kolb !

Erik Osterman

I will be done with calls around 4:30p PST and can help you out!

dustinvb

@Michael Kolb If you can make the call, I’d love to attend to. Let us know.

Michael Kolb

sorry i am not able to take the call today, but my calendar is open tomorrow

dustinvb

Okay, Erik’s calendar has availability at 2:15pm I won’t be able to make if you can you it can be booked here. https://calendly.com/cloudposse/30min?back=1

2019-02-22

Codefresh Status - Incident History
08:40:11 PM

http://status.codefresh.io/incidents/b05stwrlnbzv\|DockerHub - Issues pushing/pulling from registry Feb 22, 20:35 UTC Monitoring - Docker has reported a new issues with Docker Hub

https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5c705ae105b76a35c3765c30

Update from February 22, 2019 8:26PM UTC


Issues pushing/pulling from registryPartial Service Disruption Components Docker Hub Registry

Locations IAD3

February…

DockerHub - Issues pushing/pulling from registry

Codefresh’s Status Page - DockerHub - Issues pushing/pulling from registry.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Codefresh Status - Incident History
08:55:11 PM

http://status.codefresh.io/incidents/b05stwrlnbzv\|DockerHub - Issues pushing/pulling from registry Feb 22, 20:48 UTC Resolved - Dockerhub has reported the incident as Resolved:

February 22, 2019 12:44PM PST February 22, 2019 8:44PM UTC [Resolved] This issue has been resolved.

More info here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5c705ae105b76a35c3765c30 Feb 22, 20:35 UTC Monitoring - Docker has reported a new issues with Docker Hub

https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5c705ae105b76a35c3765c30

Update from February 22, 2019 8:26PM…

2019-02-19

Codefresh Status - Incident History
03:20:09 PM

http://status.codefresh.io/incidents/jl14ql9hpsf8\|GitHub's Incident Feb 19, 15:18 UTC Investigating - There’s an incident at GitHub. Specifically related to their Notifications component. This could impact your builds and normal operations flow (some webhooks may not be delivered).

More information here: https://www.githubstatus.com/

GitHub's Incident

Codefresh’s Status Page - GitHub’s Incident.

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

Codefresh Status - Incident History
07:09:59 PM

http://status.codefresh.io/incidents/jl14ql9hpsf8\|GitHub's Incident Feb 19, 18:50 UTC Monitoring - Update from GitHub: “Update - We are continuing to monitor webhook delivery times as they return to normal.”

More information here: https://www.githubstatus.com/ Feb 19, 15:18 UTC Investigating - There’s an incident at GitHub. Specifically related to their Notifications component. This could impact your builds and normal operations flow (some webhooks may not be delivered).

More information here: https://www.githubstatus.com/

GitHub's Incident

Codefresh’s Status Page - GitHub’s Incident.

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

Codefresh Status - Incident History
03:40:05 AM

http://status.codefresh.io/incidents/jl14ql9hpsf8\|GitHub's Incident Feb 20, 03:26 UTC Resolved - This incident has been resolved. Feb 19, 18:50 UTC Monitoring - Update from GitHub: “Update - We are continuing to monitor webhook delivery times as they return to normal.”

More information here: https://www.githubstatus.com/ Feb 19, 15:18 UTC Investigating - There’s an incident at GitHub. Specifically related to their Notifications component. This could impact your builds and normal operations flow (some webhooks may not be delivered).

More information here:…

GitHub's Incident

Codefresh’s Status Page - GitHub’s Incident.

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

2019-02-14

Erik Osterman
05:18:34 AM

@Erik Osterman set the channel purpose: Archive: https://archive.sweetops.com/codefresh/

2019-02-13

Erik Osterman
codefresh-io/venona

Codefresh runtime-environment agent. Contribute to codefresh-io/venona development by creating an account on GitHub.

Erik Osterman

This is a fine piece of work

Erik Osterman

Deployed it yesterday - very easy

1
1

2019-02-11

anyone ever get an error similar to this? Everything was working fine last night, now all of a sudden none of my builds are working.

[SYSTEM] Error: Error: Failed to get context bitbucket with error: 500 - {"message"<i class="em em-"Error"></i> Context bitbucket not found."}; caused by StatusCodeError: 500 - {"message"<i class="em em-"Error"></i> Context bitbucket not found."}

That’s in the main_clone step. I don’t see anything on the codefresh status page that would indicate an outage

dustinvb

Mark, this seems like a legitimate error. Can you get this into Intercom for Codefresh’s support team to review?

dustinvb

Please include the Builds URL in that conversation @

2019-02-08

dustinvb

Take advantage of the multiple git provider support and add in gitlab, I think the only difference in triggering is the support of Release.

Erik Osterman

(lol, i was demonstrating how a release to 0.1.2-staging will go to the staging cluster)

dustinvb

Well shoot…

Erik Osterman

but not a bad idea

Erik Osterman

i might add something like that as a back up. it’s a good suggestion.

Erik Osterman

and demonstrates more features

casey
05:52:59 PM

@casey has joined the channel

2019-02-07

Codefresh Status - Incident History
12:35:13 AM

http://status.codefresh.io/incidents/8rndvkv9dvp8\|GitHub's Incident Feb 7, 22:38 UTC Resolved - GitHub has reported this incident as resolved. More information here: https://www.githubstatus.com/ Feb 7, 21:57 UTC Investigating - GitHub has reported an incident related to degraded API, http://GitHub.com and Notifications performance. More information here: https://www.githubstatus.com/

GitHub's Incident

Codefresh’s Status Page - GitHub’s Incident.

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

Erik Osterman

This happened in a middle of a demo today :-)

2019-02-06

Codefresh Status - Incident History
05:40:34 PM

http://status.codefresh.io/incidents/6mxnf9c83l99\|GitHub's incident: degraded API performance. Feb 6, 17:22 UTC Update - We are continuing to investigate this issue. Feb 6, 17:21 UTC Investigating - GitHub has reported an incident regarding degraded API performance. This could affect your builds. More information here: https://www.githubstatus.com/

GitHub's incident: degraded API performance.

Codefresh’s Status Page - GitHub’s incident: degraded API performance..

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

Codefresh Status - Incident History
09:10:35 PM

http://status.codefresh.io/incidents/6mxnf9c83l99\|GitHub's incident: degraded API performance. Feb 6, 20:57 UTC Resolved - GitHub has reported this incident as resolved. More information here: https://www.githubstatus.com/ Feb 6, 17:22 UTC Update - We are continuing to investigate this issue. Feb 6, 17:21 UTC Investigating - GitHub has reported an incident regarding degraded API performance. This could affect your builds. More information here: https://www.githubstatus.com/

GitHub's incident: degraded API performance.

Codefresh’s Status Page - GitHub’s incident: degraded API performance..

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

2019-02-05

Erik Osterman

Easily implement approval steps in codefresh

Erik Osterman
Add approval steps by osterman · Pull Request #4 · cloudposse/example-app

what Add approval steps why Demonstrate flow control demo

1

2019-02-02

2019-01-31

Erik Osterman
mkj28/smashing-codefresh

smashing dashboard for Codefresh builds. Contribute to mkj28/smashing-codefresh development by creating an account on GitHub.

anyone wanting to improve this - forks are more than welcome

mkj28/smashing-codefresh

smashing dashboard for Codefresh builds. Contribute to mkj28/smashing-codefresh development by creating an account on GitHub.

Erik Osterman

Nice for office wallboards!

Oleg Sucharevich

Cool!

Erik Osterman

this is by @

dustinvb
504s and Latency on Builds and Pipeline pages in UI

Codefresh’s Status Page - 504s and Latency on Builds and Pipeline pages in UI.

Erik Osterman

thanks for the heads up

Erik Osterman

thankfully at this very moment, not building.

dustinvb

Wonder if we can just integrate the status page into the channel

Erik Osterman

done

dustinvb

Incident reported earlier has been resolved.

Erik Osterman

Wierd, we didn’t get the all lear message from the atom feed?

Codefresh Status - Incident History
08:35:37 PM

http://status.codefresh.io/incidents/qswt2278fjx7\|504s and Latency on Builds and Pipeline pages in UI Jan 31, 20:14 UTC Resolved - The issue has been identified and resolved. It was due to a high load on the system. We scaled it up and took the corresponding actions to prevent further issues. Jan 31, 20:14 UTC Monitoring - We’ve resolved the issue and are actively monitoring the system. Jan 31, 19:02 UTC Investigating - Our Team is investigating reports with 504 Gateway errors and latency in Codefresh pages.

504s and Latency on Builds and Pipeline pages in UI

Codefresh’s Status Page - 504s and Latency on Builds and Pipeline pages in UI.

dustinvb

That took a bit…

Erik Osterman

Yea, guess it doesn’t poll too often

Codefresh Status - Incident History
11:35:32 PM

http://status.codefresh.io/incidents/fflygm4tj9fn\|GitHub Incident Jan 31, 22:11 UTC Resolved - GitHub has reported this incident as Resolved. More information here: https://www.githubstatus.com/ Jan 31, 22:10 UTC Update - We are continuing to work on a fix for this issue. Jan 31, 22:01 UTC Identified - GitHub has reported an ongoing incident. This could affect your builds. More information here: https://www.githubstatus.com/

GitHub Incident

Codefresh’s Status Page - GitHub Incident.

GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

2019-01-30

10:24:09 PM

@ has joined the channel

Erik Osterman
cloudposse/codefresh

Codefresh repos and pipelines in code. Contribute to cloudposse/codefresh development by creating an account on GitHub.

Erik Osterman

We’ve updated our example-app with our pipelines for unlimited staging, automatic destroy, release promotion, and multiple cluster deploys using tags

1
Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman

This also has an example of blue/green deploy’s using Istio

Erik Osterman

we use helmfile with helm

Erik Osterman

we use variant as a “deploy cli”

Erik Osterman

@

1
Erik Osterman

what’s notable about these updated pipelines is we avoid the classic “main clone” in every pipeline other than build

Erik Osterman

this makes it faster

Erik Osterman

the build artifact always contains the helmfile to deploy. this is an evolution of our statup demo (which is now out of date)

Erik Osterman

@dustinvb I want to add now all the clair scanning and other kinds of tests

dustinvb

https://steps.codefresh.io/ should have the most up-to-date documentation some testing steps requires the user to work with vendor to install server side.

Codefresh: FreshSteps

One small step for your YAML, one giant leap for your pipeline.

2019-01-29

Anyone know how to nest an array in cfstep-helm value override? For reference I am trying to set the extraHostPathMounts for the kiam helm chart https://github.com/helm/charts/blob/master/stable/kiam/values.yaml#L182

helm/charts

Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

I wound up creating a new repo with just a codefresh.yaml and a values file

helm/charts

Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

Erik Osterman

have you looked into helmfile?

I hadn’t ever heard of it. Looking into it now

Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Erik Osterman

So we use this both to distribute backing services as well as applications

Erik Osterman

and by “applications” i mean those delivered via a codefresh pipeline (e.g. web apps)

Erik Osterman

here’s our “example app”

Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman

the deploy/ folder contains the helm releases

Erik Osterman

We use this exclusively with helm

Erik Osterman

helmfile takes the pain out of dealing with values.yaml and helm releases.

Erik Osterman

plus makes it easier to make helm more 12-factor compliant

2019-01-28

Erik Osterman

This is awesome! https://steps.codefresh.io/

Codefresh: FreshSteps

One small step for your YAML, one giant leap for your pipeline.

Erik Osterman

It would be cool if there was a way to define like a step.yaml in a repo with sufficient annotations that codefresh could just populate this automatically.

I second that ^^^

2019-01-25

Daren

Is it possible to trigger a pipeline via a webhook, and have the pipeline product an artifact that the caller has access to, say a json file

Erik Osterman

ohhhh some artifact scoped to the CF_API_KEY?

Erik Osterman

I don’t know the answer. If they did, it would probably related to this new feature: https://codefresh.io/docs/docs/testing/test-reports/

Test reports

How to create and view Test reports in Codefresh

Daren

@ said you can create a named context, then used a (currently) undocumented api call to obtain it. A cronjob pipeline can be used to clean them up

Erik Osterman

is the API call related to the test reports?

Erik Osterman

ohhhh i think i understand now

Erik Osterman

you have a pipeline that writes an artifact to the shared volume

Erik Osterman

and you want to know if you could read that artifact from that volume using the codefresh cli?

Erik Osterman

(and apparently this is possible using that undocumented call)

dustinvb
codefresh-io/eks-installer

Tool to setup a new EKS cluster and connect to Codefresh - codefresh-io/eks-installer

dustinvb
codefresh-io/eks-installer

Tool to setup a new EKS cluster and connect to Codefresh - codefresh-io/eks-installer

Erik Osterman

Oh this is cool

dustinvb

Example API call to get this context from another system: https://github.com/codefresh-io/cli/blob/master/lib/logic/api/context.js#L50-L60

codefresh-io/cli

Codefresh CLI. Contribute to codefresh-io/cli development by creating an account on GitHub.

dustinvb

Since you get the build id when you trigger a pipeline, I’d suggest setting that in the file name and then you’ll be able to have a unique per build file.

2019-01-24

I am trying to deploy an application, with the image being pulled from cfr. I added a Secret to the helm chart so that the chart would deploy everything needed for the application, but it doesn’t create the secret first. What is the best practice for uploading the secret?

Erik Osterman

good question.

Erik Osterman

so there are a few ways to handle this.

Erik Osterman

If you’re not going to be using a lot of namespaces, it would be most secure to deploy the pull secret manually as a “one off”

Erik Osterman

If you’re using kops, you can provision a cluster-wide pull secret

Erik Osterman

If you’re using helmfile, then you can deploy it by adding a release of this: https://github.com/cloudposse/charts/tree/master/incubator/dockercfg

cloudposse/charts

The “Cloud Posse” Distribution of Kubernetes Applications - cloudposse/charts

dustinvb
Manage your Kubernetes cluster in Codefresh

Codefresh is a Docker-native CI/CD platform. Instantly build , test and deploy Docker images.

dustinvb

These work with any registries you have connected to Codefresh.

dustinvb

Maybe too many options?

Erik Osterman

haha

Erik Osterman

@dustinvb would be nice if there was a build-step or something we could use to automatically provision the pull secret for a namespace

dustinvb

You can use the CLI as a build step.

lol, this is all good stuff. So if I am going to have a namespace per dev team or developer, would it be best to have a helm chart that configures the namespace with all secrets needed?

Erik Osterman

the downside with provisioning secrets with helm is that they are in clear text if you run helm get values

Erik Osterman

and are also surfaced in clear text in the helm UI in codefresh

thats good to know

dustinvb

Bah I am all off on my explanation… I mis read what you’re trying to do.

dustinvb

Use helm-secrets plugin

dustinvb

That’s what I used

Erik Osterman

the helm-secrets plugin doesn’t fix it

Erik Osterman

the values are still passed in clear-text to helm

dustinvb

Yes they are indeed but on the Kubernetes side it will create the secret for you using the Helm Chart.

Erik Osterman

hrmm

Erik Osterman

or do you mean helm-secrets will create the Secrets resource out-of-band with helm, so that it won’t be visible by helm?

dustinvb

If you don’t want these secrets showing up in a pipeline it is best to circumvent the secret creation in CI/CD something else like https://github.com/Boostport/kubernetes-vault

Boostport/kubernetes-vault

Use Vault to store secrets for Kubernetes! Contribute to Boostport/kubernetes-vault development by creating an account on GitHub.

dustinvb

It is still visible by Helm Releases and the command in the pipeline.

Erik Osterman
mumoshu/aws-secret-operator

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager. - mumoshu/aws-secret-operator

Erik Osterman

@mumoshu wrote this! I can’t wait to use it. no need to deploy vault.

dustinvb

Nice!

dustinvb

Yeah for AWS this looks great. I don’t have enough access in AWS to create an example of this. If you ever have a pipeline with a helm chart using this I’d like to check it out.

Erik Osterman

True… AWS solution.

dustinvb

Mark, do any of these options look viable for your use case?

I think for now just to get unblocked i might create manually. https://github.com/mumoshu/aws-secret-operator looks pretty cool in the long term though

3

Thanks you guys!

Anyone here used codefresh api to start pipelines? The swagger at http://codefresh.io/api is not very clear…. or FWIW - anyone integrated Spinnaker with Codefresh?

cc @Purva

dustinvb

Our CLI is built off of the API. https://github.com/codefresh-io/cli/blob/master/lib/logic/api/pipeline.js#L116-L165

I’ll get your feedback to the team about the API.

codefresh-io/cli

Codefresh CLI. Contribute to codefresh-io/cli development by creating an account on GitHub.

Erik Osterman

Could be useful if any codefresh customers are using terraform

Erik Osterman

@Dan Garfield

Dan Garfield
08:11:11 PM

@Dan Garfield has joined the channel

Erik Osterman

that’s some pretty sweeet feature velocity

2019-01-23

I’m trying to deploy to a new k8s cluster, and thinking I missed a step somewhere when provisioning it. any clues here? yaml:

title: Deploy with helmfile
stage: Deploy
image: 'cloudposse/build-harness:${{BUILD_HARNESS_VERSION}}'
working_directory: ./
environment:
  - 'IMAGE_TAG=${{SEMVERSION_COMMIT_SHORT}}'
  - 'RELEASE_NAME=${{NAMESPACE}}'
  - APP_API_ROOT=https://${{BACKEND_APP_HOST}}'
  - 'AWS_ACCESS_KEY_ID=${{CHAMBER_ACCESS_KEY_ID}}'
  - 'AWS_SECRET_ACCESS_KEY=${{CHAMBER_SECRET_ACCESS_KEY}}'
commands:
  - make helm/toolbox/upsert
  - chamber exec app -- helmfile sync

log output:

Pulling image cloudposse/build-harness:0.15.3                                                                  
Digest: sha256:48379abbffe7707ce8185839c24ad20830e45fec7092edbf4febf6b2e859df98                                
Status: Image is up to date for cloudposse/build-harness:0.15.3                                                
ERROR: kubectl not installed!                                                                                  
make: *** [//build-harness/modules/helm/Makefile<i class="em em-24"></i> helm/toolbox/upsert] Error 1
Erik Osterman

that looksl ike a really old version of the build-harness

Erik Osterman

newer versions ship kubectl

Erik Osterman

oh snap

Erik Osterman

that’s the latest

1
Erik Osterman

I can take a look in a few hours

Erik Osterman

about to give a demo of this https://github.com/cloudposse/example-app

cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman

Blue/Green deployments with Codefresh and Istio and Helm

cool. i’ll poke around.

Erik Osterman

it uses our “cartrige” style deployment methodology where everything is bundled in the deployment artifact (aka docker image)

Erik Osterman

to get unblocked, you should be able to do:

Erik Osterman

make packages/install/kubectl

Erik Osterman

as a build step

aknysh

@ i think it should be a geodesic image

image: cloudposse/geodesic:${{GEODESIC_VERSION}}
Erik Osterman

Though if he does that, then he needs to install the build-harness

Erik Osterman

so it’s neither here nor there

aknysh

split into two steps

Erik Osterman

yea, could possibly do that

aknysh
deploy_helmfile:
    title: Deploy with helmfile
    stage: "Deploy"
    image: cloudposse/geodesic:${{GEODESIC_VERSION}}
    environment:
    - AWS_REGION=us-west-2
    commands:
    # Install or upgrade tiller
    - "make init"
    - "make helm/toolbox/upsert"
    # Deploy chart to cluster using helmfile
    - "chamber exec kops -- helmfile --file config/helmfile.yaml --selector component=app sync --concurrency 1 --args '--wait --timeout=600 --force --reset-values'"
    # Add kiam annotation
    - kubectl annotate namespace ${NAMESPACE} "<http://iam.amazonaws.com/permitted=.*>" --overwrite

thanks! i’ll try those options

fyi this was the example i followed, and it did work before with the same build-harness version. https://github.com/cloudposse/statup/blob/add-helmfile/codefresh/pull-request.yml#L82

Erik Osterman

That’s really odd. I can’t explain it.

2019-01-20

Igor Rodionov

Codefresh in on thoughtworks’ tech radar

Igor Rodionov
Tools | Technology Radar | Thoughtworks

The Technology Radar quadrant that explores the latest tech trends in the tools software developers use to create, debug, maintain, and support applications