#codefresh (2019-01)

CLI can now use your local machine as the volume for more verbose debugging.

--local-volume	--lv		Use your file system as volume in local run

Wondering if anyone solved/hit these:

1) discarding concurrent builds - imagine I want to have pipeline triggered on git commit, it takes 1 hour to run, but if any other build of the pipeline is triggered we only keep the currently building one and the most recent one, all others are discarded.

2) pausing pipeline - if we have pipeline triggered on commit or on cron - is there a way to manually pause the schedule? (other than deleting the trigger? since deleting/re-creating trigger is involved). Looking for something similar to Jenkins enable/disable pipeline button

3) what do people use for sequential/unique build numbering - if we wanted to generate semantic versions for our builds?

i’ve wanted to have “locks”

so we can have a pipeline block untili it obtains the lock.

i’ve seen semver services

e.g. you can curl an API for a given service and specify if you want a new path, minor or major release

e.g. curl <https://semver.host.io?service=foobar&patch+=1>

interesting… not exactly what I wanted but makes sense

one of our problems is with helm versioning

basically different projects have pinned to different versions of our release harness

which mean different versions of helm

one option is to always check the server and then use that version locally

on an totally unrelated note, I was surprised to see how many people we’ve found who like make

make is the new hotness, until you get too deep into it. Is it a really cool project these days if it isn’t go & make ?

haha

yea, stay on the surface

the deep waters of make are dangerous

Likely not useful, but wrote https://github.com/joshmyers/dynolocker for a quick distributed lock before

oh that’s cool!

yea, that’s basically what I wanted

you sir got your first

@dustinvb have you seen #kubecost? I think this could add some extra bling to your demos.

They have a dashboard that shows exactly how much a deployment costs plus makes it easy to “right size” pods.

Bah that link requires login LOL.

Have you seen the new metrics tab on the bottom of your builds we pushed yesterday?

The main one shows cumulative and you can also see each step there or click on the step and get more details.

That link is just medium being medium. There’s a X in the upper right corner that closes at dialog. Stupid medium.

Also our first version of our behind the firewall agent has been released. https://github.com/codefresh-io/venona You can deploy to your Kubernetes cluster and use as a custom runtime.

Dustin how is that different from using the codefresh cli deploy the run time environment?

That requires you allow ingress from Codefresh SaaS

This would be where you don’t want to expose your Kubernetes API to internet.

That’s badass

Sorry API proxy is coming soon. This allows for the runtime to be behind the firewall in our Hybrid scenario.

–

we’ve released our helmfile to install the codefresh runtime environment using our monochart

https://github.com/cloudposse/helmfiles/blob/master/releases/codefresh-re.yaml

@mumoshu has joined the channel

We also have venona now which does behind the firewall runtimes with no ingress.

https://github.com/codefresh-io/venona

Cool, we will do the same for that

The manual approval, step is due out next week. You’ll already see in Slack there is an option to send notifications about approval builds and a new option on the build screen to the right of pending for builds waiting approval.

That will be awesome!!

So an engineer can kick off the deployment by cutting a release in github

that triggers a codefresh pipeline

which requests an approval via slack to the engineering manager

if he approves it, then the deployment proceeds.

Yes but permissions are not available yet around the process but the pipeline is stamped with the user who approved/denied.

@mgrube Join us here.

Ah I see you, didn’t say you joined.

Clearly, I am a ninja

With that professional incognito profile pic I would say that is 100% the case.

https://codefresh.io/docs/docs/codefresh-yaml/steps/approval/

This is awesome!

not sure if you guys have investigated how Travis does their remote exec

but it’s via tmate

I’ve reproduced the functionality with this code snippet

this assumes tmate binary is in your path

if you run this snippet, you will get this output

I haven’t personally nor heard of anything about this.

then I can exec into the container

so how does this relate to codefresh?

You should tell me.

LOL

so if a container step you add sleep inf, that step won’t finish

if the step before that runs the script I shared, then it starts a reverse tunnel out to [tmate.io](http://tmate.io)

https://codefresh.io/docs/docs/troubleshooting/common-issues/access-and-debug-the-pipeline-volume-image/

then I can effectively debug my codefresh container running on codefresh

ok, this is cool

We need to think about this as another option and see about a new Fresh Step

that’s cool!

i didn’t know about the composition terminal

though that won’t apply to pipelines, right?

general pipelines

so travis has a button that basically says, on next run, start a tmate session

i think in the codefresh world, it would almost be like setting a break point

only that the container is left running

@michal.matyjek @Daren what do you think about something like this?

HOLY MOLY! I have brilliant idea

I’m going to add tmate/session as a target in our build-harness

then just add make tmate/session in any pipeline step for debugging

and it will output a temporary session

in the build logs

it will block indefinitely until the tmate session exits or a preconfigurable timeout

@daveyu

I’ve invited Francisco here from my team. I think this debugging utility talk will interest him.

We just ran into something today this would have aided in. I’d like to get this integrated into the pipeline like we do with cf_export.

This kind of feature was really useful when I used CircleCi

Codefresh in on thoughtworks’ tech radar

https://www.thoughtworks.com/radar/tools

nice!

I’m trying to deploy to a new k8s cluster, and thinking I missed a step somewhere when provisioning it. any clues here? yaml:

title: Deploy with helmfile
stage: Deploy
image: 'cloudposse/build-harness:${{BUILD_HARNESS_VERSION}}'
working_directory: ./
environment:
  - 'IMAGE_TAG=${{SEMVERSION_COMMIT_SHORT}}'
  - 'RELEASE_NAME=${{NAMESPACE}}'
  - APP_API_ROOT=https://${{BACKEND_APP_HOST}}'
  - 'AWS_ACCESS_KEY_ID=${{CHAMBER_ACCESS_KEY_ID}}'
  - 'AWS_SECRET_ACCESS_KEY=${{CHAMBER_SECRET_ACCESS_KEY}}'
commands:
  - make helm/toolbox/upsert
  - chamber exec app -- helmfile sync

log output:

Pulling image cloudposse/build-harness:0.15.3                                                                  
Digest: sha256:48379abbffe7707ce8185839c24ad20830e45fec7092edbf4febf6b2e859df98                                
Status: Image is up to date for cloudposse/build-harness:0.15.3                                                
ERROR: kubectl not installed!                                                                                  
make: *** [//build-harness/modules/helm/Makefile:24: helm/toolbox/upsert] Error 1

that looksl ike a really old version of the build-harness

newer versions ship kubectl

oh snap

that’s the latest

I can take a look in a few hours

about to give a demo of this https://github.com/cloudposse/example-app

Blue/Green deployments with Codefresh and Istio and Helm

cool. i’ll poke around.

it uses our “cartrige” style deployment methodology where everything is bundled in the deployment artifact (aka docker image)

to get unblocked, you should be able to do:

make packages/install/kubectl

as a build step

@daveyu i think it should be a geodesic image

image: cloudposse/geodesic:${{GEODESIC_VERSION}}

Though if he does that, then he needs to install the build-harness

so it’s neither here nor there

split into two steps

yea, could possibly do that

thanks! i’ll try those options

fyi this was the example i followed, and it did work before with the same build-harness version. https://github.com/cloudposse/statup/blob/add-helmfile/codefresh/pull-request.yml#L82

That’s really odd. I can’t explain it.

I am trying to deploy an application, with the image being pulled from cfr. I added a Secret to the helm chart so that the chart would deploy everything needed for the application, but it doesn’t create the secret first. What is the best practice for uploading the secret?

good question.

so there are a few ways to handle this.

If you’re not going to be using a lot of namespaces, it would be most secure to deploy the pull secret manually as a “one off”

If you’re using kops, you can provision a cluster-wide pull secret

If you’re using helmfile, then you can deploy it by adding a release of this: https://github.com/cloudposse/charts/tree/master/incubator/dockercfg

Mock deploy a secret using our Services page: https://codefresh.io/docs/docs/deploy-to-kubernetes/manage-kubernetes/#deploy-new-service

Use Codefresh CLI: https://codefresh-io.github.io/cli/more/image-pull-secret/

These work with any registries you have connected to Codefresh.

Maybe too many options?

haha

@dustinvb would be nice if there was a build-step or something we could use to automatically provision the pull secret for a namespace

You can use the CLI as a build step.

lol, this is all good stuff. So if I am going to have a namespace per dev team or developer, would it be best to have a helm chart that configures the namespace with all secrets needed?

the downside with provisioning secrets with helm is that they are in clear text if you run helm get values

and are also surfaced in clear text in the helm UI in codefresh

thats good to know

Bah I am all off on my explanation… I mis read what you’re trying to do.

Use helm-secrets plugin

That’s what I used

the helm-secrets plugin doesn’t fix it

the values are still passed in clear-text to helm

Yes they are indeed but on the Kubernetes side it will create the secret for you using the Helm Chart.

hrmm

or do you mean helm-secrets will create the Secrets resource out-of-band with helm, so that it won’t be visible by helm?

If you don’t want these secrets showing up in a pipeline it is best to circumvent the secret creation in CI/CD something else like https://github.com/Boostport/kubernetes-vault

It is still visible by Helm Releases and the command in the pipeline.

@dustinvb check this out: https://github.com/mumoshu/aws-secret-operator

@mumoshu wrote this! I can’t wait to use it. no need to deploy vault.

Nice!

Yeah for AWS this looks great. I don’t have enough access in AWS to create an example of this. If you ever have a pipeline with a helm chart using this I’d like to check it out.

True… AWS solution.

Mark, do any of these options look viable for your use case?

I think for now just to get unblocked i might create manually. https://github.com/mumoshu/aws-secret-operator looks pretty cool in the long term though

Thanks you guys!

Anyone here used codefresh api to start pipelines? The swagger at codefresh.io/api is not very clear…. or FWIW - anyone integrated Spinnaker with Codefresh?

Our CLI is built off of the API. https://github.com/codefresh-io/cli/blob/master/lib/logic/api/pipeline.js#L116-L165

I’ll get your feedback to the team about the API.

@dustinvb https://www.reddit.com/r/Terraform/comments/ajggfw/mask_sensitive_data_in_output_from_terraform_plan/

Could be useful if any codefresh customers are using terraform

@Dan Garfield

@Dan Garfield has joined the channel

https://codefresh.io/docs/docs/whats-new/whats-new/

that’s some pretty sweeet feature velocity

Is it possible to trigger a pipeline via a webhook, and have the pipeline product an artifact that the caller has access to, say a json file

ohhhh some artifact scoped to the CF_API_KEY?

I don’t know the answer. If they did, it would probably related to this new feature: https://codefresh.io/docs/docs/testing/test-reports/

@francisco said you can create a named context, then used a (currently) undocumented api call to obtain it. A cronjob pipeline can be used to clean them up

is the API call related to the test reports?

ohhhh i think i understand now

you have a pipeline that writes an artifact to the shared volume

and you want to know if you could read that artifact from that volume using the codefresh cli?

(and apparently this is possible using that undocumented call)

Example of writing a file to context. https://github.com/codefresh-io/eks-installer/blob/master/scripts/codefresh-save-tfstate.sh

Using that at a later date https://github.com/codefresh-io/eks-installer/blob/master/scripts/codefresh-load-tfstate.sh

Oh this is cool

Example API call to get this context from another system: https://github.com/codefresh-io/cli/blob/master/lib/logic/api/context.js#L50-L60

Since you get the build id when you trigger a pipeline, I’d suggest setting that in the file name and then you’ll be able to have a unique per build file.

This is awesome! https://steps.codefresh.io/

It would be cool if there was a way to define like a step.yaml in a repo with sufficient annotations that codefresh could just populate this automatically.

I second that ^^^

Anyone know how to nest an array in cfstep-helm value override? For reference I am trying to set the extraHostPathMounts for the kiam helm chart https://github.com/helm/charts/blob/master/stable/kiam/values.yaml#L182

have you looked into helmfile?

We use this exclusively with helm

helmfile takes the pain out of dealing with values.yaml and helm releases.

plus makes it easier to make helm more 12-factor compliant

@Igor Rodionov @Maxim Mironenko (Cloud Posse) https://github.com/codefresh-contrib/cfstep-pipeline-creator

@Maxim Mironenko (Cloud Posse) has joined the channel

How does this compare to https://github.com/cloudposse/codefresh/tree/initial-implementation

We’ve updated our example-app with our pipelines for unlimited staging, automatic destroy, release promotion, and multiple cluster deploys using tags

https://github.com/cloudposse/example-app

This also has an example of blue/green deploy’s using Istio

we use helmfile with helm

we use variant as a “deploy cli”

@daveyu

what’s notable about these updated pipelines is we avoid the classic “main clone” in every pipeline other than build

this makes it faster

the build artifact always contains the helmfile to deploy. this is an evolution of our statup demo (which is now out of date)

@dustinvb I want to add now all the clair scanning and other kinds of tests

https://steps.codefresh.io/ should have the most up-to-date documentation some testing steps requires the user to work with vendor to install server side.

https://github.com/mkj28/smashing-codefresh

Nice for office wallboards!

Cool!

this is by @michal.matyjek

http://status.codefresh.io/incidents/qswt2278fjx7

thanks for the heads up

thankfully at this very moment, not building.

Wonder if we can just integrate the status page into the channel

done

Incident reported earlier has been resolved.

Wierd, we didn’t get the all lear message from the atom feed?

504s and Latency on Builds and Pipeline pages in UI Jan 31, 20:14 UTC Resolved - The issue has been identified and resolved. It was due to a high load on the system. We scaled it up and took the corresponding actions to prevent further issues. Jan 31, 20:14 UTC Monitoring - We’ve resolved the issue and are actively monitoring the system. Jan 31, 19:02 UTC Investigating - Our Team is investigating reports with 504 Gateway errors and latency in Codefresh pages.

That took a bit…

Yea, guess it doesn’t poll too often

GitHub Incident Jan 31, 22:11 UTC Resolved - GitHub has reported this incident as Resolved. More information here: https://www.githubstatus.com/ Jan 31, 22:10 UTC Update - We are continuing to work on a fix for this issue. Jan 31, 22:01 UTC Identified - GitHub has reported an ongoing incident. This could affect your builds. More information here: https://www.githubstatus.com/