#release-engineering (2020-05)
All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.
CI/CD Discussions
Archive: https://archive.sweetops.com/release-engineering/
2020-05-01

@Zachary Loeber @Erik Osterman (Cloud Posse) thanks for rec on gomplate
2020-05-05

I still cannot believe why someone thought that the default, unchangable behaviour of multibranch pipelines in jenkins should be to build all discovered branches on project creation.

Clean up your branches ya’ll!

But yeah, stampede of builds is a pain, you can change that behaviour though…

I never clean my branches. There is no reasonable reason for that

Yes there is. delete merged branches, it keeps your git history much cleaner

exceptions:
Bamboo UX shittiness we had in maybe 2014 - where in order to build a branch we had to scroll through thousands of branches in a dropdown.
Jenkins buildstorm shittiness now….

So there are at least 2 reasons

neither of them good

makes me happy that both github and gitlab have options to auto-delete merged branches

@loren we’ve been using this feature since it was introduced, but it seems to work 50/50 for me

has it been working for you?

on github, what i’ve noticed is that it works perfectly if i merge the pr that i opened. if someone else merges it, then it does not auto-delete

aha! maybe that’s it.

so we have required reviews and status checks, and after approved review, we usually defer to the teammate to merge

of course, that’s a bit trickier with community prs where the contributor does not have permissions, but then that branch is on their fork at least

oh, and this chrome extension is the total bomb! before github added the feature, this extension did it (and sooo much more)… https://github.com/sindresorhus/refined-github
Browser extension that simplifies the GitHub interface and adds useful features - sindresorhus/refined-github

I never-ever wanted that and i’ve never met anyone who would.

Yeah, Jenkins is a bit but it is a known quantity of
2020-05-07

jenkins update:
• configuring it is more difficult than writing a CI from scratch
• NEVER EVER do plugin-based architecture in your projects. It is begging for short-sightedness in design. Plugin composition will compose implicit assumptions, in case of Jenikins way out of date. Right now I’m getting a buildstorm each time I reboot Jenkins. Disabling branch scanning triggering builds disables webhooks. It’s a house of mirrors and everything is completely backwards, bolted-on, etc.
Some tasks/projects are taxing to one’s sanity. This is one of them. Compared to that, porting a large amount of badly tested Python 2 code to Python 3 with no real integration tests was relatively pleasant.

Sadly, we have no money for GitLab. But we do have money for me to waste time on Jenkins.


automating Jenkins is an afterthought to an afterthought

Configuration-as-code is a guessing game.

it’s impossible to configure projects other than with job-dsl. Last time i tried it, I couldn’t get it to acknowledge the existence if an already existing credential (by ID).


@johncblandii @Steve Boardwell this buildstorm problem might come to bite us too

I’ve been there. A buildstorm can trigger a lot of issues due to scaling needs as it’ll push the current node(s) to the brink which can cause a large scale up with a generally slow scale down.
It is a pain in the neck to manage!

Sorry. A little late to the party.
Agree with all the above, although we have been able to mitigate the problem somewhat by reducing the number of allowed branches (design decision though so may not fit everyone’s needs).
We chose to allow a single development branch, short-lived release branches, along with PRs for changes. We use branch filtering to stop other branches not matching develop
or release-x.y.z[.*]
being discovered, along with the BuildStrategyPlugin to build said branches only (also using it to stop PR’s from automatically building but that’s not scope of this).
This ultimately means that we only have branches which “should” be built anyway, with PR’s for anything else. As a side-effect the branches have been build already which means they don’t fall into the Jenkins restart/repository index scan trap because the git SHA is not null
as with non-built branches.

@johncblandii has joined the channel

@Karoline Pauls You seen https://issues.jenkins-ci.org/browse/JENKINS-57588 ?

I’ve seen all of them at this point

heh, times are hard when you fall into the depths of 3 year old Jenkins issues

eugh.

I’ve just tried to reproduce the build storm by killing a pod. It didn’t work this time.
Now I updated some plugins, I’ll see if this will trigger it.

annoyingly, it didn’t

so i don’t know what caused it in the morning

Hello, we provide CI for many teams in a big company. Unfortunately, we have a huge problem with server loads after the spin up process, after restarts and jobs modification. Multibranch jobs execu…

what’s most punishing is the “prison-grade” UX of credentials. It takes some 5 clicks to do the basic thing

and the sad part is that it’s all backwards because no one really wants to build old branches
2020-05-08

kubernetes will scale pods but those builds are useless in the first place
2020-05-09

this kind of sucks, devs largely want build-on-push
if you stop branch discovery from discovering branches, it will also filter out webhooks. Dumb but true.

I suppose it depends what you want to test. PRs are also build on push

For us, the only branches worth building/testing were either the main branches (develop, master, release-*, etc) or branches which are going to be merged into another branch. Devs are able to create branches with incomplete work without worrying about failing builds, since a build is only triggered once a PR is created, etc.
When testing a PR, we are also able to test the actual merged state as opposed to the feature branch in question, which may or may not have been rebased against the latest base branch.
Swings and roundabouts with pros and cons on both sides really. We found this method easier to handle in the end.

Everyone is saying “it depends what you want” but in different projects somehow I never had a problem with the way GitLab works.

Yes definitely. I wasn’t saying it works - it definitely doesn’t , just offering an approach/work-around which helped mitigate the problem.

what about dynamically detecting buildstorms and pausing everything with the web API? (there must be a web API right?)
2020-05-10

Hi guys! Glad to see you. I hope you are well So, I would like to ask you about a secrets delivery for application. I am using the approach where CI/CD process deliveries secrets from Secrets Storage (Vault, AWS Secret Manager, etc) to environment variables for application, then application uses these secrets. But guys from my team suggest to use approach where an application gets secrets from Secrets Storage by itself. I am not sure that it is right way, so I want to ask you about that. What the approach do you use for credentials delivery?

I like the env approach as others have mentioned, but from strictly a security point-of-view, environment variables aren’t the best option because an attacker with access to the node and inspect the environment of a process in /proc/$pid/env

However, accessing the secret storage directly introduces new problems - complicating local development and requiring more things running under docker compose

you can still use vault with envs (see envconsul
) and I think that’s a happy medium, but still suffers from the aforementioned problem.

Thank you, Eric for your explanation! Yeah, security is a pain point for any cases :)

I prefer env variables. It makes it easier to switch out the secrets storage since the application wouldn’t need to change.

Thank you! yep! It was my first thought too)
2020-05-11

So I assumed that an input
stage in a jenkins pipeline stage with agent none
does not occupy an executor while waiting for input. How wrong I was. To prevent it from idling executors, it needs a when
block with beforeAgent true
. If there is no condition you’d want in when
, you have to add a condition that always evaluates to true, e.g. expression { true }
.

it still doesn’t work, maybe because there’s a top-level agent that i needed to access GIT variables…

It may only work if the main agent block is none
. The main block defines the default agent to be used. That means that all the other stages need the agent set if they use one.

Not a very elegant solution though.

i’ve just confirmed that

i had to copy my env vars to 5 places

Can you set the env vars in a function with env.MY_VAR=bla
? This should set them globally without using the environment block mutliple times.

I’m not aware of the ability to set global environment in a declarative pipeline.

Create a function outside the pipeline block def setEnvs() { env.MY_VAR=... }
and call it in one of the first stages. Alternatively, you should also be able to use a script block script { ...env.MYVAR=bla.... }

I use this section environment{}
for defining all global envs for all stages in declarative pipeline

The script step is a way to allow scripted pipeline code when declarative doesn’t provide the solution ootb.

can i create a top-level function, outside of script
in a declarative pipeline?
is declarativeness just a convention?

also, i think i’ll still have to call setEnv everywhere to make it work with stage restarts

Declarative has a DSL to test conformity and give structure. It doesn’t limit you to just declarative though.

it’s soooo annoying that you cannot get the git hash without checking out

Git hash can be obtained from body of webhook

how do you obtain the body of the webhook?

Can receive any HTTP request, extract any values from JSON or XML and trigger a job with those values available as variables. Works with GitHub, GitLab, Bitbucket, Jira and many more. - jenkinsci/g…

i’d rather stick to what blue ocean does by default

You using a pipeline job or multibranch?

multibranch pipeline

And it doesn’t provide the git info? I would have expected it to. Have you checked the git branch source plugin docs?

you need the checkout scm step (or in declarative: no skipDefaultCheckout()
) to have any env vars

Yes, that was it. Why do you need the git hash before checkout?

i want to set it globally, without allocating a global agent

actually, there seem to be currentbuild
and scm
global objects

Even before checkout? If after checkout suffices, set top level agent to none, run Checkout stage with agent of your choice, after checkout run script { env.GIT_HASH=... }
to set it for the rest of the pipeline.

again, I don’t think this will work with re-running steps

I see your point. Forgot the restart at stage bit.
Last chance, according to https://www.jenkins.io/doc/book/pipeline/running-pipelines/#restart-from-a-stage all the build info will stay the same so you could have a function which conditionally runs scm checkout if workspace is empty, or something along those lines.

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

i think the easiest way would be to build self with parameters

but hacky

very. If you really need to avoid starting an agent, I would look adding the checkoutIfNeeded
method to run the checkout scm
step only if needed. My guess is that it should work, and you are not messing around with the general flow of things.

Good luck anyway

I need a ton of luck because essentially everything is breaking all the time in ways requiring at least 5 sentences to describe.

I should have just written my own CI, i shit you not. I’ve done something similar once

lol. Is Jenkins a requirement? Can you explain your CI requirements? Maybe we can come at it from another angle.

i posted them a while before

basically, on one hand we got spoiled by gitlab which generally does everything right, on the other we cannot afford to use it company-wise, on the other (third) hand, some business people want to be able to use the UI to run jobs, eh

no one ever got fired for picking jenkins

Makes sense. And build-wise, what do you want to happen?

which build-wise?

I have repoA I want a job to… I want the following things to happen for (a) branches, (b) PR’s, etc

at the moment I’m getting hudson.remoting.ProxyException: groovy.lang.MissingPropertyException: No such property: POD_LABEL for class: WorkflowScript
which is an arcane error message that in the past happened when a pipeline was triggered from another but wasn’t passed some parameters that didn’t have default values.
I am “lucky” that i debugged it once, so now i know what may be wrong. It was excruciating to narrow it down in the first place.

I have a project repo. I want to build a docker image, upload it to the “testing” ECR repository, run tests in parallel utilising that image, then on success push that image to the “production” ECR repo, later have parallel manual stages for (1) testing branch rollout and (2) production deploy, which wait for use input. The “testing branch rollout” stage should have a “delete testing rollout” stage afterwards, which is not dependent on the prod rollout but that’s not really possible in Jenkins because it’s not DAG-based.
I want the input steps not to consume an executor, which i solved as in https://issues.jenkins-ci.org/browse/JENKINS-62250.
Since the input step’s “Abort” button aborts the entire pipeline, I added a “SKIP” checkbox that’s later checked in when
, so you can tick “SKIP” and press “Proceed”. Sadly when having 2 of such jobs in parallel, the Blue Ocean UI locks up if you skip one and doesn’t let you deploy or skip the other.

the deploy steps trigger a subpipeline in the deploy repo and it mostly works, except for the arcane error message if you accidentally pass a null value that was required

Have you thought about using the build
step (https://www.jenkins.io/doc/pipeline/steps/pipeline-build-step/) to run individual jobs? The Jenkinsfile could be the orchestrator (timeouts for stages, etc) with the jobs handling the tasks. If not for all the steps, it would presumably help for the inputs, etc.

I do use the build step to trigger deployments in the deployment project

there is simply no answer to this, basically the Jenkins architecture is all wrong because if everything is a plugin, there’s not broader picture to how they’re developed. It’s begging for short-sightedness.

Yes, I expect you’re right.

I got that working eventually, but that’s the n-th time i change one thing to see 3 others break. Be that a cautionary tale.

Glad to hear it. How did the solution with the parallel inputs look in the end?


the deploy repo should complain if they’re selected

Hi all Jenkins experts) I set up jenkins using helm/stable to my k8s cluster. All executors are spawned each time for new job. Sometimes my jobs are blocked for 4-5 minutes with status “waiting for the next available executor”, but any jobs are running at this moment. Looks like this is some timeout, but I do not know, what is it Does someone face this issue ?

to some degree, yes

Check the jenkins logs. The k8s plugin fails silently with some things (i think it was an error in my yaml declaration when I had something similar)

if there is an error in yaml, it doesn’t eventually succeed

Wasn’t sure from the OP that it did. Could also be the resources with new nodes spinning up, etc. Would still check the logs and the k8s agent pods. Maybe waiting for something.
2020-05-12

It is quite interesting, there is a “set environment variables” step that the declarative pipeline inserts. If i do an SCM checkout explicitly (and skip the default one), there are no GIT_* variables.
2020-05-13

Ha! Talking of repository scanning, someone just commented on this ticket I created last year https://issues.jenkins-ci.org/browse/JENKINS-56878
2020-05-14


@Andriy Knysh (Cloud Posse) just shared with me that GitHub Actions finally has organizational secrets!!

it’s not GitHub Actions per se, it’s all Secrets (they could be used for GH Actions)

Can the secrets be read by things other than actions? Like GitHub applications?

oh sorry, looks like you can create a secret from an app, but you can read only the encrypted value of it https://developer.github.com/v3/actions/secrets/

only GitHub Actions (for now) can decrypt the secrets

Ah okay. I’m waiting for them to allow Gitub Applications to read the secrets :)


Oh that’s awesome
2020-05-15

yes!!

big news on org secrets

now if they will just allow us to rerun successful builds (for implementing/testing purposes) that’d be awesome
2020-05-22

Hey team

has anyone here done any setup of Jenkins on K8’s?

Yes, we are using the Jenkins operator

you have any caveats to using it? Scaling agent workloads, things like that?

It’s been non trivial to run on Kubernetes

@johncblandii and @Jeremy G (Cloud Posse) are leading the charge

I worked with a client once that told me he slammed a cluster with a build once via an uncontrolled Jenkins agent

@Callum Robertson are you setting it up on EKS?

@Callum Robertson we used https://github.com/jenkinsci/kubernetes-operator to deploy Jenkins. There are some caveats/issues with it being a “perfect” deploy, but overall it works and, once running, is pretty easy to update.
I’ve also used CloudBees so installed it with https://docs.cloudbees.com/docs/cloudbees-jenkins-distribution/latest/distro-install-guide/kubernetes#install-cjd-helm.
Kubernetes native Jenkins Operator. Contribute to jenkinsci/kubernetes-operator development by creating an account on GitHub.
CloudBees simplifies the installation of CloudBees Jenkins Distribution on Kubernetes by providing you with a Helm chart.

The Jenkins operator is only “alpha” quality right now and is likely to evolve significantly before being production-ready. Jenkins itself has been around a long time and is showing signs of old age, but is battle-tested and full of options. Still, I would take a look at alternative CI/CD systems like GitHub actions.
Any CI/CD system you deploy in your cluster is going to have the potential to slam your cluster if not properly configured or managed. This is why most people run them on dedicated, isolated clusters.

native master on K8’s and agents, etc