#release-engineering (2020-05)

Clean up your branches ya’ll!

But yeah, stampede of builds is a pain, you can change that behaviour though…

I never clean my branches. There is no reasonable reason for that

Yes there is. delete merged branches, it keeps your git history much cleaner

exceptions:

Bamboo UX shittiness we had in maybe 2014 - where in order to build a branch we had to scroll through thousands of branches in a dropdown.

Jenkins buildstorm shittiness now….

So there are at least 2 reasons

neither of them good

makes me happy that both github and gitlab have options to auto-delete merged branches

@loren we’ve been using this feature since it was introduced, but it seems to work 50/50 for me

has it been working for you?

on github, what i’ve noticed is that it works perfectly if i merge the pr that i opened. if someone else merges it, then it does not auto-delete

aha! maybe that’s it.

so we have required reviews and status checks, and after approved review, we usually defer to the teammate to merge

of course, that’s a bit trickier with community prs where the contributor does not have permissions, but then that branch is on their fork at least

oh, and this chrome extension is the total bomb! before github added the feature, this extension did it (and sooo much more)… https://github.com/sindresorhus/refined-github

I’ve been there. A buildstorm can trigger a lot of issues due to scaling needs as it’ll push the current node(s) to the brink which can cause a large scale up with a generally slow scale down.

It is a pain in the neck to manage!

Sorry. A little late to the party.

Agree with all the above, although we have been able to mitigate the problem somewhat by reducing the number of allowed branches (design decision though so may not fit everyone’s needs).

We chose to allow a single development branch, short-lived release branches, along with PRs for changes. We use branch filtering to stop other branches not matching develop or release-x.y.z[.*] being discovered, along with the BuildStrategyPlugin to build said branches only (also using it to stop PR’s from automatically building but that’s not scope of this).

This ultimately means that we only have branches which “should” be built anyway, with PR’s for anything else. As a side-effect the branches have been build already which means they don’t fall into the Jenkins restart/repository index scan trap because the git SHA is not null as with non-built branches.

I suppose it depends what you want to test. PRs are also build on push

For us, the only branches worth building/testing were either the main branches (develop, master, release-*, etc) or branches which are going to be merged into another branch. Devs are able to create branches with incomplete work without worrying about failing builds, since a build is only triggered once a PR is created, etc.

When testing a PR, we are also able to test the actual merged state as opposed to the feature branch in question, which may or may not have been rebased against the latest base branch.

Swings and roundabouts with pros and cons on both sides really. We found this method easier to handle in the end.

Everyone is saying “it depends what you want” but in different projects somehow I never had a problem with the way GitLab works.

Yes definitely. I wasn’t saying it works - it definitely doesn’t , just offering an approach/work-around which helped mitigate the problem.

I like the env approach as others have mentioned, but from strictly a security point-of-view, environment variables aren’t the best option because an attacker with access to the node and inspect the environment of a process in /proc/$pid/env

However, accessing the secret storage directly introduces new problems - complicating local development and requiring more things running under docker compose

you can still use vault with envs (see envconsul) and I think that’s a happy medium, but still suffers from the aforementioned problem.

Thank you, Eric for your explanation! Yeah, security is a pain point for any cases :)

Thank you! yep! It was my first thought too)

It may only work if the main agent block is none. The main block defines the default agent to be used. That means that all the other stages need the agent set if they use one.

Not a very elegant solution though.

i’ve just confirmed that

i had to copy my env vars to 5 places

Can you set the env vars in a function with env.MY_VAR=bla? This should set them globally without using the environment block mutliple times.

I’m not aware of the ability to set global environment in a declarative pipeline.

Create a function outside the pipeline block def setEnvs() { env.MY_VAR=... } and call it in one of the first stages. Alternatively, you should also be able to use a script block script { ...env.MYVAR=bla.... }

I use this section environment{} for defining all global envs for all stages in declarative pipeline

The script step is a way to allow scripted pipeline code when declarative doesn’t provide the solution ootb.

can i create a top-level function, outside of script in a declarative pipeline?

is declarativeness just a convention?

also, i think i’ll still have to call setEnv everywhere to make it work with stage restarts

Declarative has a DSL to test conformity and give structure. It doesn’t limit you to just declarative though.

it’s soooo annoying that you cannot get the git hash without checking out

Git hash can be obtained from body of webhook

how do you obtain the body of the webhook?

Using plugin https://github.com/jenkinsci/generic-webhook-trigger-plugin

i’d rather stick to what blue ocean does by default

You using a pipeline job or multibranch?

multibranch pipeline

And it doesn’t provide the git info? I would have expected it to. Have you checked the git branch source plugin docs?

you need the checkout scm step (or in declarative: no skipDefaultCheckout() ) to have any env vars

Yes, that was it. Why do you need the git hash before checkout?

i want to set it globally, without allocating a global agent

actually, there seem to be currentbuild and scm global objects

Even before checkout? If after checkout suffices, set top level agent to none, run Checkout stage with agent of your choice, after checkout run script { env.GIT_HASH=... } to set it for the rest of the pipeline.

again, I don’t think this will work with re-running steps

I see your point. Forgot the restart at stage bit.

Last chance, according to https://www.jenkins.io/doc/book/pipeline/running-pipelines/#restart-from-a-stage all the build info will stay the same so you could have a function which conditionally runs scm checkout if workspace is empty, or something along those lines.

i think the easiest way would be to build self with parameters

but hacky

very. If you really need to avoid starting an agent, I would look adding the checkoutIfNeeded method to run the checkout scm step only if needed. My guess is that it should work, and you are not messing around with the general flow of things.

Good luck anyway

I need a ton of luck because essentially everything is breaking all the time in ways requiring at least 5 sentences to describe.

I should have just written my own CI, i shit you not. I’ve done something similar once

lol. Is Jenkins a requirement? Can you explain your CI requirements? Maybe we can come at it from another angle.

i posted them a while before

basically, on one hand we got spoiled by gitlab which generally does everything right, on the other we cannot afford to use it company-wise, on the other (third) hand, some business people want to be able to use the UI to run jobs, eh

no one ever got fired for picking jenkins

Makes sense. And build-wise, what do you want to happen?

which build-wise?

I have repoA I want a job to… I want the following things to happen for (a) branches, (b) PR’s, etc

at the moment I’m getting hudson.remoting.ProxyException: groovy.lang.MissingPropertyException: No such property: POD_LABEL for class: WorkflowScript which is an arcane error message that in the past happened when a pipeline was triggered from another but wasn’t passed some parameters that didn’t have default values.

I am “lucky” that i debugged it once, so now i know what may be wrong. It was excruciating to narrow it down in the first place.

I have a project repo. I want to build a docker image, upload it to the “testing” ECR repository, run tests in parallel utilising that image, then on success push that image to the “production” ECR repo, later have parallel manual stages for (1) testing branch rollout and (2) production deploy, which wait for use input. The “testing branch rollout” stage should have a “delete testing rollout” stage afterwards, which is not dependent on the prod rollout but that’s not really possible in Jenkins because it’s not DAG-based.

I want the input steps not to consume an executor, which i solved as in https://issues.jenkins-ci.org/browse/JENKINS-62250.

Since the input step’s “Abort” button aborts the entire pipeline, I added a “SKIP” checkbox that’s later checked in when , so you can tick “SKIP” and press “Proceed”. Sadly when having 2 of such jobs in parallel, the Blue Ocean UI locks up if you skip one and doesn’t let you deploy or skip the other.

the deploy steps trigger a subpipeline in the deploy repo and it mostly works, except for the arcane error message if you accidentally pass a null value that was required

Have you thought about using the build step (https://www.jenkins.io/doc/pipeline/steps/pipeline-build-step/) to run individual jobs? The Jenkinsfile could be the orchestrator (timeouts for stages, etc) with the jobs handling the tasks. If not for all the steps, it would presumably help for the inputs, etc.

I do use the build step to trigger deployments in the deployment project

there is simply no answer to this, basically the Jenkins architecture is all wrong because if everything is a plugin, there’s not broader picture to how they’re developed. It’s begging for short-sightedness.

Yes, I expect you’re right.

I got that working eventually, but that’s the n-th time i change one thing to see 3 others break. Be that a cautionary tale.

Glad to hear it. How did the solution with the parallel inputs look in the end?

i merged 2 steps into one, allowing for mutually exclusive options

the deploy repo should complain if they’re selected

to some degree, yes

Check the jenkins logs. The k8s plugin fails silently with some things (i think it was an error in my yaml declaration when I had something similar)

if there is an error in yaml, it doesn’t eventually succeed

Wasn’t sure from the OP that it did. Could also be the resources with new nodes spinning up, etc. Would still check the logs and the k8s agent pods. Maybe waiting for something.

it’s not GitHub Actions per se, it’s all Secrets (they could be used for GH Actions)

Can the secrets be read by things other than actions? Like GitHub applications?

oh sorry, looks like you can create a secret from an app, but you can read only the encrypted value of it https://developer.github.com/v3/actions/secrets/

only GitHub Actions (for now) can decrypt the secrets

Ah okay. I’m waiting for them to allow Gitub Applications to read the secrets :)

Yes, we are using the Jenkins operator

you have any caveats to using it? Scaling agent workloads, things like that?

It’s been non trivial to run on Kubernetes

@johncblandii and @Jeremy G (Cloud Posse) are leading the charge

I worked with a client once that told me he slammed a cluster with a build once via an uncontrolled Jenkins agent

@Callum Robertson are you setting it up on EKS?

@Callum Robertson we used https://github.com/jenkinsci/kubernetes-operator to deploy Jenkins. There are some caveats/issues with it being a “perfect” deploy, but overall it works and, once running, is pretty easy to update.

I’ve also used CloudBees so installed it with https://docs.cloudbees.com/docs/cloudbees-jenkins-distribution/latest/distro-install-guide/kubernetes#install-cjd-helm.

The Jenkins operator is only “alpha” quality right now and is likely to evolve significantly before being production-ready. Jenkins itself has been around a long time and is showing signs of old age, but is battle-tested and full of options. Still, I would take a look at alternative CI/CD systems like GitHub actions.

Any CI/CD system you deploy in your cluster is going to have the potential to slam your cluster if not properly configured or managed. This is why most people run them on dedicated, isolated clusters.