#release-engineering (2020-07)

jenkins_ci All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.

CI/CD Discussions

Archive: https://archive.sweetops.com/release-engineering/

2020-07-24

2020-07-23

wannafly37 avatar
wannafly37

Trying to decide where to store helm values and ci/cd information for charts; Am thinking deploy to staging on merge to master and deploy to prod on github release, but that feels cumbersome with a monorepo due to having to filter on paths and put names in releases, but it feels a little wrong to have an entire repo with just 2 files in it for helm deploys.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Have you considered #helmfile?

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

With helmfile you can easily pull values stored from tons of locations. Disk, buckets, terraform state, vault, SSM, etc.

wannafly37 avatar
wannafly37

Thanks, I hadn’t - helmfile looks interesting but not sure how it’d help my scenario. Right now I’m looking at doing named github release tags [email protected] for example, and having the pipeline parse the tag to know what to run. This way all files could live in the same repo, the other “easier” option is a repo for each, but in the external-dns example the repo would literally have just a values file and the yaml for the pipeline

shamil.kashmeri avatar
shamil.kashmeri

helmfile can help with that, and, you can do that with helm-git

shamil.kashmeri avatar
shamil.kashmeri

helm-git if youre talking about deploying charts from a git repo with tags as chart versions

2020-07-21

2020-07-20

2020-07-19

Zach avatar

Folks using azure devops - with the self hosted build agents, is there a way to do this (in AWS preferrably) so that the agent is created ‘on demand’ as part of an EC2 ASG or Fargate task? We have a few very long running, and expensive compute resource jobs currently in jenkins, i wouldn’t want to just leave those instances running 24/7 if were to migrate to Azure DevOps. Their docs seem more focused on just having some permanent pool of these agents up.

Mark avatar

So what I have done quite recently which might cater for you is add an agent to our bastion/control node to be the interface for azure pipelines to interact with.

I used Terraform and for the aws_instance resource there is a property called user_data which will execute a custom cloudconfiguration script which actually creates an agent for us with a name controlled by a variable in the pipeline. So in the next job of the pipeline, I was able to set the agent dynamically based on the pipeline variable.

Zach avatar

ah ha! yes thats exactly the sort of thing I was wanting to do

Zach avatar

So you have 1 “permanent self-hosted agent” always running on a cheap node, and you use that one to spin up the others

Zach avatar

And all the agents need a public IP correct?

Mark avatar

Yeah pretty much. When I need a new environment I first use the azure provided agent to create my new agent (terraform). From then, it’s over to my agent to do whatever work it needs to do.

pool:
    name: terraformed-envs
    demands: env -equals $(TF_VAR_ENV_NAME)

Actually a fair callout regards the public ip, the instance does have one but I never thought it required a public ip, it has one for a different reason. I would imagine it would require a public ip though given the networking across IAAS’s.

Zach avatar

I assume you have some logic in the ‘init agent’ job to wait for the ‘build agent’ to actually become available?

Zach avatar

re: public ip, I think the Azure docs said the self-hosted agents have to be reachable on 443 to communicate status back

:--1:1
Mark avatar

Yeah there is classic polling logic intermediary

Mark avatar

does this agent exist…………….

Zach avatar

Ah, so you poll the azure api

Zach avatar

Super helpful thanks - this proves out some issues we had with dumping jenkins in favor of other solutions

Mark avatar

Yeah given our whole ecosystem.. this is one of the smaller tradeoffs I’m dealing with I believe we did something similar dumping jenkins for this

2020-07-17

2020-07-16

Zach avatar

what exactly are the limits of running self managed gitlab CE in our cloud environment? I can’t seem to find a good document on CE vs EE

Joe Niland avatar
Joe Niland

I have a client using it. Are there any specific limitations you’re concerned about?

Zach avatar

Ah well I finally found the page, just wasn’t googling well enough https://about.gitlab.com/pricing/self-managed/feature-comparison/

Self-Managed Feature Comparison attachment image

Code, test & deploy with GitLab. Everyone can contribute!

Zach avatar

Mostly just looking for whether there was some critical “oh this won’t work for you” feature as a CI/CD tool

Zach avatar

And it looks like the CE version won’t connect with external repos (github or bitbucket)? So that’s a non starter

Joe Niland avatar
Joe Niland

I see. I’ve built a few pipelines on CE and haven’t hit any limitations. But they are all in on GitLab CE.

Joe Niland avatar
Joe Niland

Yeah looks like Premium+ only

Zach avatar

hmm we might only need the starter plan anyways if we just hosted with them. We’re deep into Jira already so I don’t need their project/issue management piece

Joe Niland avatar
Joe Niland

I think they’re fine for git and ci/cd. The Kubernetes integration is pretty awesome.

Zach avatar

I’m trying to get us off of a (failing) jenkins setup but we have a few ‘special cases’ we’re trying to find compatibility for, and we’re all EC2 ASG deployments right now

Zach avatar

Kubernetes probably way more complicated than my org needs

Joe Niland avatar
Joe Niland

What are the special cases and do you use awscli to deploy?

Zach avatar

we have a few ‘jobs that trigger other jobs’ pipelines for a couple of apps, and some large/long running tests. We’ve been looking at travis, circle and semaphore so far. They’re close but don’t get us 100% without a lot of pain. Our deploys are AMIs into ASGs and done via terraform, when there’s a new AMI we B/G the entire ASG

Zach avatar

there’s some aspects to it I’m working to change because there’s a lot of pain in a few places

Joe Niland avatar
Joe Niland

What is the pain exactly? Lately worked on triggering other pipelines within Azure DevOps.

roth.andy avatar
roth.andy

There’s a bunch of good stuff in Starter that you don’t get in CE. One of the biggest ones for me is with CE there is really no way to make someone wait for a green pipeline run before merging an MR. There is also no way to require code review approvals. That alone is worth $4/dev/month to me.

Zach avatar

Yah I think the starter would work, I’m kicking off a trial now

Zach avatar

we’re just evaluating a lot of options right now

Zach avatar

@Joe Niland we B/G the entire system at once, with a pretty lengthy release cycle. There’s some historical reasons for that but I”m trying to change it. It requires a lot of manual effort (automated via terraform but its not true “CID”) due to some decisions on how that B/G would be accomplished, which are now difficult to engineer back out

Joe Niland avatar
Joe Niland

@ makes sense.

Zach avatar

Can’t tell if I’m reading the gitlab docs correctly, they seem to be a bit of a messy overlap between hosted/self-hosted …. If using hosted gitlab, I can use a self-hosted runner in my own cloud? Are these able to be spun up on demand by integration with gitlab?

Zach avatar

Private go modules are the devil with hosted CI services

2020-07-15

roth.andy avatar
roth.andy
01:04:49 AM

I am the Jenkins Master!!!!

jenkins_ci2
3
1
Joe Niland avatar
Joe Niland

Nice!

bradym avatar
bradym

very cool

wannafly37 avatar
wannafly37

Is this JenkinsX?

roth.andy avatar
roth.andy

No. Regular Jenkins OSS with the blue ocean plugin

Jonathan Marcus avatar
Jonathan Marcus

What are those zPlaceholder nodes?

roth.andy avatar
roth.andy

There’s a bug in the blue ocean UI where if some stages run in parallel and some don’t all of the stages will appear sequentially rather than being tiered.

roth.andy avatar
roth.andy

My fix was to just have all stages set to parallel and add a 2and placeholder stage to each one that only has 1 stage in it

Zach avatar

“Working as intended, workaround is available” Jenkins Team — probably

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@johncblandii (Cloud Posse)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I haven’t seen that bug. I’ve used s - s - p - s - p before, but maybe that was because it was CloudBees which was a lot more solid than OSS

2020-07-14

Zach avatar

Anybody using SemaphoreCI? Or have opinions on it?

2020-07-08

2020-07-07

Andrey Nazarov avatar
Andrey Nazarov

Any tips how you effectively switch off some functionality on dev/staging? Like you don’t want to send emails to real users from dev and staging envs or something similar (btw there is a nice service called Mailtrap to achieve this, but sending emails is not the only case). Recently we’ve got a requirement that a certain file with a cryptic name should be created by hand on every production instance (a set of services) to be sure that production functionality can’t be accidentally activated on dev or staging and env vars are not even an option.

Tim Birkett avatar
Tim Birkett

Why are env vars not an option? How are your applications / services configured? Typically, you’d have feature flags in your applications config to turn things on and off. You should just be able to configure services differently throughout environments.

Andrey Nazarov avatar
Andrey Nazarov

Almost everything is configured via env vars, sure. But some of the functionality is disabled/locked down if a certain file doesn’t exist (there is a method in the code verifying this), and there are people who don’t want to consider anything else, because they think env vars can be easily messed up.

The obvious idea is to have dev functionality by default and only if some certain flags are set explicitly the production features are activated. But we failed to agree upon the env vars approach.

I was wondering if somebody has such strict requirements somehow and possible solutions like double verifications or something. This manual file creation is unsuitable for dynamic infrastructures and seems a bit odd.

roth.andy avatar
roth.andy

feature flags

Lee Skillen avatar
Lee Skillen

Be careful with dev-by-default. If you release to production but have failed to override the defaults, you might end up with something like (extreme example), your production instances running in debug mode. We take the opposite approach, and everything is (endpoints aside) configured for prod-by-default. Basically the default configuration is a reasonable best practice, then we have a separate dev and staging configuration baseline. The way that developers run the service makes it load the dev baseline, and they can customise it based on environment, set by the individual developer. In terms of turning on/off functionality; a lot of prod-like things just won’t work until configured because we’ve separated the app from the services it requires. Local devs run those services alongside it. Production is already running them as separate layers, and is configured as part of IaC.

:100:1
Andrey Nazarov avatar
Andrey Nazarov

Thanks for the clarification! Yes, I was wrong in my definition. It was not literally “dev-by-default”, but let’s call it “reasonable defaults” when all stuff like debug modes are deactivated and so on.

:--1:1

2020-07-06

joshmyers avatar
joshmyers

joshmyers avatar
joshmyers
Add GHE functionality by joshmyers · Pull Request #19 · cloudposse/github-commenter

what Adds Github Enterprise functionality why So we can use the awesome github-commenter with GHE. testing Running the thing now returns the comment ID rather than memory address: 2020/07/06 13:30:…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks! LGTM - gonna have @Andriy Knysh (Cloud Posse) review

Add GHE functionality by joshmyers · Pull Request #19 · cloudposse/github-commenter

what Adds Github Enterprise functionality why So we can use the awesome github-commenter with GHE. testing Running the thing now returns the comment ID rather than memory address: 2020/07/06 13:30:…

joshmyers avatar
joshmyers

Cheers

joshmyers avatar
joshmyers

@Andriy Knysh (Cloud Posse) Let me know if there is anything I can do to help get that released

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

looking now

joshmyers avatar
joshmyers

Thank you so much!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Merged - release 0.7.0

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

images are building

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/github-commenter

Command line utility for creating GitHub comments on Commits, Pull Request Reviews or Issues - cloudposse/github-commenter

joshmyers avatar
joshmyers

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thanks josh for the contribution! glad you were able to use our github-commenter ! =)

joshmyers avatar
joshmyers

Using it for a conftest Atlantis integration along with github-status-updater :–1:

joshmyers avatar
joshmyers

Thanks folks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That’s rad!

joshmyers avatar
joshmyers
07:44:37 AM
joshmyers avatar
joshmyers

github-status-updater updates the status to failed so is no longer mergeable, Atlantis won’t apply.

2020-07-03

Karoline Pauls avatar
Karoline Pauls
10:35:07 AM

If anyone is interested in the Jenkins follow-up.

Karoline Pauls avatar
Karoline Pauls

in general case - beware of solutions that look BEST on a comparison chart - they may tick every single feature but the chart won’t tell you how well those features are integrated

Karoline Pauls avatar
Karoline Pauls

https://issues.jenkins-ci.org/browse/JENKINS-62250?focusedCommentId=390581#comment-390581

this for example - the plugin-based architecture means there’s no strong unifying design

Karoline Pauls avatar
Karoline Pauls

if you push too hard, things show their limits, mostly manifesting as stack traces

Karoline Pauls avatar
Karoline Pauls

too many builds in a matrix - Groovy codegen failure

need a button to deploy any branch at any time (worked fine on GitLab) - if you use a global agent, it will remain in use… because the job is technically paused, you have to use agent none

testing a docker container built in a previous step? - great, how do you tag it? BRANCH-HASH-BUILD_ID - great, BUILD_ID will change if you restart a stage. BRANCH-HASH? - great, but you still need a global agent, otherwise you won’t be able to use the HASH part in a k8s agent pod template…

Oh, you’re a smart one, like me - you’ll do agent any to fetch the repo and set GIT variables and nest agent kubernetes inside, which can use git variables in its template! Congrats, now put it in a build matrix, so you can more reliably trigger the deadlock you’ve just constructed!

I’d really like to see https://github.com/laszlocph/woodpecker to take off. I don’t have time to write my own CI, although I’ve written a scheduler and dashboard once before.

laszlocph/woodpecker

An opinionated fork of the Drone CI system. Contribute to laszlocph/woodpecker development by creating an account on GitHub.

Karoline Pauls avatar
Karoline Pauls

also how does it manage to take several minutes to start is beyond me and i’m scared to strace it

2020-07-02

2020-07-01

sweetops avatar
sweetops

Anyone know if it’s possible to use github branch protections to ensure that a PR to branch-a must come from branch-b?

sweetops avatar
sweetops

i’m thinking not

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes & no. You can trivially create a github action that checks the source branch and target branches.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You can fail the build if they aren’t what you want them to be.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then use github branch protections to require this github action build to pass before merging..

sweetops avatar
sweetops

okay, i’m with you.

sweetops avatar
sweetops

I guess it’s finally time for me to explore github actions

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’ll be well worth it for these simple things.

    keyboard_arrow_up