#release-engineering (2020-07)

it’ll be well worth it for these simple things.

in general case - beware of solutions that look BEST on a comparison chart - they may tick every single feature but the chart won’t tell you how well those features are integrated

https://issues.jenkins-ci.org/browse/JENKINS-62250?focusedCommentId=390581#comment-390581

this for example - the plugin-based architecture means there’s no strong unifying design

if you push too hard, things show their limits, mostly manifesting as stack traces

too many builds in a matrix - Groovy codegen failure

need a button to deploy any branch at any time (worked fine on GitLab) - if you use a global agent, it will remain in use… because the job is technically paused, you have to use agent none

testing a docker container built in a previous step? - great, how do you tag it? BRANCH-HASH-BUILD_ID - great, BUILD_ID will change if you restart a stage. BRANCH-HASH? - great, but you still need a global agent, otherwise you won’t be able to use the HASH part in a k8s agent pod template…

Oh, you’re a smart one, like me - you’ll do agent any to fetch the repo and set GIT variables and nest agent kubernetes inside, which can use git variables in its template! Congrats, now put it in a build matrix, so you can more reliably trigger the deadlock you’ve just constructed!

I’d really like to see https://github.com/laszlocph/woodpecker to take off. I don’t have time to write my own CI, although I’ve written a scheduler and dashboard once before.

also how does it manage to take several minutes to start is beyond me and i’m scared to strace it

Thanks! LGTM - gonna have @Andriy Knysh (Cloud Posse) review

Cheers

@Andriy Knysh (Cloud Posse) Let me know if there is anything I can do to help get that released

looking now

Thank you so much!

Merged - release 0.7.0

images are building

https://github.com/cloudposse/github-commenter/runs/846123382

thanks josh for the contribution! glad you were able to use our github-commenter ! =)

Using it for a conftest Atlantis integration along with github-status-updater

Thanks folks

That’s rad!

github-status-updater updates the status to failed so is no longer mergeable, Atlantis won’t apply.

Why are env vars not an option? How are your applications / services configured? Typically, you’d have feature flags in your applications config to turn things on and off. You should just be able to configure services differently throughout environments.

Almost everything is configured via env vars, sure. But some of the functionality is disabled/locked down if a certain file doesn’t exist (there is a method in the code verifying this), and there are people who don’t want to consider anything else, because they think env vars can be easily messed up.

The obvious idea is to have dev functionality by default and only if some certain flags are set explicitly the production features are activated. But we failed to agree upon the env vars approach.

I was wondering if somebody has such strict requirements somehow and possible solutions like double verifications or something. This manual file creation is unsuitable for dynamic infrastructures and seems a bit odd.

feature flags

Be careful with dev-by-default. If you release to production but have failed to override the defaults, you might end up with something like (extreme example), your production instances running in debug mode. We take the opposite approach, and everything is (endpoints aside) configured for prod-by-default. Basically the default configuration is a reasonable best practice, then we have a separate dev and staging configuration baseline. The way that developers run the service makes it load the dev baseline, and they can customise it based on environment, set by the individual developer. In terms of turning on/off functionality; a lot of prod-like things just won’t work until configured because we’ve separated the app from the services it requires. Local devs run those services alongside it. Production is already running them as separate layers, and is configured as part of IaC.

Thanks for the clarification! Yes, I was wrong in my definition. It was not literally “dev-by-default”, but let’s call it “reasonable defaults” when all stuff like debug modes are deactivated and so on.

Nice!

very cool

Is this JenkinsX?

No. Regular Jenkins OSS with the blue ocean plugin

What are those zPlaceholder nodes?

There’s a bug in the blue ocean UI where if some stages run in parallel and some don’t all of the stages will appear sequentially rather than being tiered.

My fix was to just have all stages set to parallel and add a 2and placeholder stage to each one that only has 1 stage in it

“Working as intended, workaround is available” Jenkins Team — probably

@johncblandii

I haven’t seen that bug. I’ve used s - s - p - s - p before, but maybe that was because it was CloudBees which was a lot more solid than OSS

I have a client using it. Are there any specific limitations you’re concerned about?

Ah well I finally found the page, just wasn’t googling well enough https://about.gitlab.com/pricing/self-managed/feature-comparison/

Mostly just looking for whether there was some critical “oh this won’t work for you” feature as a CI/CD tool

And it looks like the CE version won’t connect with external repos (github or bitbucket)? So that’s a non starter

I see. I’ve built a few pipelines on CE and haven’t hit any limitations. But they are all in on GitLab CE.

Yeah looks like Premium+ only

hmm we might only need the starter plan anyways if we just hosted with them. We’re deep into Jira already so I don’t need their project/issue management piece

I think they’re fine for git and ci/cd. The Kubernetes integration is pretty awesome.

I’m trying to get us off of a (failing) jenkins setup but we have a few ‘special cases’ we’re trying to find compatibility for, and we’re all EC2 ASG deployments right now

Kubernetes probably way more complicated than my org needs

What are the special cases and do you use awscli to deploy?

we have a few ‘jobs that trigger other jobs’ pipelines for a couple of apps, and some large/long running tests. We’ve been looking at travis, circle and semaphore so far. They’re close but don’t get us 100% without a lot of pain. Our deploys are AMIs into ASGs and done via terraform, when there’s a new AMI we B/G the entire ASG

there’s some aspects to it I’m working to change because there’s a lot of pain in a few places

What is the pain exactly? Lately worked on triggering other pipelines within Azure DevOps.

There’s a bunch of good stuff in Starter that you don’t get in CE. One of the biggest ones for me is with CE there is really no way to make someone wait for a green pipeline run before merging an MR. There is also no way to require code review approvals. That alone is worth $4/dev/month to me.

Yah I think the starter would work, I’m kicking off a trial now

we’re just evaluating a lot of options right now

@Joe Niland we B/G the entire system at once, with a pretty lengthy release cycle. There’s some historical reasons for that but I”m trying to change it. It requires a lot of manual effort (automated via terraform but its not true “CID”) due to some decisions on how that B/G would be accomplished, which are now difficult to engineer back out

@Zach makes sense.

Can’t tell if I’m reading the gitlab docs correctly, they seem to be a bit of a messy overlap between hosted/self-hosted …. If using hosted gitlab, I can use a self-hosted runner in my own cloud? Are these able to be spun up on demand by integration with gitlab?

So what I have done quite recently which might cater for you is add an agent to our bastion/control node to be the interface for azure pipelines to interact with.

I used Terraform and for the aws_instance resource there is a property called user_data which will execute a custom cloudconfiguration script which actually creates an agent for us with a name controlled by a variable in the pipeline. So in the next job of the pipeline, I was able to set the agent dynamically based on the pipeline variable.

ah ha! yes thats exactly the sort of thing I was wanting to do

So you have 1 “permanent self-hosted agent” always running on a cheap node, and you use that one to spin up the others

And all the agents need a public IP correct?

Yeah pretty much. When I need a new environment I first use the azure provided agent to create my new agent (terraform). From then, it’s over to my agent to do whatever work it needs to do.

pool:
    name: terraformed-envs
    demands: env -equals $(TF_VAR_ENV_NAME)

Actually a fair callout regards the public ip, the instance does have one but I never thought it required a public ip, it has one for a different reason. I would imagine it would require a public ip though given the networking across IAAS’s.

I assume you have some logic in the ‘init agent’ job to wait for the ‘build agent’ to actually become available?

re: public ip, I think the Azure docs said the self-hosted agents have to be reachable on 443 to communicate status back

Yeah there is classic polling logic intermediary

does this agent exist…………….

Ah, so you poll the azure api

Super helpful thanks - this proves out some issues we had with dumping jenkins in favor of other solutions

Yeah given our whole ecosystem.. this is one of the smaller tradeoffs I’m dealing with I believe we did something similar dumping jenkins for this

Have you considered #helmfile?

With helmfile you can easily pull values stored from tons of locations. Disk, buckets, terraform state, vault, SSM, etc.

Thanks, I hadn’t - helmfile looks interesting but not sure how it’d help my scenario. Right now I’m looking at doing named github release tags [email protected] for example, and having the pipeline parse the tag to know what to run. This way all files could live in the same repo, the other “easier” option is a repo for each, but in the external-dns example the repo would literally have just a values file and the yaml for the pipeline

helmfile can help with that, and, you can do that with helm-git

helm-git if youre talking about deploying charts from a git repo with tags as chart versions