#general (2020-02)

Thanks for the shout out! Really enjoyed the zoom session yesterday - will be back next week!

Thanks for the welcome! Hello

Thanks @Mike Martin! Ya, yesterday’s #office-hours was a good one. See you next week!

Hey @Zack Hewison!

Thank you! Happy to join!

Thanks for letting me in. I am still on my learning journey. so please be kind :)

@Tom Howarth all skill levels welcome!

and as a result still red zone for twilighting as per Googles norman operation procedures

This was very concisely said, I try and formulate such a sentence when someone throws Kubernetes into the conversation, even them having never operated it

probably a copy & mistake problem, but — is wrong

should be apk --update --no-cache helm2@cloudposse etc..

https://github.com/cloudposse/build-harness/blob/master/modules/packages/Makefile

When using this target, you would do something like make packages/install/helm HELM_VERSION=....

Hmm

I’m adding a few more binaries to build harness from private GitHub repo which was working fine while on .27 but helm 2.14 didn’t work for us anymore

I was basing off of the buildharness image and baking a new one after using the private GitHub release target

aha

gotcha

Now I just want to make sure helm2 is used

As “helm”

Bumping to .33 defaults to helm3 :(

ok, so yes, you’re on the right track. You’ll probably want to uninstall helm@cloudposse

and then use helm2@cloudposse

Ok, makes sense because helm3 is fully unlinked so I might as well get rid of it properly and I don’t have to muck around with update-alternatives directly then

Thanks!

ya, and when you start moving to helm3, we have a helm3@cloudposse package so you can keep both installed

Cool cool - were deprecating helm though so probably won’t do that

what are you moving towards?

not a text renderer for structured documents

i can understand for your own apps, but if you depend on any open source charts, you’re undertaking a monumental effort and an exercise in how to manage technical debt at a colossal scale

How many charts move to helm3?

i would never want to manage prometheus, grafana, kiam, cert-manager and the 2 dozen other charts

all helm2 charts are compatible with helm3

Hmm maybe I don’t have to version pin helm at all then

Didn’t test it, haven’t looked into helm3

but you need need to “upgrade” helm2 releases to “helm3” releases

or uninstall (with helm2) / reinstall (with helm3)

Overall I have had mostly headaches from the upstream helm-charts repo

Oh we don’t use tiller

Only render and send manifests to kube through ArgoCD

aha

So probably don’t need to do anything then

ok, so the “package” concept of helm is not something you’re leveraging

You mean the releases through config maps?

if we think about helm being like rpm , then what you’re doing is like:

which is fine, but then it’s like managing a linux distro without any package manager

it’s not really the same thing … think about it more like spinnaker helm bake phase

i don’t think of helm as a template system

i think of it as a package system

Helm shouldn’t try to play CD

i guess it’s a perspective on where the “CD” starts and ends.

Which depends on how much control your other tools need over traffic routing within and automated rollover

that’s a fair point.

Why shouldn’t helm do CD? Arguably that is its entire purpose.

For most deployments it works perfectly fine. If you are doing some nuanced rollout then of course you will need to plan out a strategy first and consider an appropriate tool to suit your requirements.

To me, helm primarily solved the issue of managing related Kubernetes resources which are tightly coupled and have few variable configurations across different use cases

True that for the examples listed above (externalDNS / cluster autoscaler / monitoring agents / … ) the Deployment strategies are fairly simple to let it handle by Helm. helm 2 was horrible security and rbac wise due to the tiller component though.

requiring more fine grained patching of custom paths to handle secret management as per company requirements (vault service accounts / sealed secrets / chamber entry points) … helm charts just become overly generic …

We’re currently doing the last mile customisation (as some ppl call it) with some custom patching on TOP of the library of open source “packages” which often contain operational experience of the many contributors to it

But I’ve also found that many helm charts are lift and shifted into kube early on and completely missing latest evolution of the tools they package actually adopting Kube api to leverage a lot of the distributed complexity issues - so sometimes I strip 90% of the stable chart (and I do create issues and have my charts public for ppl who search for the same search terms and come across my comments)

I disliked tiller a whole lot as well, it made pipelines stupid hard to work with and maintain any level of security. But that’s really no longer relevant. I’m not wholely advocating helm for everything but I’ve come to accept it as a tool in my belt for certain tasks.

given the above issues:

• outdated charts with slow adoption of consumer patches
• last mile customisations highly dependent on individual use cases either resulting in overly generic charts of inflexible charts
• poor handling of modern CD practices (it’s designed for VERY simple upgrade strategies)

I’ve found that almost every public chart needs some form of manipulation to suit my client’s needs as well. Pulling them down into localized chart repos is all to common but also is technically part of best practice of eliminating outside dependencies anyway.

See also istio deprecating it’s official helm charts, it’s basically a consensus in the community

Pulling down with poorly documented patching leaves you in a much bigger mess of managing upgrades

We still heavily leverage helm, albeit some of my colleagues do it very begrudgingly

Not true at all. Pulling random charts out in the wild into production clusters leaves you in a far more likely position to create technical debt though.

And we are trying hardest to get rid of it (mostly for internal deployments for now)

in a modern CICD pipeline, how do you share common elements across dozens of projects effectively?

Istio also doesn’t define the industry standards

though it is really exciting and fun stuff, I see a very small portion of businesses adopting it

sorry, I’m playing devil’s advocate here because I’ve had a few people categorically make some of the same claims you have based purely on prior distaste for the tool.

I’m one of those people that begrudgingly use it and see its value in the right places.

adopting what? Service meshes?

I also don’t push secrets through it at all

I do pull models for that on my clusters so I flat out work around some of the stickier elements of using it.

I’m also not categorically against it, but I’ve adopted severe abused helm charts with undocumented patches from upstream

helm, sry

And my colleagues who inherited this from the person I replaced hate it a lot more than me

The fact so many projects have been packaged with Helm shouldn’t lock us into it though

I think fundamentally we kind of agree :)

right, so for now, if you have to package a common set of deployment code for shared use across multiple development efforts what well thought out tools are there to use?

kustomize looks pretty cool. That’s one I should look into?

for our internal deployments we integrated templates into a binary to avoid abuse

ugh… must be a better way

Now that kustomize was pushed into kubectl … it’s idd something that seems to solve our patching issue

Myeah I’m not 100% with it either, more so because I really want a structure aware template that doesn’t fall over a missing space

And simply catches syntax errors way earlier

I think jsonnet + kustomize

But haven’t played with enough myself

TF .12 HCL is quite interesting, just migrated all resources not managed by kube to that - I was very sceptical at first

I never tried TF to manage kube deploys … I don’t think the TF model fits kube control loops model?

I do think my friend’s company moved or TF to manage to kube deployments though

If you always know your cluster state then tf is ideal for it I’d think.

I’m still spitting out base clusters and configuring them after the fact via stacks of helmfiles for baseline config

I used to pre-create namespaces and shove secrets into the cluster via TF though. I’m not against using it for kube deployments but then I have to connect developer pipelines to terraform which kind of scares me (probably more than it should)

I’m now leaning heavily into the deployment/environment git repo approach that will allow me to leverage gitops if I like (or even just stand-alone pipeline as code in the repo itself to do push deployments)

My direct supervisor kind of designed an internal replacement for helmfile+helm with that binary I mentioned

• it can fetch and render helm charts
• it overlays patches in a crude way (not as fine grained as kustomize)
• it takes a map like helmfile to do all of the above
• it pushes to a sync branch which ArgoCD tracks
• it does basic cluster bootstrap and sets up argoCD tracking for all other deploy repos (team specific) related to the cluster

And it has those compiled in templates for non-helm internal microservices which have defaults and some globally + locally overwritable values

your super sounds like a person I could get along with.

how many repos/deployments are you supporting?

(for what its worth I’m eyeing over jx for some of this for opinionated deployment into kube, they have some pretty sound ideas around teams/projects/environments)

I haven’t heard about jx- I have to say the design of the tool is mostly by my supervisor Chris Kolenko - I helped implement the functionality … I know he’s dying to talk publicly about it but we haven’t had the time yet to finalise it - currently handling about 3 Brandings of the same set of microservices across dev / test / staging / prod - so 12 repos of that and 2 cluster repos per env (for ephemeral cluster upgrades, we just stand up a new version cluster, boostrap it and switch over) we are mid way of this migration while also adding new business requirements all the time… so it’s very in flux, cluster repos retire when clusters do

It’s not so big and already quite complex, we come from a monorepo for deployments that was using templated helmfile templating values for helm templates - a templating nightmare rendered by argoCD with zero insight in obscure error messages

helm templates templating more helm templates?

Sorry for digging (and feel free to not answer) but is it just you two handling all that?

I work mostly in a bubble as of late so hearing how other devops teams are operating is like my drug or something.

It grew from a team of DevOps and Dev (outsourced) to an in house team being built, i was 3rd to join 6 months ago and team is now 5 “devops”

It’s not our job title just trying to keep things simple

I’m currently supporting a mostly offshored team of developers for approximately 45 repos across 4 teams and have had anywhere from 3 to 8 environments (each with their own fully loaded kube clusters and pipelines) at once.

I converted all their pipelines to pipeline as code early on or there’s no way I’d be able to take on such load.

@Zachary Loeber are you also using flux?

(i forget - you told me before)

I’m curious about this one: Use DNS-based service discovery (instead of IPs or depend on consul); use short-dns names with search domains rather than FQHN.

firstly I think you meant FQDN

but aside from the typo and the fact that I currently follow this practice I’m wondering about cross-cluster services and such.

consul is being considered for a springboot microservice deployment (they currently use Zuul or something like that) and I’m under the impression that using Consul may be able to help with high availability and service discovery across regions and such

but that’s more of a hunch than that I’ve actually deployed such a construct

Maybe adding that configuration parameters should avoid environment ‘stamping’ would be a good thing to add (though there is likely a better way to word that)

this is kind of like the URL or URI ambiguity. IMO, FQHN is “fully qualified hostname” and “hostnames” are for hosts.

vs FQDN is a “fully qualified domain name”, which is the domain for the hosts. basically, it’s a zone. Like [us-west-2.prod.example.co](http://us-west-2.prod.example.co), whereby the FQHN might be [api.us-west-2.prod.example.co](http://api.us-west-2.prod.example.co)

I get your point about consul ; it’s not wrong. it’s also like using ASM (aws secrets manager). do you embed it in your code? that’s how you get maximal benefit from it, but also, maximally limit how/where it can operate. is the secrets interface external or intrinsic to the application logic?

I think what’s interesting about the new hashicorp service mesh based on consul is how it’s implemented with sidecars (like most service meshes)

this means your application does not need to be aware of that implementation detail. it simplifies local development but still works well in a complex multi-cloud environment

actually, the way i initially interpreted Health checks should not depend on the health of the backing services is to not alert based on cause, but i’m not really sure what this sentence means the more I look at it

Nevermind it is what I thought it was https://12factor.net/backing-services

my point is services should gracefully degrade. So if a service depends on the health of it’s backing services, and the backing services, go down, then my service would go down. this is a great way to create site-wide blackouts in a heartbeat.

also, my my service can’t talk to one database, but it can to another? what should happen. i think the service is still “healthy” & “ready”, but might return 503 for requests that are impacted.

the downstream service should then decide on how to handle the 503

@Erik Osterman (Cloud Posse) so maybe have specific 500-series responses based on backing service outages? And a monitoring service with an intelligent ruleset would check the most upstream service and be able to detect these issues?

ideally your app should be aware of both when it is healthy, and when it is able to serve requests based on its backing services - if i can’t write to a database then i shouldn’t be trying to take requests, especially if there is a node that can

yea, definitely have monitoring/escalations on this; my point is more that what do we want kubernetes to do when the upstream backend is offline and my service depends on it? do we want kubernetes to kill my service (restart the pod, fail service health checks), or keep it online. i argue that since the problem is not my service, we should keep my service online (by passing health checks), even if my service might respond with 50x to other API requests (e.g. ``GET /healthz = 200, but POST /api/delete/user = 503` )

if i can’t write to a database then i shouldn’t be trying to take requests, especially if there is a node that can

health check should say - “i’m all good, don’t try and restart me” readiness check should say - “i can take requests, send me some traffic!”

but is it that binary (black/white)?

so if 1% of functionality is impacted by not being able to reach some upstream, should my service stop accepting requests?

if you can gracefully degrade then you are “ready”

but you should be able to say “i’m running fine, but i can’t service traffic” without getting all your pods killed

especially helpful for avoiding a startup storm

yea, agree with that

hmmm that’s an interesting point @Erik Osterman (Cloud Posse). Maybe that’s why microservices have a better place on k8s than monoliths never thought about it but now I have an argument for it

like if you had bs-monolith-app and it does a bunch of crap, including processing images and writing them back to S3, and in the impossible case that S3 goes down, it’ll lose 10% of its functionality

versus, if you had bs-image-processing-service and bs-document-processing-service , then the backing service going down will completely screw up the former but not the latter

and in this case bs-image-processing-servicecan just say “I CANT DO ANYTHING” and its pods will get constantly recreated, the entire service will flap and the SREs/Admins/operators will see that much more clearly than it just losing 10% of its functionality

I’ve heard the jargon “blah blah microservices kubernetes” countless times before and took it blindly but never pictured this very convincing scenario

Maybe backend services are one of a few criteria used for scoping the roles of microservices. As some of us may know, we may get a head of ourselves and make arbitrarily small microservices, sometimes known as nanoservices, where the overhead of managing them doesn’t meet the value of having them scoped that specifically in the first place

yea, this is the decoupling that we want to achieve between services. how that decoupling is defined, is a bit arbitrary.

especially helpful for avoiding a startup storm yea, this is a good real-world example. a common problem I’ve seen is a Java app (for example) on startup will throw an exception if it cannot connect to the database and exit. this can lead to a startup storm that @Chris Fowles talks about. you have all these services in a crash loop until the database comes back up. then the database comes online, and everything slams it all at once, taking out the database. then the java apps crash again.

eventually, the crashloop backoffs reduce the effects of the storm and the system comes back online (if you’re lucky)

oh so startup storm = thundering herd

i thought startup storm had something to do with startup companies

This looks cool. Could help with my “devtools” container. Wonder how it compares to the “alternatives” way @Erik Osterman (Cloud Posse) mentioned

Still looking into it but it’s pretty generic. Mostly just scripts with shims (reminds me of the oh-my-zsh plugin system but targeted towards various app versions of cli tools instead)

Yep. That’s the thing it has felt like Brew has always been missing.

only reason I looked into it was zero desire to deal with installing a specific version of golang on a new(ish) linux laptop of mine. I’d used gvm before for such task

or pyenv for python

programming languages suck like that right?

I just used nvm in dadsgarage. It would be great to standardize with something like this

anyway, the asdf tool plugin list was WAY larger than I expected

but it works more like direnv

yeah, it even has stuff like Terraform and Helm

looking for a .tools-version file and sourcing the right shim if found

Apart from the fact that git is made as a decentralised version control system and you could theoretically pull and push directly between computers or a central repo on a server over ssh, you could setup gittea or gitlab and configure a repo to sync another repo on both pull and push. (ps. bonus points if you make a working helm chart for gittea :))