#general (2020-02)

General conversations related to DevOps/Automation

General Discussions

2020-02-29

SweetOps avatar
SweetOps
08:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Victor Wong

Good to have you here =)

2020-02-28

James Huffman avatar
James Huffman

got a question for anyone using GCP. if you have an app running under a service account in project A, but you want to access resources in project B (from that app), what’s the best way to do this? i’ve seen API keys, service account impersonation, etc. thrown out there. curious what people are actually doing in real applications, though.

SweetOps avatar
SweetOps
08:00:16 PM

Hey everyone, give a warm welcome to our newest members!

  • @Jawwad Yunus
  • @Castro Mbithii
  • @nishgupta29
  • @randomy
  • @Adam Perry
  • @Marcin Brański
  • @Tan Quach

Good to have you here =)

1

2020-02-27

erik-stephens avatar
erik-stephens

Wondering if something about chart versions I’m not grok’ing… I’ve seen some charts where its version looks intentionally locked/coupled to its appVersion . Seems like a lot of potential to muck with the semver in my helm values. Hasn’t happened to me personally yet, but in a forked chart that I maintain, I’m debating whether to break that coupling to appVersion. Curious if anyone hear has thoughts on that.

RB avatar

github is having issues. does anyone setup a readonly internal mirror for when this happens?

Nikola Velkovski avatar
Nikola Velkovski

Nah, the alternative is much better, pack your things and go home

1
RB avatar

im a remote engineer, so im already home.

RB avatar

ive been googling and havent seem to find anything. maybe github up time has been so good that no one had to worry abou tit

Nikola Velkovski avatar
Nikola Velkovski

We had it once in a company

Nikola Velkovski avatar
Nikola Velkovski

I dont remember the details since it was setup before I came

loren avatar
loren

i usually have all my repos pretty up-to-date locally, and write tests and build scripts in a way they can be run locally instead of only through CI

Nikola Velkovski avatar
Nikola Velkovski

but it was a cron that synced one critical repo

RB avatar

sounds super bespoke

Nikola Velkovski avatar
Nikola Velkovski

should be straight forward

RB avatar

ah ok. i was hoping for something more cookie cutter, off-the-shelf. basically i give it our enterprise org name and an auth token and it grabs all our repos and has an endpoint

loren avatar
loren

could run a cron that just loops through your local clones and runs git fetch periodically?

RB avatar

yea but then thats a solution only for myself

loren avatar
loren

ahh

loren avatar
loren

well gitlab has some pretty great webhook based options to mirror from github

loren avatar
loren

every push to github is then synced to gitlab

1
Nikola Velkovski avatar
Nikola Velkovski
GitHub Status

Welcome to GitHub’s home for real-time and historical data on system performance.

RB avatar

lol… it was causing issues this morning for us on the east coast so i was looking at alternatives in case it happens again

RB avatar

would be nice to have an artifactory like approach for git* service providers like github

RB avatar

so you first clone/pr/commit/etc to the mirror, which then redirects the changes upstream to github. if github is having issues, it won’t stop developers. we can simply wait for github to come back, and resync our mirror with upstream

Roderik van der Veer avatar
Roderik van der Veer

Apart from the fact that git is made as a decentralised version control system and you could theoretically pull and push directly between computers or a central repo on a server over ssh, you could setup gittea or gitlab and configure a repo to sync another repo on both pull and push. (ps. bonus points if you make a working helm chart for gittea :))

RB avatar

but it doesn’t seem like this solution exists… so if anyone wants to make a lot of money or contribute to the open source community by hedging that github will have issues like this in the future, then ill be your customer

loren avatar
loren

git proxy or git cache turn up a couple options. not sure how robust they would be for a team, or if availability would be any better than GH

loren avatar
loren
rohanpm/ngitcached

A caching git proxy. Contribute to rohanpm/ngitcached development by creating an account on GitHub.

loren avatar
loren
sitaramc/gitpod

local caching server for git when the actual server is on the other side of a (possibly slow) WAN link - sitaramc/gitpod

SweetOps avatar
SweetOps
08:00:20 PM

Hey everyone, give a warm welcome to our newest members!

  • @Steve Neuschotz
  • @SATYA PATI
  • @Jesse
  • @Bogdan Lata
  • @Steen
  • @Martin Leopold
  • @igal
  • @hari
  • @Alan Rickman
  • @scott866

Good to have you here =)

2020-02-26

SweetOps avatar
SweetOps
08:49:42 PM

Hey everyone, give a warm welcome to our newest members!

  • @bougyman
  • @joey
  • @Oliver Smit
  • @RB
  • @Kendall Link

Good to have you here =)

3
Jesse avatar
Jesse

Hello all. I keep coming across SweetOps archives in Google searches, and thought I would check out the Slack. Judging from my brief skimming of #random… I’m glad I did.

2020-02-25

SweetOps avatar
SweetOps
08:00:07 PM

Hey everyone, give a warm welcome to our newest members!

  • @Milosb
  • @Dragos Andronache
  • @Jeremy Schuller
  • @dan
  • @Thuong
  • @Moritz S

Good to have you here =)

4
Jeremy Schuller avatar
Jeremy Schuller

TY TY

1

2020-02-24

Mohamed Meabed avatar
Mohamed Meabed

Hey! awesomes - glad to be here

1
Zachary Loeber avatar
Zachary Loeber

asdf-vm, a wicked cool project I tinkered with last night comes with a backstory written as a ballad -> https://github.com/asdf-vm/asdf/blob/master/ballad-of-asdf.md

asdf-vm/asdf

Extendable version manager with support for Ruby, Node.js, Elixir, Erlang & more - asdf-vm/asdf

roth.andy avatar
roth.andy

This looks cool. Could help with my “devtools” container. Wonder how it compares to the “alternatives” way @Erik Osterman (Cloud Posse) mentioned

asdf-vm/asdf

Extendable version manager with support for Ruby, Node.js, Elixir, Erlang & more - asdf-vm/asdf

Zachary Loeber avatar
Zachary Loeber

Still looking into it but it’s pretty generic. Mostly just scripts with shims (reminds me of the oh-my-zsh plugin system but targeted towards various app versions of cli tools instead)

roth.andy avatar
roth.andy

Yep. That’s the thing it has felt like Brew has always been missing.

Zachary Loeber avatar
Zachary Loeber

only reason I looked into it was zero desire to deal with installing a specific version of golang on a new(ish) linux laptop of mine. I’d used gvm before for such task

Zachary Loeber avatar
Zachary Loeber

or pyenv for python

Zachary Loeber avatar
Zachary Loeber

programming languages suck like that right?

roth.andy avatar
roth.andy

I just used nvm in dadsgarage. It would be great to standardize with something like this

Zachary Loeber avatar
Zachary Loeber

anyway, the asdf tool plugin list was WAY larger than I expected

Zachary Loeber avatar
Zachary Loeber

but it works more like direnv

roth.andy avatar
roth.andy

yeah, it even has stuff like Terraform and Helm

Zachary Loeber avatar
Zachary Loeber

looking for a .tools-version file and sourcing the right shim if found

SweetOps avatar
SweetOps
08:00:11 PM

Hey everyone, give a warm welcome to our newest members!

  • @Stef
  • @sia.s.saj
  • @kimxogus
  • @Jey Ayyalu
  • @Zorin Wade
  • @Francisco Montada
  • @Sreekumar
  • @Andy To
  • @Abhishek Gupta

Good to have you here =)

2020-02-23

SweetOps avatar
SweetOps
08:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Mohamed Meabed
  • @Richard Gomes

Good to have you here =)

2020-02-21

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@scorebot can you keep tabs?

:--1:2
:100:1
1
scorebot avatar
scorebot
05:20:31 PM

@scorebot has joined the channel

scorebot avatar
scorebot
05:20:32 PM

Thanks for adding me emojis used in this channel are now worth points.

scorebot avatar
scorebot
05:20:33 PM

Wondering what I can do? try @scorebot help

SweetOps avatar
SweetOps
08:00:04 PM

Hey everyone, give a warm welcome to our newest members!

  • @Chuck B
  • @Marco Ceppi
  • @nafas muhammad
  • @scorebot
  • @Nabil Becker

Good to have you here =)

3

2020-02-20

SweetOps avatar
SweetOps
08:00:47 PM

Hey everyone, give a warm welcome to our newest members!

  • @Mario Gonzales
  • @George Kontridze
  • @vinayaks
  • @anthonygiza
  • @Daniel Blue

Good to have you here =)

1
tomkinson avatar
tomkinson

Is this the place to ask a trouble shooting devOps technical question. I don’t want to pollute

MattyB avatar
MattyB

Yep! There are different channels for specific issues (e.g. #aws, #release-engineering, #terraform, #terragrunt, and so many more)

tomkinson avatar
tomkinson

Hi Matty, yes I didn’t see the one that might be appropriate for a Rancher, RMQ port issue

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ya, maybe #kubernetes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

some rancher users there

2020-02-19

2020-02-18

SweetOps avatar
SweetOps
08:00:48 PM

Hey everyone, give a warm welcome to our newest members!

  • @ngan nguyen
  • @vincent.drl
  • @Rico
  • @Ievgenii Shepeliuk

Good to have you here =)

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I whipped up a list of practical recommendations for “12 factor” apps on Kubernetes. Appreciate any feedback. Did I miss anything?

https://cloudposse.com/12-factor-app/

Zachary Loeber avatar
Zachary Loeber

I’m curious about this one: Use DNS-based service discovery (instead of IPs or depend on consul); use short-dns names with search domains rather than FQHN.

Zachary Loeber avatar
Zachary Loeber

firstly I think you meant FQDN

Zachary Loeber avatar
Zachary Loeber

but aside from the typo and the fact that I currently follow this practice I’m wondering about cross-cluster services and such.

Zachary Loeber avatar
Zachary Loeber

consul is being considered for a springboot microservice deployment (they currently use Zuul or something like that) and I’m under the impression that using Consul may be able to help with high availability and service discovery across regions and such

Zachary Loeber avatar
Zachary Loeber

but that’s more of a hunch than that I’ve actually deployed such a construct

Zachary Loeber avatar
Zachary Loeber

Maybe adding that configuration parameters should avoid environment ‘stamping’ would be a good thing to add (though there is likely a better way to word that)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is kind of like the URL or URI ambiguity. IMO, FQHN is “fully qualified hostname” and “hostnames” are for hosts.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

vs FQDN is a “fully qualified domain name”, which is the domain for the hosts. basically, it’s a zone. Like [us-west-2.prod.example.co>, whereby the FQHN might be <http://api.us-west-2.prod.example.co|api.us-west-2.prod.example.co](http://us\-west\-2\.prod\.example\.co)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I get your point about consul ; it’s not wrong. it’s also like using ASM (aws secrets manager). do you embed it in your code? that’s how you get maximal benefit from it, but also, maximally limit how/where it can operate. is the secrets interface external or intrinsic to the application logic?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think what’s interesting about the new hashicorp service mesh based on consul is how it’s implemented with sidecars (like most service meshes)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this means your application does not need to be aware of that implementation detail. it simplifies local development but still works well in a complex multi-cloud environment

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(the poly repo thing is opinionated - realize that’s not a requirement; this list is meant for our customers)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

:--1:3
Yonatan Koren avatar
Yonatan Koren
Flags should enable/disable functionality without knowledge of stage or environment (e.g. do not use if ($environment == 'dev') { ... })

THANK YOU!

Chris Fowles avatar
Chris Fowles

it’s a good opinion in my book - mono repo is an overhead, doesn’t make sense for most systems - and when it does you should know that it does

Chris Fowles avatar
Chris Fowles

i do think that things that are versioned together should be in the same repo though - if you’ve got two services that are tightly coupled and need to be released together (you shouldn’t paint yourself into this corner) then it will probably be easier to control that if they’re in the same repo

Yonatan Koren avatar
Yonatan Koren

https://www.hashicorp.com/resources/closing-keynote-terraform-at-google Kind of an interesting take on the monorepo

(see Single version policy in the transcript)

Closing Keynote: Terraform at Google attachment image

Learn about the workflow Google engineers use for Terraform plan-and-applies, and hear how the company migrated from Terraform 0.11 to 0.12.

:--1:1
Chris Fowles avatar
Chris Fowles

Health checks should not depend on the health of the backing services - this is an interesting conversation to have. usually boils down to the difference between “healthy” and “ready”

Yonatan Koren avatar
Yonatan Koren

I’ve seen something along those lines here https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q - Paper written by a former Google SRE

Yonatan Koren avatar
Yonatan Koren

actually, the way i initially interpreted Health checks should not depend on the health of the backing services is to not alert based on cause, but i’m not really sure what this sentence means the more I look at it

Yonatan Koren avatar
Yonatan Koren

Nevermind it is what I thought it was https://12factor.net/backing-services

The Twelve-Factor App

A methodology for building modern, scalable, maintainable software-as-a-service apps.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

my point is services should gracefully degrade. So if a service depends on the health of it’s backing services, and the backing services, go down, then my service would go down. this is a great way to create site-wide blackouts in a heartbeat.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, my my service can’t talk to one database, but it can to another? what should happen. i think the service is still “healthy” & “ready”, but might return 503 for requests that are impacted.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the downstream service should then decide on how to handle the 503

Yonatan Koren avatar
Yonatan Koren

@Erik Osterman (Cloud Posse) so maybe have specific 500-series responses based on backing service outages? And a monitoring service with an intelligent ruleset would check the most upstream service and be able to detect these issues?

Chris Fowles avatar
Chris Fowles

ideally your app should be aware of both when it is healthy, and when it is able to serve requests based on its backing services - if i can’t write to a database then i shouldn’t be trying to take requests, especially if there is a node that can

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, definitely have monitoring/escalations on this; my point is more that what do we want kubernetes to do when the upstream backend is offline and my service depends on it? do we want kubernetes to kill my service (restart the pod, fail service health checks), or keep it online. i argue that since the problem is not my service, we should keep my service online (by passing health checks), even if my service might respond with 50x to other API requests (e.g. ``GET /healthz = 200, but POST /api/delete/user = 503` )

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


if i can’t write to a database then i shouldn’t be trying to take requests, especially if there is a node that can

Chris Fowles avatar
Chris Fowles

health check should say - “i’m all good, don’t try and restart me” readiness check should say - “i can take requests, send me some traffic!”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but is it that binary (black/white)?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so if 1% of functionality is impacted by not being able to reach some upstream, should my service stop accepting requests?

Chris Fowles avatar
Chris Fowles

if you can gracefully degrade then you are “ready”

Chris Fowles avatar
Chris Fowles

but you should be able to say “i’m running fine, but i can’t service traffic” without getting all your pods killed

:--1:1
Chris Fowles avatar
Chris Fowles

especially helpful for avoiding a startup storm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, agree with that

Yonatan Koren avatar
Yonatan Koren

hmmm that’s an interesting point @Erik Osterman (Cloud Posse). Maybe that’s why microservices have a better place on k8s than monoliths never thought about it but now I have an argument for it

Yonatan Koren avatar
Yonatan Koren

like if you had bs-monolith-app and it does a bunch of crap, including processing images and writing them back to S3, and in the impossible case that S3 goes down, it’ll lose 10% of its functionality

Yonatan Koren avatar
Yonatan Koren

versus, if you had bs-image-processing-service and bs-document-processing-service , then the backing service going down will completely screw up the former but not the latter

Yonatan Koren avatar
Yonatan Koren

and in this case bs-image-processing-servicecan just say “I CANT DO ANYTHING” and its pods will get constantly recreated, the entire service will flap and the SREs/Admins/operators will see that much more clearly than it just losing 10% of its functionality

Yonatan Koren avatar
Yonatan Koren

I’ve heard the jargon “blah blah microservices kubernetes” countless times before and took it blindly but never pictured this very convincing scenario

Yonatan Koren avatar
Yonatan Koren

Maybe backend services are one of a few criteria used for scoping the roles of microservices. As some of us may know, we may get a head of ourselves and make arbitrarily small microservices, sometimes known as nanoservices, where the overhead of managing them doesn’t meet the value of having them scoped that specifically in the first place

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, this is the decoupling that we want to achieve between services. how that decoupling is defined, is a bit arbitrary.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


especially helpful for avoiding a startup storm
yea, this is a good real-world example. a common problem I’ve seen is a Java app (for example) on startup will throw an exception if it cannot connect to the database and exit. this can lead to a startup storm that @Chris Fowles talks about. you have all these services in a crash loop until the database comes back up. then the database comes online, and everything slams it all at once, taking out the database. then the java apps crash again.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

eventually, the crashloop backoffs reduce the effects of the storm and the system comes back online (if you’re lucky)

Yonatan Koren avatar
Yonatan Koren

oh so startup storm = thundering herd

:100:1
Yonatan Koren avatar
Yonatan Koren

i thought startup storm had something to do with startup companies

Chris Fowles avatar
Chris Fowles

@Yonatan Koren is apparently the human encyclopedia of relevant links at a moments notice

1
Yonatan Koren avatar
Yonatan Koren

lol this is what i get paid for

Yonatan Koren avatar
Yonatan Koren

jk

2020-02-17

SweetOps avatar
SweetOps
08:02:00 PM

Hey everyone, give a warm welcome to our newest members!

  • @Frankie.li
  • @CC
  • @Pawel
  • @Alex Kiss

Good to have you here =)

CC avatar

Thanks!

Alex Kiss avatar
Alex Kiss

good to be here

vincent.drl avatar
vincent.drl

Is this the right way to version pin helm to v2 in buildharness .33?

apk —update —no-cache [email protected] && update-alternatives —set helm /use/share/helm/2/bin/helm2

?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

probably a copy & mistake problem, but is wrong

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

should be apk --update --no-cache [email protected] etc..

vincent.drl avatar
vincent.drl

After this helm version -c correctly shows version 2.16.1

vincent.drl avatar
vincent.drl

Couldn’t figure out if there was a make target that would allow packages to overwrite helm symlink to the specific one I want

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@vincent.drl there are a few different concepts that are getting mixed together

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

there’s the build-harness docker image that has packages installed

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then there are the install targets. the install targets are designed to work across platforms, thus do not use apk which is specific to alpine linux

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/build-harness

Collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more - cloudposse/build-harness

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

When using this target, you would do something like make packages/install/helm HELM_VERSION=....

vincent.drl avatar
vincent.drl

Hmm

vincent.drl avatar
vincent.drl

I’m adding a few more binaries to build harness from private GitHub repo which was working fine while on .27 but helm 2.14 didn’t work for us anymore

vincent.drl avatar
vincent.drl

I was basing off of the buildharness image and baking a new one after using the private GitHub release target

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

aha

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

gotcha

vincent.drl avatar
vincent.drl

Now I just want to make sure helm2 is used

vincent.drl avatar
vincent.drl

As “helm”

vincent.drl avatar
vincent.drl

Bumping to .33 defaults to helm3 :(

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, so yes, you’re on the right track. You’ll probably want to uninstall [email protected]

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and then use [email protected]

vincent.drl avatar
vincent.drl

Ok, makes sense because helm3 is fully unlinked so I might as well get rid of it properly and I don’t have to muck around with update-alternatives directly then

vincent.drl avatar
vincent.drl

Thanks!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ya, and when you start moving to helm3, we have a [email protected] package so you can keep both installed

vincent.drl avatar
vincent.drl

Cool cool - were deprecating helm though so probably won’t do that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what are you moving towards?

vincent.drl avatar
vincent.drl

not a text renderer for structured documents

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i can understand for your own apps, but if you depend on any open source charts, you’re undertaking a monumental effort and an exercise in how to manage technical debt at a colossal scale

vincent.drl avatar
vincent.drl

How many charts move to helm3?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i would never want to manage prometheus, grafana, kiam, cert-manager and the 2 dozen other charts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

all helm2 charts are compatible with helm3

vincent.drl avatar
vincent.drl

Hmm maybe I don’t have to version pin helm at all then

vincent.drl avatar
vincent.drl

Didn’t test it, haven’t looked into helm3

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but you need need to “upgrade” helm2 releases to “helm3” releases

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or uninstall (with helm2) / reinstall (with helm3)

vincent.drl avatar
vincent.drl

Overall I have had mostly headaches from the upstream helm-charts repo

vincent.drl avatar
vincent.drl

Oh we don’t use tiller

vincent.drl avatar
vincent.drl

Only render and send manifests to kube through ArgoCD

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

aha

vincent.drl avatar
vincent.drl

So probably don’t need to do anything then

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, so the “package” concept of helm is not something you’re leveraging

vincent.drl avatar
vincent.drl

You mean the releases through config maps?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if we think about helm being like rpm , then what you’re doing is like:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
rpm2cpio myrpmfile.rpm | cpio -idmv
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

which is fine, but then it’s like managing a linux distro without any package manager

vincent.drl avatar
vincent.drl

it’s not really the same thing … think about it more like spinnaker helm bake phase

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i don’t think of helm as a template system

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think of it as a package system

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

vincent.drl avatar
vincent.drl

Helm shouldn’t try to play CD

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i guess it’s a perspective on where the “CD” starts and ends.

vincent.drl avatar
vincent.drl

Which depends on how much control your other tools need over traffic routing within and automated rollover

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s a fair point.

Zachary Loeber avatar
Zachary Loeber

Why shouldn’t helm do CD? Arguably that is its entire purpose.

Zachary Loeber avatar
Zachary Loeber

For most deployments it works perfectly fine. If you are doing some nuanced rollout then of course you will need to plan out a strategy first and consider an appropriate tool to suit your requirements.

vincent.drl avatar
vincent.drl

To me, helm primarily solved the issue of managing related Kubernetes resources which are tightly coupled and have few variable configurations across different use cases

vincent.drl avatar
vincent.drl

True that for the examples listed above (externalDNS / cluster autoscaler / monitoring agents / … ) the Deployment strategies are fairly simple to let it handle by Helm. helm 2 was horrible security and rbac wise due to the tiller component though.

requiring more fine grained patching of custom paths to handle secret management as per company requirements (vault service accounts / sealed secrets / chamber entry points) … helm charts just become overly generic …

vincent.drl avatar
vincent.drl

We’re currently doing the last mile customisation (as some ppl call it) with some custom patching on TOP of the library of open source “packages” which often contain operational experience of the many contributors to it

But I’ve also found that many helm charts are lift and shifted into kube early on and completely missing latest evolution of the tools they package actually adopting Kube api to leverage a lot of the distributed complexity issues - so sometimes I strip 90% of the stable chart (and I do create issues and have my charts public for ppl who search for the same search terms and come across my comments)

Zachary Loeber avatar
Zachary Loeber

I disliked tiller a whole lot as well, it made pipelines stupid hard to work with and maintain any level of security. But that’s really no longer relevant. I’m not wholely advocating helm for everything but I’ve come to accept it as a tool in my belt for certain tasks.

vincent.drl avatar
vincent.drl

given the above issues:

  • outdated charts with slow adoption of consumer patches
  • last mile customisations highly dependent on individual use cases either resulting in overly generic charts of inflexible charts
  • poor handling of modern CD practices (it’s designed for VERY simple upgrade strategies)
Zachary Loeber avatar
Zachary Loeber

I’ve found that almost every public chart needs some form of manipulation to suit my client’s needs as well. Pulling them down into localized chart repos is all to common but also is technically part of best practice of eliminating outside dependencies anyway.

vincent.drl avatar
vincent.drl

See also istio deprecating it’s official helm charts, it’s basically a consensus in the community

vincent.drl avatar
vincent.drl

Pulling down with poorly documented patching leaves you in a much bigger mess of managing upgrades

vincent.drl avatar
vincent.drl

We still heavily leverage helm, albeit some of my colleagues do it very begrudgingly

Zachary Loeber avatar
Zachary Loeber

Not true at all. Pulling random charts out in the wild into production clusters leaves you in a far more likely position to create technical debt though.

vincent.drl avatar
vincent.drl

And we are trying hardest to get rid of it (mostly for internal deployments for now)

Zachary Loeber avatar
Zachary Loeber

in a modern CICD pipeline, how do you share common elements across dozens of projects effectively?

Zachary Loeber avatar
Zachary Loeber

Istio also doesn’t define the industry standards

Zachary Loeber avatar
Zachary Loeber

though it is really exciting and fun stuff, I see a very small portion of businesses adopting it

Zachary Loeber avatar
Zachary Loeber

sorry, I’m playing devil’s advocate here because I’ve had a few people categorically make some of the same claims you have based purely on prior distaste for the tool.

Zachary Loeber avatar
Zachary Loeber

I’m one of those people that begrudgingly use it and see its value in the right places.

vincent.drl avatar
vincent.drl

adopting what? Service meshes?

Zachary Loeber avatar
Zachary Loeber

I also don’t push secrets through it at all

Zachary Loeber avatar
Zachary Loeber

I do pull models for that on my clusters so I flat out work around some of the stickier elements of using it.

vincent.drl avatar
vincent.drl

I’m also not categorically against it, but I’ve adopted severe abused helm charts with undocumented patches from upstream

Zachary Loeber avatar
Zachary Loeber

helm, sry

vincent.drl avatar
vincent.drl

And my colleagues who inherited this from the person I replaced hate it a lot more than me

The fact so many projects have been packaged with Helm shouldn’t lock us into it though

vincent.drl avatar
vincent.drl

I think fundamentally we kind of agree :)

Zachary Loeber avatar
Zachary Loeber

right, so for now, if you have to package a common set of deployment code for shared use across multiple development efforts what well thought out tools are there to use?

Zachary Loeber avatar
Zachary Loeber

kustomize looks pretty cool. That’s one I should look into?

vincent.drl avatar
vincent.drl

for our internal deployments we integrated templates into a binary to avoid abuse

Zachary Loeber avatar
Zachary Loeber

ugh… must be a better way

vincent.drl avatar
vincent.drl

Now that kustomize was pushed into kubectl … it’s idd something that seems to solve our patching issue

vincent.drl avatar
vincent.drl

Myeah I’m not 100% with it either, more so because I really want a structure aware template that doesn’t fall over a missing space

vincent.drl avatar
vincent.drl

And simply catches syntax errors way earlier

vincent.drl avatar
vincent.drl

I think jsonnet + kustomize

vincent.drl avatar
vincent.drl

But haven’t played with enough myself

vincent.drl avatar
vincent.drl

TF .12 HCL is quite interesting, just migrated all resources not managed by kube to that - I was very sceptical at first

vincent.drl avatar
vincent.drl

I never tried TF to manage kube deploys … I don’t think the TF model fits kube control loops model?

vincent.drl avatar
vincent.drl

I do think my friend’s company moved or TF to manage to kube deployments though

Zachary Loeber avatar
Zachary Loeber

If you always know your cluster state then tf is ideal for it I’d think.

Zachary Loeber avatar
Zachary Loeber

I’m still spitting out base clusters and configuring them after the fact via stacks of helmfiles for baseline config

Zachary Loeber avatar
Zachary Loeber

I used to pre-create namespaces and shove secrets into the cluster via TF though. I’m not against using it for kube deployments but then I have to connect developer pipelines to terraform which kind of scares me (probably more than it should)

Zachary Loeber avatar
Zachary Loeber

I’m now leaning heavily into the deployment/environment git repo approach that will allow me to leverage gitops if I like (or even just stand-alone pipeline as code in the repo itself to do push deployments)

vincent.drl avatar
vincent.drl

My direct supervisor kind of designed an internal replacement for helmfile+helm with that binary I mentioned

  • it can fetch and render helm charts
  • it overlays patches in a crude way (not as fine grained as kustomize)
  • it takes a map like helmfile to do all of the above
  • it pushes to a sync branch which ArgoCD tracks
  • it does basic cluster bootstrap and sets up argoCD tracking for all other deploy repos (team specific) related to the cluster
vincent.drl avatar
vincent.drl

And it has those compiled in templates for non-helm internal microservices which have defaults and some globally + locally overwritable values

Zachary Loeber avatar
Zachary Loeber

your super sounds like a person I could get along with.

Zachary Loeber avatar
Zachary Loeber

how many repos/deployments are you supporting?

Zachary Loeber avatar
Zachary Loeber

(for what its worth I’m eyeing over jx for some of this for opinionated deployment into kube, they have some pretty sound ideas around teams/projects/environments)

vincent.drl avatar
vincent.drl

I haven’t heard about jx- I have to say the design of the tool is mostly by my supervisor Chris Kolenko - I helped implement the functionality … I know he’s dying to talk publicly about it but we haven’t had the time yet to finalise it - currently handling about 3 Brandings of the same set of microservices across dev / test / staging / prod - so 12 repos of that and 2 cluster repos per env (for ephemeral cluster upgrades, we just stand up a new version cluster, boostrap it and switch over) we are mid way of this migration while also adding new business requirements all the time… so it’s very in flux, cluster repos retire when clusters do

vincent.drl avatar
vincent.drl

It’s not so big and already quite complex, we come from a monorepo for deployments that was using templated helmfile templating values for helm templates - a templating nightmare rendered by argoCD with zero insight in obscure error messages

Zachary Loeber avatar
Zachary Loeber

helm templates templating more helm templates?

Zachary Loeber avatar
Zachary Loeber

Sorry for digging (and feel free to not answer) but is it just you two handling all that?

Zachary Loeber avatar
Zachary Loeber

I work mostly in a bubble as of late so hearing how other devops teams are operating is like my drug or something.

vincent.drl avatar
vincent.drl

It grew from a team of DevOps and Dev (outsourced) to an in house team being built, i was 3rd to join 6 months ago and team is now 5 “devops”

vincent.drl avatar
vincent.drl

It’s not our job title just trying to keep things simple

Zachary Loeber avatar
Zachary Loeber

I’m currently supporting a mostly offshored team of developers for approximately 45 repos across 4 teams and have had anywhere from 3 to 8 environments (each with their own fully loaded kube clusters and pipelines) at once.

Zachary Loeber avatar
Zachary Loeber

I converted all their pipelines to pipeline as code early on or there’s no way I’d be able to take on such load.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Zachary Loeber are you also using flux?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(i forget - you told me before)

2020-02-16

SweetOps avatar
SweetOps
08:00:08 PM

Hey everyone, give a warm welcome to our newest members!

  • @21042jim
  • @Christoph Gerkens

Good to have you here =)

2020-02-15

SweetOps avatar
SweetOps
08:00:09 PM

Hey everyone, give a warm welcome to our newest members!

  • @Kevin Hetman
  • @krismulica
  • @acs0508
  • @Örjan Sjöholm

Good to have you here =)

2020-02-14

SweetOps avatar
SweetOps
08:00:13 PM

Hey everyone, give a warm welcome to our newest members!

  • @Sudesh Lalmal Pathirana
  • @mattia.bertorello
  • @Olivier

Good to have you here =)

1

2020-02-13

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what are everyones favorite videos for learning terraform, helm, kubernetes, etc? please share with me!

Zachary Loeber avatar
Zachary Loeber

I don’t use videos for learning much but thus far I think that Kubernetes In Action is the current bible of kubernetes.

Alex Siegman avatar
Alex Siegman

Same, videos tend to be too slow-paced for me, and I’m a quick reader. I’d rather read, pause where I want, test, play, search for answers…

A coworker of mine has enjoyed the LinuxAcademy content, I can’t vouge for it but it’s helped him learn stuff.

grv avatar
kelseyhightower/kubernetes-the-hard-way

Bootstrap Kubernetes the hard way on Google Cloud Platform. No scripts. - kelseyhightower/kubernetes-the-hard-way

grv avatar

Also, courses on udemy from Edward Viaene, for Kubernetes, terraform and Prometheus are good ones I can think of

grv avatar
Edward Viaene | DevOps, Cloud, Big Data Specialist| Udemy

Edward Viaene is a Udemy instructor with educational courses available for enrollment. Check out the latest courses taught by Edward Viaene

SweetOps avatar
SweetOps
08:00:17 PM

Hey everyone, give a warm welcome to our newest members!

  • @Siraj Rauff
  • @ericyang879
  • @sekhar modu
  • @Vidhya Vijayakumar

Good to have you here =)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Helpful feedback @Zachary Loeber @Alex Siegman

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Let me rephrase this to what learning materials have ya’ll found the most helpful

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am trying to curate some content to help newcomers to grok the concepts and give them a place to get started

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am personally searching for content and overwhelmed with how bad most of the stuff is (well, to be kind - just not the way I would explain it), so it’s not surprising that if someone is a newcomer to this stuff how hard it is to find good learning materials

Alex Siegman avatar
Alex Siegman

For people brand new to kubernetes, I’ve found “Kubernetes Up and Running” pretty invaluable for learning the basic concepts. It’s available for free from a few places, I haven’t found a give-information-free link in my quick google, but https://azure.microsoft.com/en-us/resources/kubernetes-up-and-running/

Kubernetes: Up and Running, Second Edition

Improve the agility, reliability, and efficiency of your distributed systems by using Kubernetes. Get the practical Kubernetes deployment skills you need in this O’Reilly e-book. You’ll learn how to:Develop and deploy real-world applications.Create and run a simple cluster.Integrate storage in…

Alex Siegman avatar
Alex Siegman

Or you can buy it from o’reilly / amazon

Alex Siegman avatar
Alex Siegman

I think it helps to be focused on what you’re trying to learn, and more importantly, for what purpose. Example being, Up and Running is a great book for anyone who has need to understand basic kubernetes concepts. But if you’re a developer who has to make the program and stuff it in to a container, will that necessarily help you? I find it harder to find learning materials at the level and “depth” around a given concept than materials for a tool or concept in general

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s a good point.

Alex Siegman avatar
Alex Siegman

Another great example is python books. There’s a million “learn python” books, but all of them cover the same boring stuff. What about a “learn python for someone who already knows python” book - where it goes more in to style, architecture choices for python programs, good designs for whatever, etc. I don’t need to be taught python, I want to learn/explore idiomatic approaches to problems in python, learn to use the language elegantly and efficiently, etc.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

“here’s what you need to know as a developer…:”

:--1:1
grv avatar

@Erik Osterman (Cloud Posse) can i be added to the terraform channel, kicked myself out by mistake

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmmm anyone can join any channel :-)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Even if you leave one…

grv avatar

good to know, thanks

Sudesh Lalmal Pathirana avatar
Sudesh Lalmal Pathirana

hii

1

2020-02-12

2020-02-11

SweetOps avatar
SweetOps
08:00:09 PM

Hey everyone, give a warm welcome to our newest members!

  • @Geoff Weinhold
  • @julius.blank
  • @Conti Mattia
  • @Meg Yahl

Good to have you here =)

Zachary Loeber avatar
Zachary Loeber

Dang, the daily new member list is getting larger and larger. The cult of devops is spreading….

:--1:2
Geoff Weinhold avatar
Geoff Weinhold

come closer to the light!

Patrick M. Slattery avatar
Patrick M. Slattery

Question on secrets for things such as DB passwords etc. Does anyone keep the originals in a Git repo (encrypted of course)? We do and I’m questioning myself on whether we should.

Pro: We have the master copy offline from the K8S cluster and in the event of a total failure we still have access to the original passwords.

Cons: Do we really need to keep yet another copy of the passwords? They are in the Azure Keyvault/Google Secret Manager (where apps pull them them from)

Anybody have any good reasons to go either way?

Yonatan Koren avatar
Yonatan Koren

@Patrick M. Slattery I mean, it sounds like you’re facing the burden of keeping passwords in more than place. What about a centralized secrets management solution such as HashiCorp Vault?

Pros:

• Secrets are in one place

• Failure scenario: Vault’s storage can be HA, for example with Consul as the data store. Even in total failure, you will be okay if you have enabled Consul backups to AWS S3.

• Cloud agnostic - i.e. you’re not relying on AWS KMS, GSM, etc

• Dynamic Secrets Engines (You can give admin credentials to Vault and it will issue and automatically revoke temporary DB credentials)

• A k8s mutation webhook controller which essentially acts as an operator to allow injection of secrets to pods https://www.vaultproject.io/docs/platform/k8s/injector/index.html Cons:

• Expensive to run HashiCorp’s reference architecture of 5 Consul nodes + 3 Vault nodes (https://learn.hashicorp.com/vault/operations/ops-reference-architecture) - Vault integrated Raft storage is still in beta until 1.4.0

• You’re running a third party application on top of everything you’re already running

• Re: Failure scenario and automated backups - for non-Enterprise Consul you need to manage automated backups yourself

Patrick M. Slattery avatar
Patrick M. Slattery

Yeah, I’ve heard a couple of Vault horror stories that keep me away from it. We ourselves initially used Consul for state management in our app and had several disasters with it. It would be hard to persuade anyone here to run Consul again.

Patrick M. Slattery avatar
Patrick M. Slattery

That said the pros are all very much what I really want…

grv avatar

You can use AWS secrets manager (if on AWS), but it becomes really expensive if there are large number of secrets to be stored (we are also in the same boat right now on which secrets manager to use though)

Patrick M. Slattery avatar
Patrick M. Slattery

We are currently using Azure Keyvault / Google Secrets Manager (We are moving away from Azure though) Price for either is not anything crazy but then again we only have a few dozen secrets in each instance at most

Tom Howarth avatar
Tom Howarth

and as a result still red zone for twilighting as per Googles norman operation procedures

Patrick M. Slattery avatar
Patrick M. Slattery

Google Secrets Manager is nice but is pretty light on features being so new (Still in beta)

Chris Fowles avatar
Chris Fowles

as long as the encryption is well protected i (depending on the workload) don’t see a massive issue with using git as the source of truth as long as they are that. things like needing to rotate at short intervals and needing to audit retrieval of secrets would change that recommendation

Chris Fowles avatar
Chris Fowles

vault and consul are great tools, but they’re not without operational overheads - they need to be kept running and maintained

Chris Fowles avatar
Chris Fowles

if you can’t commit to the overhead required to run them you’re introducing a weakness into the system rather than a strength as they’ll quickly end up in the critical path

:--1:2
Yonatan Koren avatar
Yonatan Koren

This was very concisely said, I try and formulate such a sentence when someone throws Kubernetes into the conversation, even them having never operated it

2020-02-10

SweetOps avatar
SweetOps
08:00:19 PM

Hey everyone, give a warm welcome to our newest members!

  • @Daniel Stoor

Good to have you here =)

2020-02-09

SweetOps avatar
SweetOps
08:00:08 PM

Hey everyone, give a warm welcome to our newest members!

  • @Aaron Lennon

Good to have you here =)

2020-02-08

SweetOps avatar
SweetOps
08:00:21 PM

Hey everyone, give a warm welcome to our newest members!

  • @rustemabd
  • @Andrew Cameron-Douglas
  • @Tom Howarth

Good to have you here =)

rustemabd avatar
rustemabd

Thank you! Happy to join! :–1:

Tom Howarth avatar
Tom Howarth

Thanks for letting me in. I am still on my learning journey. so please be kind :)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Tom Howarth all skill levels welcome!

2020-02-07

SweetOps avatar
SweetOps
08:00:26 PM

Hey everyone, give a warm welcome to our newest members!

  • @Norbert Fenk
  • @Dhrumil Patel

Good to have you here =)

2020-02-06

SweetOps avatar
SweetOps
08:00:13 PM

Hey everyone, give a warm welcome to our newest members!

  • @Mike Martin
  • @Zack Hewison
  • @Miranda Pearson
  • @Silke Van den Broeck
  • @hugomelo97

Good to have you here =)

:--1:1
Mike Martin avatar
Mike Martin

Thanks for the shout out! Really enjoyed the zoom session yesterday - will be back next week!

Zack Hewison avatar
Zack Hewison

Thanks for the welcome! Hello

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks @Mike Martin! Ya, yesterday’s #office-hours was a good one. See you next week!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hey @Zack Hewison!

2020-02-04

SweetOps avatar
SweetOps
08:00:14 PM

Hey everyone, give a warm welcome to our newest members!

  • @cia
  • @Josh Hudson
  • @tomkinson
  • @Maciej Kozlowski
  • @Jason Carter

Good to have you here =)

2
tomkinson avatar
tomkinson

thanks!

Jason Carter avatar
Jason Carter

Thanks

2020-02-03

wannafly37 avatar
wannafly37

2nd interview for my first DevOps type role today! Anyone want to share tips?

MattyB avatar
MattyB

#jobs had some good discussion on it recently. i’m not sure how much is relevant to a first timer though

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ya some good interview questions were posted in #jobs. Honestly, it comes down to the interviewer and their style of interviewing.

SweetOps avatar
SweetOps
08:00:12 PM

Hey everyone, give a warm welcome to our newest members!

  • @wattiez.morgan
  • @Viktors D
  • @Prasanna Pawar
  • @Almas Hilman Muhtadi
  • @Dan Griffiths
  • @Sharanya reddy pagidi

Good to have you here =)

:--1:2
gyoza avatar
gyoza

thx for the hard work on that elasticsearch module thats really nice.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks @gyoza!

    keyboard_arrow_up