#kubernetes (2018-11)

kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2018-11-04

rohit.verma avatar
rohit.verma

hi all, what are you opinion on different networking option in kubernetes on aws. Which is more preferred and felt robust. We did tried with aws-vpc-cni but felt that its not stable enough even with 1.1.0 for kuberntes 1.10.6. This becomes more unstable when all your worker nodes are unstable and started giving exception as sandox ip changed etc..

rohit.verma avatar
rohit.verma

we then switched to calico, but somehow its observed that its impacting the way pods terminate. If we delete a deployment, pods remain in terminating state for 5+ minutes.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The pods stuck in a terminating state is a very frequently observed problem. Could it be related to the network layer? Maybe - but I would explore other possibilities. To me the network culprit seems like a red herring.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Lots of posts/issues on it. Usually related to zombies.

2018-11-06

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@rohit.verma we saw something like that with some of k8s pods, in particular kiam- when deleted, the pods take many minutes to terminate

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so maybe it’s an issue with some deployments, not the network itself?

rohit.verma avatar
rohit.verma

But the pods I am referring here is generic like nginx or spring boot app

rohit.verma avatar
rohit.verma

Anyways more concerned about a general opinion on different kubernetes networks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We haven’t had the opportunity to explore/optimize the network layer in k8s

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also are you familiar with the dumb-init “fix” ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is to address the same symptoms

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Yelp/dumb-init

A minimal init system for Linux containers. Contribute to Yelp/dumb-init development by creating an account on GitHub.

onzyone avatar
onzyone

@Andriy Knysh (Cloud Posse) hello again, do you have any doco’s and best practices for promoting kube, within nonp? … ie … dev to staging?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for clarification, are you talking about promoting images and helm charts? or promoting usage of kubernetes within a company

onzyone avatar
onzyone

right now we are using diff name spaces in k8s

onzyone avatar
onzyone

currently it is within company

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So the same cluster for staging and production?

onzyone avatar
onzyone

dev and staging

onzyone avatar
onzyone

bump

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

sorry, i let this fall through the cracks.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we don’t have a well documented process for what you want. we’ve implemented and documented it internally for customers, but still need to document it on our site.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, looks like the video was taken down =/

onzyone avatar
onzyone

nice

onzyone avatar
onzyone

this is the same thing that I had in mind …

onzyone avatar
onzyone

do what is your view databases with persistent volumes?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Use fully managed databases for anything you care about

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Use database containers for disposable environments

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So for example, when we deploy environments for every PR we use containers

onzyone avatar
onzyone

thumbsup_all

onzyone avatar
onzyone

what are your thoughts on some of the work that Kelsey Hightower has done in this space? https://github.com/kelseyhightower/pipeline

kelseyhightower/pipeline

A step by step guide on creating build and deployment pipelines for Kubernetes. - kelseyhightower/pipeline

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

have’t taken a look at it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Nonp?

onzyone avatar
onzyone

ya we run a two accounts … were one is prod and one is none-prod

onzyone avatar
onzyone

and all our none-prod stuff happens in nonp

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Aha gotcha

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We don’t have the promotion process documented but I can share how it looks (we use Codefresh)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am currently on my phone so will share a little later

onzyone avatar
onzyone

np sounds good thumbsup_all

Tee avatar

using kops or terraform for creating kubernetes Production. What is better and cons ??

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Tee we use terraform to create kops resources, e.g.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

Collection of Terraform root module invocations for provisioning reference architectures - cloudposse/terraform-root-modules

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

Collection of Terraform root module invocations for provisioning reference architectures - cloudposse/terraform-root-modules

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and then use kops to provision k8s clusters

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

there was some discussion earlier in #terraform I think related to EKS

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we also have TF modules for EKS

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Tee are you thinking GCP or AWS?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so on AWS, my opinion is that it’s more work than than necessary to manage EKS with terraform. the challenge comes down to upgrading. there’s some discussions on strategies for that.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Andriy Knysh (Cloud Posse) are ya’ll using it with kops? Looks like it. How does TF generation fit in if at all?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

with kops, the ability to do rolling-updates is built in; it’s a purpose built tool like kops will do a better job at managing lifecycles.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if fargate announces EKS support at the end of the month, I might change my stance

Tee avatar

But the EKS and FARGATE gets pretty expensive

Tee avatar

as far i think

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

humans aren’t cheap either

Tee avatar

Right

Tee avatar

So what do you suggest for longterm. Not considering the cost. With less bottlenecks and nightmares

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

not an easy question

Tee avatar

I mean in terms of stability

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

kops is well established and works well, and does lifecycle management

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

EKS is new and lacks a lot of features, but it will stay and they will improve it

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Fargate will improve and cost will be reduced

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(we’re not using EKS in production yet, so our story will be biased towards kops)

Tee avatar

Oh ok. Thanks @Erik Osterman (Cloud Posse) & @Andriy Knysh (Cloud Posse) for your suggestions.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yea, the point is that with the current state of EKS, you need to do and provision even more resources than using kops

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and it does not support many features

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Fargate could improve it, but as many mentioned it’s costly (and it does not exists yet)

Tee avatar

That makes sense

Matthew avatar
Matthew

I am currently moving all of our infrastructure off Mesosphere DC/OS onto EKS and EKS has been phenomenal in my opinion - just lots of support from many different aspects such as AWS and the Kubernetes community

Matthew avatar
Matthew

as well as great folks like Cloud Posse

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yea thanks @Matthew

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

the point is that with EKS, if for example you need to perform a rolling update, it’s not supported out of the gate

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so a lot of friction with many things

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

with kops it just works

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

but sure for longterm EKS/Fargate would be better

Matthew avatar
Matthew

Yeah i’ve talked with EKS specialist from AWS and they currently suggest a blue/green strategy for upgrading which can be tedious and at times break backwards compatibility

btai avatar

how do you export a single context of your kubeconfig?

btai avatar

say my local kubeconfig has a dev qa prod context

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@btai we don’t have multiple contexts. We use containers + ENV vars pattern (implemented in geodesic + repo per env + Dockerfile(s)). So in each container (prod, staging, dev, etc), when we run it, we have all ENV vars defined for that particular env (ENV vars come from Dockerfiles or from SSM if they are secrets). That includes everything for Terraform, kops, k8s, etc.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

So when we do for example kops export kubecfg, the environment knows what context we want

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and we can run those geodesic containers locally and also in CI/CD pipelines (for which we use Codefresh since it can run each pipeline step as a Docker container)

btai avatar

nice thanks

rms1000watt avatar
rms1000watt

Have you guys used Codefresh enterprise? I know you’re all big into codefresh here. Just curious of any pitfalls or bits of advice you guys have

rms1000watt avatar
rms1000watt

(enterprise to run on-prem)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So Codefresh enterprise has 3 variations: full SaaS, hybrid and on-prem

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ve been working exclusively with the enterprise SaaS

rms1000watt avatar
rms1000watt

Ooo

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what’s the primary driver for going on-prem?

rms1000watt avatar
rms1000watt

compliance requiring no dependence on external SaaS providers

rms1000watt avatar
rms1000watt

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

which compliance certification?

rms1000watt avatar
rms1000watt

oh wow, you haven’t even taken me out on a date yet to be asking such risquè questions.

rms1000watt avatar
rms1000watt

lol jk, I think fedramp

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

lol

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok - that’s a whole ’nother cup of tea

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not familiar with

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but sounds like you’d need full on-prem.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so i probably wouldn’t enlighten you more than you already know

rms1000watt avatar
rms1000watt

No worries. We’re new to codefresh–so just probing for any gotcha’s really

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@dustinvb can definitely elaborate

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

are you using the helm based install?

rms1000watt avatar
rms1000watt

I think at the moment, yes

rms1000watt avatar
rms1000watt

debating about the release of terraform 0.12 and using all the templating stuff

rms1000watt avatar
rms1000watt

rather than 2 templating engines.. tiller.. and all that jazz

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so i get where you’re coming from - but from what i’ve gleaned the current helm provider is too basic to handle all kinds of helm charts. maybe with 0.12 it’s better off

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you’ve seen our helmfiles repo? basically you can’t do half of what we do with helmfile using that provider

rms1000watt avatar
rms1000watt

https://marketplace.fedramp.gov/#/product/aiware-government?sort=productName

Just as a cover my ass that I’m not saying providing any confidential information.. it’s publicly available that we’re on fedramp ^^^ lol

rms1000watt avatar
rms1000watt

Oh, my bad. I didn’t mean the helm provider.. I meant generating the k8s.yml files on the fly based on the infra-state.. no helm installation anywhere

rms1000watt avatar
rms1000watt

just a thought at the moment–not necessarily going that direction for sure

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but yea, you could basically create terraform modules in place of helm charts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

… if terraform templating is sufficient

rms1000watt avatar
rms1000watt

hehe, yeah, big “if”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s been my experience, the “simple” case always works well regardless of the technology

rms1000watt avatar
rms1000watt

ah

rms1000watt avatar
rms1000watt

how have you guys been liking helm? any complaints with the tiller stuff, or you guys are experienced enough with it all–nothing really bugs you?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i mean, it sucks about the tiller and all

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but i look at helm more like an interface

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and the interface won’t change dramatically, but the underlying implementation is getting a big overhaul as you’re probably aware

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

as part of that tiller is going away and the template engine going pluggable

rms1000watt avatar
rms1000watt

“tillerless helm” is the buzz

rms1000watt avatar
rms1000watt

yea

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

as a way to manage a complex apps it’s great

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and app dependencies

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i say (and with some humility) that those before us have invested a lot of time in what it takes to manage software releases

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

deb, rpm, apk, etc.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we tried to avoid that with just a Makefile; it worked well until it didn’t. in the end, we needed all that a package manager provides and conceded to package .apk alpine packages

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

my point is that just templatizing raw kubernetes resources and applying them seems easy enough and i’m sure you can get away with it for a long time

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but then you realize you want to have dependencies, triggers on deployment or uninstall, and rollbacks, etc. then you’re on your own.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the more homegrown/spun, the more the solution diverges from the trajectory the community is taking

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

because the community is solving problems around a standardized toolset

rms1000watt avatar
rms1000watt

all true

rms1000watt avatar
rms1000watt

so i’m curious.. you bring up rollbacks

rms1000watt avatar
rms1000watt

codefresh/spinnaker’s solutions didn’t offer enough in that aspect?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

codefresh relies on the fact that helm does rollbacks automatically

rms1000watt avatar
rms1000watt

ah

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and even bakes that into the UI with one-click rollbacks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

they also have some even more cool stuff in the works - but you’ll have to ask them to see it

rms1000watt avatar
rms1000watt

For sure

rms1000watt avatar
rms1000watt

we have meetings setup with them

rms1000watt avatar
rms1000watt

We’ll probe

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

very cool! hit me afterwards and let me know how it goes

1
rms1000watt avatar
rms1000watt

does all this reveal a well needed niche (product offering) in the CI/CD process for k8s?

rms1000watt avatar
rms1000watt

since there always ends up being handrolled stuff?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha, not sure - there are more CI/CD platforms today than ever

rms1000watt avatar
rms1000watt

https://github.com/gaia-pipeline/gaia I like their philosophy at that in particular

gaia-pipeline/gaia

Build powerful pipelines in any programming language. - gaia-pipeline/gaia

rms1000watt avatar
rms1000watt

yea

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i can’t keep them straight anymore

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

spinnaker is now coming out with an enterprise offering too

rms1000watt avatar
rms1000watt

haha nice. Well, after the bloodbath, hopefully the best solution reigns supreme

rms1000watt avatar
rms1000watt

ah

rms1000watt avatar
rms1000watt

halyard was surprising when I first played with it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and then github actions

rms1000watt avatar
rms1000watt

then I looked at the helm chart for spinnaker.. and it was just a bunch of hal commands

rms1000watt avatar
rms1000watt

yea

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but i agree that there’s still big room for improvement

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the fact there is so much handrolling and independent tooling

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think codefresh is well poised to do that as it relates to cicd+kubernetes+helm

rms1000watt avatar
rms1000watt

does your gut think helm isn’t going anywhere?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

until I see an alternative that has anywhere near the critical mass of helm, yes - i think it’s here for the foreseeable future

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for example, there’s ksonnet (based on jsonnet) which looks interesting

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but i think some variation of that could be used as a pluggable engine for helm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, i don’t want to see proliferation of more packaging systems right now - it’s too early

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
rimusz/helm-tiller

Helm tiller plugin aka Tillerless Helm. Contribute to rimusz/helm-tiller development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

have you seen this plugin?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is pretty smart.

rms1000watt avatar
rms1000watt

I thiiiink I’ve seen this one.. if not it was something similar

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

basically, it’s a drop in replacement. it still stores all configs in the cluster (per namespace if you want)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you run a temporary tiller locally

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this can be run as part of CI

rms1000watt avatar
rms1000watt

interesting.. hmm.. nice actually!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(though would break the codefresh helm UI, since it would need to talk to the tiller and there would be none running)

rms1000watt avatar
rms1000watt

ah, right

2018-11-07

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
technosophos/helm-ksonnet

Experimental ksonnet plugin for Helm. Contribute to technosophos/helm-ksonnet development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Dig it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Proposal: Jsonnet template integration · Issue #2577 · helm/helm

In order to provide jsonnet rendering for helm charts a new ReleaseModule similar to the Rudder ReleaseModule should be developed. This module would take charts and render them as Jsonnet templates…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Guess my hopes of seeing ksonnet as a template engine in helm were misguided

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I know Lua is coming. I’d heard such great things about jsonnet, that I assumed it would be well suited. But Lua I guess is a better understood embeddable language

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Last I had to write Lua was 14 years ago when dealing with Nginx

2018-11-08

rms1000watt avatar
rms1000watt

For the codefresh peeps out there.. does it matter what/how the ingress controller looks when using codefresh for deployments?

rms1000watt avatar
rms1000watt

2018-11-09

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Nope, we use for example the CloudFlare Acesss/Argo ingress and nginx-ingress controller in the same cluster

Ryan Ryke avatar
Ryan Ryke
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I love his presentations and he’s definitely the best evangelist for kubernetes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and i think he’s presenting the simple side that should be presented

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and here comes the but…..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but in the real world of deploying complex applications with interdependencies, secrets, configurations, etc… it devolves into something much more complicated

Ryan Ryke avatar
Ryan Ryke

his presentations are always awesome

Ryan Ryke avatar
Ryan Ryke

for sure

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and the gap to cross from the hello world examples to customer apps is huge

Ryan Ryke avatar
Ryan Ryke

he makes it look so “easy button”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

PLEASE SOMEONE SHOW ME HOW TO MAKE THIS EASIER

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i want to

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i hate this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and here’s the rest of all the other apps

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Ryan Ryke avatar
Ryan Ryke

so with one of my customers we are working on two distinct steps… once to build the app, then a seperate one to update (deploy) the app in an ongoing fashion

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we’re using helm, and some hate on helm for one reason or another. but one things for sure, this is hiding an even more enormous pile of YAML/go templating on the backend.

Ryan Ryke avatar
Ryan Ryke

i hate kelsey hightower in the best way possible

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Continuous Delivery with Amazon EKS and Jenkins X | Amazon Web Servicesattachment image

Amazon Elastic Container Service for Kubernetes (Amazon EKS) provides a container orchestration platform for building and deploying modern cloud applications using Kubernetes. Jenkins X is built on Kubernetes to provide automated CI/CD for such applications. Together, Amazon EKS and Jenkins X provide a continuous delivery platform that allows developers to focus on their applications. This […]

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

i did not think it could do so much, it creates pipelines for infrastructure itself (prod and staging), and pipelines for the app, and even spawns a separate testing/staging env in k8s for each PR, and comments on GitHub on PRs (like atlantis), and creates GitHub repos with Helm charts for the infrastructure (prod and staging)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

https://github.com/jenkins-x/sso-operator (@Erik Osterman (Cloud Posse) already posted it before)

jenkins-x/sso-operator

Single Sign-On Kubernetes operator for Dex identity provider - jenkins-x/sso-operator

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

one thing it can’t do is to upgrade the k8s cluster b/c it itself sits in the same cluster

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

wow

ramesh.mimit avatar
ramesh.mimit

@here Any recommendations for learning distributed systems from basics to advance?

ramesh.mimit avatar
ramesh.mimit

noticed, lot of people knows the tools but not the concepts…

2018-11-11

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@ramesh.mimit I found this site very interesting and with lots of resources about distributed systems, and real-life examples from many companies

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
ramesh.mimit avatar
ramesh.mimit

@Andriy Knysh (Cloud Posse) thanks..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

“Google Kubernetes Engine’s third consecutive day of service disruption”

2018-11-12

btai avatar

anyone use the official python kube library?

btai avatar

can you load the config from a dict?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

~why not use config profiles instead?~

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

~e.g. AWS_DEFAULT_PROFILE=cp-prod-admin~

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

~the underlying aws SDK should then handle everything automatically~

btai avatar

the kube config?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

heh, my bad @btai

btai avatar

how would i run kubectl within a container running from a job?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

here’s an example doing it from a deployment: https://github.com/onfido/k8s-rabbit-pod-autoscaler

onfido/k8s-rabbit-pod-autoscaler

Kubernetes autoscaler for pods that consume RabbitMQ - onfido/k8s-rabbit-pod-autoscaler

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

doing it from a job wouldn’t be any different

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

just need the proper role bindings

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in this case, kubectl is gettin called from in the autoscale.sh

btai avatar

so if i have the wrong role bindings

btai avatar

would i be getting this error:

btai avatar
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

all i know is when we implemented it for redis using the strategy above (for rabbit), we didn’t need to specify any hosts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it just autodiscovers it

btai avatar

the pod autodiscovers

btai avatar

ok thats what i was hoping for

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it also provides a kube context

btai avatar

so the pod itself didnt have any kubeconfig or kube api secrets

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, it didn’t have anythign like that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
vanvalenlab/kiosk-autoscaler

Contribute to vanvalenlab/kiosk-autoscaler development by creating an account on GitHub.

btai avatar

yeah i have a job basically doing the same thing

btai avatar

executing a shell script that makes a kubectl call

btai avatar

but i get the above error

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

kops cluster?

btai avatar

aks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we tested it on gke and kops

btai avatar

yeah it works in kops

btai avatar

that job

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh interesting!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you have rbac enabled in kops?

btai avatar

although the kops

btai avatar

doesnt have rbac enabled

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha

btai avatar

yeah

btai avatar

so do i create a clusterrolebinding for the job?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
onfido/k8s-rabbit-pod-autoscaler

Kubernetes autoscaler for pods that consume RabbitMQ - onfido/k8s-rabbit-pod-autoscaler

2018-11-13

btai avatar

thanks

btai avatar

sorry, still new to k8s

btai avatar

hypothetically if i create a cluster role binding with the namespace and name that matches the job, that should work?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

More or less

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I don’t know the specific matching selectors that are available

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is near. Copy secrets from a centralized system of record. https://github.com/mittwald/kubernetes-replicator/

mittwald/kubernetes-replicator

Kubernetes controller for synchronizing secrets & config maps across namespaces - mittwald/kubernetes-replicator

2018-11-14

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

How are you guys handling busy helm deployments where the tiller is busy attending to other deployments…

                                                                                                                             
Error: could not find a ready tiller pod  
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Max Moon @dustinvb

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add --replicas option for Tiller HA fixes #2334 by onorua · Pull Request #3464 · helm/helm

Introduce –replicas option to configure amount of Tiller instances on the cluster. Fixes #2334. The next PR will be about distributed lock, this one is just exterior.

dustinvb avatar
dustinvb

I haven’t ran into this scaling issue yet.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

—replicas option looks nice

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@michal.matyjek @Daren have you run into this?

Max Moon avatar
Max Moon

I have not run into this yet either

Daren avatar

I have not

michal.matyjek avatar
michal.matyjek

not yet

michal.matyjek avatar
michal.matyjek

how many deployments are we talking about?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

just concurrency

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we’re running helm on every PR synchronization for unlimited staging environments

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we’re getting it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. 2 developers push at around the same time

2018-11-19

sarkis avatar

hey all - curious what the verdict is on kiam vs kube2iam… it seems like kiam was created to address some issues with kube2iam - is kiam the way to go these days?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@sarkis yea, kube2iam is dead and should not be used. It’s a massive liability to even deploy in an AWS account. If you run more than N hosts (N ~10), you’ll DoS AWS APIs and they rate limit you.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

kiam addresses this by having a client/server model. clients run on all nodes (agents), and talk to the server. the server is responsible for fetching the credentials which reduces rate of requests

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it also caches

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think there’s been some frustration related to the rate of development on Kiam, but the worse bugs are fixed.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, I don’t know of any alternatives to kiam and kube2iam for AWS

sarkis avatar

thanks @Erik Osterman (Cloud Posse)!

2018-11-21

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
futuresimple/helm-secrets

A helm plugin that help manage secrets with Git workflow and store them anywhere - futuresimple/helm-secrets

2018-11-28

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


You can use a PodPreset object to inject information like secrets, volume mounts, and environment variables etc into pods at creation time.

    keyboard_arrow_up