#kubernetes (2018-08)

kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2018-08-28

pericdaniel avatar
pericdaniel

what are some interview questions that come up for k8?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m not quite sure - haven’t interviewed

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but I can share the line of questioning I use

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I like to start broad - first ask about the kubernetes architecture. what are all the daemons used to create a kubernetes cluster. the objective here is to see if the candidate is just a user or operator.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if they don’t know, then they are a user. if they answer correctly (e.g. api server, controller, kubelet, proxy, etc), then I dig down into each one of those to see what level of depth they have.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if they are just a user, that’s cool too - with kops, you barely need to know the underlying services any more

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then I ask them to rattle off as many resource types as they can. it shows what they’ve used.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then when to use what kind of resource type and when.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I like to ask what kinds of problems they’ve encountered in the past and how they solved them.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I like to ask what kinds of apps they’ve deployed and how they deployed it. if they deployed stateful apps, i’m always curious if the risks are well understood.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i like to know what integrations they’ve used with kubernetes. for example, if they haven’t used helm, that would be a redflag.

pericdaniel avatar
pericdaniel

thank you!

pericdaniel avatar
pericdaniel

this helps!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Interesting….

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

<https://github.com/kubernetes/charts> now redirects to <https://github.com/helm/charts>

2018-08-24

Max Moon avatar
Max Moon

Kind of an edge case scenario, but lead me to finding some (older) issues for nginx-ingress, but other folks might run into this as well.. There is a known race condition in the 0.11 release of nginx-ingress that leads to a race condition that causes the mechanism that retrieves secrets to fail, if and only if, you are creating multiple ingresses w/ TLS enabled.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is with kube-lego?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@rohit.verma

Max Moon avatar
Max Moon

correct

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

btw, rohit contributed cert manager

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but I haven’t tried it out yet

Max Moon avatar
Max Moon

i haven’t gotten around to moving to cert manager yet either

Max Moon avatar
Max Moon

From the git issues I was seeing, the main guy that works on nginx-ingress (git user: aledbf) fixed it pretty quickly in later releases, apparently should be stable in >= 0.13

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think this issue is what led @dave.yu and @jonathan.olson to ultimately switch to ACM

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

less fancy and not dynamic

dave.yu avatar
dave.yu
07:29:08 PM

@dave.yu has joined the channel

Max Moon avatar
Max Moon

yeah, i think that for staging (especially unlimited staging env) cert-manager or kube-lego is fine, but for prod, ACM (for now) is the way to go

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

have you set that up?

Max Moon avatar
Max Moon

in the process of

Max Moon avatar
Max Moon

Also, learned a bit about this, this morning, very cool. https://cilium.io/ Saw it featured on the TGI Kube thing that Joe Beda/Heptio does

Cilium

Linux-Native, API-Aware Networking and Security for Containers. Open source project, Fork me on Github

Max Moon avatar
Max Moon

FWIW: Updated to use nginx-ingress 0.15.0 and all my services are still available and the race condition is fixed

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
[nginx-ingress] Upgrade to 0.15.0 · Issue #31 · cloudposse/helmfiles

what Upgrade nginx-ingress to use docker image 0.15.0 why Fix race condition when using TLS and kube-lego references https://sweetops.slack.com/archives/CBW699XE0/p1535138518000100

Max Moon avatar
Max Moon

thanks!

rohit.verma avatar
rohit.verma

@Erik Osterman (Cloud Posse) We have even stopped using certmanager and nginx ingress controller , we are currently using ACM and kube-aws-ingress-controller

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thanks for the update

rohit.verma avatar
rohit.verma

This is a better setup if we are not require to rewrite targets

rohit.verma avatar
rohit.verma

I would also recommend to switch to CoreDns

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

how was kube-aws-ingress-controller to setup? is this the same are the original coreos alb ingress?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(does kube-aws-ingress-controller use ALBs?)

rohit.verma avatar
rohit.verma

yes kube-aws-ingress-controller use ALB

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

nice

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

did you need to specify the subnets?

rohit.verma avatar
rohit.verma

but its very different than coreos alb ingress

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or security groups

rohit.verma avatar
rohit.verma

no, it doesn’t work that way

rohit.verma avatar
rohit.verma

it takes only region

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(the coreos one didn’t autodiscover a lot - at least when we tried it, and ultimate felt it wasn’t worth it)

rohit.verma avatar
rohit.verma

but then discover autodiscover all components

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

sweet!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

do you have a helmfile you can share?

rohit.verma avatar
rohit.verma

coreos had only single alb ingress controller, they have another forwarder component as skipper

rohit.verma avatar
rohit.verma

i don’t have helmfile but can share my manifest.

rohit.verma avatar
rohit.verma
01:38:48 AM
rohit.verma avatar
rohit.verma

it automatically discover the acm also which we have created as part of root modules

rohit.verma avatar
rohit.verma

another cool thing would be to enable aws-iam-authenticator by default

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so you’re not using the zalando helm chart?

rohit.verma avatar
rohit.verma

nopes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

how come?

rohit.verma avatar
rohit.verma

my policy is, if the setup is not too complicated kubectl apply is preferred

rohit.verma avatar
rohit.verma

this a very simple setup

rohit.verma avatar
rohit.verma

and if the chart didn’t work in 3 trials , use kubectly lack of time

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(haha, you had me googling kubectly)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, a bit of time goes to maintaining charts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

interested to see what other engines will be introduced in helm 3

rohit.verma avatar
rohit.verma

i am awaiting when helm3 will be launched, specially as of namespace limitation

rohit.verma avatar
rohit.verma

i couldn’t use helm to manage our internal service

rohit.verma avatar
rohit.verma

hey did you got chance to try with .dockerenv in xx.cloudpose.co modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re doing something similar for Caltech. Building a kubernetes-in-a-box distro with geodesic as the base image

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

And a pretty setup menu.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re writing the env to /localhost/.geodesic/env and then sourcing it on load.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s similar in design, but not using a .dockerenv

rohit.verma avatar
rohit.verma

nice

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, we’ve got a poc running #geodesic containers in CI/CD

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(codefresh)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

lots of PRs in the past week to polish up the geodesic base image for that

rohit.verma avatar
rohit.verma

if I got it right, you are saying you did setup a ci/cd for complete dev.cp.co environment, correct?

rohit.verma avatar
rohit.verma

that is very cool, i was trying to samething at some point of time,

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, we got it as far as running init-terraform and terraform plan

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

nothing precluding apply

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

rohit.verma avatar
rohit.verma

but doing that with Codefresh, isn’t unsecure, cause geodesic require nearly adminacess to setup env

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, going to start adding some testing

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

CD of infrastructure requires admin access pretty much

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

either that be CodeBuild, Jenkins, or any other system.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

our strategy is multipronged

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

use multiple pipelines for different kinds of CI and CD

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and use multiple codefresh accounts, one per stage

rohit.verma avatar
rohit.verma

I agree but I will rely on instance profile more than a exposing admin credentials

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, just codebuild is more teadious iMO to work with

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

codefresh enterprise supports running agents on prem

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, the way we’re currently pursuing this is still running aws-vault inside of codefresh

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

to generate short-lived sessions

rohit.verma avatar
rohit.verma

that is much better, i would probably continue with codebuild since we are not using aws-vault also

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea… you’re diverging but that’s cool

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

rohit.verma avatar
rohit.verma

once you are finished, if you decided to openup I will translate to a codebuild pipeline for you

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that would be cool

rohit.verma avatar
rohit.verma

anyways since this is very limited build minutes, it would be easily covered under build plane

rohit.verma avatar
rohit.verma

by the way do you come across this project https://github.com/GoogleContainerTools/kaniko

GoogleContainerTools/kaniko

kaniko - Build Container Images In Kubernetes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Haven’t seen it

rohit.verma avatar
rohit.verma
Similar tools include:

img
orca-build
umoci
buildah
FTL
Bazel rules_docker
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

is this something you’re researching to implement?

rohit.verma avatar
rohit.verma

yes, we are using ci agents in kubernetes. Since most of them use dind and rely on host docker

rohit.verma avatar
rohit.verma

its insecure as well as we experienced that its not good to have more than 1 agent per machine

rohit.verma avatar
rohit.verma

docker build is actually a single threaded command and use some intrinsic locking

rohit.verma avatar
rohit.verma

if we are able to use these projects with our agents, we can actually run many agents with our cluster

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, good point

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

just came across img the other day

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not something though we’re optimizing for

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

honestly, there’s too much to solve

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i’d rather not also have to solve building

rohit.verma avatar
rohit.verma

just checking

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

have you used probot/hubot?

rohit.verma avatar
rohit.verma

nope, never heard actually

rohit.verma avatar
rohit.verma

we are mostly on gitlab

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i like the way the kubernetes/charts PRs work. how authorized users can issue commands via comments.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

want to do that via slack and github

rohit.verma avatar
rohit.verma

this is also very cool, github is actually coolest of all. I wish my org wasn’t such a miser to use the free gitlab

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha, these days it feels like i just hear how awesome gitlab is

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
04:29:49 AM

Geodesic “Kiosk” Mode

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Using dialog we’ve created a simple menu system to spin up kubernetes clusters with the full ML stack they need.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

One-click create/destroy cluster

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Uses helmfile to install all charts

2018-08-21

tarrall avatar
tarrall
01:23:14 AM

@tarrall has joined the channel

2018-08-20

Daren avatar
Daren

@Erik Osterman (Cloud Posse) No. Id wait for an official upgrade path., but it will be welcome

2018-08-18

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
10:23:50 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren @michal.matyjek are you using this? I didn’t know about this.

michal.matyjek avatar
michal.matyjek

I do not believe we do, first time I see this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Helm - The Package Manager for Kubernetes.

Documentation for Helm - The Kubernetes Package Manager.

2018-08-16

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
kubernetes/kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management

2018-08-15

Dylan avatar
Dylan
07:38:42 PM

@Dylan has joined the channel

2018-08-08

pericdaniel avatar
pericdaniel
03:05:30 PM

@pericdaniel has joined the channel

mholttech avatar
mholttech
01:54:59 AM

@mholttech has joined the channel

2018-08-05

jylee avatar
jylee
04:24:24 PM

@jylee has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cert-manager and/or kube-lego hit Let's Encrypt too aggressively · Issue #407 · jetstack/cert-manager

/kind bug What happened: Hi, I&#39;m an engineer at Let&#39;s Encrypt. I think you may also have heard from my colleague @cpu. We&#39;re finding that a lot of our top clients (21 out of 25, by log …

2018-08-03

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
weaveworks/eksctl

eksctl - a CLI for Amazon EKS

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
eksctl - a CLI for Amazon EKS attachment image

a CLI for Amazon EKS

2018-08-01

Phil avatar
Phil
08:13:14 AM

@Phil has joined the channel

my-janala avatar
my-janala
02:23:42 PM

@my-janala has joined the channel

    keyboard_arrow_up