#kubernetes (2019-06)

Intresting tool that checks K8s best practices https://github.com/reactiveops/polaris

what are you guy’s strategy for memory requests? for example looking at my historical data, my api pods use about 700Mi memory on average. I believe it’s better to set that memory request down to around that number, which will allow for more excess memory in the pool. I have it currently overallocated (1000Mi per api pod) and it adds up how much memory is being reserved but unusable by others that may need it.

Other considerations to take into account is (a) how much memory volatility there is… perhaps 30% variance is a bit high (b) disruptions - how bad is it if the service is evicted to another node?

I would suspect the more pods of a given service you run, the more insulated you are from disruptions of pod evictions

which means you can get by with a a 5-10% limit. make sure you monitor pod restarts.

so long as that number stays at or near 0, you’re good.

how are you all connecting kubectl into the k8s cluster these days?

via teleport

teleport supports both ssh and kubectl

SAML authentication

what they call proxy is ~ a bastion, for a centralized entry point

https://gravitational.com/teleport

Interesting ty

thanks @Erik Osterman (Cloud Posse) i think i can get away with closer to 5-10%. dont have that much memory volatility looking at my metrics

hi

Alex Co, [Jun 6, 2019 at 143 PM]: i’m having an issue while looping the helm template

env: {{- range .Values.app.configParams }} - name: {{ . | title }} valueFrom: secretKeyRef: name: “{{ .Values.app.secretName }}” key: {{ . | title }} {{- end }}

this is my code in the template to generate the environment var from the values.yaml

but when i ran the helm lint, it complaints like this

executing “uiza-api-v4/templates/deployment.yaml” at <.Values.app.secretName>: can’t evaluate field Values in type interface {}

i guess that helm template does not allow me to put the secretName value inside a loop

is there anyway to solve this ?

nvm, it’s because i did not declare .Values.app.secretName as global variable

I encountered an issue with eks ebs volume provisioning, with small worker groups (less than 3) the pv was created before the pod and in the wrong AZ.

is settting volumeBindingMode: WaitForFirstConsumer enough on v1.12 to fix this problem?

Yes @nutellinoit It works. But I would also suggest to set an affinity policy for one AZ only, to ensure in case of pod restart or eviction, the pod be scheduled on the same AZ of the PVC

Hi People , do you know of a best/sane way to install k8s on AWS. I see that there are multiple ways to do it. I am eyeing kops because terraform duh but before creating the cluster there’s still a lot of preparation to do like:

• creating vpc
• kops state bucket
• route53 record And than all of it has to be passed on to kops as a cli command. This is all fine but to me it looks like a bit too much. Is there any other way of doing it ?

I use EKS

with terraform

ok that’s a way

and how do you handle upgrades, i read somewhere that it’s a bit tricky with EKS

You mean master version upgrade ?

like k8s 1.2 -> 1.3 upgrade

Yeah, well its a pretty new cluster

right now my cluster is running on version 1.2

what are the challenges that you have heard of ?

I don’t remember the details but I think Erik mentioned something about the upgrade in EKS is not as easy

I might be wrong though

https://aws.amazon.com/blogs/compute/making-cluster-updates-easy-with-amazon-eks/

aws blog says its easy

:)))

fair enough

offcourse we have multiple environments

so we can upgrade the lower environment and check if it works

and proceed with upgrade

niice

they made it much easier recently - you can do it via the AWS console, just change the version

then upgrade your worker nodes afterwards

(but yes you’ll want to do it in non-prod first just in case)

Terraform has a parameter for version

oh that sounds promising

https://www.terraform.io/docs/providers/aws/r/eks_cluster.html

and what about installing it, EKS gives you the master nodes only, what about getting the other nodes in EC2, is it just a matter of using cloud-init ?

We do with with auto-scaling-group

You can follow this

https://learn.hashicorp.com/terraform/aws/eks-intro

Thanks a lot people

I will try it out

i tried an upgrade with eks and terraform

from 1.11 to 1.12

is pretty smooth

control plane upgrades without downtime

to upgrade the workers the only thing to do is to update amis

and replace workers

and follow the directions on aws documentation to patch system deployment with new container versions

https://docs.aws.amazon.com/en_us/eks/latest/userguide/update-cluster.html

ok, so I guess there’s a posibility to automate replacing the instances in the AS somehow

I will look into it

Thanks for your support people!

You can simply terminate one old instance at time and wait for autoscaling group to launch replacements

arhg ye gute ole click-ops

we’ve developed a lambda with step functions that does the instance replacement, step functions serving as a waiter

so it’s fire and forget

takes a while but it’s atomic

@Nikola Velkovski I’ve found the k8s upgrades to be a bit slow. it increases in time (by like 5~7 minutes per worker node) so upgrades can take a long time. For me, I wouldn’t be comfortable letting the upgrade for a production cluster run unattended (i.e. overnight while im sleeping) and naturally your production cluster probably has the most worker nodes. What I’ve found works for me pretty well is just using terraform to spin up a new cluster, deploy to the new cluster, and doing the cutover at the DNS level. food for thought

that sounds a lot like aws Elasticsearch @btai terraform apply usually times out when upgrading the ES cluster

thanks!

@Nikola Velkovski give a try with rancher. It is the most easiest way to spin up k8s on multiple clouds as per our experience with the tool. https://rancher.com/

@rj I saw that as well but I didn’t know if it was any good.

Hey Guys Does anyone configured SMTP as a grafana config-map for kubernetes?

apiVersion: v1 kind: ConfigMap metadata: labels: app: grafana name: grafana-smtp-config-map namespace: monitoring data: grafana.ini: | enabled =true host=<host> user=<user> password=<password> skip_verify= false from_address=<email> from_name=Grafana welcome_email_on_sign_up=false

Ex: something like this

and adding this config map in kubernetes grafana deployment - configMap: defaultMode: 420 name: grafana-smtp-config-map name: grafana-smtp-config-map

i am trying using above methods to add smtp to grafana.ini

but i am unable to add smtp to grafana.ini, is there any documentation/suggestions which can help me here?

Does anyone have any experience scaling with custom metrics from Datadog across namespaces (or the external metrics API in general)?

hi, anyone here is using Gloo Gateway on K8s ?

i’m having a problem that the virtual service stopped accepting traffic after awhile, and status on the ELB to gloo gateway proxy show that it ’s OutOfService

wonder if anyone here got the same problem

Hi all! Has someone faced this error before?

kernel:[22989972.720097] unregister_netdevice: waiting for eth0 to become free. Usage count = 1

https://github.com/hjacobs/kubernetes-failure-stories

Public #office-hours starting now! Join us on Zoom if you have any questions. https://zoom.us/j/684901853

what ingress controller are you guys using? it seems like alb-ingress-controller isnt quite robust enough for me. things that i feel like its missing:

1. new ingress object = new ALB so there would be a one-to-one mapping of ALBs to services for me (multi-tenant cluster)
2. provisioned resources don’t get cleaned up, at this point i feel like i might want to terraform the load balancer resources i need with the cluster

Im using Ambassador.. a lot of features regarding routing of traffic based on any kind headers, regex matching, Jaeger tracing. Name it :)

@maarten does ambassador spin up cloud resources for u? (load balancers, security groups, etc)

i realized that might not be a feature i want as of now in k8s. since terraform is better at managing cloud resource state

Can someone point me to best practices for setting up Traefik/Nginx-Proxy/etc as an ingress for Kubernetes running on 80? Everything is running but ClusterIP is internal and NodePort doesn’t allow ports below 30000. What am I missing?

I was really excited about this post https://medium.com/localz-engineering/kubernetes-traefik-locally-with-a-wildcard-certificate-e15219e5255d

but he’s using a LoadBalancer w/ Docker for Mac Kubernetes which doesn’t make sense.

Hello @davidvasandani thanks for the article that you’ve written. Could you please tell me the main capabilities that trafeik have as ingress-controller? Do you have any article with this capabilities?

Hi @Hugo Lesta. Not my article but Traefik has many capabilities. https://docs.traefik.io/configuration/backends/kubernetes/

This previous article you sent me seems worthy for me, I’ll try to improve my knowledhe about traefik over k8s.

Its good and helped me out but its incomplete. The author mentions using LoadBalancer locally but doesn’t describe how. With a lot of additional work I’ve gotten it working with MetalLB locally. This was a very useful article: https://medium.com/@JockDaRock/kubernetes-metal-lb-for-docker-for-mac-windows-in-10-minutes-23e22f54d1c8

anyone using this on their clusters? https://github.com/buzzfeed/sso

Looks interesting

Doesn’t support websockets, so it was a deal breaker for us

things like the k8s dashboard or grafana require that

bite the bullet. just deploy KeyCloak with Gatekeepers

havent heard of keycloak/gatekeeper

I can give you a demo

it’s open source, by redhat

we have the helmfiles for it too

does it integrate w/google saml?

yup, that’s the beauty with keycloak

it basically supports every saml provider

yeah it looks like

and we use it with gsuite

not only that, you an use it with https://github.com/mulesoft-labs/aws-keycloak

with aws

it can become the central auth service for everything

nice

we use it with kubernetes, teleport, atlantis, grafana, etc

yeah super nice, fully integrated

you can even integrate it with multiple auth providers at the same time

do helmfiles have a remote “helm chart” that is used as the base?

not sure, better to check in #helmfile

ah yeah it does

https://github.com/cloudposse/helmfiles/blob/master/releases/keycloak-gatekeeper.yaml#L4

oh, “base” is a loaded term now

since helmfile has the concept of bases

this is not a base in that sense

like uh

base docker image base

aha

yes, we use the community image

but guess it would be more secure to run our own

given the role this plays

i guess im not being clear enough

i was curious if the helm chart is just abstracted out

for a helmfile

for the gatekeeper, we’re doing something a bit unusual/clever

we are defining an environment

then using that to generate a release for each service in the environment

the alternative is to use sidecars or automatic sidecar injetion

https://github.com/stakater/ProxyInjector

ah

Here’s what our environments file looks like

i see

Have you people seen this ? https://github.com/hjacobs/kubernetes-failure-stories

I got to the spotify video, which is kinda cool, they admit their rookie mistakes around terraform

Hi guys, how to connect vpn from inside pods? any recommendation out there? So basically, I have to connect client data on premise, only using 1 ip

is anyone here using aws-okta with EKS? I’m having trouble granting additional roles access to the cluster.

Hey @cabrinha I’d be interested in hearing anything you ran into with aws-okta and EKS. I’m going to be starting down that road this week.

HI guys, Please could someone help me . I just created an EKS cluster and its unable to apply some changes to the cluster. I keep getting that error log.. I am using the same user I used to create the cluster. I am also using auth account. So the Users are not exactly in that account they assume role. Am not sure what am missing here being trying this for days now.. Thanks

How did you bring up the cluster? Are you using terraform?

https://github.com/kubernetes/community/tree/master/icons

I am stuck while installing Kubeadm in AWS on Amazon linux

Below is the error i get after running sudo install -y kubeadm

http://yum.kubernetes.io/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for kubernetes

Please help me to get over it.