#kubernetes (2020-02)

@rms1000watt @Erik Osterman (Cloud Posse) I’m late to the convo but:

1. I think aws-service-operator was just renamed (old one archived), not that the project was canceled: https://github.com/aws/aws-service-operator-k8s
2. Crossplane looks interesting: https://crossplane.io/

crossplane looks interesting for sure

You see this too? https://github.com/hashicorp/terraform-k8s/ https://www.hashicorp.com/blog/creating-workspaces-with-the-hashicorp-terraform-operator-for-kubernetes/

(official one by hashicorp - but only works in combination with terraform cloud)

i havent

yeah, i was considering terraform workspaces

https://github.com/aws/containers-roadmap/issues/596

in short, i’m trying to build docker images on our new jenkins stack built 100% with node groups now and the docker builds fail due to networking

FYI: https://github.com/aws/containers-roadmap/issues/596

@Erik Osterman (Cloud Posse) @Andriy Knysh (Cloud Posse) I thought we could set the extra args for enabling the docker bridge, but it seems like that’s not how the https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh#L341 is setup

Are you doing docker builds under jenkins on kubernetes?

yuppers

planning to look at img, etc again, but the new jenkins needs to match the old for now

aha

yea, b/c I think getting away from dnd or mounting docker sock is a necessity

there’s seemingly a dozen ways now to build images without docker daemon.

are you mounting the host docker socket?

agreed. that’s the goal in an upcoming sprint

yuppers

they build fine. they can’t communicate externally

simple pipeline:

pipeline {
   agent {
       label 'xolvci'
   }

   stages {
      stage('docker') {
         steps {
            container('jnlp-slave') {
                writeFile file: 'Dockerfile', text: '''FROM openjdk:8-jre
                    RUN apt update && apt upgrade -y && apt install -y libtcnative-1'''
                script {
                    docker.build('jbtest:latest')
                }
            }
         }
      }
   }
}

hrmmm but they can talk to other containers in the cluster?

seems so

do you also have artifactory?

(thinking you could set HTTP_PROXY env)

and get caching to boot!

yes on artifactory

artifactory runnign in the cluster?

not yet

what would i set HTTP_PROXY to? cluster endpoint?

some other service in the cluster (assuming that the docker build can talk to services in the cluster)

e.g. deploy squid proxy or artifactory

that said, I did not have this problem when I did my prototype on digital ocean, so I am guessing whatever you are encountering is due to the EKS networkign

yeah, it is def’ eks. it isn’t an issue when using rancher either

ok

is rancher using ENIs?

not sure. just tinkered w/ it mostly

might can downgrade to 0.10.0 since it supports inline in userdata.tpl

worst day to fool w/ getting disparate nodes on a cluster.

lol

yea, trying to get this working with managed node groups might be wishful thinking

yeah, i ditched that. i’m back for this group to a custom one with https://github.com/cloudposse/terraform-aws-eks-cluster

AWS support said it literally is not possibly with node groups

are you back to using terraform-aws-eks-workers?

for this one group of workers, yes

it isn’t connecting, though

i say that and it connects

lol

it wouldn’t work with the --kubelet-extra-args='--node-labels=node_type=cloudbees-jobs', though

specifically:

bootstrap_extra_args          = "--enable-docker-bridge=true --kubelet-extra-args='--node-labels=node_type=cloudbees-jobs'"

manually tweaked the launch template to not use = and that seemed to work for the bridge

trying again w/ the extra args

ugh…yup. frickin’ =

^

^ booooo

have you seen this https://kubedex.com/90-days-of-aws-eks-in-production/

and yes, you can connect managed nodes (terraform-aws-eks-node-group), unmanaged nodes (terraform-aws-eks-workers) and Fargate nodes (terraform-aws-eks-fargate-profile) to the same cluster (terraform-aws-eks-cluster)

an example here https://github.com/cloudposse/terraform-aws-eks-fargate-profile/blob/master/examples/complete/main.tf

yeah, got managed and unmanaged now. waiting on fargate in us-west-2

i saw a link to that 90 days earlier. gonna peep

annnnnnnnd the builds are working

phew. come on saturday!

https://github.com/weaveworks/flagger

Flagger is the first controller to automate the way canaries “should” be done

Oh yeah, this is the one istio integrates with

with prometheus metrics

it supports nginx-ingress too

https://docs.flagger.app/usage/nginx-progressive-delivery

hmm.. still requires 2 services. Not a deal breaker. Just interesting.

in a perfect world, I can still just helmfile apply and win

fwiw, we’re doing some blue/green deployments with helmfile

and using hooks to query kube api to determine the color

Ah, nice

that is pretty interesting

Ingress-nginx? That project has recently changed termination behavior. Details in change log.

So not actually ingress-nginx, we are just using an nginx container behind ambassador. This is a legacy port to Kubernetes from an ECS system

Our ingress comes through Ambassador

Okay. Hard to say, having a preStop sleep and termination grace should allow single requests enough time to finish. Websocket clients will need to reconnect though (onto new pod). So just server side isn’t enough.

Yeah I’ll have to play around with termination grace periods a bit more I think

Thanks!

Also make sure to tune maxUnavailable

Ok…had a permission issue. We set fsGroup to 995 on our old cluster. It is 994 on the new one. All is well.

Now to get away from using the docker socket all together.

Nifty. I’ll give it a whirl (as I’ve been looking for an excuse to load me up some rancher). Thanks for sharing

got it installed the other night in minikube but the rancher chart defaults to using a loadbalancer IP that doesn’t work for me. I’ll tinker with it more. Either way, thx for sharing.

hmm. For me (Docker Desktop on Mac) it created an Ingress mapped to localhost, which worked great.

I don’t think I did anything special for it to do that.

Can you make this markdown or a snippet instead?

Ya my bad. The Yaml file went all wonky on mobile which is where I posted this from. I’ll post the Yaml from Mac in a snippet

I was looking at https://www.rabbitmq.com/web-stomp.html and notice it says if enabling the webstomp plugin use port 15674, also seems with TLS and SSL they are using 15673 not 15674. But maybe I am confused if this plugin is required

How long have you been using Rancher? How do you like it?

Using Rancher for over 18months. Mixed reviews. Lot of open issues. Support is scattered. Product is getting better now though I feel they try to focus on their own products like k3s

Getting SSL was like pulling teeth.

No but I did look into it out of curiosity. It has a krew plugin and only a few actual implementations but looks fairly promising for scaffolding out new operators (or that’s what I think its supposed to be used for….)

last time I checked it out there were only 2 apps in the repo

looks like the may have added a few though. You looking to make an operator?

we’re looking into defining a CRD to describe an opinionated “microservice” that meets a specific customer’s needs

we’re looking at it less as a way to share existing operators - so don’t mind if there’s a limited selection.

That’s sort of what drove me towards it before. Sorry for my non-answer but I definitely have interest in it if anyone else pipes up on this one.

hey - if it’s on your radar, it’s added validation in my

It just seem that there should be an easier way to create CRDs right?

I’ll revisit and try to deploy the spark operator one for the heck of it, I have a partial need for such a thing anyway

(spark on kube is awful migraine inducing yuckiness of technology….)

it’s really early in our exploration. basically, we want to be able to define something like

buildpack: microservice   # the type of application lifecycle
deployment: { strategy: rolling }
public: true 
healthchecks:
  liveness: /healthz
  readiness: /readyz
replias: [2, 10]
port: 8080
resources: { limits: { memory: 2g, cpu: 300m }, requests: { memory: … }}
variables:              # standard environment variables 
  DEBUG: false
environments: [ preview, prod, staging ]  
addons:
  - redis       # or memcache
  - postgres    # or mysql
  - s3-bucket
  

and it will automatically configure our pipelines for a type of microservice and setup the backing services.

we want to be untethered to the CI/CD technology (E.g. use codefresh if we want to), use helm or helmfile if we want to, and not be vendor locked into some rancherish solution - which albeit is nice - locks us into technology decisions which we don’t get to make.

we’d like to be able to use Elasticache Redis in production, while a simple redis container in preview environments.

we’d like to have the settings to connect to these automatically populated in the namespace so the service can just “mount” them as envs

so will the addons be then deployed via service catalog then (or will you abstract calls to sc rather)?

something like that. so we can deploy the operator in dev (for preview environments for example), and it would use containers for postgres and redis

or we deploy the same operator in staging and then it uses RDS and elasticache

etc

really just riffing out loud right now. very early stages.

right right, I’m following. I so wish I were in a position to offer actual help on this but it’s been on my mind as well. It seems that complex helm charts simply isn’t enough for a holistic solution

it’s a building block. i still stand by helm.

i don’t want something that (we didn’t write - so to say) that tells me how to do everything

crossplane does some abstraction of elements for cluster deployments but doesn’t address the microservice/team needs

because the minute some customer says they want X instead of Y the entire solution is null and void

I don’t NOT stand by helm, it serves a purpose for certain

so the operator-as-a-pattern is interesting for these reasons

but then maybe not as powerful?

I don’t know about less powerful, but anything that requires an operator to function inherently makes it less maneuverable? Maybe not the right word.

That being said, have you done anything with OLM at all?

https://github.com/operator-framework/operator-lifecycle-manager

Hey, so right off the top, kubernetes 15.x seems to be expected to install an operator that I want to run:

though it seems operator specific, zookeeper installed without issue (because of course dumb ol’ zookeeper will work….)

Huh, so you’re looking to basically make a CRD version of monochart?

I imagine it as a more contextually aware version of it but I’m probably not seeing the whole picture

I’ve been wanting to build a kubernetes operator just as a learning experience, but I haven’t had a proper idea that also seems feasible for my level of programming knowledge~

everytime i think i have an idea, i find a better way to do it with built in stuff

yea, we always finding something else that works well enough.

I’m always looking for more glue though

this thread is basically my own inner monologue echoed back at me

we’re never going to get away from go-template+sprig though are we

Thank you for your response. Writing out custom code to handle both is definitely an option. I just wanted to be sure I wasn’t missing out on anything that’s already available OOB today. This helps.

its a great question either way. I’m still digging into it a little out of personal curiosity

How does HPA get/read the metric to scale up on queue messages?

Is there a way to more deterministically say something like “For every 50 messages in the queue, spin up a pod”?

That way when it checks, it can spin down to an appropriate amount as the backlog is handled

The prometheus adapter deployment allows for feeding custom metrics into HPA

If your deployment emits the right metrics, you should be able to scale on it I’d think

There’s a regular attendee of office hours who plays with this a lot, I can’t definitively remember his name though. I think maybe @dalekurt?

Hey

And @roth.andy

Sup

Cool, how many pi’s do i need ? raspberry pi 3 or should i got for the raspberry pi 4 ?

I have 6 3B’s. I wish they were 4s

The extra RAM is really nice. Once etcd and kubelet and whatever else you need is running there isn’t much room for anything real

If I could do it over it would be with the 4s with 4gb of ram

Noted

Power Over Ethernet (PoE) is worth it. Much less hassle than having to figure out USB C power distribution

I went with kubeadm so I could have multi-master, but now that k3s supports multi master I would use that

https://github.com/alexellis/k3sup

Thanks for the link

I’ve got 7 rock64 4gb and in the end back using 3 as masters and workers as VM with KVM (x86). I would just VM the lot next time (I tend to go in circles from full virt to metal and back)

https://www.engadget.com/2020/02/27/35-raspberry-pi-4-double-ram-2-GB/?guccounter=1

I don’t know the answer to this, but maybe you could share what you want to accomplish?

sry, have a hard time remembering to start threads

afaik autoscaler does not auto-apply node taints from the last time I looked into it on aks clusters.

[10:29 AM] but I was not being extremely exhaustive on what I was doing either, I found that using taints for steering deployments was not super intuitive and leaned into using node affinities to force preferences for workload node pools/groups instead. [10:30 AM] annotations seemed to carry over to autoscaled nodes just fine

https://github.com/kubernetes/autoscaler/issues/2434#issuecomment-592494391 https://github.com/kubernetes/autoscaler/issues/2434#issuecomment-592546084

answers there

The reason I want to understand the use-case is that it sounds like it might be related to working with spot instances.

https://github.com/pusher/k8s-spot-rescheduler/

there’s another controller to manage the node taints - just struggling to remember what it was called

https://github.com/aws/aws-node-termination-handler is the other one I was thinking off

@Erik Osterman (Cloud Posse) So, since i didn’t want to play with webhooks (however threy’re called), i decided to:

• add a taint and a label to ondemand nodes

• add a label to spot nodes (unused but is there for completeness) Now, pods that must run on ondemand have:

• a toleration for the ondemand taint

• a nodeSelector targetting the ondemand label (except for daemonsets) Pods that do not have these, won’t schedule on ondemand.

(the “webhooks” i meant are called admission control)

I know about base ondemand capacity but I think that making sure critical workloads are scheduled on the ondemand nodes with a rescheduler would be “eventually consistent”.

also my ondemand nodes are 4 times smaller than the spot ones, they’re just to run critical things like cluster-autoscaler, alb ingress controller, metrics server, ingress merge (is it still necessary even?), job schedulers…

That’s a nice solution @Karoline Pauls! I like the approach and it makes a lot of sense the way you did it.

thanks

yep, spin up cluster with baseline “cluster services” which include logging daemonset, monitoring, etc. then kick off deployment pipelines to deploy our stateless applications to the new cluster. (and we do a route53 cutover once that is all done)

so do you prestage the vpcs and such?

thanks for your time btw

@Zachary Loeber the cluster and its vpc get provisioned together. During the provisioning stage, it gets vpc peered with other VPCs (rds vpc for example). I don’t mind chatting about this at all. We have a unique setup and I sometimes wish I had more peeps to bounce ideas off of. While its been great for doing things like CVE patches and cluster version upgrades (esp the ones that aren’t exactly push buttom i.e 1.11 -> 1.12), it definitely requires more creativity sometimes because many turnkey solutions for things we want to implement don’t usually take into account k8s cluster that aren’t long lived.