#kubernetes (2020-05)

thanks guys

that’s awesome! let’s hear more about it on #office-hours next week if you’re around

sure, I missed the last office hours as I was taking the exam then

Yep, check out https://github.com/RothAndrew/istio-practice/tree/master/eks

uses Gateway, VirtualService, Cert-Manager with LetsEncrypt. Would love any feedback you might have

quick question then, if you deploy the operator via istioctl and use the demo profile, the ingress gateway it creates is usable?

or should you create a gateway for your published apps like you have done and have multiple gateways?

yep, for sure

it is usable

Also, each profile is just a set of defaults that get set, that you are free to override. In my deployments I have deployed demo with half a dozen or so overrides, like enabling HTTPS and SDS

so, should there be a gateway per namespace?

It’s really up to you. You can totally get away with just one gateway for the whole cluster. That way you have a standard set of rules, like always redirecting HTTP to HTTPS for example.

The limitation that I have discovered is, that if you are using SDS with cert-manager, then each Gateway gets one-and-only-one Certificate, and the Gateway and Certificate must be in the same namespace as the ingress deployment, which is almost always istio-system

You can assign as many dnsNames as you want in the Certificate resource

And thanks for this istio guide. I dig it for certain. With minor modifications it works brilliantly with a local kind cluster running metallb and defreitas/dns-proxy-server

Nice. Did you document any of those mods? I’d love to include that as another option. MetalLB looks awesome for LoadBalancers in other clouds such as Hetzner

https://github.com/zloeber/CICDHelper -> I just finished up my first round of making this hacky thing work

I do perform an istioctl operator deployment with a custom manifest generated from the demo profile to enable the ingress for prometheus, grafana, and kiali but that never seems to work properly from what I can tell.

so I gave up on it and just used your example to craft a quick helmfile to do the bookinfo deployment as an example instead

as always, thanks for the inspiration good sir.

(the istio profile example used was only tested for a kind cluster thus far, need to give k3d a whirl at some point too…)

And MetalLB is pretty nifty for testing, not entirely certain how performant it is but generically it does work

i don’t doubt there would be some connection errors, but I don’t consider that downtime

that’s failover working the way it should

you can also throttle the rate of the roll out

What are your concerns? … e.g. why would email click tracking need to be handled differently.

e.g. architecturally, i’ve seen this implemented with lambdas + kinesis, but that might not need to be an optimization most need

These are configs for things that aren’t tied to a specific app of ours, so it’s just a little unclear where to put them.

With that in mind I’m just thinking about what would be easiest to maintain and add new stuff to as needed.

like

          env:
            - name: DD_AGENT_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP

in your container env?

Right but Kong does not support referencing the Downward API (that declaration) so looking for an alternative solution outside of forking the supported Helm Chart for DD agent or the codebase for Kong

ah dang, sorry, i am of no immediate use here

no worries, thanks for helping!

@roth.andy

@roth.andy was the inspiration for that post

is the app a server app or browser app?

if a Docker container starts and the process exits right away, you would see the same behavior you described

for example, for a server app, you start an HTTP listener which prevents the app from exiting

-it mode starts an interactive session with the container, so even if the app exits, the container is being kept alive

I believe it’s a server app @Andriy Knysh (Cloud Posse)

I’ll try to find the error hold on

Then the app never launches, vs if I do -it in the docker run command it launches the app accordingly.

but I can’t find a way to specify -it when running the pod through the kubernetes dash board @Andriy Knysh (Cloud Posse)

you don’t need to specify that

something is wrong with the app when it runs in a container

a Node app should work in a container, and you should be able to access it from your local computer via the port binding (should be able to open a browser on the host port and see the app)

did you test that?

Yeah

It’s literally just the default create-react-app app

npm start opens it in localhost:3000

did you test it in a docker container on your local computer?

for example, using Docker compose like this:

yeah I have a docker compose file taken from the tutorial in my first link

and when you run the composition, you can see your app on localhost:3000 (or 3001 in your case)?

works fine for me

I’ll see what port hold on

or come to think of it, it just builds another image

which has the same effect of only working when docker run has the -it command after

my docker compose file:

version: '3.7'
  
services:

  sample:
    container_name: sample
    build:
      context: .
      dockerfile: Dockerfile
    volumes:
      - '.:/app'
      - '/app/node_modules'
    ports:
      - 3001:3000
    environment:
      - CHOKIDAR_USEPOLLING=true
    stdin_open: true
    tty: true

docker-compose up should build the image and start the container https://docs.docker.com/compose/gettingstarted/ https://stackoverflow.com/questions/36249744/interactive-shell-using-docker-compose

in your example, it just builds the image and exits

(that’s why you have to run docker run after)

so something is wrong with your app or docker compose

try to remove

stdin_open: true
tty: true

and run again

you should be able to run docker-compose up (it should build the image and start the container, with the -d option it should start the container and exit, but you should be able to see the container running) and then open a browser and see your app running on localhost:3001

only after that you need to think about deploying to Kubernetes (it has nothing to do with docker -it arguments)

oohhh gotcha, thanks

so I guess that there’s something up with create-react-app’s default application that doesn’t allow it to be dockerized

Thanks! I’ll try it when I get the chance. Sounds promising

thanks @maarten for finding the reason why the app exits after start

@Joey you should put the app in the CI mode, then run docker-compose up -d

and then see the app running in the browser at localhost:3001

@Andriy Knysh (Cloud Posse) @maarten it worked!! Thank you guys so much!

simply just adding CI=true before npm start

nice catch @maarten!

use the external-dns controller with route53

This is a cool project. We’ve deployed it and it works well.

We didn’t end up doing much with it yet, but have a PR open for it here: https://github.com/cloudposse/helmfiles/pull/186

yes, we do this on EKS, but we still use nginx-ingress

do you need to use the ALB with other AWS services?

If not, why not just use nginx-ingress with NLBs

Use NLB with random ports?

to use NLB with nginx-ingress, you just need to set 1 annotation. It’s a very quick win.

No need to worry about random ports. Or are you talking about some application requiremnet you have to use random ports?

I will have a deep look into that option.

you could use nginx-ingress as a reverse proxy to your application ports.

the vicious cycle of switching from writing code to configuration language back to writing code

ya, seriously

jenkins feels like an interesting case study related to this:

1. first there were freestyle jobs that everyone created with clickops
2. then came the Jenkinsfile littered with imperative groovy script
3. people loved it. wrote “pipelines as applications” so complicated that no one knew how they worked and they couldn’t be tested. people revolted.
4. then jenkins came out with declarative pipelines. people loved it. they converted all their groovy pipelines to the declarative style.
5. then came gocd, argocd, flux, spinnaker, etc. everyone started to hate the declarative jenkins pipelines.
6. yadda yadda yadda

in infrastructure as code. first we had things like boto (python) and wrote infrastructure as code in a pure language. that became untennable. so we all gathered around terraform and cloudformation based on the lessons learned. then that became limiting. so pulumi came out to show that we really need is to return to pure programming of infrastructure as code. aws followed suit with CDK.

everything comes full circle.

i suspect you’re probably more interested in the ‘source’ of the traffic rather than the destination

@joey the inbound rules don’t seem to be making a difference

I can still access the page after I delete them

you can access the page from something other than your ip?

actually… it’s buffering now I think there was just a delay

oh man @joeys for the win!!!

yeah it definitely makes a difference changing the inbound

working on my computer but not my phone now (as expected)

Do you know if there’s a way to add other people’s IP addresses?

Or would they just have to log into the console themselves?

destination is for your server connecting to other stuff, e.g. twitter api or onlyfans.com api

source is for anything coming IN to your server

gotcha

once a source-based tcp connection is established, your server can return data to it

makes sense

to add other people.. well.. depends on how creative you want to get

on second thought I think it’s totally doable just to type it in manually

then add /32

about to try with my phone

yes

easiest way is just get their ip and add the /32 (cidr for ipv4 single ip address) for their ip’s yourself

boom! It works!!!

beyond that, google has your answers

Thanks man! I really appreciate it

alright

you could get all sorts of creative and create an api with lambda that has access to update the security group when some criteria is met on your page that someone updates or something crazy stupid like that

good stuff to think about for the future

my project is incredibly rushed though lol

it really depends on what you’re trying to do, and thus no one here can necessarily tell you what you need to do in those cases

Have you looked at https://github.com/roboll/helmfile ? (and here is our collection of them https://github.com/cloudposse/helmfiles/tree/master/releases)

it solves a lot of issues from your list (but not all, and not “with the worst templating language I’ve ever seen” )

but I agree with your points that having just one tool (Terraform, and it’s a very good tool) offers many advantages

i kind of expect in the best case it will be “i had 8 problems with 1 tool” to “i have 5 problems with 2 tools”

I’ve actually deployed full team projects using pure terraform before

Honestly, there was a certain elegance to using metadata for implicit dependency chaining that was hard to deny.

But the devs hated HCL and almost immediately demanded some additional simplistic yaml formatting/pipelines for their dev environment to deploy configmap changes from

My use case was to deploy the cluster with workload in a single pipeline to facilitate an ephemeral cluster deployment

almost everything I read about doing such a thing told me that the cluster deployment and workload deployment to the cluster should be separate states (I ignored all the warnings and it worked fine for me though…)

Where I found that helm excelled was the post deployment tests I was able to run. Both tf and helm can deploy a workload that never actually works and be considered successful. I suppose that post-deployment e2e testing functionality could easily be done any number of other ways. I know that it is not baked into terraform the way it is into a purpose built Kubernetes tool is.

I’m having a hard time understanding your first statement though. ‘…if an initial deployment fails, none after will succeed..’

usually that is what you would want in your pipeline right?

perhaps you are overcomplicating your charts?

for instance, I almost never use subcharts to reduce overall complexity and downstream issues.

helm also can do diffs

and it stores state in secrets within kubernetes then does three way merging. I don’t think that terraform remote state and helm three-way merging is a like-for-like comparison. Remote state is offset by the fact that it is an additional outside dependency

I’ve already shared this in another workspace but: My biggest pain-point with the TF K8s provider is dealing with any project with CRDs (Istio/Spinnaker et al). To work around that I had to use the Helm TF provider which is excellent! I am still struggling with tools that come packed with their own CLIs and I hope the community shifts to embrace operators ASAP

Excellent point, any CRDs inherently are not supported in the terraform kubernetes provider. You only get common kubernetes resources to work with.

I’m having a hard time understanding your first statement though. ‘…if an initial deployment fails, none after will succeed..’ The reality looks like:

1. A developer presses the deploy button in the CI to deploy their branch
2. Release fails but the app runs
3. They don’t look and keep deploying when needed
4. Nothing happens

But the devs hated HCL and almost immediately demanded some additional simplistic yaml formatting/pipelines for their dev environment to deploy configmap changes from I also hate HCL but to prefer YAML over HCL is some serious stockholm syndrome.

If you add a helm test to your chart and subsequent deployment pipeline code you should be able to capture some of the false positives (if I read things right)

I ran into that issue at least. Deployments worked just fine but there were actual underlying container or service issues so no one was the wiser when some component wouldn’t be working. The helm test can be a simple pod that wget’s the service endpoint (or similar)

helm tests are an option but i may as well turn them into monitors once i have them

I’d say both are important

my idea is that if someone goes so far as to write a helm test (i had one.. once), it can as well be a monitor cronjob running every minute, sending metrics

I’d put the test directly in a pipeline for validation of deployment success and if a cronjob were to be used as the monitoring solution (not what I’d personally recommend) I’d make that part of the chart itself.

otherwise, including a prometheus rule as part of the chart to trigger alerts would be more holistic I’d think.

@Karoline Pauls, You always spark super interesting conversations btw

Do I? I thought I was just one person pissed at Jenkins.

what better reason to innovate better solutions than being riled up at the current ones?

regarding the cronjob, in my Salt days I would sometimes write custom Datadog healthchecks (in Python) that would run on instances, hence the idea to have an analogue on kubernetes.

The issue is, as I’m trying to get Jenkins to do what I need (currently a manual deploy step with a toggle for online/offline migrations, dev/prod, etc), it is burning me out. My external ALB ingress controller experiment saw no commits in a month.

I never got to actually deploy custom healthchecks in datadog (it seems to require a local agent which I didn’t have available to me in the last environment I worked in). I used terraform to setup metrics based alerts instead though. There are also some datadog integration project out there for auto-creating some alerts for kube based deployments I thought (https://github.com/fairwindsops/astro)

not what you are likely looking for though, sorry

I was thinking to change the location of the core dump to write to a persistent volume, but running into an issue because /proc/sys/kernel/core_pattern is read-only and I don’t know why. I’m running on EKS.

Locally I can edit that file only if i run docker with the “–privileged” flag, so my question is how do I do the equivalent with deployments/pods in EKS

Kubernetes handles the log rotation automatically

There are a few tunable parameters for this

Note, that containers started outside of kubernetes (e.g. docker in docker with jenkins) are not automatically log rotated

Thanks. Yes, thas is correct when we talk about containers that are able to redirect logs to stdout/stderror. But That application is not able to send logs to stdout/stderror. That application stores its logs on the container filesystem. Just writes into files /var/log/php/application_errors.log and /var/log/php/application_events.log So, I’ve mounted hostPath volume from node to each container to directory /var/log/php Now, I can scrape log using FluentD, that was deployed to my k8s cluster (DaemonSet). And everything is OK but the files are growing each day and should be rotated. It was what I asked about Legacy application is pain

Sounds like you need to setup logrotate jobs on the nodes.

https://www.digitalocean.com/community/tutorials/how-to-manage-logfiles-with-logrotate-on-ubuntu-16-04

But you told the application to log to /var/log/php/application_errors.log, right? In that case you can tell it to log to /dev/stdout or /dev/stderr

that way it will play nicely with the logging platform

you can also ln -s /var/log/php/application_errors.log /dev/stderr

inside the Dockerfile

I’ve done that exact thing before and it works well. Completely forgot about it earlier.

man 90% of keeping up with modern tech is remembering what to forget and trying not to forget what to remember

Great! Thank you, guys! Will try it today and get back with additional questions

Excellent list, highly recommended to run through this if you are just getting into kubernetes for your workloads.

Have a look at the events when you describe the pod, in a case I’ve dealt with in the past it was that it couldn’t allocate an IP

you should write a script to send your cluster a bunch of traffic or use something like ab or gatling or vegeta or any number of load testing tools

They’re having an outage: https://status.quay.io/

Consider implementing this https://docs.docker.com/registry/recipes/mirror/

yea I know that they have an outage I am just wondering how people mitigate it in general

so for me I am removing a node every hour from our dev cluster and adding a new one just to see how it would feel to have short lived nodes and now quay was down (500’s) and the node could get the load up and running

so it is my fault but was just curious to hear how people handle docker images are they all hosting them themselves or would they be affected by this as well

use https://www.sonatype.com/nexus-repository-oss

this outage has been going on for awhile now https://status.quay.io/incidents/kw2627bsdwd9

yeah… ouch.

i’m curious what the root cause was. they had ~19 hour outage

ya, it seems like it

sometimes I swear that the matrix is glitching on me, I’m deep diving into argocd right now

If I go outside for a walk and see 3 instances where someone is walking a pet cat or iguana or something I’ll know I’m in the matrix…

you happen to find direct links to the CRDs for this thing?

nm, just used my gcloud account to grab the files

Haven’t looked past the news article yet

https://github.com/kelseyhightower/kubernetes-the-hard-way

https://github.com/jamiehannaford/what-happens-when-k8s

https://kind.sigs.k8s.io/docs/user/quick-start/

Hrm… so I think what you’ll find is there is a wide gap between how it’s done on AWS vs Azure vs GKE vs bare metal, etc

Did you have one in mind?

Huh interesting. Yeah, I’d say I’d probably focus on AWS.

Your suggestion would be to tailor learning towards k8s on EKS?

I think there are two sides of it. On the one side, you need to learn k8s the platform (how to run stuff on k8s). that will be the most similar across cloud providers (but not identical).

@joey — Good stuff, I’ll definitely check those out. Thanks for sharing.

Then there’s operating k8s as a platform. that’s where you need to pick one cloud provider and kick the tires.

this is where you get the operational experience.

Makes sense.

and if you’re going to choose a specific cloud provider and do it like that, you might as well be using terraform and/or terragrunt

then I think the “Best Practice” will be determined if you want to go the AWS-native approach with cloud formation, or use the eksctl. Or if you want to use #terraform.

to use k8s effectively, you’ll inevitably need to provision a bunch of stuff that isn’t handled by EKS (e.g. IAM roles). so using terraform is advisable.

Yeah, I’m already very bought in on Terraform so that would be my approach for sure.

@Erik Osterman (Cloud Posse) Do you have a resource you’d suggest for the “you need to learn k8s the platform (how to run stuff on k8s)” side of things? I think that’s really what I’m looking for.

If you are on a linux platform you can look at using libvirt+terraform to get running pretty quickly with your own local cluster. https://github.com/zloeber/k8s-lab-terraform-libvirt

for getting comfy with kube deployments and getting around I’d just start up a kind,k3d,minikube, or microk8s local cluster and start looking to deploy things to it that you might find yourself deploying for work.

a sufficiently complex app that you could start with and feasibly could be in several types of environments might be airflow

You don’t need cloud resources to dive into the deep end pretty quickly with k8s

@Zachary Loeber Good stuff, I’ll check those out and keep that in mind. Thanks for sharing!

let us know how it goes, glad you are diving deeper into kube, take a deep swig of the koolaide….whatever they laced the kube-koolaide with is addicting

Yup that looks good

As an extra safety precaution you can use iptables to firewall direct access to the real metadata api

Also implicitly, you’ll need to provision the IAM roles that you will need

those are provided. In the account this will be in I don’t have permission to manage IAM

as of 3.5 on kiam you can use service account iam roles instead of instance profiles: https://github.com/uswitch/kiam/blob/master/CHANGELOG.md#v35

Not using EKS

allegedly you can do it without eks: https://github.com/aws/amazon-eks-pod-identity-webhook/

i’ve not seen or deployed this tho

Interesting

i’ve got kiam running on service accounts and it works pretty well - i only did it that way because .net core credential provider in the aws sdk didn’t support webidenties for a while so we couldn’t go full service account roles.

i’m probably going to rip it out soon, as the .net sdk supports it now.

but it’s been solid enough

Something far easier that I’ve seen done is to just add a Validating Admission Controller to make sure containers come from the registry you want.

But your description is obviously much more elegant than that

That is part of the preventative portion of a holistic solution for certain

I suppose the tool I’m thinking of would be for migration of outside dependencies to local registries only

this would be pretty cool, would love to hear if you find something before i set off on a path to try to do something related

Just make sure the juice is worth the squeeze. Unless you have compliance/regulatory requirements, something like that will add a huge operational headache

@btai @Pierre Humberdroz

Oh awesome !

It may be useful for figuring out how to eliminate core service outside dependencies

outside deps == evil

this looks awesome

reminds me a little bit of that uber project too

Three devops holy war creeds; 1. latest tag is evil, 2. outside dependencies are our enemy; 3. incremental improvement of all things….

kube-fledged use case #4: If a cluster administrator or operator needs to roll-out upgrades to an application and wants to verify before-hand if the new images can be pulled successfully.

cool beans

I was wrong about the chart not being available for this: https://github.com/senthilrch/kube-fledged/tree/master/deploy/kubefledged-operator/helm-charts/kubefledged

the name for “that uber project” is kraken https://github.com/uber/kraken

I ran across that one as well, the name certainly is apropos…

(this is all rampant, unqualified speculation. I have no inside knowledge of what’s going on.)