#kubernetes (2020-08)

external secrets operator

https://github.com/ContainerSolutions/externalsecret-operator

Thank you, Erik!

Do you have any experience or opinion about https://github.com/godaddy/kubernetes-external-secrets ?

Oh right, the reason I was looking at the one by ContainerSolutions was it supported 1Password.

I would choose the godaddy one instead if strictly using SSM

(1Password support is nice from a developer UX perspective and doesn’t require providing any access to AWS, period.)

it returns a 200 and some fancy response like this ;

{ "kind": "Deployment", "apiVersion": "apps/v1", "metadata": { "name": "zipkin", "namespace": "monitoring", "selfLink": "/apis/apps/v1/namespaces/monitoring/deployments/zipkin", "uid": "33341330-5dca-4f45-8645-7e74ac8c9464", "resourceVersion": "26994061", "generation": 2, "creationTimestamp": "2020-08-05T19:16:28Z", "deletionTimestamp": "2020-08-05T19:16:47Z", "deletionGracePeriodSeconds": 0, "labels": { "name": "zipkin" }, "annotations": { "[deployment.kubernetes.io/revision](http://deployment.kubernetes.io/revision)": "1" }, "finalizers": [ "foregroundDeletion" ] }, "spec": { "replicas": 1, "selector": { "matchLabels": { "name": "zipkin" } }, "template": { "metadata": { "creationTimestamp": null, "labels": { "name": "zipkin" } }, "spec": { "containers": [ { "name": "zipkin", "image": "openzipkin/zipkin", "resources": {

        `},`
        `"terminationMessagePath": "/dev/termination-log",`
        `"terminationMessagePolicy": "File",`
        `"imagePullPolicy": "Always"`
      `}`
    `],`
    `"restartPolicy": "Always",`
    `"terminationGracePeriodSeconds": 10,`
    `"dnsPolicy": "ClusterFirst",`
    `"nodeSelector": {`
      `"nodegroup": "eks-monitoring"`
    `},`
    `"automountServiceAccountToken": false,`
    `"shareProcessNamespace": false,`
    `"securityContext": {`

    `},`
    `"schedulerName": "default-scheduler"`
  `}`
`},`
`"strategy": {`
  `"type": "RollingUpdate",`
  `"rollingUpdate": {`
    `"maxUnavailable": 1,`
    `"maxSurge": 2`
  `}`
`},`
`"revisionHistoryLimit": 10,`
`"progressDeadlineSeconds": 600`   `},`   `"status": {`
`"observedGeneration": 2,`
`"replicas": 1,`
`"updatedReplicas": 1,`
`"readyReplicas": 1,`
`"availableReplicas": 1,`
`"conditions": [`
  `{`
    `"type": "Available",`
    `"status": "True",`
    `"lastUpdateTime": "2020-08-05T19:16:28Z",`
    `"lastTransitionTime": "2020-08-05T19:16:28Z",`
    `"reason": "MinimumReplicasAvailable",`
    `"message": "Deployment has minimum availability."`
  `},`
  `{`
    `"type": "Progressing",`
    `"status": "True",`
    `"lastUpdateTime": "2020-08-05T19:16:29Z",`
    `"lastTransitionTime": "2020-08-05T19:16:28Z",`
    `"reason": "NewReplicaSetAvailable",`
    `"message": "ReplicaSet \"zipkin-5d6d665688\" has successfully progressed."`
  `}`
`]`   `}` `}`

What is this about replicaset has succesfully progressed ? I want it gone xD

The deployment, the replicaset under it, and the pods under it, cascading from the deployment DELETE request

is this some kind of bug ?

Look into helm hooks

That’s how we typically do it.

Sweet. Will do.

Helm hooks are just jobs that have a special annotation and are executed at certain life cycle events

Sounds perfect. That’s the answer I was looking for. Thank man!

For out staging environment, we use pre-install hook in helm for creating a database and import a dump defore deploying for example

this is what ours looks like

apiVersion: batch/v1
kind: Job
metadata:
  name: "api-migration-{{ now | unixEpoch }}"
  annotations:
    # This is what defines this resource as a hook. Without this line, the
    # job is considered part of the release.
    "helm.sh/hook": post-install, post-upgrade
...

I’d also look into doing backwards and forwards compatible database migrations and https://aws.amazon.com/blogs/database/building-a-cross-account-continuous-delivery-pipeline-for-database-migrations/

But thats for… more intense requirements

@Vlad Ionescu (he/him) Yeah, looked into solutions like flyway. My client is a node / typescript shop and they’re using some up / down migration tool. But one thing at a time

We’ve been trying to answer the same question. Our client is a Java shop and they are using Liquibase extensively, migration scripts are located within the app and managed by the app during its startup. It worked pretty well in their pre-Kube env. The problem emerged with the migration to K8s. Now we’ve got locks here and there and unlocking all this stuff manually is a pain as there are quite some microservices with their own databases and a lot of envs containing all this stuff. It happens because we have several replicas, because we have cluster autoscaler, because a scheduler decides to reschedule the pod, because sometimes it takes too long for migration scripts to complete, etc. And it’s obvious that the current approach should be reconsidered and we should find a K8s-aware way.

Consider maybe a continuous delivery framework like ArgoCD

There are some projects trying to do something with liquibase locks

https://github.com/szintezisnet/Kubernetes-Liquibase-Lock-Release

and

https://github.com/oridool/liquibase-locking

Haven’t tried any of these. To me they don’t look mature, but who knows

And there are these folks trying to define your db structure as K8s objects:

https://schemahero.io/

Let’s not forget (and I’m joking…)

https://news.ycombinator.com/item?id=24618598

I’m wondering about this comment in ECS

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#network_mode

and how that could relate to K8s

I have no idea on your actual question — but just wondering @jose.amengual, you / your org moving away from ECS and towards k8s?

we are thinking

but we were having a discussion on latency

and I think Bridge mode is slow than having a NIC+IP on a host but even more so in docker

Aha interesting. Sounds like you have a large investment in ECS. Let me know where you land — I’m interested if you end up taking the plunge.

but since in K8s bridge is widely used I was wondering how latency plays in

we are going to move or I quit

I hate ECS

Hahaha damn, strong opinion! Gotcha man.

there is sooooo many little things that adds up

it is WAY easier to run docker-compose on instances behind a alb

using cloud-init

ECS was supposed to help you to run containers, but the learning curve is huge

is like learning chef or ansible

Hahah I’d say the learning curve for k8s is larger though… it just gives you a better, open source platform.

yes, maybe the curve is about the same but then the deployments and such is easier

not yet another vendor locking way of doing things etc

I think gke does this too to get better perfomance, so in your gke cluster the pods get an ip address of the real subnet, so you don’t have an overlay network inside k8s

gke uses calico

interesting

Haha, hate ecs I’ve been following ECS rebalancing issue (https://github.com/aws/containers-roadmap/issues/105) for few years but recently unsubscribed. What a relief it was

Are you load testing the k8 bridge anyhow? It will be interesting to use something like Apache AB from more than 6 IP addresses to cover the different Zones.

no I have not done any testing yet, just wondering if this was a thing

are all of the nodes getting restarted at the same time? can’t you stagger them out so pods can shuffle around and normalize between reboots?

all of nodes restart at same time … i was thinking probably its broking cluster network

is someone mandating that all of the nodes have to start at the same time?

that just sounds like poor decision making

if you don’t want stuff to break don’t restart everything at the same time

our infrastructure is managed by a different team and they do weekend patching for almost 500-1000 machines at the same time. so it’s difficult to request them to reboot servers in sequence

idk. i did something like this when i needed to do this exercise by hand.

https://gist.github.com/jfreeland/e937248a298e7d8e506b92b6bbc8ebad

there were some oddball cases where i couldn’t let the cluster-autoscaler do it for me so

“we have to restart 500-1000 machines and we don’t know how to do it properly” is a pretty nutty reason to risk taking a service down.

that particular script was focused on an aws_autoscaling_group but it’d be trivial to modify that to ssh to the node and reboot it as well. lots of options. is your kubernetes control plane on some of the nodes that are being recycled too?

They have to find a way to execute a “Rolling restart”. It should be possible in every cloud provider. For exmaple, in AWS Autoscaling group you have the instance refresh utility

And it’s callable via the AWS API (could be done right after the patch is applied, in your example)

It’s one way to tinker your way through it, but you guys gonna find your own hehe

haha yup I got it Thanks a lot for your inputs @JMC & @joey

https://github.com/stefanprodan/k8s-prom-hpa

https://github.com/DataDog/watermarkpodautoscaler

if i’m not mistaken https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#example-limit-scale-down-rate was added in v1.17 or v1.18.. i had to write a custom autoscaler in the past because kubernetes hpa didn’t have a cool down factor to not scale things down too fast

oh yeah, https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-cooldown-delay

This is great joey, thank you!

See this chart for inspiration and patterns:

https://github.com/helm/charts/tree/master/incubator/common

Awesome — Will check that out.

Does CP use the common charts?

I ask as I’m digging into it now and I’m wondering how widely they’re used by the community. Even though they’re going to make my charts more DRY… they are also going introduce some fun rabbit holes in where things are coming from since those common templates are doing a good bit.

We have until recently. See https://github.com/cloudposse/charts/pull/256

Interesting… so you’re pulling back on using it and bringing it in because they’re deprecating that repo.

Is monochart your --starter chart or is it an older version of a library chart?

It doesn’t seem to be marked a library chart and doesn’t seem to be using the library chart patterns, but then again I don’t know how folks were accomplishing that prior to helm3.

the monochart is our universal helm chart we use in nearly every customer engagement.

Using the monochart, we can deploy 99% of customer applications we’ve come across. It means we don’t need to develop an original chart for every application and thus reduces the techdebt and chart sprawl.

Gotcha. Seems I need to do some reading to grok that a bit more.

We look at the values.yaml as a custom schema that lets you define a custom, declarative specification for how an app should work.

Then we make the chart implement that specification.

then we make developers choose a specification.

Gotcha. And I’m guessing you then you use Helmfile to just provide values.yaml files to monochart for each application image your deploying?

That sounds like a pattern I want! Tomorrow I’m jumping into the old ref architecture so I’ll see what I find while digging into that.

Wow, I knew about the recent pricing changes and how orphaned images were going to be deleted.

I didn’t know about this:

Why is this operator required?
The primary reason for creating this operator, is to make it easier for users of Kubernetes to consume images from the Docker Hub after recent pricing and rate-limiting changes were brought in, an authenticated account is now required to pull images.
Unauthenticated users: 100 layers / 6 hours
Authenticated users: 200 layers / 6 hours
Paying, authenticated users: unlimited downloads
See also//www.docker.com/pricing)

DockerHub rate limits seems like a huge liability!

Right? I was also surprised by that!

more reasons to build up a pull thru registry

uff. You know what I just now see as an Issue? CI/CD !

Sorry to drag up this oldish thread but the great limit switching time is approaching… Just wondering if anyone has come up with a reasonably elegant way to tackle getting ready for this?

Sure, you can docker login or manage your docker config on nodes with user-data or you can create a Secret in every namespace and then rewrite all of your charts, deployment YAMLs etc to include the relevant imagePullSecrets oh, and re-deploy or kubectl patch them all… and update Jenkins Kubernetes plugin configs… Probably have to hunt down any special implementations in Jenkinsfiles … I’m sure you get the point.

Any experiences to share?

@tim.j.birkett did you see @Pierre Humberdroz’s recommendation? https://github.com/alexellis/registry-creds

Indeed I did see that @Erik Osterman (Cloud Posse) - it looks quite interesting and fairly similar to: https://github.com/zakkg3/ClusterSecret

For anyone else who stumbles across this during a search of Slack or the Cloudposse Slack archives, @zadkiel kindly enlightened me to this project: https://github.com/titansoft-pte-ltd/imagepullsecret-patcher - It manages the image pull secrets and patches the default service account in each namespace, keeping everything up to date on changes.

Not really sure how i feel about that. I’d rather manage things via terraform, or am I missing something?

ya im still looking into it. it does look promising. and it doesnt do all aws like terraform does so it’s not going to be a replacement

I don’t even know if Fluentbit + Firelens on EKS is a thing after looking for a few… but I would be interested if anyone got DataDog logging working properly with EKS Fargate services. This seems like a real pain.

Can’t DD just pull from Cloudwatch ?

Haha but still need to ship logs to cloudwatch and then also pay them as well.

I believe I found the solution. I need to mount a emptyDir volume to both agent and application and then configure DD to look at that location. It’s just tedious and I’m surprised it’s not documented well.

I POC’d the Firelens + Fluentbit with ECS Fargate, I got it working pretty quickly – didn’t move forward with it though because we’d have to recreate a handful of Monitors/Log Views that are currently based on the CloudWatch Log Group Name

I’m probably going to end up creating a Fluentbit configuration that also pushes to CloudWatch. Which is a little silly, but then again I don’t know that I want to rely solely on DD.

the only thing I’m going to say about DD log collection is that is SUPER expensive

far more than other SaaS products out there

we forbid the use of it in our company for that reason, just my 2 cents

K9s

@roth.andy, this was one of the best tool recommendations ever! Thank you!!!