#kubernetes (2020-04)

This..is..amazing.

darn. I had at one point tried to build an internal cluster management tool and wanted to call it fleet. Because it was essentially managing a fleet of kube clusters (and keeping with the ocean/ship theme) never got around to buildint it out completely

my proof

figures right? well fleet looks open source maybe you can use it anyway

what module do you use for Go log output anyway?

@Zachary Loeber i used https://github.com/sirupsen/logrus but i havent been doing a ton of Go development in the last few years so I’m prob not the best person to ask

We’ve encountered something that sounds similar when the subnets aren’t correctly configured with route tables or the wrong subnets are passed to EKS. In this case, pod(1) is on node A, pod (2) is on node B; node A and node B are on different subnets.

do you have network policies? that could be different for both namespaces?

Erik: same node, same subnet

Btai: no NACLs in AWS. But I’m guessing you’re referring to k8s network policies.. uhhh no clue, but I’ll look for it

I want to try and look at CNI/SNAT failures or something.. but I trying to field for suggestions first before going down that rabbit hole

no network policies in k8s

Unless you are doing unconventional things, I would look for more obvious, user error type problems. Just based on my own experiences, I am usually at fault 99% of the time.

i agree. In this case.. I think i got wrapped around the axel in playing with Service Endpoints.. opposed to the service itself

I was hitting the endpoint defined here..

kubectl -n dev-1 describe svc my-svc-name | grep -i endpoints:

Instead of just making the service fully available where I need it

the technical issue still stands, but the need to solve it diminished.. since I’m just going to hit the service (as I should have all along)

Depends wildly on your app. What’s the terminationGracePeriodSeconds set for that app? Think of it like this: pod is alive and ready and serving requests. Pod gets notified to stop work. How long does it serve requests? What happens to in-flight requests? How does that affect the app?

For starting time, it’s the same problem in reverse. How deep are your aliveness and more importantly your readiness checks? How do you know a pod is ready to serve requests? Does it serve requests in the first 10 minutes with super-high latency cause it’s still populating some caches?

More importantly, do your pods get replaced often? If not, you may not even need to stress about this

They get replaced a few times day (multiple deployments a day). There’s 100+ pods in the deployment.

terminationGracePeriodSeconds: 30

i guess i need to do my homework more on when the deregistration delay timer begins

Yeah, sounds like you do need to worry about it

When a pod has to be replaced, the following flow happens: • SIGTERM is sent to the pod. Apps should get that as “dude, I got a notice to stop work graciously so I will start doing that”. Finish in progress-work, try to clean up nicely, and so on. At this time the ALBs should be set so no new connections are sent to these podsw • we wait for terminationGracePeriodSeconds

• SIGKILL is sent to the pod which kills all the containers inside by force

Meanwhile, ALB Ingress Controller runs a loop every say 10s and checks for any new pods or any new LB changes and updates the ALBs accordingly

All of these have to make out and kiss in sync

It helps a lot to draw this out and mock scenarios

re: make out and kiss in sync

https://67.media.tumblr.com/668927139d282654dee7df5b1f715f93/tumblr_inline_o21a2aH9He1szrmgb_500.gif

I’m on the same page with you with your analysis so far. The dark spot in my mind is how Deregistration Delay works after a pod is marked as terminating

(proc is killed by itself naturally by SIGTERM or forced by SIGKILL)

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html helps

actually, I think the TG stops routing to a terminating/deregistering instance

but the deregistration delay keeps the current connections alive for up to XXseconds

If that’s the case, no big deal

I think the most important thing is aws-alb-ingress-controller to update ASAP once a pod is marked as terminatng

so it can be marked as deregistering in the TG

When a pod is marked as Terminating, we’re in between that SIGTERM and SIGKILL limbo. We still have connections from client-ALB-pod.

Now, after Deregistration Delay the ALB forcefully kills all connections from the client to the pod.

That’s helpful if say your app cannot die gracefully if there are still active connections

Or at least that’s how I understand it based on the above link

yeah

makes sense

terminating grace period length > deregistration delay length

Since there’s a lag waiting for AWS alb ingress controller to tell the TG that a target is deregistering

I think that is correct, yup.

Also, don’t forget about the ALB Ingress Controller loop.

That happens every 10s I think. And if it ran just before your pod switched to Terminating…

Again, drawing and testing all the situations( or the most important ones) helps a lot

yeah, I agree

(so has kops historically)

Oh okay did not know. I am new to aws after all and never had to deal with unmanaged clusters

which version of k8s is running eks ?

1.15

ho that’s very old.

Yep. It also only was just added in march

This is the issue regarding that: https://github.com/aws/containers-roadmap/issues/487

no, but you’ve piqued my interest.

the SG rules for the NLB specifically are getting added to node-port-level? how were you specifying your SG rules for this NLB?

these are the only annotations I use

https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#aws

list of annotations

i want to disable the sg rule addition somehow

this is what its doing to the SG for the EKS nodes

interesting. i had not stumbled upon this yet but i haven’t been using security groups on my (public) NLB. great to know though.

these are private NLBs

hmm, i wonder if that makse a difference

All I can do is try

i suspect not. logically it makes sense to me though that if you’re applying a SG, it’s going to lock down the port on the node that’s frontending the service.

https://github.com/kubernetes/legacy-cloud-providers/blob/36f88b5f5220796e2770093888cee7cd4ee61d19/aws/aws.go#L3540

here’s the problem tho, if you don’t specify an SG, it’ll grab one anyways

Oh, the terminology is NodePort

We use the nlb mode by default with that annotation. Haven’t been bothered by the rule additions. Why fight it?

we maxed out on inbound security group rules

lol

only 60 rules per SG

each NLB’s nodeport is making 2 entries in there

https://github.com/kubernetes/kubernetes/pull/74692/files#diff-298a224837f7a3edc5b5f37ddb8fa47aR671

this looks kind of promising tho

what is random for you?

Here u go Pierre: https://github.com/helm/charts/issues/21973

This is what I am talking about

but it is indented for you ?

Yeah yea of course… Within my myvalues.yaml file its indented right

This is how the service name gets defined you need to set fullnameOverride

Ok… SO the service name is defined with an override. In this case defining the “grafana.fullname” within service section should fix the issue…

Am i saying it right?

no you can not set the service name on its own.

You can only override the name for all manifests

I am wondering why you would not just take the default?

Well, the reason why I dont want to take the defalut is, I am having an issue when I am setting the Ingress for the same (Grafana Service). I do see a name mismatch here cause I need to define the service name within the Ingress configuration before I deploy the service

Why would you not enabled the ingress of the helm chart?

https://github.com/helm/charts/blob/efd0f2cfa46471c888dbfc6ab30975f2d74d5790/stable/grafana/values.yaml#L130

service: name: svc-grafana namespace: kube-system type: ClusterIP port: 80 targetPort: 3000 annotations: {} labels: {} portName: service

ingress: enabled: true annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/load-balancer-attributes: ‘routing.http2.enabled=true,idle_timeout.timeout_seconds=600,deletion_protection.enabled=true’ alb.ingress.kubernetes.io/certificate-arn: certname alb.ingress.kubernetes.io/listen-ports: ‘[{“HTTP”: 80}, {“HTTPS”:443}]’ alb.ingress.kubernetes.io/actions.ssl-redirect: ‘{“Type”: “redirect”, “RedirectConfig”: { “Protocol”: “HTTPS”, “Port”: “443”, “StatusCode”: “HTTP_301”}}’ name: grafana-ingress namespace: kube-system service: annotations: alb.ingress.kubernetes.io/target-type: ip labels: {} path: /* hosts: - grafana.company.com ## Extra paths to prepend to every host configuration. This is useful when working with annotation based services. extraPaths: - path: backend: serviceName: ssl-redirect servicePort: use-annotation - path: /* backend: serviceName: svc-grafana servicePort: 80

"service: name: svc-grafana namespace: kube-system type: ClusterIP port: 80 targetPort: 3000 annotations: {} labels: {} portName: service

ingress: enabled: true annotations: [kubernetes.io/ingress.class](http://kubernetes.io/ingress.class): alb [alb.ingress.kubernetes.io/scheme](http://alb.ingress.kubernetes.io/scheme): internet-facing [alb.ingress.kubernetes.io/load-balancer-attributes](http://alb.ingress.kubernetes.io/load-balancer-attributes): 'routing.http2.enabled=true,idle_timeout.timeout_seconds=600,deletion_protection.enabled=true' [alb.ingress.kubernetes.io/certificate-arn](http://alb.ingress.kubernetes.io/certificate-arn): certname [alb.ingress.kubernetes.io/listen-ports](http://alb.ingress.kubernetes.io/listen-ports): '[{"HTTP": 80}, {"HTTPS":443}]' [alb.ingress.kubernetes.io/actions.ssl-redirect](http://alb.ingress.kubernetes.io/actions.ssl-redirect): '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}' name: grafana-ingress namespace: kube-system service: annotations: [alb.ingress.kubernetes.io/target-type](http://alb.ingress.kubernetes.io/target-type): ip labels: {} path: /* hosts: - [grafana.company.com](http://grafana.company.com) ## Extra paths to prepend to every host configuration. This is useful when working with annotation based services. extraPaths: - path: backend: serviceName: ssl-redirect servicePort: use-annotation - path: /* backend: serviceName: svc-grafana servicePort: 80"

can you wrap that in

`

``

Sorry,, here u go

and where are you trying to reference the service name?

If u look at the “name” under Service configuration and the “servicename” under Ingress configuration, the parameters are different and thats why I want to control the service name so that it can be set under the Ingress.

I am trying to refer the servicename under:

service:
  name: svc-grafana

you do not need to define the grafana path afaik

After I deploy the configuration, here is the error I am getting in the alb-ingress-configuration pod logs:

E0416 19:39:58.180001       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile targetGroups due to failed to load serviceAnnotation due to no object matching key \"kube-system/svc-grafana\" in local store"  "controller"="alb-ingress-controller" "request"={"Namespace":"kube-system","Name":"grafana-1587065956"}

You should be able to remove the grafana service declaration

since https://github.com/helm/charts/blob/efd0f2cfa46471c888dbfc6ab30975f2d74d5790/stable/grafana/templates/ingress.yaml#L37-L40

It is added by default

ho….

So, I am guessing to get rid of the whole section:

  extraPaths:
    - path:
      backend:
        serviceName: ssl-redirect
        servicePort: use-annotation
    - path: /*
      backend:
        serviceName: svc-grafana
        servicePort: 80

no need for it then….

I am not sure why you have added it.

So it might be yes

I was configuring it based on templates I got from github and AWS ALB Ingress sections, Pierre…

Let me remove it and will see if it deploys

sure let me know. Happy to help.

This time, it throwed a new error:

I0416 20:00:22.317225       1 tags.go:43] kube-system/grafana-1587067180: modifying tags {  ingress.k8s.aws/stack: "kube-system/grafana-1587067180",  kubernetes.io/service-name: "grafana-1587067180",  kubernetes.io/service-port: "80",  ingress.k8s.aws/resource: "kube-system/grafana-1587067180-grafana-1587067180:80",  kubernetes.io/cluster/cluster_name: "owned",  kubernetes.io/namespace: "kube-system",  kubernetes.io/ingress-name: "grafana-1587067180",  ingress.k8s.aws/cluster: "cluster_name"} on arn:aws:elasticloadbalancing:AWSSetup

ooh sorry, nevermind. Thats not an error. However, the original error still persists

hard to say / judge what might be going on ..

sorry Pierre. the above one is the error I am getting

I will dig in more…

The error says that you can not use server.type: ClusterIP so if you would like to use a load balancer you have to change the type

So it might work if you set it to: LoadBalancer

But, without defining the ClusterIP, how does the service will be created with the proper setup for exterrnal or even internal accessing??

either way, let me try to change the server.type and will see what it does

Actually, I am guessing I need to add this

service.loadBalancerIP	IP address to assign to load balancer (if supported)	nil

you can not create an ALB for a ClusterIP Service if you would like to use a Load Balancer you will need to switch the service type.

ooh wow..

ok ok

let me change the server type to loadbalancer and see what it does then

Son a gun… It worked…

Boy, u r amazing…!!!

I truly apprecate your help here Pierre!!!!

no worries

you should be using an IP address within the kubernetes network range right?

this is cool! Maybe also something for #community-projects

thanks Pierre, I was surprised at how well it works

DO u mean the “hosts” section, Joey?

I just checked and I see the domain url I used under “hosts” section is the one I used and its the one beoing loaded

However there are multiple redirects are happening

no, i mean the prometheus server dashboard or whatever service it is you’re hitting when you hit that ingress

i’m just wondering if the things that aren’t loading aren’t loading because you’re getting an incorrect url

Yes, the prometheus server dashboard will be accessable by the url defined in hosts section and thats how you access it

And thats where the issue is

prometheus server dashboard is not getting loaded fully

if you open inspect mode in chrome or ff or whatever browser you’re using, for the objects that are not being loaded, is the host being requested the same as all the other assets?

Yes, I used the developer tools and verified the domain names and its all using the proper domain name

I fixed the issue… thanks joey

what was it?

Follow

I’ve recieved some interesting comments in serverless-forum.slack.com about this

When is it too much… 5? 10? 100?

It’s hard to say without knowing what those variables all mutate, which I assume is what they do.

If variables are mostly the same across many different deployments, having something that “generates” them on the fly based on some external source could be an abstraction which may or may not suit your taste …

If you prefer explicitness to carry all the way into the deployment manifests however, you probably want the dynamic generation to happen outside of Kubernetes - in some form of manifest generation..

I may be biased, but I really like chamber, regardless of Kubernetes or not.

https://github.com/segmentio/chamber

chamber exec path/to/global/variables path/to/deployment/specific/variables -- my-binary

If you convert the individual env vars into a config map your eyes will thank you when you have to look over the deployment manifests

Plus, you can then look at controllers to auto restart your deployments when/if the configmaps change

That I can agree with for sure .

chamber looks sweet, too bad it is provider specific

also.. you can combine it from several configmaps, right?

And then you could (if several deployment shares “environment globals” so to speak) - combine environment variables in a nice way. I think it’s described in the doc somewhere, let me see.

Totally

This, envFrom: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#configure-all-key-value-pairs-in-a-configmap-as-container-environment-variables

combined with multiple, from: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#define-container-environment-variables-with-data-from-multiple-configmaps

weirdly enough there are cases where you can benefit from multiple config maps

Thinking about something specific other than this “hierarchical” combination of globals/env/service?

Nah, it technically boils down to your succinct statement

I had one where the base deployment was an app that needed some variables based on the cluster which was getting deployed within the pipeline but they later wanted to push out specific updates to config elements that were client specific

so, yeah, basically hierarchical combo

Thanks Guys, it was really helpfull

@Zachary Loeber Did you use any controller which tracks config-maps/secrets change and restart pods if there is a change?

I did not unfortunately, seems pretty easy to do though.

in lower environments I just didn’t see the need (my pipelines would always push more recent deployments based on either build id or git commit hash tagged containers)

I was able to utilize this one: https://github.com/pusher/wave

maybe there are alternatives

store your secrets in vault and grab them on startup

Im not familiar with vault. Is that something that runs in k8s?

https://www.vaultproject.io/

If that runs on k8s, does that mean vault becomes the source of truth and you dont keep a copy anywhere?

I dont want to commit the secrets in git (unless they are properly encrypted)

the way i’ve done it is running vault in a separate cluster/as it’s own piece of infrastructure, and then in my pods i have a vault agent init container that authenticates to vault, grabs secrets, and passes them in files or as environment variables to relevant containers

it’s not a trivial exercise, but it’s clean and can be provider agnostic, as opposed to using secret manager from $cloud_provider

Im not as concerned about them being secured in k8s as having a secure copy elsewhere…

Im using help charts for my apps…

so I have a configmaps.yaml in that chart

have you checked out git-crypt for storing encrypted secrets in git?

Not yet. I looked at sealed-secrets, helm-secrets or Kamus so far. I’ll check that out

thanks

Also using Hashicorp Vault and very happy with it, though pulling secrets into static files or environment variables with Kustomize has gotten a bit complicated.

I’m using AWS SSM Parameter Store ( https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) which I like as it gives me a source of truth outside of the cluster.

We’re currently switching over to using helmfile (https://github.com/roboll/helmfile) which has built in support for retrieving values from SSM parameter store (and other systems, including vault) by using vals (https://github.com/variantdev/vals)

nice, hadn’t seen vals.

Thanks

Sure, let me know if you decide to look into any of that stuff and have questions. I’m still pretty new to helmfile myself but happy to help if I can.

As for vault, someone just released some Terraform to deploy it on AWS: https://github.com/jcolemorrison/vault-on-aws

It is quite complicated though, so the folks at Segment created chamber as a result: https://github.com/segmentio/chamber

When on Azure and without a centralized Hashicorp Vault deployment I leaned on using keyvault with this project to auto-inject secrets using mutating admission webhooks: https://github.com/SparebankenVest/azure-key-vault-to-kubernetes

the concept is similar for other operators as well though (hashicorp’s vault operator does the same I believe). That way you are not putting your secrets anywhere at all except in your secret store and pulling them into deployments if the cluster is authorized to do so.

or you can pre-seed cluster secrets as well I suppose. I’ve done this as well but you are then pushing the responsibility for secrets deployment to an upstream pipeline (the one that creates your cluster generally).

How do you authenticate to vault and how to store those credentials?

service accounts

How do you protect/secure it?

I am working last couple od days with GoDaddy External secret implementation. You can integrate aws secret manager, parameter store or even vault with it.

I am pretty happy so far

@Pierre Humberdroz thanks for sharing

that is a long list

anything jump out at you? mostly looks like bug fixes to me. no enhancements stand out that I want to try

Helm chart stable/nginx-ingress is now maintained in the ingress-nginx repository According to https://github.com/kubernetes/ingress-nginx/issues/5161 there is documentation in the works for migrating from chats/stable to the new location.

yea stable/incubator helm repo are not longer supported

Wow, I hadn’t seen that yet. That’s a big change.