#kubernetes (2019-05)

kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2019-05-02

nutellinoit avatar
nutellinoit

@mgrube we use Rancher for users permission management in front of EKS

1

2019-05-06

pericdaniel avatar
pericdaniel

Has anyone made their GKE nodes static with Terraform?

johncblandii avatar
johncblandii

I think I’m missing a step with this new eks cluster. applying the configmap keeps giving me an unauthorized.

The aws-iam-authenticator call is working as expected so I have access, but applying the file does not work

johncblandii avatar
johncblandii

error: You must be logged in to the server (the server has asked for the client to provide credentials)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

has your session expired?

johncblandii avatar
johncblandii

doesn’t aws-iam-auth create a new one every time?

johncblandii avatar
johncblandii

i used the CP module to do this before and it is working fine still on that end

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you using geodesic? can you exit the shell, run it again, and assume role?

johncblandii avatar
johncblandii

nopers. direct

johncblandii avatar
johncblandii

trying in a fresh terminal

johncblandii avatar
johncblandii

error: You must be logged in to the server (the server has asked for the client to provide credentials)

johncblandii avatar
johncblandii

aws-iam-authenticator token -i... works just fine

johncblandii avatar
johncblandii

i’ve confirmed the configmap matches the first one i did

johncblandii avatar
johncblandii

(sans names, of course)

johncblandii avatar
johncblandii

maybe i need to update my core ~/.kube/config?

johncblandii avatar
johncblandii

no go there

johncblandii avatar
johncblandii

any debug tips?

johncblandii avatar
johncblandii

ahhh….might be this:
This could be because the cluster was created with one set of AWS credentials (from an IAM user or role), and kubectl is using a different set of credentials.

I created it via CI

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

this was the issue?

johncblandii avatar
johncblandii

I think so

johncblandii avatar
johncblandii
Amazon EKS Troubleshooting - Amazon EKS

This chapter covers some common errors that you may see while using Amazon EKS and how to work around them.

johncblandii avatar
johncblandii

and the kubectl apply on CI failed because kc isn’t available on there (yet)

johncblandii avatar
johncblandii

FYI @wbrown43 ^

johncblandii avatar
johncblandii

confirmed. used the CI users creds and it worked as expected

btai avatar

@johncblandii you’ll need to update your authenticator configmap to allow other roles/users

johncblandii avatar
johncblandii

just got through that part, @btai.

johncblandii avatar
johncblandii

my last eks was a local install i did so i did not realize this was a rule

btai avatar

@johncblandii yeah took me half a day just trying to figure out how to get aws-iam-authenticator working and I ran into the same issues as you did haha

btai avatar

but no problems since

johncblandii avatar
johncblandii

to which I spent the same half-day (while in 5 hours of meetings back-to-back-to-back)

johncblandii avatar
johncblandii

johncblandii avatar
johncblandii

now my nodes aren’t connecting so onto issue #3; lol

btai avatar

did you add the role for the worker nodes to your config map as well?

johncblandii avatar
johncblandii

yup. the cp module does it

johncblandii avatar
johncblandii

going to nix one and let the scaling kick off a fresh one now that the map is applied

johncblandii avatar
johncblandii

so no public ip seems to have been the issue

johncblandii avatar
johncblandii

started one w/ a public ip and voila

btai avatar

@johncblandii are you talking about public ip for your worker nodes?

johncblandii avatar
johncblandii

yup

johncblandii avatar
johncblandii

our other cluster didn’t have public, but when I upgraded them to 1.12 i had to do the same

(unsure if that’s related, but i did notice that)

btai avatar

@johncblandii fwiw, i didnt have to make my worker nodes public. im still on 1.11 but i cant imagine that would change in 1.12

johncblandii avatar
johncblandii

i hear you. that’s just what i noticed when i moved to .12

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

I have a cluster in AWS in 3 availability zones, with 3 masters, but only 2 nodes. kops put both nodes in the same AZ? Is this a bug? How do I get kops to spread the nodes evenly across AZs?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s not a kops thing

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

compare how the master node pools are created to how the worker node pools are created

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s how to ensure more even distribution

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

AWS will make “best effort” to allocate instances evenly, but no guarantee

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the only way to have a “guarantee” is to create node pools tied to exactly one AZ

Issif avatar

for this purpose, we create 3 ASG with only one master inside

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

precisely…

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

Yes, kops creates an instance group per zone for the masters, but just 1 instance group for all the nodes.

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

So it turns out the bigger issue is that AWS autoscale group does launching and zone balancing separately, and to do zone balancing it has to launch a new instance before deleting the old one. Well, we had run up against our instance/type limit for the region, so it could not do zone balancing.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh fascinating

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

good sluething

wbrown43 avatar
wbrown43
06:50:41 PM

@wbrown43 has joined the channel

2019-05-07

johncblandii avatar
johncblandii

Is there a clean way to get the security group created for an LB so I can assign it to the workers SG to approve traffic?

johncblandii avatar
johncblandii

The LB is created through the helm deploy.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
AWS: aws_security_group - Terraform by HashiCorp

Provides details about a specific Security Group

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
stakater/Whitelister

A tool to white list node and developer IPs for kubernetes. - stakater/Whitelister

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I would pursue a k8s native solution rather than trying to fuse terraform with helm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, IP whitelisting should be used as a last resort. Identity Aware Proxies is ala keycloak is a better approach

btai avatar

alb ingress controller creates you an ALB and the necessary security groups and assigns them to access your workers

johncblandii avatar
johncblandii

@Andriy Knysh (Cloud Posse) there isn’t enough on the SG to query that way. it has the [k8s.io/](http://k8s.io/)… tag, but it is not specific.

johncblandii avatar
johncblandii

@Erik Osterman (Cloud Posse) this isn’t fusing helm and tf. it is the SG created by TF, but I’m mainly just adding an SG record so it is mainly AWS infrastructure networking.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if you go that route, you can filter by name (the resource has some name) and not tags

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

or add your own specific tag

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think I lack context of where you are trying to do this?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


“Is there a clean way from XXXXXXX to get the security group created for an LB by ZZZZZZ so I can assign it to the workers SG in YYYYYYY to approve traffic?”

johncblandii avatar
johncblandii

you may be technically right w/ fusing them. i’m technically wanting a value from k8s so i can configure the AWS SG to allow communication.

The SG is handled within TF manually

btai avatar

@johncblandii what type of LB are you using, if youre using an ALB I would suggest alb ingress controller as it does all that for you. (the downside is when you tear down your cluster, it wont clean up for you)

johncblandii avatar
johncblandii

it automatically used a classic elb (helm install)

btai avatar

what helm chart @johncblandii

johncblandii avatar
johncblandii

Twistlock

johncblandii avatar
johncblandii

welp…bitten by the “providers cannot be dynamically initialized” issue

2019-05-08

2019-05-09

rms1000watt avatar
rms1000watt

Dang, how do we get Curtis Mattoon into cloud posse slack? https://github.com/cmattoon/aws-ssm/pull/29

Added log-level functionality by rms1000watt · Pull Request #29 · cmattoon/aws-ssm

I didn’t see any other way to set the log level. So here it is!

rms1000watt avatar
rms1000watt

This tool works pretty good. But just curious if you peeps have any other methods for dynamically added k8s secrets from SSM

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not from SSM

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

have you seen @mumoshu’s ASM operator?

rms1000watt avatar
rms1000watt

nope, I shall take a look-see

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think extending that to support SSM would be nice

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or creating a separte one

rms1000watt avatar
rms1000watt

https://github.com/mumoshu/aws-secret-operator (for the others in the channel)

mumoshu/aws-secret-operator

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager. - mumoshu/aws-secret-operator

1
rms1000watt avatar
rms1000watt
Why not use AWS SSM Parameter Store as a primary source of secrets?

Pros:

Parameter Store has an efficient API to batch get multiple secrets sharing a same prefix.

Cons:

Its API rate limit is way too low. This has been discussed in several places in the Internet:
rms1000watt avatar
rms1000watt

However, they just updated the rate limit to 1k req/s

rms1000watt avatar
rms1000watt

so it might be a non-issue now

rms1000watt avatar
rms1000watt

Also, you can set the limit and incur costs. Haven’t actually clicked this before.. lets see what happens

rms1000watt avatar
rms1000watt

rms1000watt avatar
rms1000watt

Ohhh, this is how to you get 1k: https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-throughput.html

You can increase the limit to 1,000 TPS on the Settings tab. Increasing the throughput limit incurs a charge on your AWS account.
rms1000watt avatar
rms1000watt

$0.05 per 10,000 Parameter Store API interactions k.. I’ll stop spamming

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s great; didn’t know they increased the limit

mumoshu avatar
mumoshu

I thought secretsmanager had the same amount of charge

rms1000watt avatar
rms1000watt

secretmanager i think is $1/mo/secret. Lemme google a littttle

rms1000watt avatar
rms1000watt

whoops.. $0.40/mo

rms1000watt avatar
rms1000watt
PER SECRET PER MONTH
$0.40 per secret per month. For secrets that are stored for less than a month, the price is prorated (based on the number of hours.)

PER 10,000 API CALLS
$0.05 per 10,000 API calls.
rms1000watt avatar
rms1000watt

loren avatar

Also good to cache your secrets, to avoid extra API calls and rate limits… https://aws.amazon.com/about-aws/whats-new/2019/05/Secrets-Manager-Client-Side-Caching-Libraries-in-Python-NET-Go/

rms1000watt avatar
rms1000watt

this is interesting

rms1000watt avatar
rms1000watt

curious how it works in detail. Like, does it make your microservice stateful? Or does it put the cache local to your cluster? Or is aws handling all the caching for us automagically?

The go SDK code looks straight forward though. Awesome find!

rms1000watt avatar
rms1000watt

https://github.com/cmattoon/aws-ssm/pull/30 fixing a bug in aws-ssm if anyone else was considering to use it

Added next token to getparameterbypath for secrets > 10 by rms1000watt · Pull Request #30 · cmattoon/aws-ssm

The Go SDK for GetParameterByPath limits to 10 values in the response. This should grab them all.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

how does it look when you want many parameters?

kind: Secret
metadata:
  name: my-secret
  annotations:
    aws-ssm/k8s-secret-name: my-secret
    aws-ssm/aws-param-name: my-db-password
    aws-ssm/aws-param-type: SecureString

e.g. /db/*

Added next token to getparameterbypath for secrets > 10 by rms1000watt · Pull Request #30 · cmattoon/aws-ssm

The Go SDK for GetParameterByPath limits to 10 values in the response. This should grab them all.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


The name of the AWS SSM Parameter. May be a path.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i guess that answers it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but still curious. i never really kicked the tires on aws-ssm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(ultimately, client wanted per-service access controls so we went with Chamber +S3 + IAM + KIAM)

rms1000watt avatar
rms1000watt

@Erik Osterman (Cloud Posse)

apiVersion: v1
kind: Secret
metadata:
  name: my-secret-name
  annotations:
    aws-ssm/k8s-secret-name: my-secret-name
    aws-ssm/aws-param-name: {{ .Values.ssm_path }}
    aws-ssm/aws-param-type: Directory
data: {}

Where ` .Values.ssm_path == /directory/within/ssm`

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ah, thx!

rms1000watt avatar
rms1000watt

(lol, sorry about the delay)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

how’s the helmfile PR coming along?

rms1000watt avatar
rms1000watt

stale at the moment. been a bit busy. basically I didn’t consider multiple files

rms1000watt avatar
rms1000watt

and there’s some chicken/egg issue about when the template-rendering happens and when to reference a file

rms1000watt avatar
rms1000watt

so I just need to hit my head a little harder on it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

maybe that will be simpler if they decouple the multi-phase rendering

rms1000watt avatar
rms1000watt

Possibly. I thought multi-phase rendering was needed for template in template situations

2019-05-10

2019-05-11

Exequiel Barrirero avatar
Exequiel Barrirero
Exequiel Barrirero avatar
Exequiel Barrirero

Interesting approach for -> Deploying API Gateway in front of EKS / K8s Kops Clusters inside VPC private subnets And many other useful info about Integrating EKS with other AWS Services

2019-05-15

davidvasandani avatar
davidvasandani
containership/konstellate

Free and Open Source GUI to Visualize Kubernetes Applications. - containership/konstellate

oscarsullivan_old avatar
oscarsullivan_old

Thanks I like this. For #terraform there’s also https://github.com/camptocamp/terraboard

camptocamp/terraboard

A web dashboard to inspect Terraform States - camptocamp/terraboard

davidvasandani avatar
davidvasandani

Thanks for sharing @oscarsullivan_old! This looks really neat. You should share it in the #terraform channel.

camptocamp/terraboard

A web dashboard to inspect Terraform States - camptocamp/terraboard

Vidhi Virmani avatar
Vidhi Virmani

I am trying to setup kubernetes dashboard on AWS EKS cluster. I am able to setup the dashboard but facing a small issue with certs. I want to use aws certificate arn with the dashoard as an argument with command

kubectl apply -f <https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml>

is this possible?

2019-05-16

nutellinoit avatar
nutellinoit

To anyone that is tempted to use t3a or m5a instances on an EKS cluster, don’t

nutellinoit avatar
nutellinoit
Support for t3a, m5ad and r5ad instance types is missing · Issue #262 · awslabs/amazon-eks-ami

What would you like to be added: Support for t3a, m5ad and r5ad instance types. Why is this needed: AWS had added new instance types, and the AMI does not currently support them.

nutellinoit avatar
nutellinoit

there is an incompatibility on calculating number of eni available

2
endofcake avatar
endofcake
Introducing Polaris: Keeping your Kubernetes Clusters Healthy - Reactive Ops

We started ReactiveOps with a simple vision: transform infrastructure operations by leveraging decades of large-scale operations and product experience.

timduhenchanter avatar
timduhenchanter
kedacore/keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes - kedacore/keda

fb-wow1

2019-05-17

Issif avatar
reactiveops/polaris

Validation of best practices in your Kubernetes clusters - reactiveops/polaris

Sandeep Kumar avatar
Sandeep Kumar

How do we generate a wildcard certificate using kubernetes kind:managedCertificate, trying with below method but not successful apiVersion: networking.gke.io/v1beta1 kind: ManagedCertificate metadata: name: example-certificate spec: domains: - *.example.net

Please let me know if there is any documentation/suggestions to create a wild card certificate with expiry date mentioned in it

sarkis avatar

^ Polaris looks really interesting… I’m going to try to get it going this weekend see if it’s useful… any thoughts on it yet if someones already set it up?

sarkis avatar

Couldn’t wait for the weekend testing it now… it offers some nice checks… I can see this becoming more and more useful as more checks/best practices are added …

2019-05-19

James D. Bohrman avatar
James D. Bohrman

Hey all! I’m having an issue building my example-voting-app with Codefresh.

I added the variable for KUBE_CONTEXT but I keep getting an error that throws:


error: no context exists with the name: "gke_example-voting-app-240610_us-east1-c_example-votin
g-app".                                                                                        
[SYSTEM] Error: Failed to run freestyle step: Running Helm Upgrade; caused by NonZeroExitCodeEr
ror: Container for step title: Running Helm Upgrade, step type: freestyle, operation: Freestyle
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

KUBE_CONTEXT should be the name of a kubernetes integration in codefresh

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it would seldom, if ever have the app name in it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add Kubernetes Cluster

How to connect your Kubernetes cluster to the Codefresh dashboard

James D. Bohrman avatar
James D. Bohrman

Got it thanks!

James D. Bohrman avatar
James D. Bohrman

I ran kubectl get context in my GKE shell and got:

gke_example-voting-app-240610_us-east1-c_example-voting-app
James D. Bohrman avatar
James D. Bohrman

I put that as my KUBE_CONTEXT variable and can’t figure what I’m doing wrong. The docs say to put KUBE_CONTEXT as “Your friendly Kubernetes Cluster Name” I’ve also tried “example-voting-app” as the context variable. Which is the EKS cluster name. No dice there either.

2019-05-20

aaratn avatar

Can anyone help me with aws alb loadbalancer with helm chart ? Any samples that I can refer ?

sarkis avatar
flant/werf

Werf (previously known as dapp) helps to implement and support Continuous Integration and Continuous Delivery - flant/werf

2019-05-22

Pablo Costa avatar
Pablo Costa
2
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

finally

btai avatar

ergg.. i spent a good part of a day understanding how it works/getting it to work w/my eks cluster spun up in tf

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Public/Free Office Hours with Cloud Posse starting now!!

https://zoom.us/j/684901853

btai avatar

anyone try federation yet?

2019-05-24

Kevin Gimbel avatar
Kevin Gimbel

Hey all, I’ve a question and I can’t seem to find an answer. I’m running an AWS EKS cluster with two Nodes, each Node in EKS has a restriction of 20 Pods per Node. The Nodes are auto scaled and shut down each night and started in the morning since it’s just a test / staging system at the moment. However, one Node is always full (20/20 Capacity) while the other runs 4/20. We want to run a DaemonSet with filebeat for log aggregation but cannot ensure it runs on both nodes because one is full.

Is there a way I can (easily) ensure the DaemonSet is scheduled before all other pods? Or can I reserve a spot / space on a Node for a specific Pod, Deployment, or DaemonSet?

Kevin Gimbel avatar
Kevin Gimbel

I would like to avoid configuration overhead. I’ve already read about Affinity and Anti-Affinity but I’m not sure if this can help me

Kevin Gimbel avatar
Kevin Gimbel

Someone in the Kubernetes Slack answered my question, looks like this is it: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, this is what you want to look into.

2019-05-29

Vidhi Virmani avatar
Vidhi Virmani

Hi all,

Is there anyone who has setup kubernetes dashboard on EKS using istio ingress gateway? I am facing some issues where my dashboard crash after 4 mins. I am not sure if its a good idea to use istio ingress gateway to run kubernetes-dashboard. Any help is appreciated

Vidhi Virmani avatar
Vidhi Virmani

It is fixed now. I had to provide few configs in istio

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Vidhi Virmani how are you securing it?

johncblandii avatar
johncblandii

(comment just to monitor response)

Vidhi Virmani avatar
Vidhi Virmani

@Erik Osterman (Cloud Posse) I am currently allowing very few users to access the dashboard using aws-iam-authenticator.

1

2019-05-31

Issif avatar
claranet/kcs

Select which kubeconfig.yaml to use in an easy way. KCS means kubeconfig switcher. - claranet/kcs

3
    keyboard_arrow_up