#kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2019-05-31

Issif
claranet/kcs

Select which kubeconfig.yaml to use in an easy way. KCS means kubeconfig switcher. - claranet/kcs

3

2019-05-29

Vidhi Virmani

Hi all,

Is there anyone who has setup kubernetes dashboard on EKS using istio ingress gateway? I am facing some issues where my dashboard crash after 4 mins. I am not sure if its a good idea to use istio ingress gateway to run kubernetes-dashboard. Any help is appreciated

Vidhi Virmani

It is fixed now. I had to provide few configs in istio

Erik Osterman

@Vidhi Virmani how are you securing it?

johncblandii

(comment just to monitor response)

Vidhi Virmani

@Erik Osterman I am currently allowing very few users to access the dashboard using aws-iam-authenticator.

1

2019-05-24

Kevin Gimbel

Hey all, I’ve a question and I can’t seem to find an answer. I’m running an AWS EKS cluster with two Nodes, each Node in EKS has a restriction of 20 Pods per Node. The Nodes are auto scaled and shut down each night and started in the morning since it’s just a test / staging system at the moment. However, one Node is always full (20/20 Capacity) while the other runs 4/20. We want to run a DaemonSet with filebeat for log aggregation but cannot ensure it runs on both nodes because one is full.

Is there a way I can (easily) ensure the DaemonSet is scheduled before all other pods? Or can I reserve a spot / space on a Node for a specific Pod, Deployment, or DaemonSet?

Kevin Gimbel

I would like to avoid configuration overhead. I’ve already read about Affinity and Anti-Affinity but I’m not sure if this can help me

Kevin Gimbel

Someone in the Kubernetes Slack answered my question, looks like this is it: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/

1
Erik Osterman

Yes, this is what you want to look into.

2019-05-22

Pablo Costa
2
aknysh

finally

ergg.. i spent a good part of a day understanding how it works/getting it to work w/my eks cluster spun up in tf

Erik Osterman

Public/Free Office Hours with Cloud Posse starting now!!

https://zoom.us/j/684901853

anyone try federation yet?

2019-05-20

aaratn

Can anyone help me with aws alb loadbalancer with helm chart ? Any samples that I can refer ?

sarkis
flant/werf

Werf (previously known as dapp) helps to implement and support Continuous Integration and Continuous Delivery - flant/werf

2019-05-19

James D. Bohrman

Hey all! I’m having an issue building my example-voting-app with Codefresh.

I added the variable for KUBE_CONTEXT but I keep getting an error that throws:


error: no context exists with the name: "gke_example-voting-app-240610_us-east1-c_example-votin
g-app".                                                                                        
[SYSTEM] Error: Failed to run freestyle step: Running Helm Upgrade; caused by NonZeroExitCodeEr
ror: Container for step title: Running Helm Upgrade, step type: freestyle, operation: Freestyle
Erik Osterman

KUBE_CONTEXT should be the name of a kubernetes integration in codefresh

Erik Osterman

it would seldom, if ever have the app name in it

Erik Osterman
Add Kubernetes Cluster

How to connect your Kubernetes cluster to the Codefresh dashboard

James D. Bohrman

Got it thanks!

James D. Bohrman

I ran kubectl get context in my GKE shell and got:

gke_example-voting-app-240610_us-east1-c_example-voting-app
James D. Bohrman

I put that as my KUBE_CONTEXT variable and can’t figure what I’m doing wrong. The docs say to put KUBE_CONTEXT as “Your friendly Kubernetes Cluster Name” I’ve also tried “example-voting-app” as the context variable. Which is the EKS cluster name. No dice there either.

2019-05-17

Issif
reactiveops/polaris

Validation of best practices in your Kubernetes clusters - reactiveops/polaris

Sandeep Kumar

How do we generate a wildcard certificate using kubernetes kind:managedCertificate, trying with below method but not successful apiVersion: networking.gke.io/v1beta1 kind: ManagedCertificate metadata: name: example-certificate spec: domains: - *.example.net

Please let me know if there is any documentation/suggestions to create a wild card certificate with expiry date mentioned in it

sarkis

^ Polaris looks really interesting… I’m going to try to get it going this weekend see if it’s useful… any thoughts on it yet if someones already set it up?

sarkis

Couldn’t wait for the weekend testing it now… it offers some nice checks… I can see this becoming more and more useful as more checks/best practices are added …

2019-05-16

nutellinoit

To anyone that is tempted to use t3a or m5a instances on an EKS cluster, don’t

nutellinoit
Support for t3a, m5ad and r5ad instance types is missing · Issue #262 · awslabs/amazon-eks-ami

What would you like to be added: Support for t3a, m5ad and r5ad instance types. Why is this needed: AWS had added new instance types, and the AMI does not currently support them.

nutellinoit

there is an incompatibility on calculating number of eni available

2
endofcake
Introducing Polaris: Keeping your Kubernetes Clusters Healthy - Reactive Ops

We started ReactiveOps with a simple vision: transform infrastructure operations by leveraging decades of large-scale operations and product experience.

timduhenchanter

Scale on queue depth

https://github.com/kedacore/keda

kedacore/keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes - kedacore/keda

1

2019-05-15

davidvasandani
containership/konstellate

Free and Open Source GUI to Visualize Kubernetes Applications. - containership/konstellate

oscarsullivan_old

Thanks I like this. For #terraform there’s also https://github.com/camptocamp/terraboard

camptocamp/terraboard

A web dashboard to inspect Terraform States - camptocamp/terraboard

davidvasandani

Thanks for sharing @oscarsullivan_old! This looks really neat. You should share it in the #terraform channel.

camptocamp/terraboard

A web dashboard to inspect Terraform States - camptocamp/terraboard

Vidhi Virmani

I am trying to setup kubernetes dashboard on AWS EKS cluster. I am able to setup the dashboard but facing a small issue with certs. I want to use aws certificate arn with the dashoard as an argument with command

kubectl apply -f <https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml>

is this possible?

2019-05-11

Exequiel Barrirero
Exequiel Barrirero

Interesting approach for -> Deploying API Gateway in front of EKS / K8s Kops Clusters inside VPC private subnets And many other useful info about Integrating EKS with other AWS Services

2019-05-10

2019-05-09

rms1000watt

Dang, how do we get Curtis Mattoon into cloud posse slack? https://github.com/cmattoon/aws-ssm/pull/29

Added log-level functionality by rms1000watt · Pull Request #29 · cmattoon/aws-ssm

I didn’t see any other way to set the log level. So here it is!

rms1000watt

This tool works pretty good. But just curious if you peeps have any other methods for dynamically added k8s secrets from SSM

Erik Osterman

not from SSM

Erik Osterman

have you seen @mumoshu’s ASM operator?

rms1000watt

nope, I shall take a look-see

Erik Osterman

i think extending that to support SSM would be nice

Erik Osterman

or creating a separte one

rms1000watt

https://github.com/mumoshu/aws-secret-operator (for the others in the channel)

mumoshu/aws-secret-operator

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager. - mumoshu/aws-secret-operator

1
rms1000watt
Why not use AWS SSM Parameter Store as a primary source of secrets?

Pros:

Parameter Store has an efficient API to batch get multiple secrets sharing a same prefix.

Cons:

Its API rate limit is way too low. This has been discussed in several places in the Internet:
rms1000watt

However, they just updated the rate limit to 1k req/s

rms1000watt

so it might be a non-issue now

rms1000watt

Also, you can set the limit and incur costs. Haven’t actually clicked this before.. lets see what happens

rms1000watt

rms1000watt

Ohhh, this is how to you get 1k: https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-throughput.html

You can increase the limit to 1,000 TPS on the Settings tab. Increasing the throughput limit incurs a charge on your AWS account.
rms1000watt

$0.05 per 10,000 Parameter Store API interactions k.. I’ll stop spamming

Erik Osterman

that’s great; didn’t know they increased the limit

mumoshu

I thought secretsmanager had the same amount of charge

rms1000watt

secretmanager i think is $1/mo/secret. Lemme google a littttle

rms1000watt

whoops.. $0.40/mo

rms1000watt
PER SECRET PER MONTH
$0.40 per secret per month. For secrets that are stored for less than a month, the price is prorated (based on the number of hours.)

PER 10,000 API CALLS
$0.05 per 10,000 API calls.
rms1000watt

loren

Also good to cache your secrets, to avoid extra API calls and rate limits… https://aws.amazon.com/about-aws/whats-new/2019/05/Secrets-Manager-Client-Side-Caching-Libraries-in-Python-NET-Go/

rms1000watt

this is interesting

rms1000watt

curious how it works in detail. Like, does it make your microservice stateful? Or does it put the cache local to your cluster? Or is aws handling all the caching for us automagically?

The go SDK code looks straight forward though. Awesome find!

rms1000watt

https://github.com/cmattoon/aws-ssm/pull/30 fixing a bug in aws-ssm if anyone else was considering to use it

Added next token to getparameterbypath for secrets > 10 by rms1000watt · Pull Request #30 · cmattoon/aws-ssm

The Go SDK for GetParameterByPath limits to 10 values in the response. This should grab them all.

Erik Osterman

how does it look when you want many parameters?

kind: Secret
metadata:
  name: my-secret
  annotations:
    aws-ssm/k8s-secret-name: my-secret
    aws-ssm/aws-param-name: my-db-password
    aws-ssm/aws-param-type: SecureString

e.g. /db/*

Added next token to getparameterbypath for secrets > 10 by rms1000watt · Pull Request #30 · cmattoon/aws-ssm

The Go SDK for GetParameterByPath limits to 10 values in the response. This should grab them all.

Erik Osterman


The name of the AWS SSM Parameter. May be a path.

Erik Osterman

i guess that answers it

Erik Osterman

but still curious. i never really kicked the tires on aws-ssm

Erik Osterman

(ultimately, client wanted per-service access controls so we went with Chamber +S3 + IAM + KIAM)

rms1000watt

@Erik Osterman

apiVersion: v1
kind: Secret
metadata:
  name: my-secret-name
  annotations:
    aws-ssm/k8s-secret-name: my-secret-name
    aws-ssm/aws-param-name: {{ .Values.ssm_path }}
    aws-ssm/aws-param-type: Directory
data: {}

Where ` .Values.ssm_path == /directory/within/ssm`

Erik Osterman

Ah, thx!

rms1000watt

(lol, sorry about the delay)

Erik Osterman

how’s the helmfile PR coming along?

rms1000watt

stale at the moment. been a bit busy. basically I didn’t consider multiple files

rms1000watt

and there’s some chicken/egg issue about when the template-rendering happens and when to reference a file

rms1000watt

so I just need to hit my head a little harder on it

Erik Osterman

maybe that will be simpler if they decouple the multi-phase rendering

rms1000watt

Possibly. I thought multi-phase rendering was needed for template in template situations

2019-05-08

2019-05-07

johncblandii

Is there a clean way to get the security group created for an LB so I can assign it to the workers SG to approve traffic?

johncblandii

The LB is created through the helm deploy.

aknysh
AWS: aws_security_group - Terraform by HashiCorp

Provides details about a specific Security Group

Erik Osterman
stakater/Whitelister

A tool to white list node and developer IPs for kubernetes. - stakater/Whitelister

Erik Osterman

I would pursue a k8s native solution rather than trying to fuse terraform with helm

Erik Osterman

Also, IP whitelisting should be used as a last resort. Identity Aware Proxies is ala keycloak is a better approach

alb ingress controller creates you an ALB and the necessary security groups and assigns them to access your workers

johncblandii

@aknysh there isn’t enough on the SG to query that way. it has the [k8s.io/](http://k8s.io/)… tag, but it is not specific.

johncblandii

@Erik Osterman this isn’t fusing helm and tf. it is the SG created by TF, but I’m mainly just adding an SG record so it is mainly AWS infrastructure networking.

aknysh

if you go that route, you can filter by name (the resource has some name) and not tags

1
aknysh

or add your own specific tag

Erik Osterman

I think I lack context of where you are trying to do this?

Erik Osterman


“Is there a clean way from XXXXXXX to get the security group created for an LB by ZZZZZZ so I can assign it to the workers SG in YYYYYYY to approve traffic?”

johncblandii

you may be technically right w/ fusing them. i’m technically wanting a value from k8s so i can configure the AWS SG to allow communication.

The SG is handled within TF manually

@johncblandii what type of LB are you using, if youre using an ALB I would suggest alb ingress controller as it does all that for you. (the downside is when you tear down your cluster, it wont clean up for you)

johncblandii

it automatically used a classic elb (helm install)

what helm chart @johncblandii

johncblandii

Twistlock

johncblandii

welp…bitten by the “providers cannot be dynamically initialized” issue

2019-05-06

pericdaniel

Has anyone made their GKE nodes static with Terraform?

johncblandii

I think I’m missing a step with this new eks cluster. applying the configmap keeps giving me an unauthorized.

The aws-iam-authenticator call is working as expected so I have access, but applying the file does not work

johncblandii

error: You must be logged in to the server (the server has asked for the client to provide credentials)

aknysh

has your session expired?

johncblandii

doesn’t aws-iam-auth create a new one every time?

johncblandii

i used the CP module to do this before and it is working fine still on that end

aknysh

you using geodesic? can you exit the shell, run it again, and assume role?

johncblandii

nopers. direct

johncblandii

trying in a fresh terminal

johncblandii

error: You must be logged in to the server (the server has asked for the client to provide credentials)

johncblandii

aws-iam-authenticator token -i... works just fine

johncblandii

i’ve confirmed the configmap matches the first one i did

johncblandii

(sans names, of course)

johncblandii

maybe i need to update my core ~/.kube/config?

johncblandii

no go there

johncblandii

any debug tips?

johncblandii

ahhh….might be this:
This could be because the cluster was created with one set of AWS credentials (from an IAM user or role), and kubectl is using a different set of credentials.

I created it via CI

aknysh

this was the issue?

johncblandii

I think so

johncblandii
Amazon EKS Troubleshooting - Amazon EKS

This chapter covers some common errors that you may see while using Amazon EKS and how to work around them.

johncblandii

and the kubectl apply on CI failed because kc isn’t available on there (yet)

johncblandii

FYI @wbrown43 ^

johncblandii

confirmed. used the CI users creds and it worked as expected

@johncblandii you’ll need to update your authenticator configmap to allow other roles/users

johncblandii

just got through that part, @btai.

johncblandii

my last eks was a local install i did so i did not realize this was a rule

@johncblandii yeah took me half a day just trying to figure out how to get aws-iam-authenticator working and I ran into the same issues as you did haha

but no problems since

johncblandii

to which I spent the same half-day (while in 5 hours of meetings back-to-back-to-back)

johncblandii

johncblandii

now my nodes aren’t connecting so onto issue #3; lol

did you add the role for the worker nodes to your config map as well?

johncblandii

yup. the cp module does it

johncblandii

going to nix one and let the scaling kick off a fresh one now that the map is applied

johncblandii

so no public ip seems to have been the issue

johncblandii

started one w/ a public ip and voila

@johncblandii are you talking about public ip for your worker nodes?

johncblandii

yup

johncblandii

our other cluster didn’t have public, but when I upgraded them to 1.12 i had to do the same

(unsure if that’s related, but i did notice that)

@johncblandii fwiw, i didnt have to make my worker nodes public. im still on 1.11 but i cant imagine that would change in 1.12

johncblandii

i hear you. that’s just what i noticed when i moved to .12

Jeremy Grodberg

I have a cluster in AWS in 3 availability zones, with 3 masters, but only 2 nodes. kops put both nodes in the same AZ? Is this a bug? How do I get kops to spread the nodes evenly across AZs?

Erik Osterman

it’s not a kops thing

Erik Osterman

compare how the master node pools are created to how the worker node pools are created

Erik Osterman

that’s how to ensure more even distribution

Erik Osterman

AWS will make “best effort” to allocate instances evenly, but no guarantee

Erik Osterman

the only way to have a “guarantee” is to create node pools tied to exactly one AZ

Issif

for this purpose, we create 3 ASG with only one master inside

1
Erik Osterman

precisely…

Jeremy Grodberg

Yes, kops creates an instance group per zone for the masters, but just 1 instance group for all the nodes.

Jeremy Grodberg

So it turns out the bigger issue is that AWS autoscale group does launching and zone balancing separately, and to do zone balancing it has to launch a new instance before deleting the old one. Well, we had run up against our instance/type limit for the region, so it could not do zone balancing.

Erik Osterman

oh fascinating

Erik Osterman

good sluething

wbrown43
06:50:41 PM

@wbrown43 has joined the channel

2019-05-02

nutellinoit

@mgrube we use Rancher for users permission management in front of EKS

1
    keyboard_arrow_up