#kubernetes (2019-05)

error: You must be logged in to the server (the server has asked for the client to provide credentials)

has your session expired?

doesn’t aws-iam-auth create a new one every time?

i used the CP module to do this before and it is working fine still on that end

you using geodesic? can you exit the shell, run it again, and assume role?

nopers. direct

trying in a fresh terminal

error: You must be logged in to the server (the server has asked for the client to provide credentials)

aws-iam-authenticator token -i... works just fine

i’ve confirmed the configmap matches the first one i did

(sans names, of course)

maybe i need to update my core ~/.kube/config?

no go there

any debug tips?

ahhh….might be this:
This could be because the cluster was created with one set of AWS credentials (from an IAM user or role), and kubectl is using a different set of credentials.

I created it via CI

this was the issue?

I think so

https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html#unauthorized

and the kubectl apply on CI failed because kc isn’t available on there (yet)

FYI @wbrown43 ^

confirmed. used the CI users creds and it worked as expected

@johncblandii you’ll need to update your authenticator configmap to allow other roles/users

just got through that part, @btai.

my last eks was a local install i did so i did not realize this was a rule

@johncblandii yeah took me half a day just trying to figure out how to get aws-iam-authenticator working and I ran into the same issues as you did haha

but no problems since

to which I spent the same half-day (while in 5 hours of meetings back-to-back-to-back)

now my nodes aren’t connecting so onto issue #3; lol

did you add the role for the worker nodes to your config map as well?

yup. the cp module does it

going to nix one and let the scaling kick off a fresh one now that the map is applied

so no public ip seems to have been the issue

started one w/ a public ip and voila

@johncblandii are you talking about public ip for your worker nodes?

yup

our other cluster didn’t have public, but when I upgraded them to 1.12 i had to do the same

(unsure if that’s related, but i did notice that)

@johncblandii fwiw, i didnt have to make my worker nodes public. im still on 1.11 but i cant imagine that would change in 1.12

i hear you. that’s just what i noticed when i moved to .12

it’s not a kops thing

compare how the master node pools are created to how the worker node pools are created

that’s how to ensure more even distribution

AWS will make “best effort” to allocate instances evenly, but no guarantee

the only way to have a “guarantee” is to create node pools tied to exactly one AZ

for this purpose, we create 3 ASG with only one master inside

precisely…

Yes, kops creates an instance group per zone for the masters, but just 1 instance group for all the nodes.

So it turns out the bigger issue is that AWS autoscale group does launching and zone balancing separately, and to do zone balancing it has to launch a new instance before deleting the old one. Well, we had run up against our instance/type limit for the region, so it could not do zone balancing.

oh fascinating

good sluething

The LB is created through the helm deploy.

using this https://www.terraform.io/docs/providers/aws/d/security_group.html and query by filter or tags?

https://github.com/stakater/Whitelister

I would pursue a k8s native solution rather than trying to fuse terraform with helm

Also, IP whitelisting should be used as a last resort. Identity Aware Proxies is ala keycloak is a better approach

alb ingress controller creates you an ALB and the necessary security groups and assigns them to access your workers

@Andriy Knysh (Cloud Posse) there isn’t enough on the SG to query that way. it has the [k8s.io/](http://k8s.io/)… tag, but it is not specific.

@Erik Osterman (Cloud Posse) this isn’t fusing helm and tf. it is the SG created by TF, but I’m mainly just adding an SG record so it is mainly AWS infrastructure networking.

if you go that route, you can filter by name (the resource has some name) and not tags

or add your own specific tag

I think I lack context of where you are trying to do this?

“Is there a clean way from XXXXXXX to get the security group created for an LB by ZZZZZZ so I can assign it to the workers SG in YYYYYYY to approve traffic?”

you may be technically right w/ fusing them. i’m technically wanting a value from k8s so i can configure the AWS SG to allow communication.

The SG is handled within TF manually

@johncblandii what type of LB are you using, if youre using an ALB I would suggest alb ingress controller as it does all that for you. (the downside is when you tear down your cluster, it wont clean up for you)

it automatically used a classic elb (helm install)

what helm chart @johncblandii

Twistlock

welp…bitten by the “providers cannot be dynamically initialized” issue

you reminded me that we had the exact issue for it! https://github.com/mumoshu/aws-secret-operator/issues/14

this is interesting

curious how it works in detail. Like, does it make your microservice stateful? Or does it put the cache local to your cluster? Or is aws handling all the caching for us automagically?

The go SDK code looks straight forward though. Awesome find!

how does it look when you want many parameters?

kind: Secret
metadata:
  name: my-secret
  annotations:
    aws-ssm/k8s-secret-name: my-secret
    aws-ssm/aws-param-name: my-db-password
    aws-ssm/aws-param-type: SecureString

e.g. /db/*

The name of the AWS SSM Parameter. May be a path.

i guess that answers it

but still curious. i never really kicked the tires on aws-ssm

(ultimately, client wanted per-service access controls so we went with Chamber +S3 + IAM + KIAM)

@Erik Osterman (Cloud Posse)

apiVersion: v1
kind: Secret
metadata:
  name: my-secret-name
  annotations:
    aws-ssm/k8s-secret-name: my-secret-name
    aws-ssm/aws-param-name: {{ .Values.ssm_path }}
    aws-ssm/aws-param-type: Directory
data: {}

Where ` .Values.ssm_path == /directory/within/ssm`

Ah, thx!

(lol, sorry about the delay)

how’s the helmfile PR coming along?

stale at the moment. been a bit busy. basically I didn’t consider multiple files

and there’s some chicken/egg issue about when the template-rendering happens and when to reference a file

so I just need to hit my head a little harder on it

maybe that will be simpler if they decouple the multi-phase rendering

Possibly. I thought multi-phase rendering was needed for template in template situations

Thanks for sharing @oscarsullivan_old! This looks really neat. You should share it in the #terraform channel.

KUBE_CONTEXT should be the name of a kubernetes integration in codefresh

it would seldom, if ever have the app name in it

https://codefresh.io/docs/docs/deploy-to-kubernetes/add-kubernetes-cluster/

Got it thanks!

finally

ergg.. i spent a good part of a day understanding how it works/getting it to work w/my eks cluster spun up in tf

I would like to avoid configuration overhead. I’ve already read about Affinity and Anti-Affinity but I’m not sure if this can help me

Someone in the Kubernetes Slack answered my question, looks like this is it: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/

Yes, this is what you want to look into.

(comment just to monitor response)

@Erik Osterman (Cloud Posse) I am currently allowing very few users to access the dashboard using aws-iam-authenticator.