#kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2019-10-18

Brandon Shutter
Brandon Shutter

Just deployed k8s via the k8s-workers module, everything is working great. Being able to add iam users and roles via terraform is amazing.

Brandon Shutter

Attempting to deploy a gitlab helm chart results in

Error creating load balancer (will retry): failed to ensure load balancer for service default/gitlab-nginx-ingress-controller: could not find any suitable subnets for creating the ELB
Brandon Shutter

I used CloudPosse’s VPC, Subnets, EKS, local.tag and EKS Workers modules

Brandon Shutter

I figured it out

4
Brandon Shutter

I needed to add the var.tags to the subnet module

aknysh

@Brandon Shutter thanks! Have you looked at this working example https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/main.tf

cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

aknysh
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

aknysh

I believe you are talking about these tags https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/main.tf#L19 (shared is required by EKS)

cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

2019-10-16

Chris Fowles

interested in thoughts - my thoughts are it sounds like it’s trying to separate dev and ops which i do not like

Erik Osterman
Announcing Cloud Native Application Bundle (CNAB) - Docker Blog

Learn from Docker experts to simplify and advance your app development and management with Docker. Stay up to date on Docker events and new version announcements!

Erik Osterman
CNAB: a spec for packaging distributed apps.

Cloud Native Application Bundles facilitate the bundling, installing and managing of container-native apps — and their coupled services.

Chris Fowles

sort of - it seems more like a way to implement an abstraction layer between teams of dev/ops/infra teams. cnab feels like more of a packaging tool kit to me, where this feels more like enterprise service catalogish kind of stuff (insert hand-waving)

Chris Fowles

while i understand the pain that’s driving the need, i’m not sure i’d like to deal with an environment where that was required

Chris Fowles

i’m also a little sick of abstractions over the kube apis that just look like the kube apis

1

2019-10-12

Erik Osterman
kubernetes/kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management - kubernetes/kops

Erik Osterman

this is what I was referring to

Erik Osterman

then there are some other terraform modules (not by us) that leverage this (i think)

2019-10-11

@Erik Osterman Thanks

I’m trying to pass encrypted values to secrets and use them as variables, will that work?

{{ (tpl (.Files.Glob “configs/*“).AsSecrets . ) | indent 2 }}

Hey all, trying to set up kops in a new environment set up with the reference-architectures repo, so right now trying to run kops-aws-platform (https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops-aws-platform) and it seems it expects IAM roles like masters.us-west-2.testing.ryanjarv.sh and nodes.us-west-2.testing.ryanjarv.sh to be set up. Wondering if there is some step I missed that handles that.

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

Erik Osterman

those are provisioned by kops

Ok thanks will look into that. It did run ok but might need a more recent version or something.

Think I got it figured out, missed the extra steps here before. (https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops)

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

Erik Osterman

just so there’s no confusion we’re not using the terraform mode for kops

Erik Osterman

there are some other modules out there by others that do that

Erik Osterman

our module is for setting up the aws integration points that kops expects.

Terraform mode? Suppose I don’t know much to much about managing kops/k8s. Is that just managing individual pods with terraform? k8s in general still gets set up with the kops-aws-platform module right?

Edit: ok nvm seems the cluster itself is set up with kops.

2019-10-10

Austin Cawley-Edwards

Awesome, thank you both!

sarkis

cross posting from #security because it is relevant here: https://sweetops.slack.com/archives/CBXSAR45B/p1570720099000200

Kubernetes 'Billion Laughs' Vulnerability Is No Laughing Matter - The New Stack

A new vulnerability has been discovered within the Kubernetes API. This flaw is centered around the parsing of YAML manifests by the Kubernetes API server. During this process the API server is open to potential Denial of Service (DoS) attacks. The issue (CVE-2019-11253 — which has yet to have any details fleshed out on the page) has been labeled a ‘Billion Laughs’ attack because it targets the parsers to carry out the attack.

Michael Cram

This is why you always use a bastion host and isolate your cluster from everyone.

Kubernetes 'Billion Laughs' Vulnerability Is No Laughing Matter - The New Stack

A new vulnerability has been discovered within the Kubernetes API. This flaw is centered around the parsing of YAML manifests by the Kubernetes API server. During this process the API server is open to potential Denial of Service (DoS) attacks. The issue (CVE-2019-11253 — which has yet to have any details fleshed out on the page) has been labeled a ‘Billion Laughs’ attack because it targets the parsers to carry out the attack.

how to encrypt passwords in helm values.yaml, any good documents is appreciated. Thanks

Erik Osterman

I assume you’re referring to helm’s values.yaml

right

I used helm secrets to make sure passwords are hidden when pushed to code repositories

I was not sure about helm get values

can you please let me know other startegies

@Erik Osterman ^

Erik Osterman

@AG there’s the helm-secrets plugin that tries to address this

1
Erik Osterman

but secrets will still be clear-text in the if you run helm get values

Erik Osterman

(which is why you just can’t pass any secrets via helm that you truly care about)

Erik Osterman

instead, the better pattern is to assume the secrets have been installed some other way…. basically assume the resource already exists and don’t provision with helm

Erik Osterman

then when you install the chart release, it will block until that secret exists.

Erik Osterman

there are a few strategies for populating secrets

Erik Osterman

basically, you want to decouple the lifecycle of secrets with the lifecycle of helm releases

2019-10-09

Austin Cawley-Edwards

Hey all, not sure if this belongs in this channel so please let me know if it’s not the place, but I just opened up a neat feature PR for the cloudposse/prometheus-to-cloudwatch app - if anyone uses that and has some time to give some feedback I would really appreciate it, thanks! https://github.com/cloudposse/prometheus-to-cloudwatch/pull/28

feat: add ability to exclude dimensions per-metric by austince · Pull Request #28 · cloudposse/prometheus-to-cloudwatch

Closes #27 This feature allows users to exclude a set of dimensions from metrics. It should be easy enough to add a dimensions whitelist as well, which seems to be in the style of this application,…

2
Erik Osterman

@aknysh will review

Erik Osterman

@Austin Cawley-Edwards thanks for the contribution

2019-10-07

@PePe @Cameron Boulton I love you

5

like.. project is done already..

1

global accelerator is amazing

1
1
1
Cameron Boulton

Yea, 80% of infra solutions are like this: people fall back on what they know and build these Rube Goldberg machines that have already been solved.

3

I’m glad it worked for you

1

2019-10-05

Erik Osterman

This should be possible today using simple nginx ingress with the right annotations

it’s not available on k8s 1.14 which is the highest eks version

Erik Osterman

<http nlb <http “eipalloc-07e3afcd4b7b5d644,eipalloc-0d9cb0154be5ab55d,eipalloc-0e4e5ec3df81aa3ea”

Erik Osterman

Ah so need to run a newer version of k8s not supported by eks

Erik Osterman
EIP allocation for NLB Nginx-ingress · Issue #81421 · kubernetes/kubernetes

The issue points to the reported closed issue here : #63959 I tested this but its not working correctly and ingress is not respecting the annotations : I have hard time getting this working with NL…

Using static IP addresses for Application Load Balancers | Amazon Web Services

Introduction In August 2016, Elastic Load Balancing launched Application Load Balancer (ALB), which enable many layer 7 features for your HTTP traffic. People use Application Load Balancers because they scale automatically to adapt to changes in your traffic. This makes planning for growth easy, but it has a side effect of changing the IP addresses […]

I referenced this one initially. It is an option i’m considering

Using static IP addresses for Application Load Balancers | Amazon Web Services

Introduction In August 2016, Elastic Load Balancing launched Application Load Balancer (ALB), which enable many layer 7 features for your HTTP traffic. People use Application Load Balancers because they scale automatically to adapt to changes in your traffic. This makes planning for growth easy, but it has a side effect of changing the IP addresses […]

it’s pretty gnarly, but definitely last resort

I appreciate you sharing this

it is so much easier to use global accelerator

1

thank you, I’m taking a look. I haven’t heard of it before

1

2019-10-04

I got a tricky one for you peeps.. At a high level, I need a static IP (Elastic IP) in front of a k8s service or ing.

aws-alb-ingress-controller doesn’t help since ALBs can’t use EIPs out of the box.. (yes, you can put an NLB in front of it.. and have a lambda function keep the NLB target group up to date the ALB IPs.. https://aws.amazon.com/blogs/networking-and-content-delivery/using-static-ip-addresses-for-application-load-balancers/)

Using nlb annotations in a svc is feature poor even with the latest version of EKS (k8s 1.14) and doesn’t properly attach EIPs to the NLB.

What else should I look at? Things that sound nice but I’ve never touched before (CRDs, Operators, etc..) could maybe help.. or not? What do you think?

Does it have to be an IP? Can it be a domain name? nginx-ingress controller works really well. Set up a domain in Route53 and use nginx-ingress controller, so your service is http://myservice.example.com, or whatever you want it to be.

Yeah, IP. Someone needs to whitelist our IP for an integration.

Cameron Boulton

For inbound traffic @Ryan? As in the integration is going to PUSH to your IP?

@Cameron Boulton exactly

Cameron Boulton

Huh. I agree with Pepe: Global Accelerator is probably your best bet.

Was going to just stand up NLB -> ECS (with Traefik) -> ALB (DNS)

interesting

lemme take a look at that.. haven’t heard of it

Alternatively.. I can use terraform to stand up an NLB + EIPs.. then use a lambda function or some code somewhere to constantly update the NLB target group with the results from kubectl get nodes

2019-10-03

2019-10-02

sohel2020

Does sweetops has any terraform module to create Kubernetes cluster using kops?

davidvasandani

No they use kops from the cli to provision kubernetes.

1
Alex Siegman

That’s true, however they still set up a lot of dependent resources with terraform. See:

https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops

and

https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops-aws-platform

and there’s other modules in that same repo to assist kops with some stuff.

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

1
Alex Siegman

but correct, no automation of kops itself

1
Erik Osterman

Ya we haven’t automated kops because what kops does it does better than terraform

1
Erik Osterman

It’s purpose built for managing the lifecycle of the cluster with the business logic of how to do updates. Terraform is more like a bulldozer.

1

2019-10-01

ruan.arcega

i am using terraform-aws-elasticsearch module in my stack and im loved it

ruan.arcega

from cloudposse repository congratulations to those involved!!

Erik Osterman

Awesome! We use that one all the time

Erik Osterman

It’s great with fluentd and k8s

ruan.arcega

yeah, so, i got some trouble, when kibana record the CNAME on route53, the path /_plugin/kibanamust not be part of the record.

there is a issue for it to fix: https://github.com/cloudposse/terraform-aws-elasticsearch/issues/14

kibana_hostname contains invalid records · Issue #14 · cloudposse/terraform-aws-elasticsearch

When dns_zone_id is supplied, the module attempts to create a CNAME Route53 record for the domain&#39;s Kibana endpoints. These endpoints look like &quot;xxx.<region>.http://es.amazonaws.com/_plugin

ruan.arcega

must be just <http://vpc-sb-shared-elasticsearch-6m6ftgtu6n74l3dh3drw3vwmvq.us-east-1.es.amazonaws.com>

Erik Osterman

@aknysh this looks like a bug

Erik Osterman

that’s odd though since we deploy this regularly

aknysh

this is a feature

aknysh
cloudposse/terraform-aws-elasticsearch

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash. - cloudposse/terraform-aws-elasticsearch

aknysh

and

aknysh
cloudposse/terraform-aws-elasticsearch

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash. - cloudposse/terraform-aws-elasticsearch

aknysh

use the same domain name <http://testing.cloudposse.co>

aknysh
TestExamplesComplete 2019-07-28T22<i class="em em-37"></i>01Z command.go<i class="em em-121"></i> domain_hostname = <http://es-test.testing.cloudposse.co>                                                        
TestExamplesComplete 2019-07-28T22<i class="em em-37"></i>01Z command.go<i class="em em-121"></i> kibana_hostname = <http://kibana-es-test.testing.cloudposse.co>                                                 
aknysh

we don’t add /_plugin/kibana to it

aknysh

we add it in the helmfiles

aknysh

one of those could be removed since they point to the same thing

aknysh

<http://es-test.testing.cloudposse.co> is the ES domain endpoint

Erik Osterman

right, but I think @ruan.arcega is saying the cname was created automatically with the /_plugin/kibana which is wrong

Erik Osterman
aknysh

<http://es-test.testing.cloudposse.co> /_plugin/kibana would be the Kibana URL

Erik Osterman

right, but look at his screenshot from route53

aknysh

i see it. Maybe something is changed already in AWS. We deployed it last time a few months ago

Erik Osterman
Erik Osterman
Erik Osterman

so our DNS is pointing to the wrong output

Erik Osterman

should it be using domain_name

Erik Osterman

?

aknysh

domain_name is not URL

aknysh

it’s just the name of ES domain

aknysh

we have

<http://vpc-xxx-xxxxx-elasticsearch-xxxx.eu-west-2.es.amazonaws.com/_plugin/kibana/>

as CNAME and it’s working

aknysh

(I mean AWS accepted the record before and accepting it now)

Erik Osterman

yea, so it’s accepting the record

Erik Osterman

but the record is still garbage

Erik Osterman

/ is invalid in DNS

aknysh
Type	Domain Name	Canonical Name	TTL
CNAME	<http://kibana-elasticsearch.eu-west-2.xxx.xxx.io>	<http://vpc-xxx-xxx-elasticsearch-xxxx.eu-west-2.es.amazonaws.com/_plugin/kibana/>
aknysh

resolution works too

aknysh

but I agree since those are the same, one could be removed

2019-09-28

jetstreamin

has anyone used kubeless before?

2019-09-27

2019-09-26

Erik Osterman
A Practical Guide to Setting Kubernetes Requests and Limits

Setting Kubernetes requests and limits effectively has a major impact on application performance, stability, and cost. And yet working with many teams over the past year has shown us that determining the right values for these parameters is hard. For this reason, we have created this short guide and are launching a new product to help teams more accurately set Kubernetes requests and limits for their applications.

Matthew Cascio

Wow. I had just begun research to make a tool that does exactly this. Saves me some time, I guess. Thanks for the post.

A Practical Guide to Setting Kubernetes Requests and Limits

Setting Kubernetes requests and limits effectively has a major impact on application performance, stability, and cost. And yet working with many teams over the past year has shown us that determining the right values for these parameters is hard. For this reason, we have created this short guide and are launching a new product to help teams more accurately set Kubernetes requests and limits for their applications.

Erik Osterman

Do you use helmfile by chance?

Matthew Cascio

Planning on using it more. Why do you ask?

Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Erik Osterman

here’s how we deploy kubecost

Erik Osterman

Matthew Cascio

Thanks, I’ll take a look

2019-09-24

Erik Osterman
2
ruan.arcega

hi guys,

quick question….

i have 2 configMap, A and B, and i have a bunch of key value in both situations, but, in the middle of this bunch i have some key name duplicated with values differents, when configure the specification using envFrom in kuberentes, i call configMap A and in the sequencia configMap B.

Which value will persistent in my container when the pod going up and running ?

I’m guessing the last one wins. But this seems straight forward enough to just test and find out. I don’t actually know off the top of my head

ruan.arcega

yeah @ you are right, i just took a test now, and the last one wins….

i had this question because in the project that i am working one developers want to use the same environment variable with values different.

i know it was not possible, but never had some tests… i suggested use a prefix when call configMap…

- prefix: VALUE_
  configMapRef:
    name: configmap

and treat this situation in the code…

Have you ever heard of anything (in EKS) where you containers just sit idle? Like, we had a 10 minute period in time where the containers didn’t do anything (no logs, no http responses). Datadog said the containers were still running and consuming memory.. but just no application activity. When we looked at the container logs.. it’s like nothing happened.. i mean, there was a 10 minute gap in time between logs, but other than that.. it looked normal

Cameron Boulton

Are logs/responses the only metrics you are reviewing? You said Datadog shows memory consumption but does CPU or memory for that same time period show any changes or is it literally flat?

Cameron Boulton

Do you have request metrics (rate, errors, etc.) for this time period?

the datadog metrics are flat, but non-zero.. idling around 1-5% in both cases

request metrics, yea. Like, the ALB gets a ton of 4xx

since the containers aren’t returning HTTP results

Cameron Boulton

Sounds like they aren’t getting any requests either though

right

Cameron Boulton

ALB health graph looks like …?

like, i wouldn’t be surprised if the readiness probe failed.. so the k8s service stopped routing requests.. but then the livlieness probe would have died too and caused a restart.. which we had 0 of

lemme look at ALB graph

like.. an order of magnitude more 4xx and 5xx

but health seemed fine, unless I’m not looking at the right spot

Cameron Boulton

You’ll have to look at Target Groups monitoring specifically for [Un]Healty Hosts metric

Cameron Boulton

And how are you configuring your ALB(s)? Directly? Indirectly with a Kubernetes controller like alb-ingress-controller?

lol, 8 out of 350 healthy hosts

alb-ingress-controller

Cameron Boulton

Check that controller’s logs

Cameron Boulton

Maybe an reconciliation loop is stuck from bad config and triggering every few minutes?

i can look at that

this only happened once out of like.. a few months of having alb-ingress-controller

@Cameron Boulton you’re a god

yea alb-ingress-controller

11 minute gap

Cameron Boulton

will you marry me?

this is amazing work

Cameron Boulton

Yea I think alb-ingress-controller reconsil loop is 10 mins +/- some seconds (imprecise scheduler)

Cameron Boulton

I’ll send my consultation bill to Calm, Attention: Ryan Smith

so whyy in the heck has this never happened before

Cameron Boulton

hahahaha nice

“payment: 1 big hug”

Cameron Boulton

Ha

Cameron Boulton

Well, depending on what the logs are showing you that it can’t reconcile; maybe this is the first time someone pushed a config change that it couldn’t handle.

what’s lame is the the deployments only change the k8s deployment image

which trigger a deploy

so like.. uhh.. there shouldn’t really have been anything gnarly that killed it

Cameron Boulton

Maybe you’ve hit a bug/are on an older version of alb-ingress-controller?

ah.. guessing older version

<http<i class="em em-//docker.io/amazon/aws-alb-ingress-controller"></i>v1.1.2>

yeah, a patch behind

Cameron Boulton

I’m skeptical that’s it then

Cameron Boulton

What are the logs telling you that the controller is doing/failing?

only info level logs.. but they look like..

I0924 18<i class="em em-55"></i>33.158007       1 targets.go:95] service-a: Removing targets from arn<img src="/assets/images/custom_emojis/aws.png" class="em em-aws">elasticloadbalancing

to remove a big chunk of them.. then

I0924 19<i class="em em-06"></i>30.101251       1 targets.go:80] service-a: Adding targets to arn<img src="/assets/images/custom_emojis/aws.png" class="em em-aws">elasticloadbalancing

adding the big chunk (11min later)

and it was during this timeframe all the activity ceased

Cameron Boulton

Right (as you would expect)

Cameron Boulton

How recently was that version deployed?

the pod alb-ingress-controller is 43 days old

Cameron Boulton

Any chance you’ve been experiencing this issue for 43 days or is it newer?

brand spanking new

caused an outage.. alerted everyone

would have known if it happened before

Cameron Boulton

Okay

Cameron Boulton

How old is/are the Ingress(es) that are annotated for this controller?

the ing is 154d old

Cameron Boulton

Actually, that’s creation time not last modified so nevermind

lol whoops

Cameron Boulton

If you describe your Ingress(es) do you see anything under the events?

<none> events

Cameron Boulton

Hmm

stupid question.. should probably just google it.. but do you have a replica > 1 when you run alb-ingress-controller

Cameron Boulton

Maybe keep going back in your Ingress Controller logs until you find something else or the start of the “Removing/Adding targets” loop?

Cameron Boulton

Did you launch the alb-ingress-controller into a new namespace/cluster recently?

Theres just some detaching and attaching of SGs to ENIs

I0924 18<i class="em em-55"></i>35.660049       1 instance_attachment.go:120] service-a: detaching securityGroup sg-redacted from ENI eni-redacted

hmmm, yeah, but different cluster and different sub account

this one has been stable for a few months

Cameron Boulton

The new cluster/sub-account recent?

but this is the 11 min gap.. and the logs

I0924 18<i class="em em-55"></i>59.734800       1 instance_attachment.go:120] service-a: detaching securityGroup sg-readacted from ENI eni-redated
I0924 19<i class="em em-06"></i>30.101251       1 targets.go:80] prod/app-api: Adding targets to arn<img src="/assets/images/custom_emojis/aws.png" class="em em-aws">elasticloadbalancing

nothing in between

unless EKS had a fart or something for a little

Cameron Boulton

I0924 19<i class="em em-06"></i>30.101251 is the beginning/first instance of “Adding targets” loop?

new cluster/sub account. yeah, but not today.. like a week ago

I0924 18<i class="em em-55"></i>33.158007       1 targets.go:95] service-a: Removing targets 
I0924 18<i class="em em-55"></i>58.344701       1 targets.go:95] service-a: Removing targets 
I0924 19<i class="em em-06"></i>30.101251       1 targets.go:80] service-a: Adding targets 
Cameron Boulton

What are the “args” key for the only container in the alb-ingress-controller pod (if any)?

      - args:
        - --cluster-name=k8s
        - --ingress-class=alb
Cameron Boulton

Okay, so if you have this controller running anywhere else on this cluster or any other cluster in the same account and that one is also using --cluster-name=k8s the controllers are going to fight over ALBs/Target Groups.

Cameron Boulton

Any chance that’s possible?

lemme iterate regions.. but I’m fairly certain 1 cluster in this account

yeah.. 1 cluster in the account

incubator/aws-alb-ingress-controller

helm deployed

Cameron Boulton

And no possibility of it in another namespace or something?

➜  ~ kubectl get pods --all-namespaces \| grep alb
default       alb-aws-alb-ingress-controller-6b9cfd997f-b99zz                   1/1     Running     1          43d

afk for a smidge.. father duties

i really appreciate all your help on this btw. you’re a rare breed and it’s incredibly invaluable

Cameron Boulton

Sure thing. I think that’s all I can spare today though. The behavior you describe sure feels like an solation failure/reconciliation competition.

2019-09-18

Alex Siegman
Deploying to Kubernetes with Helm and GitHub Actions

This tutorial will go through the basics of GitHub actions as well as deploying to Kubernetes using a pre-built Helm action

1
Erik Osterman
Kubernetes 1.16: Custom Resources, Overhauled Metrics, and Volume Extensions

Authors: Kubernetes 1.16 Release Team We’re pleased to announce the delivery of Kubernetes 1.16, our third release of 2019! Kubernetes 1.16 consists of 31 enhancements: 8 enhancements moving to stable, 8 enhancements in beta, and 15 enhancements in alpha. Major Themes Custom resources CRDs are in widespread use as a Kubernetes extensibility mechanism and have been available in beta since the 1.7 release. The 1.16 release marks the graduation of CRDs to general availability (GA).

Erik Osterman
getsentry/sentry-kubernetes

Kubernetes event reporter for Sentry. Contribute to getsentry/sentry-kubernetes development by creating an account on GitHub.

Erik Osterman

2019-09-16

cabrinha

Anyone use https://github.com/kubernetes-sigs/aws-efs-csi-driver yet? I’m having some trouble getting the volume mounted.

kubernetes-sigs/aws-efs-csi-driver

CSI Driver for Amazon EFS https://aws.amazon.com/efs/ - kubernetes-sigs/aws-efs-csi-driver

2019-09-12

joshmyers

Anyone done multi region EKS?

Nikola Velkovski

Not but I was doing some research

Nikola Velkovski

To me it looks like it’s possible with a Latency record combined with ExternalDNS

Nikola Velkovski

but that’s possible if this PR is merged

Nikola Velkovski
Add support for latency-based routing on AWS · Issue #571 · kubernetes-incubator/external-dns

Route53 on AWS supports &quot;latency-based routing&quot; for DNS records. You can have multiple DNS records for the same hostname, having different ALIAS to regional ELBs. This is usually the pref…

Nikola Velkovski

I was doing a blog post about it

Nikola Velkovski

and decided to cut it short at the DNS/app level.

2019-09-11

2019-09-10

johncblandii

I’m not learned in the area of k8s scheduling so this is destroying my day. LOL. Anyone have any helpful insights?
Warning FailedScheduling 50s (x9 over 8m7s) default-scheduler 0/1 nodes are available: 1 node(s) had volume node affinity conflict.

johncblandii

This is a 1-node cluster on EKS

Alex Siegman

the scheduler is trying to tell you it has no nodes to work on, I believe. I’m no expert here either, but i’d start by investigating the node itself to see why it’s busto.

what does kubectl get nodes show?

Alex Siegman

I wish the error was better than “volume node affinity conflict”

Alex Siegman

volume node affinity makes me think that some pod and some persistant volume out on EBS can’t connect to each other. A PV backed by EBS will limit a POD to a specific AZ. That AZ will match that of the worker that created/hosts the PV. Is somehow an EBS volume of a PV being created in the wrong AZ?

Alex Siegman

not sure how you launched your 1 node in EKS, or why this would be an issue thereon

johncblandii

at hashiconf so was moving around, but catching up on the read

johncblandii

get nodes shows the node is healthy and not maxed out

johncblandii

it is just 1 node, though. i guess it is time to grow this a bit

johncblandii

ah, it does have a pvc. let me check the AZ

Maycon Santos

kubectl describe pv PV_NAME kubectl describe node NODE_NAME

Maycon Santos

check its Labels or VolumeId and ProviderID

Maycon Santos

could be a relaunch of your single node on another AZ

johncblandii

nailed it. pv in 2c and node in 2a

johncblandii

my other nodes aren’t joining the cluster anymore

johncblandii

not sure why my eks nodes no longer work on private IPs, but i had this node problem with another cluster too

johncblandii

got the nodes back and that resolved it

2019-09-05

Jonathan Le
Amazon EKS Cluster OIDC Issuer URL · Issue #9995 · terraform-providers/terraform-provider-aws

Community Note Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request Please do not leave &quot;+1&quot; or &quot;me to…

Jonathan Le

Anyone wanna add their thumbs up to that issue? EKS IAM POD roles TF

4
Jonathan Le

It should hit 2.28.0 coming out in Thursday on the AWS Provider

4

2019-09-04

Erik Osterman
Introducing Fine-Grained IAM Roles for Service Accounts | Amazon Web Services

Here at AWS we focus first and foremost on customer needs. In the context of access control in Amazon EKS, you asked in issue #23 of our public container roadmap for fine-grained IAM roles in EKS. To address this need, the community came up with a number of open source solutions, such as kube2iam, kiam, […]

wow that’s noiceeee

2019-08-29

Robert
07:19:12 PM

@Robert has joined the channel

2019-08-28

Alex Siegman

I haven’t looked at the material, but I saw it advertised as “agnostic” which is really nice

Alex Siegman

Linked it to my team this morning, I’ll pass on any feedback I get if they try it

Erik Osterman

cool, lmk!

Erik Osterman
HashiCorp looks into easier secret management for Kubernetes • DEVCLASS

HashiCorp has finished work on Consul 1.6 and offered a first insight on upcoming Vault features aimed at users of container orchestrator Kubernetes.

2

2019-08-27

What are people using using to terraform custom resource definitions for kubernetes?

Null resource local exec, with a life cycle hook for destroy

aaratn

Null resources

Alejandro Rivera

Hi, I’m using nginx ingress controller to expose thanos sidecar, I have validated that service is setup correctly and it’s responding as expected, but when using Nginx I get 400 error:

Alejandro Rivera
00.00.00.00 - [00.00.00.00] - - [27/Aug/2019<i class="em em-23"></i>58:28 +0000] "PROXY TCP4 00.00.00.00 00.00.00.00 44782 30226" 400 163 "-" "-" 0 0.000 [] [] - - - -...

(edited out the ip addresses)

this is the ingress config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    <http<i class="em em-//external-dns.alpha.kubernetes.io/hostname\|external-dns.alpha.kubernetes.io/hostname>"></i> foo.bar
    <http<i class="em em-//kubernetes.io/ingress.class\|kubernetes.io/ingress.class>"></i> nginx
    <http<i class="em em-//nginx.ingress.kubernetes.io/backend-protocol\|nginx.ingress.kubernetes.io/backend-protocol>"></i> GRPC
    <http<i class="em em-//nginx.ingress.kubernetes.io/ssl-redirect\|nginx.ingress.kubernetes.io/ssl-redirect>"></i> "true"
    <http<i class="em em-//service.beta.kubernetes.io/aws-load-balancer-internal\|service.beta.kubernetes.io/aws-load-balancer-internal>"></i> "true"
    <http<i class="em em-//service.beta.kubernetes.io/aws-load-balancer-ssl-cert\|service.beta.kubernetes.io/aws-load-balancer-ssl-cert>"></i> foo
  creationTimestamp: "2019-08-27T23<i class="em em-52"></i>59Z"
  generation: 1
  labels:
    service: thanos-sidecar
  name: thanos-sidecar
  namespace: monitoring
  resourceVersion: "foo"
  selfLink: /apis/extensions/v1beta1/namespaces/monitoring/ingresses/thanos-sidecar
  uid: foo
spec:
  rules:
  - host: foo.bar
    http:
      paths:
      - backend:
          serviceName: thanos-sidecar
          servicePort: grpc
status:
  loadBalancer:
    ingress:
    - hostname: foo.bar
Alejandro Rivera

Enabled TLS and still getting 400 errors

Alejandro Rivera
00.00.00.00 - [00.00.00.00] - - [28/Aug/2019<i class="em em-02"></i>13:40 +0000] "PRI * HTTP/2.0" 400 163 "-" "-" 0 0.002 [] [] - - - -

2019-08-26

Sandeep Kumar

is it possible to Clone existing Google cloud Kubernetes cluster using gcloud command line options? I see the documentation available for cloning existing cluster manually from GCP console (https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster#clone-existing-cluster)

2019-08-22

how was the upgrade from 1.11 to 1.12?

Been testing that recently

Fairly mixed results in terms of predictably, at least on my side thus far

I had planned to focus on building in support for automated https://github.com/heptio/velero as a fall back plan

heptio/velero

Backup and migrate Kubernetes applications and their persistent volumes - heptio/velero

Though mit something I will be able to focus on, at least in the current company I’m at

2019-08-21

anyone not running k8s 1.13+ in production? now that CVEs aren’t being fixed in 1.11 what are y’all strategies? I feel like everyone I talk to is still on 1.11

Erik Osterman

EkS or kops?

Erik Osterman

We just upgraded our first kops clusters from 1.12 to the latest release in 1.13

2019-08-20

guigo2k
Security release of Kubernetes v1.15.3, v1.14.6, v1.13.10 - CVE-2019-9512 and CVE-2019-9514

Hello Kubernetes Community, A security issue has been found in the net/http library of the Go language that affects all versions and all components of Kubernetes. The vulnerabilities can result in a DoS against any process with an HTTP or HTTPS listener. Am I vulnerable? Yes. All versions of Kubernetes are affected. Go has released versions go1.12.8 and go1.11.13, and we have released the following versions of Kubernetes built using patched versions of Go. Kubernetes v1.15.3 - go1.12.9 Kub…

Erik Osterman
vitaly.markov

@Erik Osterman but why?

ruan.arcega

and how do we have to treat? what’s the best way to do this? i think he meant that if you are on this level you are already accommodated…

Erik Osterman

kelsey has a theme going on right now to remove some of the sugar coating around kubernetes. it’s been touted up as this magical container platform that solve all our problems. the reality is that like any other piece of software you run, there are tradeoffs. one of the common best practices is to toss traditional DR out the window; no more treating servers (and services) as “pets”, instead treat them as cattle. the crude analogy is with cattle is if they get sick you put them down rather than spend thousands at the vet making them well again. with servers, it’s a little less crude. terminate them and move on. kubernetes makes that very easy, however, there’s still an operator responsible for kubernetes. it’s not “serverless”. So like the Rancher responsible for the cattle, we are in the end responsible the cluster. Not everything will be fully automated in an unattended fashion (or should be).

Erik Osterman

2019-08-17

Erik Osterman

Like for observability?

Erik Osterman

—-

Erik Osterman
vmware/octant

A web-based, highly extensible platform for developers to better understand the complexity of Kubernetes clusters. - vmware/octant

2019-08-16

is there an easy way to grab the headers on the http request from one service to another in a k8s cluster

2019-08-12

i5okie

Hi, anyone here have experience with Flux where it keeps re-applying manifests even if nothing was changed?

Does anyone have experience with oauth2 and kubernetes dashboard?

kskewes

Joel Speed from Pusher did a good KubeCon video on what they do with oauth2_proxy, Dex and dashboard

thank you foor the tip!

Erik Osterman

We’ve integrated it with Keycloak + Gatekeeper (kops + k8s dashboard)

2019-08-08

Hetal S

I am facing issue in TF 0.11.14 when I am creating multiple cluster

Hetal S
 * module.eks.local.kubeconfig: local.kubeconfig: Resource 'aws_eks_cluster.eks' does not have attribute 'certificate_authority.0.data' for variable 'aws_eks_cluster.eks.*.certificate_authority.0.data'

2019-08-06

Announcing the HashiCorp Vault Helm Chart

This week we’re releasing an official Helm Chart for Vault. Using the Helm Chart, you can start a Vault cluster running on Kubernetes in just minutes. This Helm chart will also be …

3
5
Erik Osterman
Announcing the HashiCorp Vault Helm Chart

This week we’re releasing an official Helm Chart for Vault. Using the Helm Chart, you can start a Vault cluster running on Kubernetes in just minutes. This Helm chart will also be …

3
5

2019-08-05

Erik Osterman
2
1
2
1
Tega McKinney

I’m still explaining that to folks in my org. Every few weeks; lets switch providers b/c we won’t have much work

2
1
2
1

2019-08-02

2019-07-31

jafow

I’m looking for a minimally-hacky way to restart running pods in order pickup a change in config data

1
jafow

here’s the use case: pods bootup and source some envrionment vars from SSM

1
jafow

now I change a config value in SSM and would like pods to pick up that value

1
jafow


I’m looking for a minimally-hacky way
I’m okay with some amount of hack tbh

1
jafow

1 idea is to delete pods 1 by 1. They get restarted and when they run they fetch the data fresh from SSM and viola, config data is up to date

1
Erik Osterman
stakater/Reloader

A Kubernetes controller to watch changes in ConfigMap and Secrets and then restart pods for Deployment, StatefulSet, DaemonSet and DeploymentConfig – [✩Star] if you&#39;re using it! - stakater/Relo…

1
jafow

cool thanks @Erik Osterman I’ve heard of it and will check it out.

1
Erik Osterman

it was very easy to setup

1
Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

1
endofcake

I wonder if there’s a way to hook it into Prometheus somehow, so it would only restart a pod if the app overall is healthy. A cursory look at this suggests I’d have to roll my own

1
Erik Osterman

assuming health is only determined by prometheus…

1
Erik Osterman

however, services should have a healthcheck endpoint

1
Erik Osterman

per the Reloader README.md, it says:
then perform a rolling upgrade

1
Erik Osterman

the only way to do a rolling upgrade is to wait for new pods to become healthy before moving on

1
Erik Osterman

thus if the new secrets cause problems, that should cause the rollout to hang

1
Erik Osterman

then the prometheus alerts for a pod crash loop should fire on the unhealthy pod

1
Erik Osterman
stakater/Reloader

A Kubernetes controller to watch changes in ConfigMap and Secrets and then restart pods for Deployment, StatefulSet, DaemonSet and DeploymentConfig – [✩Star] if you&#39;re using it! - stakater/Relo…

1
Erik Osterman

to me, it looks like what they do is update an environment variable which causes k8s to do the rolling update

1
Erik Osterman

thus all rolling update semantics are handled by k8s.

1
endofcake

Yeah, this gets more complicated with distributed applications such as Kafka, hence the need for external monitoring

1

2019-07-30

sohel2020

is it possible to create a EKS cluster in my Existing VPC?

1
Steven

yes

sohel2020
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

jober
module "eks_cluster" {
  source                  = "git:<i class="em em-<https"></i>//github.com/cloudposse/terraform-aws-eks-cluster.git?ref=master>"
  namespace               = "eg"
  stage                   = "testing"
  name                    = "cluster"
  tags                    = "${var.tags}"
  vpc_id                  = "<YOUR VPC ID>"
  subnet_ids              = ["<YOUR PUBLIC SUBNET ID'S>"]

  # `workers_security_group_count` is needed to prevent `count can't be computed` errors
  workers_security_group_ids   = ["${module.eks_workers.security_group_id}"]
  workers_security_group_count = 1
}

2019-07-29

jober

I am looking at starting to use Kubernetes for the first time for a small system that I would eventually grow. Would you suggest using Kubernetes directly or using AWS EKS?

Erik Osterman

I would give eksctl a shot

Erik Osterman

Probably the most turn key way to get up and running with EKS for a small project.

Erik Osterman

(we still use kops)

jober

i was playing around with kops over the weekend

jober

worked really well, just unsure at this point how to manage everything

Erik Osterman

kops is easy to get up and running.

Erik Osterman

the challenge with kubernetes is updates between major releases can be tricky

Erik Osterman

e.g. 1.11 -> 1.12 upgraded to etcd3 and there was no automated way to easily upgrade

Erik Osterman

while on EKS, those kinds of upgrade challenges are handled by the platform

Erik Osterman

also, i believe the upgrade from 1.14 to 1.15 is also that way

Erik Osterman
1
jober

Thanks that is really good to know. I’ll give EKS a go. Much appreciated

aknysh

we also have TF modules for EKS

aknysh
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

aknysh
cloudposse/terraform-aws-eks-workers

Terraform module to provision an AWS AutoScaling Group, IAM Role, and Security Group for EKS Workers - cloudposse/terraform-aws-eks-workers

aknysh
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

jober

Awesome

jober

Thanks so much!

jober

Just found this as well, seems like a good beginner resource https://eksworkshop.com/introduction/

EKSworkshop.com

Amazon EKS Workshop

Erik Osterman

yes, I saw that - looks amazing

2019-07-25

i5okie

last year spent a few months troubleshooting and improving apps running in ElasticBeanstalk. with help from guys here (thank you). This year I’m deleting those ElasticBeanstalk stacks one by one after moving the apps to K8s. #lifeofadevopsengineer

Erik Osterman

haha too true. Thanks @i5okie for the update.

2019-07-23

cabrinha

I’m interested to hear how people are using CD in Kubernetes. Is anyone doing Canary deployments on their EKS cluster? How?

Alex Siegman

i can’t answer your question, but is that commander keen as your profile pic?

Alex Siegman

I’d imagine a lot of people are using a service mesh like istio to manage the networking side of the canary deployment, but how they wrap all that in CD I’ve got no experience with

James D. Bohrman

Rio by Rancher. Check it out.

James D. Bohrman
Rio

The MicroPaaS for Kubernetes

2019-07-22

Is there a way to have helm chart create external resources?(RDS/Elasticache)

Erik Osterman
awslabs/aws-service-operator

AWS Service Operator allows you to create AWS resources using kubectl. - awslabs/aws-service-operator

Erik Osterman
awslabs/aws-servicebroker

AWS Service Broker. Contribute to awslabs/aws-servicebroker development by creating an account on GitHub.

Erik Osterman

Then use the Helm raw chart to provision

Erik Osterman

The CRDs for RDS

Thank you!

What are people using as an oauth2 provider to login to their apps like k8 dashboard?

Erik Osterman

keycloak

Erik Osterman
  • gatekeeper proxies

keycloak too but not for k8 but for pretty much anything

2019-07-19

Erik Osterman
Draft | Easy Kubernetes Development

Draft is a tool for developers to create cloud-native applications on Kubernetes.

saw this at kubecon

my gripes wit this is the feedback loop for rebuilding an image is slow

Erik Osterman

what were your thoughts?

so during development esp. if you save alot assuming youre doing some web dev you’d typically want to refresh your browser right after you save

and see the changes right away

even during their demo at kubecon there was a bit of waiting

for the demo image (prob really small) to rebuild

Erik Osterman

I think developers mostly want live reloading

Erik Osterman

e.g. what you get with telepresence

yep

most prob use docker compose to spin up dependencies

and run their app locally

i think itd be awesome if you could have that type of live reloading but your app is hosted on a k8s cluster in the cloud

each developer would have their own namespace

Erik Osterman

yep - that’s what we’re working on

Erik Osterman

but haven’t yet tackled telepresence

are you able to do live reloading comparable to running it locally

cause thats a gamebreaker i think for most devs

Erik Osterman
Home - Telepresence

Telepresence: a local development environment for a remote Kubernetes cluster

Erik Osterman

so with telepresence you run “the” service locally

Erik Osterman

but all your backing services run in k8s (e.g. in a developer namespace)

ah

Erik Osterman

telepresence is like a reverse proxy. it sits in k8s and any requests it gets it sends back to the service on your laptop

Erik Osterman

so it’s like teleporting your local service into the cluster

Erik Osterman

plus your local service can access everything running in k8s (e.g. database or other backing services)

Erik Osterman

since it runs on your local laptop, you get all the benefits

Erik Osterman

easier debugging, attaching debuggers, live reloads, etc

ah yeah

but i guess other than the fact that dependent services hosted in cloud

is there any other benefit than running those in a docker compose

Erik Osterman

you’re testing your services in an environment that’s closer to staging/prod

Erik Osterman

if you have 30-40 microservices as part of your stack, good luck doing that on your laptop

Erik Osterman

if you need large datasets for development, can’t do that easily locally

Erik Osterman

maybe it’s nicer to have all the data stay in AWS from a security perspective

truee

Erik Osterman

multiple developers can be working on pieces of the project at the same time

Erik Osterman

and using a shared environment

so you host the db remotely as well

assume youre working on an API

Erik Osterman

its easier for others to QA changes in a public environment (e.g. if your laptop is offline, no one can review)

Erik Osterman

yea, usually run db as a container for these env

nice

yeah the large dataset hosted remotely one is actually really useful

allowing multiple devs on same project

cabrinha

Has anyone here been able to get Istio installed to EKS? I’m trying to get it installed with my worker nodes all residing in private subnets and I’m running into weird issues.

Erik Osterman

@Vidhi Virmani I think did

cabrinha

Most of the examples I come across online assume you’re in public subnets with ELBs and security groups open to the world

cabrinha

@Vidhi Virmani if you’ve installed Istio on EKS in private subnets, ping me!

2019-07-18

timduhenchanter

Anyone familiar enough with Kubernetes API to maybe know a way to get the resourceVersion of the parent for a given pod? Trying to ignore events in the watch if the change was initiated by its parent (I am not the controller). This is related to a ReplicaSet

2019-07-17

Hi, I have an idea about how I’m going to implement config templating for our containers and I will like some feedback :

  • Secrets and non secrets will be stored in aws Parameter Store+KMS
  • Chamber will be used to update/create secrets
  • Path IAM roles will be created for every environment /dev/secrets /prod/secrets etc
  • ECS Task per environment will have access to the /env path that have secrets and not secrets ( no shared configs)
  • Dev will use chamber to set ENV variable to run their local containers
  • confd might be use to create the config templates on Docker build

most Jenkins will be doing the initial push and CodeDeploy will do the rest for Blue Green

any comments ?

2019-07-14

Erik Osterman
ibuildthecloud/k3v

Virtual Kubernetes. Contribute to ibuildthecloud/k3v development by creating an account on GitHub.

2019-07-11

Erik Osterman
[EKS]: Next Generation AWS VPC CNI Plugin · Issue #398 · aws/containers-roadmap

We are working on the next version of the Kubernetes networking plugin for AWS. We&#39;ve gotten a lot of feedback around the need for adding Kubenet and support for other CNI plugins in EKS. This …

1
Erik Osterman

Higher container density is coming

2019-07-10

Alex Co

hi, anyone here knows about use boolean value with k8s ?

Alex Co

i’m trying to read the value from Environment variable, and i realize that when setting the env var inside a pod, it’s always format it as a string

Alex Co

for example MY_VAR=abc will be MY_VAR=‘abc’ inside a pod

Alex Co

if any program is expecting the boolean type, it will throw error

Alex Co

is there a way to solve this ?

Alex Co

thanks

environment variables are always strings

it needs to be handled by your program or whatever library you are using

Alex Co

thanks @MiLk

2019-07-09

Erik Osterman
more examples of config files · Issue #508 · weaveworks/eksctl

Here is an example which should be documented, it uses only pre-existing IAM and VPC resources: apiVersion: http://eksctl.io/v1alpha4 kind: ClusterConfig metadata: name: test-cluster-c-1 region: eu-north-…

Erik Osterman

Anyone know if it’s possible to pass settings in the config to run commands on startup (like with kops)?

Erik Osterman

…in eksctl

Erik Osterman

(@mumoshu)

Erik Osterman

e.g. in kops, we can do:

  hooks:
  # Mitigate CVE-2019-5736
  - before:
    - docker.service
    manifest: \|
      Type=oneshot
      ExecStart=/usr/bin/chattr +i /usr/bin/docker-runc
mumoshu

There’s no easy way to add systemd units like that in eksctl. Technically there’re two options though -

  1. use preBootstrapCommands to write systemd units(https://github.com/weaveworks/eksctl/blob/cf5e078273d8d0d8fa802ae704f038d9c56ad8d7/pkg/apis/eksctl.io/v1alpha5/types.go#L457) or

  2. deploy a privileged daemonset that mount host volumes and writes unit files(https://github.com/mumoshu/kube-node-init)

weaveworks/eksctl

a CLI for Amazon EKS. Contribute to weaveworks/eksctl development by creating an account on GitHub.

mumoshu/kube-node-init

Kubernetes daemonset for node initial configuration. Currently for modifying files and systemd services on eksctl nodes without changing userdata - mumoshu/kube-node-init

i5okie

Hi, has anyone tried the bitnami http://kubeprod.io stack? (https://github.com/bitnami/kube-prod-runtime)

bitnami/kube-prod-runtime

A standard infrastructure environment for Kubernetes - bitnami/kube-prod-runtime

2019-07-08

2019-07-06

2019-07-04

2019-07-03

anyone running a container with 200GB+ memory ?

1

bad idea ?

1
leonardo.miranda

just curious, why do you need that size of container?

1

is a in memory key value store

1

somebody mentioned a tool for setting ENV variables for docker with support for different environments ?

with some template support

for the life of me I can’t recall the name….

dockerize?

it was something like systemenv ro something with env

ankur

env consule?

nop

mumoshu/variant

Wrap up your bash scripts into a modern CLI today. Graduate to a full-blown golang app tomorrow. - mumoshu/variant

I think so…

2019-07-02

davidvasandani
davidvasandani


Sanic is an omni-tool which lets you build, deploy, and manage Kubernetes clusters.

reminds me of #geodesic

2
Erik Osterman

Looks like a go cli tool that can be added to geodesic

2

2019-06-30

I am stuck while installing Kubeadm in AWS on Amazon linux

Below is the error i get after running sudo install -y kubeadm

<http [Errno -1] repomd.xml signature could not be verified for kubernetes

Please help me to get over it.

Glenn J. Mason
Got "repomd.xml signature could not be verified for kubernetes" error when installing Kubernetes from yum repo on Amazon Linux 2 · Issue #60134 · kubernetes/kubernetes

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I&#39;m trying to install Kubernetes on Amazon Linux 2 as described here, but I get error: [[email protected] ~]$ sudo yum install …

@Glenn J. Mason Thanks a lot .. It’s exactly what i am searching for

2019-06-28

Erik Osterman
kubernetes/community

Kubernetes community content. Contribute to kubernetes/community development by creating an account on GitHub.

Erik Osterman
4

2019-06-25

2019-06-24

Ayo Bami
11:08:17 PM

HI guys, Please could someone help me . I just created an EKS cluster and its unable to apply some changes to the cluster. I keep getting that error log.. I am using the same user I used to create the cluster. I am also using auth account. So the Users are not exactly in that account they assume role. Am not sure what am missing here being trying this for days now.. Thanks

Erik Osterman

How did you bring up the cluster? Are you using terraform?

Ayo Bami

@Erik Osterman with terraform

Ayo Bami
howdio/terraform-aws-eks

Terraform module which creates EKS resources on AWS - howdio/terraform-aws-eks

nutellinoit
Managing Users or IAM Roles for your Cluster - Amazon EKS

The aws-auth ConfigMap is applied as part of the guide which provides a complete end-to-end walkthrough from creating an Amazon EKS cluster to deploying a sample Kubernetes application. It is initially created to allow your worker nodes to join your cluster, but you also use this ConfigMap to add RBAC access to IAM users and roles. If you have not launched worker nodes and applied the

nutellinoit

point 3

Ayo Bami

@nutellinoit Its a new cluster. If my user can’t access the cluster, not sure if it can run aws-auth on the cluster if it can’t access it

Ayo Bami

We use Auth to manage IAM, The accounts are not directly in the cluster.

2019-06-23

Hey @cabrinha I’d be interested in hearing anything you ran into with aws-okta and EKS. I’m going to be starting down that road this week.

2019-06-20

Nikola Velkovski
hjacobs/kubernetes-failure-stories

Compilation of public failure/horror stories related to Kubernetes - hjacobs/kubernetes-failure-stories

Erik Osterman

it’s worth resharing

hjacobs/kubernetes-failure-stories

Compilation of public failure/horror stories related to Kubernetes - hjacobs/kubernetes-failure-stories

Nikola Velkovski

Nikola Velkovski

I got to the spotify video, which is kinda cool, they admit their rookie mistakes around terraform

Ribhararnus Pracutiar

Hi guys, how to connect vpn from inside pods? any recommendation out there? So basically, I have to connect client data on premise, only using 1 ip

cabrinha

is anyone here using aws-okta with EKS? I’m having trouble granting additional roles access to the cluster.

2019-06-19

anyone using this on their clusters? https://github.com/buzzfeed/sso

buzzfeed/sso

sso, aka S.S.Octopus, aka octoboi, is a single sign-on solution for securing internal services - buzzfeed/sso

Looks interesting

Erik Osterman

Doesn’t support websockets, so it was a deal breaker for us

Erik Osterman

things like the k8s dashboard or grafana require that

Erik Osterman

bite the bullet. just deploy KeyCloak with Gatekeepers

havent heard of keycloak/gatekeeper

Erik Osterman

I can give you a demo

Erik Osterman

it’s open source, by redhat

Erik Osterman

we have the helmfiles for it too

does it integrate w/google saml?

Erik Osterman

yup, that’s the beauty with keycloak

Erik Osterman

it basically supports every saml provider

yeah it looks like

Erik Osterman

and we use it with gsuite

Erik Osterman

not only that, you an use it with https://github.com/mulesoft-labs/aws-keycloak

mulesoft-labs/aws-keycloak

aws-vault like tool for Keycloak authentication. Contribute to mulesoft-labs/aws-keycloak development by creating an account on GitHub.

Erik Osterman

with aws

Erik Osterman

it can become the central auth service for everything

nice

Erik Osterman

we use it with kubernetes, teleport, atlantis, grafana, etc

yeah super nice, fully integrated

Erik Osterman

you can even integrate it with multiple auth providers at the same time

do helmfiles have a remote “helm chart” that is used as the base?

Erik Osterman

not sure, better to check in #helmfile

ah yeah it does

cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Erik Osterman

oh, “base” is a loaded term now

Erik Osterman

since helmfile has the concept of bases

Erik Osterman

this is not a base in that sense

Erik Osterman

like uh

base docker image base

Erik Osterman

aha

Erik Osterman

yes, we use the community image

Erik Osterman

but guess it would be more secure to run our own

Erik Osterman

given the role this plays

i guess im not being clear enough

i was curious if the helm chart is just abstracted out

for a helmfile

Erik Osterman

for the gatekeeper, we’re doing something a bit unusual/clever

Erik Osterman

we are defining an environment

Erik Osterman

then using that to generate a release for each service in the environment

Erik Osterman

the alternative is to use sidecars or automatic sidecar injetion

Erik Osterman
stakater/ProxyInjector

A Kubernetes controller to inject an authentication proxy container to relevant pods - [✩Star] if you’re using it! - stakater/ProxyInjector

ah

Erik Osterman

Here’s what our environments file looks like

Erik Osterman
services:
  - name: dashboard
    portalName: "Kubernetes Dashboard - Staging"
    host: <http://dashboard.xx-xxxxx-2.staging.xxxxxx.io>
    useTLS: true
    skipUpstreamTlsVerify: true
    upstream: <https://kubernetes-dashboard.kube-system.svc.cluster.local>
    rules:
      - "uri=/*\|roles=kube-admin,dashboard\|require-any-role=true"
    debug: false
    replicas: 1
  - name: forecastle
    host: <http://portal.xx-xxxx-2.xxxx.xxxx.io>
    useTLS: true
    upstream: <http://forecastle.kube-system.svc.cluster.local>
    rules:
      - "uri=/*\|roles=kube-admin,user,portal\|require-any-role=true"
...

i see

2019-06-18

Hugo Lesta

Hello @davidvasandani thanks for the article that you’ve written. Could you please tell me the main capabilities that trafeik have as ingress-controller? Do you have any article with this capabilities?

davidvasandani

Hi @Hugo Lesta. Not my article but Traefik has many capabilities. https://docs.traefik.io/configuration/backends/kubernetes/

Hugo Lesta

This previous article you sent me seems worthy for me, I’ll try to improve my knowledhe about traefik over k8s.

1
davidvasandani

Its good and helped me out but its incomplete. The author mentions using LoadBalancer locally but doesn’t describe how. With a lot of additional work I’ve gotten it working with MetalLB locally. This was a very useful article: https://medium.com/@JockDaRock/kubernetes-metal-lb-for-docker-for-mac-windows-in-10-minutes-23e22f54d1c8

2019-06-17

2019-06-16

2019-06-13

what ingress controller are you guys using? it seems like alb-ingress-controller isnt quite robust enough for me. things that i feel like its missing:

  1. new ingress object = new ALB so there would be a one-to-one mapping of ALBs to services for me (multi-tenant cluster)
  2. provisioned resources don’t get cleaned up, at this point i feel like i might want to terraform the load balancer resources i need with the cluster
Erik Osterman

yea, the 1:1 mapping between ingress an ALB sucks!

Im using Ambassador.. a lot of features regarding routing of traffic based on any kind headers, regex matching, Jaeger tracing. Name it :)

@ does ambassador spin up cloud resources for u? (load balancers, security groups, etc)

i realized that might not be a feature i want as of now in k8s. since terraform is better at managing cloud resource state

davidvasandani

Can someone point me to best practices for setting up Traefik/Nginx-Proxy/etc as an ingress for Kubernetes running on 80? Everything is running but ClusterIP is internal and NodePort doesn’t allow ports below 30000. What am I missing?

kskewes

Service of type Loadbalancer. Then cloud provider gives you IP or use something like metallb on bare metal. Deployment nginx ingress or whatever. Can replicate per AZ.

davidvasandani

metallb was exactly what I needed. Thanks @kskewes

davidvasandani
Kubernetes & Traefik locally with a wildcard certificate

As a passionate software engineer at Localz, I get to tinker with fancy new tools (in my own time) and then annoy my coworkers by…

davidvasandani

but he’s using a LoadBalancer w/ Docker for Mac Kubernetes which doesn’t make sense.

2019-06-12

Hi all! Has someone faced this error before?

kernel:[22989972.720097] unregister_netdevice: waiting for eth0 to become free. Usage count = 1

Erik Osterman

Public #office-hours starting now! Join us on Zoom if you have any questions. https://zoom.us/j/684901853

2019-06-11

Sandeep Kumar

Hey Guys Does anyone configured SMTP as a grafana config-map for kubernetes?

Erik Osterman

Don’t have first hand experience

Erik Osterman

Let me know if you get it working though. We should setup the same in our helmfile.

Sandeep Kumar

Sure Erik

Sandeep Kumar

apiVersion: v1 kind: ConfigMap metadata: labels: app: grafana name: grafana-smtp-config-map namespace: monitoring data: grafana.ini: | enabled =true host=<host> user=<user> password=<password> skip_verify= false from_address=<email> from_name=Grafana welcome_email_on_sign_up=false

Sandeep Kumar

Ex: something like this

Sandeep Kumar

and adding this config map in kubernetes grafana deployment - configMap: defaultMode: 420 name: grafana-smtp-config-map name: grafana-smtp-config-map

Sandeep Kumar

i am trying using above methods to add smtp to grafana.ini

Sandeep Kumar

but i am unable to add smtp to grafana.ini, is there any documentation/suggestions which can help me here?

timduhenchanter

Does anyone have any experience scaling with custom metrics from Datadog across namespaces (or the external metrics API in general)?

timduhenchanter
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: service-template
spec:
  minReplicas: 1
  maxReplicas: 3
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: service-template
  metrics:
  - type: External
    external:
      metricName: k8s.kong.default_service_template_80.request.count
      metricSelector:
        matchLabels:
            app: kong
      targetAverageValue: 5
timduhenchanter
  Warning  FailedGetExternalMetric       117s (x40 over 11m)  horizontal-pod-autoscaler  unable to get external metric default/k8s.kong.default_service_template_80.request.count/&LabelSelector{MatchLabels<i class="em em-map[string]string{app"></i> service-template,},MatchExpressions<i class="em em-[],}"></i> no metrics returned from external metrics API
timduhenchanter

^ perm issue with Datadog API in the cluster-agent

Alex Co

hi, anyone here is using Gloo Gateway on K8s ?

Alex Co

i’m having a problem that the virtual service stopped accepting traffic after awhile, and status on the ELB to gloo gateway proxy show that it ’s OutOfService

Alex Co

wonder if anyone here got the same problem

2019-06-10

Nikola Velkovski

@rj I saw that as well but I didn’t know if it was any good.

1

2019-06-09

@Nikola Velkovski give a try with rancher. It is the most easiest way to spin up k8s on multiple clouds as per our experience with the tool. https://rancher.com/

Run Kubernetes Everywhere

Rancher, open source multi cluster management platform, makes it easy for operations teams to deploy, manage and secure Kubernetes everywhere. Request a demo!

2019-06-08

Nikola Velkovski

that sounds a lot like aws Elasticsearch @btai terraform apply usually times out when upgrading the ES cluster

Nikola Velkovski

thanks!

2019-06-07

Nikola Velkovski

Hi People , do you know of a best/sane way to install k8s on AWS. I see that there are multiple ways to do it. I am eyeing kops because terraform duh but before creating the cluster there’s still a lot of preparation to do like:

  • creating vpc
  • kops state bucket
  • route53 record And than all of it has to be passed on to kops as a cli command. This is all fine but to me it looks like a bit too much. Is there any other way of doing it ?
aaratn

I use EKS

aaratn

with terraform

Nikola Velkovski

ok that’s a way

Nikola Velkovski

and how do you handle upgrades, i read somewhere that it’s a bit tricky with EKS

aaratn

You mean master version upgrade ?

Nikola Velkovski

like k8s 1.2 -> 1.3 upgrade

aaratn

Yeah, well its a pretty new cluster

aaratn

right now my cluster is running on version 1.2

aaratn

what are the challenges that you have heard of ?

Nikola Velkovski

I don’t remember the details but I think Erik mentioned something about the upgrade in EKS is not as easy

Nikola Velkovski

I might be wrong though

aaratn
Making Cluster Updates Easy with Amazon EKS | Amazon Web Services

Kubernetes is rapidly evolving, with frequent feature releases, functionality updates, and bug fixes. Additionally, AWS periodically changes the way it configures Amazon Elastic Container Service for Kubernetes (Amazon EKS) to improve performance, support bug fixes, and enable new functionality. Previously, moving to a new Kubernetes version required you to re-create your cluster and migrate your […]

aaratn

aws blog says its easy

Nikola Velkovski

:)))

Nikola Velkovski

fair enough

aaratn

offcourse we have multiple environments

aaratn

so we can upgrade the lower environment and check if it works

aaratn

and proceed with upgrade

Nikola Velkovski

niice

Tim Malone

they made it much easier recently - you can do it via the AWS console, just change the version

Tim Malone

then upgrade your worker nodes afterwards

Tim Malone

(but yes you’ll want to do it in non-prod first just in case)

aaratn

Terraform has a parameter for version

Nikola Velkovski

oh that sounds promising

aaratn
version – (Optional) Desired Kubernetes master version. If you do not specify a value, the latest available version at resource creation is used and no upgrades will occur except those automatically triggered by EKS. The value must be configured and increased to upgrade the version when desired. Downgrades are not supported by EKS.
Nikola Velkovski

and what about installing it, EKS gives you the master nodes only, what about getting the other nodes in EC2, is it just a matter of using cloud-init ?

aaratn

We do with with auto-scaling-group

aaratn

You can follow this

Nikola Velkovski

Thanks a lot people

Nikola Velkovski

I will try it out

nutellinoit

i tried an upgrade with eks and terraform

nutellinoit

from 1.11 to 1.12

nutellinoit

is pretty smooth

nutellinoit

control plane upgrades without downtime

nutellinoit

to upgrade the workers the only thing to do is to update amis

nutellinoit

and replace workers

nutellinoit

and follow the directions on aws documentation to patch system deployment with new container versions

nutellinoit
Updating an Amazon EKS Cluster Kubernetes Version - Amazon EKS

When a new Kubernetes version is available in Amazon EKS, you can update your cluster to the latest version. New Kubernetes versions introduce significant changes, so we recommend that you test the behavior of your applications against a new Kubernetes version before performing the update on your production clusters. You can achieve this by building a continuous integration workflow to test your application behavior end-to-end before moving to a new Kubernetes version.

Nikola Velkovski

ok, so I guess there’s a posibility to automate replacing the instances in the AS somehow

Nikola Velkovski

I will look into it

Nikola Velkovski

Thanks for your support people!

Nikola Velkovski

nutellinoit

You can simply terminate one old instance at time and wait for autoscaling group to launch replacements

Nikola Velkovski

arhg ye gute ole click-ops

Nikola Velkovski

we’ve developed a lambda with step functions that does the instance replacement, step functions serving as a waiter

1
Nikola Velkovski

so it’s fire and forget

Nikola Velkovski

takes a while but it’s atomic

@Nikola Velkovski I’ve found the k8s upgrades to be a bit slow. it increases in time (by like 5~7 minutes per worker node) so upgrades can take a long time. For me, I wouldn’t be comfortable letting the upgrade for a production cluster run unattended (i.e. overnight while im sleeping) and naturally your production cluster probably has the most worker nodes. What I’ve found works for me pretty well is just using terraform to spin up a new cluster, deploy to the new cluster, and doing the cutover at the DNS level. food for thought

2
Erik Osterman

I think an elegant approach is to spin up an additional node pool

2
Erik Osterman

Then cordon and drain the nodes in the old one

2

2019-06-06

Alex Co

nvm, it’s because i did not declare .Values.app.secretName as global variable

nutellinoit

I encountered an issue with eks ebs volume provisioning, with small worker groups (less than 3) the pv was created before the pod and in the wrong AZ.

nutellinoit
10:20:35 AM

is settting volumeBindingMode: WaitForFirstConsumer enough on v1.12 to fix this problem?

Pablo Costa

Yes @nutellinoit It works. But I would also suggest to set an affinity policy for one AZ only, to ensure in case of pod restart or eviction, the pod be scheduled on the same AZ of the PVC

1

2019-06-05

what are you guy’s strategy for memory requests? for example looking at my historical data, my api pods use about 700Mi memory on average. I believe it’s better to set that memory request down to around that number, which will allow for more excess memory in the pool. I have it currently overallocated (1000Mi per api pod) and it adds up how much memory is being reserved but unusable by others that may need it.

Erik Osterman

Other considerations to take into account is (a) how much memory volatility there is… perhaps 30% variance is a bit high (b) disruptions - how bad is it if the service is evicted to another node?

Erik Osterman

I would suspect the more pods of a given service you run, the more insulated you are from disruptions of pod evictions

Erik Osterman

which means you can get by with a a 5-10% limit. make sure you monitor pod restarts.

Erik Osterman

so long as that number stays at or near 0, you’re good.

sarkis

how are you all connecting kubectl into the k8s cluster these days?

Erik Osterman

via teleport

Erik Osterman

teleport supports both ssh and kubectl

Erik Osterman

SAML authentication

Erik Osterman

what they call proxy is ~ a bastion, for a centralized entry point

Erik Osterman
Modern Privileged Access Management | Teleport | Gravitational

Make it easy for users to securely access infrastructure, while meeting the toughest compliance requirements.

sarkis

Interesting ty

thanks @Erik Osterman i think i can get away with closer to 5-10%. dont have that much memory volatility looking at my metrics

Alex Co

hi

Alex Co

Alex Co, [Jun 6, 2019 at 143 PM]: i’m having an issue while looping the helm template

env: {{- range .Values.app.configParams }} - name: {{ . | title }} valueFrom: secretKeyRef: name: “{{ .Values.app.secretName }}” key: {{ . | title }} {{- end }}

this is my code in the template to generate the environment var from the values.yaml

but when i ran the helm lint, it complaints like this

executing “uiza-api-v4/templates/deployment.yaml” at <.Values.app.secretName>: can’t evaluate field Values in type interface {}

i guess that helm template does not allow me to put the secretName value inside a loop

is there anyway to solve this ?

2019-06-04

Igor Rodionov

Intresting tool that checks K8s best practices https://github.com/reactiveops/polaris

reactiveops/polaris

Validation of best practices in your Kubernetes clusters - reactiveops/polaris

1

Thanks

reactiveops/polaris

Validation of best practices in your Kubernetes clusters - reactiveops/polaris

1
sarkis

It’s pretty nice .. for now it points out things like if you have set resource limits and it’s pretty basic, but I think this can be useful the more they add to it.

1

2019-06-03

2019-05-31

claranet/kcs

Select which kubeconfig.yaml to use in an easy way. KCS means kubeconfig switcher. - claranet/kcs

3

2019-05-29

Vidhi Virmani

Hi all,

Is there anyone who has setup kubernetes dashboard on EKS using istio ingress gateway? I am facing some issues where my dashboard crash after 4 mins. I am not sure if its a good idea to use istio ingress gateway to run kubernetes-dashboard. Any help is appreciated

Vidhi Virmani

It is fixed now. I had to provide few configs in istio

Erik Osterman

@Vidhi Virmani how are you securing it?

johncblandii

(comment just to monitor response)

Vidhi Virmani

@Erik Osterman I am currently allowing very few users to access the dashboard using aws-iam-authenticator.

2019-05-24

Kevin Gimbel

Hey all, I’ve a question and I can’t seem to find an answer. I’m running an AWS EKS cluster with two Nodes, each Node in EKS has a restriction of 20 Pods per Node. The Nodes are auto scaled and shut down each night and started in the morning since it’s just a test / staging system at the moment. However, one Node is always full (20/20 Capacity) while the other runs 4/20. We want to run a DaemonSet with filebeat for log aggregation but cannot ensure it runs on both nodes because one is full.

Is there a way I can (easily) ensure the DaemonSet is scheduled before all other pods? Or can I reserve a spot / space on a Node for a specific Pod, Deployment, or DaemonSet?

Kevin Gimbel

I would like to avoid configuration overhead. I’ve already read about Affinity and Anti-Affinity but I’m not sure if this can help me

Kevin Gimbel

Someone in the Kubernetes Slack answered my question, looks like this is it: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/

Erik Osterman

Yes, this is what you want to look into.

2019-05-22

Pablo Costa
2
aknysh

finally

2

ergg.. i spent a good part of a day understanding how it works/getting it to work w/my eks cluster spun up in tf

2
Erik Osterman

Public/Free Office Hours with Cloud Posse starting now!!

https://zoom.us/j/684901853

anyone try federation yet?

2019-05-20

aaratn

Can anyone help me with aws alb loadbalancer with helm chart ? Any samples that I can refer ?

sarkis
flant/werf

Werf (previously known as dapp) helps to implement and support Continuous Integration and Continuous Delivery - flant/werf

2019-05-19

James D. Bohrman

Hey all! I’m having an issue building my example-voting-app with Codefresh.

I added the variable for KUBE_CONTEXT but I keep getting an error that throws:


error: no context exists with the name: "gke_example-voting-app-240610_us-east1-c_example-votin
g-app".                                                                                        
[SYSTEM] Error: Failed to run freestyle step: Running Helm Upgrade; caused by NonZeroExitCodeEr
ror: Container for step title: Running Helm Upgrade, step type: freestyle, operation: Freestyle
Erik Osterman

KUBE_CONTEXT should be the name of a kubernetes integration in codefresh

Erik Osterman

it would seldom, if ever have the app name in it

Erik Osterman
Add Kubernetes Cluster

How to connect your Kubernetes cluster to the Codefresh dashboard

James D. Bohrman

Got it thanks!

James D. Bohrman

I ran kubectl get context in my GKE shell and got:

gke_example-voting-app-240610_us-east1-c_example-voting-app
James D. Bohrman

I put that as my KUBE_CONTEXT variable and can’t figure what I’m doing wrong. The docs say to put KUBE_CONTEXT as “Your friendly Kubernetes Cluster Name” I’ve also tried “example-voting-app” as the context variable. Which is the EKS cluster name. No dice there either.

2019-05-17

reactiveops/polaris

Validation of best practices in your Kubernetes clusters - reactiveops/polaris

Sandeep Kumar

How do we generate a wildcard certificate using kubernetes kind:managedCertificate, trying with below method but not successful apiVersion: http://networking.gke.io/v1beta1 kind: ManagedCertificate metadata: name: example-certificate spec: domains: - *.http://example.net

Please let me know if there is any documentation/suggestions to create a wild card certificate with expiry date mentioned in it

sarkis

^ Polaris looks really interesting… I’m going to try to get it going this weekend see if it’s useful… any thoughts on it yet if someones already set it up?

sarkis

Couldn’t wait for the weekend testing it now… it offers some nice checks… I can see this becoming more and more useful as more checks/best practices are added …

2019-05-16

nutellinoit

To anyone that is tempted to use t3a or m5a instances on an EKS cluster, don’t

nutellinoit
Support for t3a, m5ad and r5ad instance types is missing · Issue #262 · awslabs/amazon-eks-ami

What would you like to be added: Support for t3a, m5ad and r5ad instance types. Why is this needed: AWS had added new instance types, and the AMI does not currently support them.

nutellinoit

there is an incompatibility on calculating number of eni available

2
endofcake
Introducing Polaris: Keeping your Kubernetes Clusters Healthy - Reactive Ops

We started ReactiveOps with a simple vision: transform infrastructure operations by leveraging decades of large-scale operations and product experience.

timduhenchanter

Scale on queue depth

https://github.com/kedacore/keda

kedacore/keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes - kedacore/keda

1

2019-05-15

davidvasandani
containership/konstellate

Free and Open Source GUI to Visualize Kubernetes Applications. - containership/konstellate

oscarsullivan_old

Thanks I like this. For #terraform there’s also https://github.com/camptocamp/terraboard

camptocamp/terraboard

A web dashboard to inspect Terraform States - camptocamp/terraboard

davidvasandani

Thanks for sharing @oscarsullivan_old! This looks really neat. You should share it in the #terraform channel.

camptocamp/terraboard

A web dashboard to inspect Terraform States - camptocamp/terraboard

Vidhi Virmani

I am trying to setup kubernetes dashboard on AWS EKS cluster. I am able to setup the dashboard but facing a small issue with certs. I want to use aws certificate arn with the dashoard as an argument with command

kubectl apply -f <https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml>

is this possible?

2019-05-11

Exequiel Barrirero
Exequiel Barrirero

Interesting approach for -> Deploying API Gateway in front of EKS / K8s Kops Clusters inside VPC private subnets And many other useful info about Integrating EKS with other AWS Services

2019-05-10

2019-05-09

Dang, how do we get Curtis Mattoon into cloud posse slack? https://github.com/cmattoon/aws-ssm/pull/29

Added log-level functionality by rms1000watt · Pull Request #29 · cmattoon/aws-ssm

I didn’t see any other way to set the log level. So here it is!

This tool works pretty good. But just curious if you peeps have any other methods for dynamically added k8s secrets from SSM

Erik Osterman

not from SSM

Erik Osterman

have you seen @mumoshu’s ASM operator?

nope, I shall take a look-see

Erik Osterman

i think extending that to support SSM would be nice

Erik Osterman

or creating a separte one

https://github.com/mumoshu/aws-secret-operator (for the others in the channel)

mumoshu/aws-secret-operator

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager. - mumoshu/aws-secret-operator

1
Why not use AWS SSM Parameter Store as a primary source of secrets?

Pros:

Parameter Store has an efficient API to batch get multiple secrets sharing a same prefix.

Cons:

Its API rate limit is way too low. This has been discussed in several places in the Internet:

However, they just updated the rate limit to 1k req/s

so it might be a non-issue now

Also, you can set the limit and incur costs. Haven’t actually clicked this before.. lets see what happens

Ohhh, this is how to you get 1k: https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-throughput.html

You can increase the limit to 1,000 TPS on the Settings tab. Increasing the throughput limit incurs a charge on your AWS account.

$0.05 per 10,000 Parameter Store API interactions k.. I’ll stop spamming

Erik Osterman

that’s great; didn’t know they increased the limit

mumoshu

I thought secretsmanager had the same amount of charge

secretmanager i think is $1/mo/secret. Lemme google a littttle

whoops.. $0.40/mo

PER SECRET PER MONTH
$0.40 per secret per month. For secrets that are stored for less than a month, the price is prorated (based on the number of hours.)

PER 10,000 API CALLS
$0.05 per 10,000 API calls.
loren

Also good to cache your secrets, to avoid extra API calls and rate limits… https://aws.amazon.com/about-aws/whats-new/2019/05/Secrets-Manager-Client-Side-Caching-Libraries-in-Python-NET-Go/

this is interesting

curious how it works in detail. Like, does it make your microservice stateful? Or does it put the cache local to your cluster? Or is aws handling all the caching for us automagically?

The go SDK code looks straight forward though. Awesome find!

https://github.com/cmattoon/aws-ssm/pull/30 fixing a bug in aws-ssm if anyone else was considering to use it

Added next token to getparameterbypath for secrets > 10 by rms1000watt · Pull Request #30 · cmattoon/aws-ssm

The Go SDK for GetParameterByPath limits to 10 values in the response. This should grab them all.

Erik Osterman

how does it look when you want many parameters?

kind: Secret
metadata:
  name: my-secret
  annotations:
    aws-ssm/k8s-secret-name: my-secret
    aws-ssm/aws-param-name: my-db-password
    aws-ssm/aws-param-type: SecureString

e.g. /db/*

Added next token to getparameterbypath for secrets > 10 by rms1000watt · Pull Request #30 · cmattoon/aws-ssm

The Go SDK for GetParameterByPath limits to 10 values in the response. This should grab them all.

Erik Osterman


The name of the AWS SSM Parameter. May be a path.

Erik Osterman

i guess that answers it

Erik Osterman

but still curious. i never really kicked the tires on aws-ssm

Erik Osterman

(ultimately, client wanted per-service access controls so we went with Chamber +S3 + IAM + KIAM)

@Erik Osterman

apiVersion: v1
kind: Secret
metadata:
  name: my-secret-name
  annotations:
    aws-ssm/k8s-secret-name: my-secret-name
    aws-ssm/aws-param-name: {{ .Values.ssm_path }}
    aws-ssm/aws-param-type: Directory
data: {}

Where ` .Values.ssm_path == /directory/within/ssm`

Erik Osterman

Ah, thx!

(lol, sorry about the delay)

Erik Osterman

how’s the helmfile PR coming along?

stale at the moment. been a bit busy. basically I didn’t consider multiple files

and there’s some chicken/egg issue about when the template-rendering happens and when to reference a file

so I just need to hit my head a little harder on it

Erik Osterman

maybe that will be simpler if they decouple the multi-phase rendering

Possibly. I thought multi-phase rendering was needed for template in template situations

2019-05-08

2019-05-07

johncblandii

Is there a clean way to get the security group created for an LB so I can assign it to the workers SG to approve traffic?

johncblandii

The LB is created through the helm deploy.

aknysh
AWS: aws_security_group - Terraform by HashiCorp

Provides details about a specific Security Group

Erik Osterman
stakater/Whitelister

A tool to white list node and developer IPs for kubernetes. - stakater/Whitelister

Erik Osterman

I would pursue a k8s native solution rather than trying to fuse terraform with helm

Erik Osterman

Also, IP whitelisting should be used as a last resort. Identity Aware Proxies is ala keycloak is a better approach

alb ingress controller creates you an ALB and the necessary security groups and assigns them to access your workers

johncblandii

@aknysh there isn’t enough on the SG to query that way. it has the <http://k8s.io/>… tag, but it is not specific.

johncblandii

@Erik Osterman this isn’t fusing helm and tf. it is the SG created by TF, but I’m mainly just adding an SG record so it is mainly AWS infrastructure networking.

aknysh

if you go that route, you can filter by name (the resource has some name) and not tags

aknysh

or add your own specific tag

Erik Osterman

I think I lack context of where you are trying to do this?

Erik Osterman


“Is there a clean way from XXXXXXX to get the security group created for an LB by ZZZZZZ so I can assign it to the workers SG in YYYYYYY to approve traffic?”

johncblandii

you may be technically right w/ fusing them. i’m technically wanting a value from k8s so i can configure the AWS SG to allow communication.

The SG is handled within TF manually

@johncblandii what type of LB are you using, if youre using an ALB I would suggest alb ingress controller as it does all that for you. (the downside is when you tear down your cluster, it wont clean up for you)

johncblandii

it automatically used a classic elb (helm install)

what helm chart @johncblandii

johncblandii

Twistlock

johncblandii

welp…bitten by the “providers cannot be dynamically initialized” issue

2019-05-06

Has anyone made their GKE nodes static with Terraform?

johncblandii

I think I’m missing a step with this new eks cluster. applying the configmap keeps giving me an unauthorized.

The aws-iam-authenticator call is working as expected so I have access, but applying the file does not work

johncblandii

error: You must be logged in to the server (the server has asked for the client to provide credentials)

aknysh

has your session expired?

johncblandii

doesn’t aws-iam-auth create a new one every time?

johncblandii

i used the CP module to do this before and it is working fine still on that end

aknysh

you using geodesic? can you exit the shell, run it again, and assume role?

johncblandii

nopers. direct

johncblandii

trying in a fresh terminal

johncblandii

error: You must be logged in to the server (the server has asked for the client to provide credentials)

johncblandii

aws-iam-authenticator token -i... works just fine

johncblandii

i’ve confirmed the configmap matches the first one i did

johncblandii

(sans names, of course)

johncblandii

maybe i need to update my core ~/.kube/config?

johncblandii

no go there

johncblandii

any debug tips?

johncblandii

ahhh….might be this:
This could be because the cluster was created with one set of AWS credentials (from an IAM user or role), and kubectl is using a different set of credentials.

I created it via CI

aknysh

this was the issue?

johncblandii

I think so

johncblandii
Amazon EKS Troubleshooting - Amazon EKS

This chapter covers some common errors that you may see while using Amazon EKS and how to work around them.

johncblandii

and the kubectl apply on CI failed because kc isn’t available on there (yet)

johncblandii

FYI @wbrown43 ^

johncblandii

confirmed. used the CI users creds and it worked as expected

@johncblandii you’ll need to update your authenticator configmap to allow other roles/users

johncblandii

just got through that part, @btai.

johncblandii

my last eks was a local install i did so i did not realize this was a rule

@johncblandii yeah took me half a day just trying to figure out how to get aws-iam-authenticator working and I ran into the same issues as you did haha

but no problems since

johncblandii

to which I spent the same half-day (while in 5 hours of meetings back-to-back-to-back)

johncblandii

johncblandii

now my nodes aren’t connecting so onto issue #3; lol

did you add the role for the worker nodes to your config map as well?

johncblandii

yup. the cp module does it

johncblandii

going to nix one and let the scaling kick off a fresh one now that the map is applied

johncblandii

so no public ip seems to have been the issue

johncblandii

started one w/ a public ip and voila

@johncblandii are you talking about public ip for your worker nodes?

johncblandii

yup

johncblandii

our other cluster didn’t have public, but when I upgraded them to 1.12 i had to do the same

(unsure if that’s related, but i did notice that)

@johncblandii fwiw, i didnt have to make my worker nodes public. im still on 1.11 but i cant imagine that would change in 1.12

johncblandii

i hear you. that’s just what i noticed when i moved to .12

Jeremy Grodberg

I have a cluster in AWS in 3 availability zones, with 3 masters, but only 2 nodes. kops put both nodes in the same AZ? Is this a bug? How do I get kops to spread the nodes evenly across AZs?

Erik Osterman

it’s not a kops thing

Erik Osterman

compare how the master node pools are created to how the worker node pools are created

Erik Osterman

that’s how to ensure more even distribution

Erik Osterman

AWS will make “best effort” to allocate instances evenly, but no guarantee

Erik Osterman

the only way to have a “guarantee” is to create node pools tied to exactly one AZ

for this purpose, we create 3 ASG with only one master inside

Erik Osterman

precisely…

Jeremy Grodberg

Yes, kops creates an instance group per zone for the masters, but just 1 instance group for all the nodes.

Jeremy Grodberg

So it turns out the bigger issue is that AWS autoscale group does launching and zone balancing separately, and to do zone balancing it has to launch a new instance before deleting the old one. Well, we had run up against our instance/type limit for the region, so it could not do zone balancing.

Erik Osterman

oh fascinating

Erik Osterman

good sluething

wbrown43
06:50:41 PM

@wbrown43 has joined the channel

2019-05-02

nutellinoit

@ we use Rancher for users permission management in front of EKS

1

2019-04-30

2019-04-29

https://github.com/roboll/helmfile/issues/392#issuecomment-455065039 @Erik Osterman I feel like I need to write some middleware for Helmfile so it can use SSM directly

Feat: Allow simple Vault integration · Issue #392 · roboll/helmfile

Currently there are many users that want to integrate Vault with Kubernetes, but there are no high level tools for this. The current Kubernetes AuthMethod for Vault is too complex and coupled to th…

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

lol think just right here

Has anyone found a good solution for user management on EKS? The best solution I can find is setting up roles that users can assume, but it doesn’t seem like an optimal solution.

Installation de aws-iam-authenticator - Amazon EKS

Amazon EKS utilise IAM pour fournir l’authentification pour votre cluster Kubernetes via l’ authentificateur AWS IAM pour Kubernetes . À partir de la version Kubernetes 1.10, vous pouvez configurer le client kubectl normal afin qu’il utilise Amazon EKS en installant l’authentificateur AWS IAM pour Kubernetes et en modifiant votre fichier de configuration

really usefull to manage by IAM

Erik Osterman

Gravitational Teleport

Erik Osterman

But you still map k8s roles to saml roles

Erik Osterman

have you looked at Gravitational Teleport?

Erik Osterman

it supports integration with SSO

Erik Osterman

and audited session logs with replay

I haven’t, but I will take a look!

Thanks @Erik Osterman

https://github.com/roboll/helmfile/pull/569 SSM integration with Helmfile. Lets see the lash-back. lol

Added SSM integration by rms1000watt · Pull Request #569 · roboll/helmfile

There&#39;s been some interest for helmfile integration with SSM. Here is an example of what it can look like. For our current workflows, we have Bash scripts that export Env Vars via aws-env then …

2019-04-26

does someone have something to say about https://www.ovh.co.uk/kubernetes/ ?

Managed Kubernetes®: orchestration of containers in the cloud - OVH

Benefit from a free, managed and highly available Kubernetes® service to orchestrate your containerised applications in the OVH cloud Free hosted master nodes

Erik Osterman

Haven’t used that in particular… I used OVH back in the day with CoreOS. Loved the service for the value. Unlimited bandwidth and beefy bare metal instances.

1

2019-04-25

what are y’all thoughts on 3 clusters/3az vs 1 cluster/3az? does the 3 cluster approach give us much more reliable availability?

for the 3 cluster approach either using federation or dns round robin

1 cluster, 3 az’s, multi-master setup

if you want more clusters, set those up in different regions instead with dns failover and/or geo/round robin load balancing between them

1

Anyone run tillerless helm? https://rimusz.net/tillerless-helm/

Tillerless Helm v2

Helm really became a de-facto as Kubernetes Package Manager. Helm is the best way to find, share, and use software built for Kubernetes as it states on https://helm.sh. That’s true and sounds very cool. Since Helm v2, helm got a server part called The Tiller Server which is

Erik Osterman

that’s interesting!

2019-04-24

Erik Osterman

@ I often use this example when talking to customers that out of the box, kubernetes supports a very basic form of canary+rolling updates. The reason for istio is to have full control over that process. Also, the “gold standard” for canary deployments is to tie it into your monitoring backplane so that you proceed to increase the level of traffic only so long as some KPIs are true. This level of controlled rollouts are more difficult to orchestrate with kubernetes primitives, which is why Istio is used. Also, it doesn’t have to be limited to blue/green. It can be a full rainbow of colors, where the traffic is spread across them.

1

Makes a ton of sense

I think we’re all saying the same thing.. or at least on the same train of thought. It’s definitely a nice thing to have as business requirements expand (as they always do). Soon enough, it’ll become a required thing to have.

1

2019-04-23

@Erik Osterman (or anyone) I got a fundamental question for ya..

What’s the difference between:

  • a canary deployment with 5% increments
  • a rolling update with maxUnavailable==0 && maxSurge==5% (with a RR Load Balancer in front)

(no rush.. food for thought)

What I’m thinking about is.. can a native rolling update be used in place of a canary deployment.. Assuming they can monitor the same metrics for health

Because.. spinnaker / istio just for the sake of canary might not be reason enough

midnight thoughts

2019-04-19

deftunix

hi all, anyone has experience with eks to assign a pool of static ip address/eni to pods based on the AWS high availability zone?

Erik Osterman

haven’t seen that done before; don’t know if it’s possible

2019-04-15

Erik Osterman

hahaha FTW!

Erik Osterman

2019-04-14

I use it, very useful : https://github.com/derailed/k9s

derailed/k9s

Kubernetes CLI To Manage Your Clusters In Style! - derailed/k9s

2

@Erik Osterman You were able to convert @stobiewankenobi lolololol

Terraform -> SSM Then aws-ssm + helm + helmfile at deploy time

SSM is beastmode.. love using http://serverless.com pulling from there also

2019-04-13

Erik Osterman
kubernetes-sigs/krew

Package manager for “kubectl plugins”. Contribute to kubernetes-sigs/krew development by creating an account on GitHub.

Erik Osterman
ernoaapa/kubectl-warp

Kubernetes CLI plugin for syncing and executing local files in Pod on Kubernetes - ernoaapa/kubectl-warp

Erik Osterman
replicatedhq/ship

A better way to deploy Kubernetes Helm charts. Contribute to replicatedhq/ship development by creating an account on GitHub.

Erik Osterman
replicatedhq/k8s-secret-generator

Contribute to replicatedhq/k8s-secret-generator development by creating an account on GitHub.

Erik Osterman

awesome idea! anytime you need a shared secret, generate it

2019-04-11

@btai you can run helm with debug/dryrun enabled - this should show you how the values are being generated which may help you work out what/why a variable value is not as expected

2019-04-08

Humberto Oliveira
CNCF Formally Adopts CRI-O Runtime for Kubernetes - Container Journal

The Cloud Native Computing Foundation has formally accepted a container runtime designed specifically for Kubernetes as an incubation project.

how do you avoid merging maps in helm?

Erik Osterman

have a more concrete example? @btai

i think it might be the way the chart is written

helm/charts

Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

that DRONE_DATABASE_DATASOURCE can be set as a postgres url i.e. <postgres<i class="em em-//username"></i>[email protected]/dbname>

but I set that as an envSecret value to pull from a k8s secret: https://github.com/helm/charts/blob/master/stable/drone/values.yaml#L159

helm/charts

Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

but i guess when the values get merged for the deployment, the secret env vars get written first then the default env vars: https://github.com/helm/charts/blob/master/stable/drone/templates/deployment-server.yaml#L74

helm/charts

Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

so the DRONE_DATABASE_DATASOURCE value gets overriden by the default sqlite value

possibly just a poorly written helm chart?

or is there a way of ignoring those default values

let me know if that makes any sense..

2019-04-07

Tim Malone

Re kops release schedule having slowed, anything to do with EKS perhaps? i.e. is pickup of kops slowing too?

Erik Osterman

I wonder… could be

Erik Osterman

Also the number of options available means there’s not as much support for anyone offering

2019-04-05

Alex Siegman

I know kops is somewhat intentionally behind Kubernetes in releases, but it looks like they are only “stable” on 1.11, which technically went EOL when 1.14 went GA if I’m not mistaken. That seems like a really slow release cadence to me. Is it still the go-to for doing home-spun K8S in AWS? EKS isn’t keeping up either.

Erik Osterman

Yea I am not sure why it’s slowed

Erik Osterman

kube-aws is also worth checking out. @mumoshu is a maintainer.

Alex Siegman

That’s the one you mentioned two weeks ago that I didn’t write down. Will take a look. Thanks!

casey

has anyone had any luck installing kiam on eks?

yeah I am using kiam on eks

casey

how did you install it? I tried with with the helmfile in cloudposses repo, but no luck.

casey

Im pretty sure it’s because I couldn’t run the kiam server on a master node since eks doesnt let you do that

I did a separate node-group for kiam-server that has the required credentials

i used the stable helm chart to install it

casey

what do you mean node-group ?

add --kubelet-extra-args --node-labels=${name_of_node_group} to your userdata

that lets you use a nodeSelector when deploying your pods

casey

ah okay so do you have one node specifically for kiam? or do you run other pods on it as well?

Erik Osterman

In our case, we run kiam servers on masters

i have a few daemonsets like node exporter, but not any other applications

Erik Osterman

And agents on all other nodes

yeah you can’t run stuff on eks masters that I am aware of

1
Erik Osterman

Oh right

Erik Osterman

Yea we use kops predominantly

The kiam-server node will have the ability to assume any role, so its best not to run anything else on it

2

2019-04-04

casey

hi all I have a quick question regarding dns zones which I am unsure of, if anyone could help it would be much appreciated.

casey

I want my domain name to be http://example.com. I have a hosted zone in aws route53 for http://example.com, which sits in my root account (I can not move it from the root account at this time, because its being used). I have another aws account called production which contains the hosted zone http://production.example.com, this account is also where my eks cluster is in. In my root account route53 zone http://example.com i have an NS record http://production.example.com so that the production account can handle those domains.

casey

If I use external-dns in my eks cluster, and allow it to create records in the http://production.example.com hosted zone, will my ssl cert hold? The SSL cert I have is a wildcard for *.http://example.com, and the records that get created from external-dns will look like *.http://production.example.com

casey

I believe that they wont, but I am not sure. Is there any common way to handle this kind of situtation?

aknysh

We request SSL certificates In each account separately

aknysh

The root certificate will not work in different accounts

aknysh

So http://prod.example.com will have its own certificate

aknysh

With wildcard

aknysh

and even if you created all environments in one account, a cert for *.<http://example.com> could be used for <http://prod.example.com>, but will not apply to *.<http://prod.example.com>

oscarsullivan_old

How do you get ..http://example.com carts.. when I Google it they cost like $1200 a hear.

oscarsullivan_old

Is this by importing your domain http://example.com into ACM and issuing within

aknysh

You mean certs? :) They are free on AWS because they can be used only with other AWS resources for which you pay, e.g. load balancers . Not with servers external to AWS

oscarsullivan_old

Yeh!

oscarsullivan_old

I did take a look a month ago

oscarsullivan_old

But that was only 3 weeks into using AWS so I was occupied with transferring all my other provider knowledge to aws

oscarsullivan_old

Eill give another shot http://and.post in AWS channel

aknysh

and they are automatically renewed on AWS (which will save you a lot of headache because we always forget to renew them )

1

2019-04-03

Erik Osterman
Kubernetes basic glossary

Must-know terminology to understand Kubernetes concepts

2019-04-01

would it be bad practice to deploy my CI tool in the same k8s cluster as what it is deploying?

Erik Osterman

Depends on what you want to accomplish with the CI tool

Erik Osterman

For example if you want the CI tool to upgrade the cluster it operates in, that won’t work

Erik Osterman

However no reason to limit yourself to one CI service

no

k8s cluster would hold the app and the CI tool that deploys the app

Tim Malone

probably ok but maybe in a different namespace?

@Tim Malone that would be the plan yeah

2019-03-30

jtbray

@Erik Osterman I was watching one of Kelsey HT’s keynote speeches, when he whips out his phone & starts using Google Assistant to manage some network services (via Istio)!! I’d love to see a well formed write-up discussing similar implementations &/or pitfalls/concerns.

Erik Osterman

Kelsey is a master of live demos

jtbray

(has me very excited to control everything by voice)!!

2019-03-29

casey

anyone have suggestions on a good rate-limiting gateway for kubernetes? I have been looking at kong, ambassador, and express-gateway, not sure which one to go with or if there is a better choice

Erik Osterman

@casey istio provides rate limiting (envoy)

Erik Osterman
Enabling Rate Limits

This task shows you how to use Istio to dynamically limit the traffic to a service.

Erik Osterman

plus with istio you get so much more than just that

2019-03-27

Erik Osterman
Boosting your kubectl productivity

If you work with Kubernetes, then kubectl is probably one of your most-used tools. This article contains a series of tips and tricks to make your usage of kubectl more efficient.

1

hello guys

somebody already have a problem with metrics using eks ?

the problem is in the hpa addon. from kubernetes-incubator

AWS EKS: cluster doesn't provide client-ca-file · Issue #119 · DirectXMan12/k8s-prometheus-adapter

Hi, I am trying to deploy on AWS EKS which now supports Kubernetes HPA AWS Blog. I successfulyy deployed Prometheus via helm but now when i try to deploy the adapter i get the following error `kube…

2019-03-26

deftunix

Hi all, is anyone of you using https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/2.3.0 to provision EKS cluster? I am experiencing an issue in the service type LoadBalancer. the cluster is not creating the balancer

aknysh

Let’s move to #terraform-aws-modules . Ping @antonbabenko , he is the creator of the module

Erik Osterman
‘AWS vs K8s’ is the new ‘Windows vs Linux’

Then…

If, like me, you’re over 40 and work in IT, you’ll probably remember a time when everyone used Windows, and a small but growing proportion of people were wasting their lives compiling Linux in their spare time.

The Windows users would look on, baffled: ‘Why would you do that, when Windows has everything you need, is supported, and is so easy to use?!’

Answers to this question varied. Some liked to tinker, some wanted an OS to be ‘free’, some wanted more control over their software, some wanted a faster system, but all had some niche reason to justify the effort.

Now…

As I stayed up for another late night trying to get some new Kubernetes add-on to work as documented, it struck me that I’m in a similar place to those days. Until a couple of years ago, Kubernetes itself was a messy horror-show for the uninitiated, with regularly-changing APIs, poor documentation if you tried to build yourself, and all the characteristics you might expect of an immature large-scale software project.

That said, Kubernetes’ governance was and is far and away ahead of most open source software projects, but the feeling then was similar to compiling Linux at the turn of the century, or dealing with your laptop crashing 50% of the time you unplugged a USB cable (yes, kids, this used to happen).

It’s not like confusion and rate of change has come down to a low level. Even those motivated to keep up struggle with the rate of change in the ecosystem, and new well-funded technologies pop up every few months that are hard to explain to others.

Take knative for example:

The first rule of the knative club is you cannot explain what knative is— Ivan Pedrazas (@ipedrazas) March 18, 2019 So my AWS-using comrades see me breaking sweat on the regular and ask ‘why would you do that, when AWS has everything you need, is supported and used by everyone, and is so easy to use!?’

AWS is Windows

Like Windows, AWS is a product. It’s not flexible, its behaviour is reliable. The APIs are well defined, the KPIs are good enough to be useful for most ‘real’ workloads. There are limits on all sorts of resources that help define what you can and can’t achieve.

Most people want this, like most people want a car that runs and doesn’t need to be fixed often. Some people like to maintain cars. Some companies retain mechanics to maintain a fleet of cars, because it’s cheaper at scale. In the same way, some orgs get to the point where they could see benefits from building their own data centres again. Think Facebook, or for a full switcher, Dropbox. (We’ll get back to this).

Like Microsoft, (and now Google) AWS embraces and extends, throwing more and more products out there as soon as they become perceived as profitable.

AWS and Kubernetes

Which brings us to AWS’s relationship with Kubernetes. It’s no secret that AWS doesn’t see the point of it. They already have ECS, which is an ugly hulking brute of a product that makes perfect sense if you are heavily bought into AWS in the first place.

But there’s EKS, I hear you say. Yes, there is. I haven’t looked at it lately, but it took a long time to come, and when it did come it was not exactly feature rich. It felt like one cloud framework (AWS) had mated with another (K8s) and a difficult adolescent dropped out. Complaints continue of deployment ‘taking too long’, for example.

Finally taking AWS’s EKS for a spin. While I’m bias for sure, this is not what I expect from a managed Kubernetes offering. It’s been 10 minutes and I’m still waiting for the control plane to come up before I can create nodes through a separate workflow. Kelsey Hightower (@kelseyhightower) January 30, 2019 Like Microsoft and Linux, AWS ignored Kubernetes for as long as it could, and like Microsoft, AWS has been forced to ’embrace and extend’ its rival to protect its market share. I’ve been in meetings with AWS folk who express mystification at why we’d want to use EKS when ECS is available.

EKS and Lock-in

Which brings us to one of the big reasons AWS was able to deliver EKS, thereby ’embracing’ Kubernetes: IAM.

EKS (like all AWS services) is heavily integrated with AWS IAM. As most people know, IAM is the true source of AWS lock-in (and Lambda is the lock-in technology par excellence. You can’t move a server if there are none you can see).

Shifting your identity management is pretty much the last thing any organisation wants to do. Asking your CTO to argue for a fundamental change to a core security system with less than zero benefit to the business in the near term and lots of risk is not a career-enhancing move.

On the other hand, similar arguments were put forward for why Linux would never threaten Windows, and while that’s true on the desktop, the advent of the phone and the Mac has reduced Windows to a secondary player in the consumer computing market. Just look at their failure to force their browsers onto people in the last 10 years.

So it only takes a few unexpected turns in the market for something else to gain momentum and knife the king of the hill. Microsoft know this, and AWS know this. It’s why Microsoft and AWS kept adding new products and features to their offering, and it’s why EKS had to come.

Microsoft eventually turned their oil tanker towards the cloud, going big on open source, and Linux and Docker, and all the things that would drag IT to their services. Oh, and you can use the same AD as your corporate network, and shift your Microsoft Windows licenses to the cloud. And the first one’s free. Microsoft don’t care about the OS anymore. Nobody does, not even RedHat, a business built around supporting a rival OS to Windows. The OS is dead, a commodity providing less and less surplus value.

Will Kubernetes force AWS to move their oil tanker towards Kubernetes? Can we expect to see them embrace Istio and Knative and whichever frameworks come after fully into their offering? (I don’t count how–to guides in their blogs).

AWS’ Competition and Cost

I don’t know. But here’s some more reasons why it might.

Like Microsoft in the heyday of Windows OS, AWS has only one competitor: the private data centre. And like Microsoft’s competitor then (Linux), adoption of that competitor is painful, expensive and risky to adopt.

But what is the OS of that data centre? Before Kubernetes the answer would have been OpenStack. OpenStack is widely regarded as a failure, but in my experience it’s alive (if not kicking) in larger organisations. I’m not an OpenStack expert, but as far as I can tell, it couldn’t cover all the ground required to become a stable product across all the infra it needed to run on and be a commodity product. Again, this is something Microsoft ruled at back in the day: you could run it on ‘any’ PC and ‘any’ hardware and it would ‘just work’. Apple fought this by limiting and controlling the hardware (and making a tidy profit in the process). Linux had such community support that it eventually covered the ground it needed to to be useful enough for its use case.

OpenStack hasn’t got there, and tried to do too much, but it’s embedded enough that it has become the default base of a Kubernetes installation for those organisations that don’t want to tie into a cloud provider.

Interestingly, the reasons AWS put forward for why private clouds fail will be just as true for themselves: enterprises can’t manage elastic demand properly, whether it’s in their own data centre or when they’re paying someone else. Command and control financial governance structures just aren’t changing overnight to suit an agile p…

2

2019-03-25

can we have an internet facing ALB that routes traffic to k8s worker nodes in private subnets?

Erik Osterman

Yes

Erik Osterman

That’s what we are doing

Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

1

man i am having lots of trouble getting alb ingress controller to work on my cluster

i know you guys are using kops though (not eks)

Cluster VPC Considerations - Amazon EKS

When you create an Amazon EKS cluster, you specify the Amazon VPC subnets for your cluster to use. Amazon EKS requires subnets in at least two Availability Zones. We recommend a network architecture that uses private subnets for your worker nodes and public subnets for Kubernetes to create internet-facing load balancers within. When you create your cluster, specify all of the subnets that will host resources for your cluster (such as worker nodes and load balancers).

Note

Internet-facing load balancers require a public subnet in your cluster. Worker nodes also require outbound internet access to the Amazon EKS APIs for cluster introspection and node registration at launch time. 
Tim Malone

how did you set up your VPC? with the cloudformation template referenced in the EKS getting started docs?

Tim Malone

(although i think that only creates public subnets… so might need some additional set up for private subnets for workers, but ALB ingress should still work once you’ve done that…)

with terraform

Tim Malone

and it’s got the relevant config mentioned in the eks getting started docs? (subnet tags and such)

yeah it does

i wrote a blurb in #sig-network I can copy/paste here

I’m trying to debug why I cannot access my service on my cluster. I went through these debugging steps and I am stuck here where I cannot curl my service by ip: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#does-the-service-work-by-ip

The service is a NodePort service:

NAME     TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
galaxy   NodePort   172.20.38.130   <none>        80:32697/TCP   3d

And when I try to curl the service IP from the k8s worker node, I get this:

node$ curl -v 172.20.38.130:80
* Rebuilt URL to: 172.20.38.130:80/
*   Trying 172.20.38.130...
* TCP_NODELAY set
* connect to 172.20.38.130 port 80 failed: Connection refused
* Failed to connect to 172.20.38.130 port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 172.20.38.130 port 80: Connection refused

nslookup from the pod:

pod$ nslookup galaxy.galaxy.svc.cluster.local
Server:        172.20.0.10
Address:    172.20.0.10#53

Name:    galaxy.galaxy.svc.cluster.local
Address: 172.20.38.130

I have this error message that I think is part of the problem in kube-proxy, but I cant find any remediation steps for it:

I0326 00<i class="em em-11"></i>09.536462       7 healthcheck.go:235] Not saving endpoints for unknown healthcheck "galaxy/galaxy"

this returns nothing:

node$ sudo ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

@Tim Malone

Erik Osterman
[cert-manager] [kiam] Provision kiam's TLS certs via cert-manager 0.7.0 by Nuru · Pull Request #107 · cloudposse/helmfiles

what Update cert-manager chart to install version 0.7.0 Update kiam to version 3.0 (helm chart version 2.1.0) and use cert-manager to automatically provision its TLS certificates why Current ver…

Erik Osterman

@Max Moon @

Erik Osterman

shout out to @Jeremy Grodberg for whipping this up

2019-03-24

Igor Rodionov

Check this up @Erik Osterman

Erik Osterman

That’s awesome!

Erik Osterman

@Jeremy Grodberg

2019-03-22

Abhsihek Jaisingh

Hey there guys! Are you still using helmfile? Had some questions regarding the same

Abhsihek Jaisingh

Sadly, couldn’t find any official community / channel for helmfile elsewhere

Erik Osterman

Yes! We use it extensively

Erik Osterman

Have you stumbled across our helmfiles? We update the weekly

Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

mmuehlberger

Here’s one in this Slack: #helmfile

Abhsihek Jaisingh

thanks!

Erik Osterman

Welcome @Abhsihek Jaisingh ! You’ve come to the right place as @mmuehlberger points out. Looks like @mumoshu is going to update the README for Helmfile so it will be easier for others to find.

2019-03-21

Erik Osterman
Kubernetes Authorization via Open Policy Agent

In a best-practice Kubernetes cluster every request to the Kubernetes APIServer is authenticated and authorized. Authorization is usually…

1

2019-03-20

any one here familair with alb ingress controlers?

Tim Malone

a tiny bit. is this in regards to your post in #sig-aws on the kubernetes Slack?

aknysh

we have a TF module to provision roles for alb ingress controller https://github.com/cloudposse/terraform-aws-kops-aws-alb-ingress

cloudposse/terraform-aws-kops-aws-alb-ingress

Terraform module to provision an IAM role for aws-alb-ingress-controller running in a Kops cluster, and attach an IAM policy to the role with permissions to manage Application Load Balancers. - clo…

aknysh
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Igor Rodionov

I do. What is the question?

I followed all the steps from the echoserver tutorial for my app deployment/service/ingress. I can see the alb, target group, and security groups are created. But I’m still getting a 502 and healthcheck failing on the alb

@Igor Rodionov

@Tim Malone yeah it is

nutellinoit

Is the alb permitted on the workers security group?

Igor Rodionov

Did you make the service ( that is backend for ingress) type of NodeIP ?

@nutellinoit yeah the alb ingress controller added the security group to the workers. @Igor Rodionov I’m using NodePort

Igor Rodionov

I do not know based on provided information

@Igor Rodionov i can provide my configs

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: galaxy-ingress
  namespace: galaxy
  annotations:
    <http<i class="em em-//kubernetes.io/ingress.class\|kubernetes.io/ingress.class>"></i> alb
    <http<i class="em em-//alb.ingress.kubernetes.io/scheme\|alb.ingress.kubernetes.io/scheme>"></i> internet-facing
    <http<i class="em em-//alb.ingress.kubernetes.io/tags\|alb.ingress.kubernetes.io/tags>"></i> Environment=dev
    <http<i class="em em-//alb.ingress.kubernetes.io/healthcheck-path\|alb.ingress.kubernetes.io/healthcheck-path>"></i> /health/
spec:
  rules:
    - host: <http://blah.example.com>
      http:
        paths:
          - path: /*
            backend:
              serviceName: galaxy
              servicePort: 80
kind: Service
apiVersion: v1
metadata:
  name: galaxy
  namespace: galaxy
  labels:
    service: galaxy
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: NodePort
  selector:
    app: galaxy
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: galaxy
  namespace: galaxy
spec:
  selector:
    matchLabels:
      app: galaxy
  replicas: {{.Values.galaxyReplicaCount \| default 3}}
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 100%
      maxUnavailable: 50%
  template:
    metadata:
      labels:
        app: galaxy
  spec:
    containers:
    - name: galaxy
      ports:
      - containerPort:8000
...
Erik Osterman

@tamsky @Igor Rodionov

tamsky
02:48:34 AM

@tamsky has joined the channel

2019-03-18

nutellinoit

Hello everyone, it is safe to update worker nodes to the latest minor kubernetes ami version? I got EKS master plane on version 1.11.8 and worker nodes on 1.11.5 and 5 autoscaling group attached to the cluster

nutellinoit

nevermind, i updated ami id, the minor version is still the same, 1.11.5

2019-03-14

endofcake

Anyone using makisu or kaniko for Docker builds?

endofcake
GoogleContainerTools/kaniko

Build Container Images In Kubernetes. Contribute to GoogleContainerTools/kaniko development by creating an account on GitHub.

endofcake
uber/makisu

Fast and flexible Docker image building tool, works in unprivileged containerized environments like Mesos and Kubernetes. - uber/makisu

2019-03-12

Hi, is there anyone who used https://www.telepresence.io/ ? I’d like to gather experiences and/or possible alternatives.

Home - Telepresence

Telepresence: a local development environment for a remote Kubernetes cluster

I haven’t used telepresence, but I was also researching tools for local dev. This one looks really interesting: https://github.com/windmilleng/tilt

windmilleng/tilt

Local Kubernetes development with no stress. Contribute to windmilleng/tilt development by creating an account on GitHub.

Any one uses helm plugins? Curious of your experiences with them. Came across this and it was interesting: https://github.com/totango/helm-ssm

totango/helm-ssm

Retrieves and injects secrets from AWS SSM. Contribute to totango/helm-ssm development by creating an account on GitHub.

Erik Osterman

Love the helm-git and helm-s3 plug-ins

Erik Osterman

Also helm-diff

2019-03-09

i5okie

I don’t know any contributors or anyone to mention as sponsor

2019-03-08

@Erik Osterman we can assume ~50 worker nodes

Erik Osterman

Did you see the Docker registry released by Uber this week that uses a p2p/torrent model for image distribution to address the problem of slow pulls?

Erik Osterman
Introducing Kraken, an Open Source Peer-to-Peer Docker Registry

Developed by Uber, Kraken is an open source peer-to-peer Docker registry capable of distributing terabytes of data in seconds.

Erik Osterman


With a focus on scalability and availability, Kraken was designed for Docker image management, replication, and distribution in a hybrid cloud environment. With pluggable back-end support, Kraken can also be plugged into existing Docker registry setups as the distribution layer.

ah i heard about it, but i didnt realize they had created a more efficient way of distributing images

ill have to check it out. i guess it works as a layer above your docker registry?

Erik Osterman

Yes that’s one mode of operation

@Erik Osterman talked to the folks maintaining kraken, sounds like it will help with the problem i have

they said they were running into the same issue

theyre also looking for help on the helm chart

Erik Osterman

@btai please post back later on how it works out for you

Erik Osterman

maybe something we should add support for in cloudposse/helmfiles

these guys @ uber on the container team have a super fast feedback loop

i asked about the helm chart earlier today for kraken, and they decided to take a first pass at it already

Simple helm chart added by apourchet · Pull Request #117 · uber/kraken

End result of helm install looks like this: $ kubectl get pods NAME READY STATUS RESTARTS AGE demo-6568968d44-g4nzw 1/1 Running 0 …

Erik Osterman

wow, impressive

i5okie

anyone know anyone for an invite to http://slack.k8s.io ?

Erik Osterman

You have filled out the form?

they had an issue where bots were spamming channels with NSFW stuff so they had to shut down slack registration

2019-03-07

does anyone here have a pattern to preload docker images into their cluster before deploys

scenario where you have a weekly cadence (i.e. release every tuesday) and you could preload all the worker nodes with the images the monday night before?

Erik Osterman

How large is the cluster? Is this something exasperated by the size of your cluster when pulling the images?

2019-03-06

casey

has anyone every tried out openshift origin? Any opinions on it?

Erik Weber

helped a client once, i wouldn’t do it again but i don’t know how it compares to other managed k8s solutions

Erik Weber

first impression is that it’s hard to get the initial configuration right if you’re not familiar with it, think we ran the installation playbook like 10 times before it worked successfully

Erik Osterman

I didn’t like that it requires modification of the host OS

Erik Osterman

I wish openshift operated entirely inside of k8s

Erik Weber

the actual installation is handled by ansible and pretty straight forward

Erik Weber

when shit hits the fan you have 3 layers to troubleshoot - plain docker, kubernetes and openshift

1

is there any way to print all the kubernetes resource objects?

johncblandii
Listing all resources in a namespace

I would like to see all resources in a namespace. Doing kubectl get all will, despite of the name, not list things like services and ingresses. If I know the the type I can explicitly ask for that

johncblandii

So, anyone running Jenkins get docker images to build without using docker.build commands from the plugin?

We need to run TF commands to get state, pass in build args, etc and our Makefile handles this. I’d rather not recreate that in a Jenkinsfile. Running 1 simple: make release ... is much easier.

Thoughts?

Erik Osterman

why not have terraform write to parameter store instead

Erik Osterman

then use chamber to read those settings

Erik Osterman

this will also let you downscope access to those parameters and not need to expose terraform state to jenkins

johncblandii

that’s the goal. we’re in the conversion of projects and they rely on env vars right now. we’re iterating to get to that place, though.

2019-03-05

casey

anyone have an opinion on kops vs eks? Do you think $144 per stage a month is worth it for eks?

Erik Osterman

I guess it just depends on what the aggregate spend will be

Erik Osterman

If a company is spending 30k/mo on amazon, then the 144/mo is a nominal amount :-)

casey

@Erik Osterman Do you have any benefits/drawbacks of using kops or eks in your experience?

Erik Osterman

An automated drain/cordon strategy of rolling updates for EKS is lacking. As such, we have preferred working with kops. Until such a strategy exists, EKS+terraform we prefer to stick with kops

1
Erik Osterman

Even eksctl does not support rolling updates yet

running into this error with external dns and unlimited staging environments. are there any strategies for dealing with this besides being more restrictive with naming? in this case I’m using the git branch name.

"InvalidChangeBatch: [FATAL problem: DomainLabelTooLong (Domain label is too long) encountered with '14bd77e7-db57-4000-8250-d89cd97cef51-staging-db-myapplication-backend']\n\tstatus code: 400, request id: c7a58d7e-3fa5-11e9-9ea2-799474e23419"
Erik Osterman

Aha, yea need normalization

Erik Osterman

What works well is to use slashes as a delimiter in your branch names

in this case, the branch is actually staging-db and the repo is myapplication${{CF_BRANCH}}-${{CF_REPO_NAME}}-backend.${{BASE_HOST}}

Erik Osterman

ohhhhhhhhhh

Erik Osterman

yes, in that case you have too much information to pack into the hostname

Erik Osterman

need zones

Erik Osterman

(dots)

ah.. so this would work? <http://14BD77E7-DB57-4000-8250-D89CD97CEF51-staging-db.myapplication-backend.us-west-2.staging.lootcrate.cc?\|14BD77E7-DB57-4000-8250-D89CD97CEF51-staging-db.myapplication-backend.us-west-2.staging.lootcrate.cc?>

thanks for the tip

Erik Osterman

something like that

Erik Osterman

you’ll need to update your ACM certs

Erik Osterman

to support the wildcard for *.<http://myapplication-backend.us-west-2.staging.lootcrate.cc>

right. still getting by with LE certs for now

Erik Osterman

aha

Erik Osterman

then no reason not to dot-it-up

Erik Osterman

E.g. epic123/my-fix

Erik Osterman

Use the first field for the service disambiguation

Erik Osterman

This also allows you to have multiple PRs deploy to the same namespace

2019-03-04

Erik Osterman
rancher/terraform-operator

Use K8s to Run Terraform. Contribute to rancher/terraform-operator development by creating an account on GitHub.

2019-03-02

2019-02-28

Erik Osterman
linki/cloudformation-operator

A Kubernetes operator for managing CloudFormation stacks via a CustomResource - linki/cloudformation-operator

endofcake
Introducing the Istio Operator for Kubernetes · Banzai Cloud

Bringing cloud native to the enterprise, simplifying the transition to microservices on Kubernetes

2
Erik Osterman

Anyone using AWS Service Mesh?

Erik Osterman

I love Istio, but it’s k8s centric; we have a upcoming use-case to create a mesh across ECS and k8s

I personally dislike the aws policy regarding opensource stealing (app-mesh is istio) so maybe you can come with an in between using true opensource project that run on both ecs and kubernetes like linkerd for example (I’m not having this use case neither use linkerd)

James D. Bohrman

I’ve read about it a bit, never used it. It seems interesting.

one day istio will be independent of k8s

2019-02-27

2019-02-25

what do you guys use for SSL certs?

@btai which cert ? the one facing our apps ? or the one needed by kube to works ? (like api server, kubelet, …)

facing your apps

endofcake
grafana/loki

Like Prometheus, but for logs. Contribute to grafana/loki development by creating an account on GitHub.

I tried it and it looks great, well integrated with grafana explore and and even better now there is a fluentd output plugin to send logs from all fluend enabled slacks (https://github.com/grafana/loki/tree/master/fluentd/fluent-plugin-loki). still it’s still in alpha and not prod ready from now

grafana/loki

Like Prometheus, but for logs. Contribute to grafana/loki development by creating an account on GitHub.

@endofcake I know that @ gave a try on this

jetstack/cert-manager

Automatically provision and manage TLS certificates in Kubernetes - jetstack/cert-manager

this is what you need

nice im looking into that right now

whats the best way to generate some certs manually in the meantime?

openssl man

can i generate some with letsencrypt ?

cloudflare/cfssl

CFSSL: Cloudflare’s PKI and TLS toolkit. Contribute to cloudflare/cfssl development by creating an account on GitHub.

yes you can

but man, certmanager is a maximum 1 hour setup for basic certificate generation

yeah?

yes !

there is an helm chart for that also in the github I linked to you

let me take a look I have some documentation for this in local

Erik Osterman
03:53:46 AM
Erik Osterman

anyone going?

2019-02-24

@James D. Bohrman i’m using jaeger with k8s

James D. Bohrman

How do you like it? I’ve been playing with it a bit and am having fun with it.

Erik Osterman

@ are you using it together with Istio?

@Erik Osterman yes and no

Let’s say not everywhere. I have tracing enabled by istio/envoy but some component are not injected by istio (lack of performances,…). So those just use the default jeager setup.

@James D. Bohrman it’s very nice and easy to implement if you use it with a service mesh. othw/ you shall implement it in yout code so k8s won’t help you with it

but I need to give a shot to the new elastic apm feature for opentracing

Erik Osterman

Has anyone looked into using AWS App Mesh (managed Envoy control plane ~ istio) with non-EKS kubernetes clusters? (e.g. #kops)

Erik Osterman


Is designed to pluggable and will support bringing your own Envoy images and Istio Mixer in the future.

Erik Osterman


Today, AWS App Mesh is available to use in preview

Erik Osterman
solo-io/supergloo

The Service Mesh Orchestration Platform. Contribute to solo-io/supergloo development by creating an account on GitHub.

Erik Osterman

@mumoshu have you seen this?

Erik Osterman
SuperGloo: The Service Mesh Orchestration Platform – solo.io – Medium

Today we are thrilled to announce the release of SuperGloo, an open-source project to manage and orchestrate service meshes at scale…

mumoshu

yep! i like the cli and their vision.

not yet sure if it worth another abstraction at this point of time

SuperGloo: The Service Mesh Orchestration Platform – solo.io – Medium

Today we are thrilled to announce the release of SuperGloo, an open-source project to manage and orchestrate service meshes at scale…

Erik Osterman

yea….

Erik Osterman

did you use it with AWS App Mesh?

mumoshu

not yet. just interestedf in istio + appmesh

2019-02-23

James D. Bohrman

Has anyone seen this yet? I haven’t played with it, but it looks really cool.

Write a Tiltfile script that describes how your services fit together. Share it with your team so that any engineer can hack on any server. See a complete view of your system, from building to deploying to logging to crashing.

https://tilt.dev/

Tilt

Local Kubernetes development with no stress

James D. Bohrman

Anyone using Jaeger with K8’s here?

2019-02-22

nutellinoit

@btai You can set a value for namespace in values.yaml eg “custom_namespace” and then you reference it the templates {{ .Values.custom_namespace }}

frednotet

Hi everyone ! Does somebody know the simplest way to enable hpa’s on a fresh new kops cluster ? metrics-server cannot connect (401 forbidden) and I can’t find the solution to retrieve metrics… maybe another solution ?

frednotet

Thanks @ but I already saw it and It didn’t help to solve it

frednotet

I’m still having same issue… it works on kube-system but not on the other namespaces

frednotet

If ever somebody reads… It’s very strange I had to rolling-out nodes & master and it works everywhere…

Did you do the steps defined in the issue ? If so those requires a rolling-update to work because kops installs kubelet on both instances and master and kubelet should be restarted.

Your case seems weird man ^^. Can you ellaborate on the issue a bit ? Is this a new cluster ? What version it is ? Did you do an update (if so which versions) ? Did you update your kops binary (if so which versions) ? How do you use kops ? (Gitops / tf / cf / nothing and prey)

frednotet

well, I have another problem actually

frednotet

maybe they’re related

frednotet

so I did several tests on a fresh new cluster

frednotet

(I have 3 clusters: “test”, “stg” and “prd”. those 3 are fresh new and are coded with terraform/kops)

frednotet

I now realize that I have 6 masters instead of 3

frednotet

if I force a rolling-update; it create new instances but they’re not healthy enough to join the cluster

frednotet

I see in their kubeconfig that they’re still configured on 127.0.0.1 instead of the k8s’s api. If I manually change this (+ restart kubelet), it will join the cluster

frednotet

but I have this error :

frednotet

Unable to perform initial IP allocation check: unable to refresh the service IP block: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1<i class="em em-4001"></i> connect: connection refused

frednotet

and the validation failed. I think that’s the reason why it ups new EC2 without releasing the old ones

frednotet

I think I will delete the full cluster and re-init it ‘cause I’m really lost and all my google is purple instead of blue now ^^

frednotet

even if I’d like to understand…

I just finish reading

what cni are you using ? if calico check that your nodes can reach the etcd cluster

it’s weird that you are using the 4001 port for etcd

what version of etcd / kubernetes are you using ? are you using etcd-manager (opt-in by default on kops w/ kube >= 1.11) ? if yes can you paste me the /etc/hosts of your masters please ?

can you type this command against your etcd cluster and paste the output => etcdctl cluster-health

frednotet

was using weave but I changed, reinstall everything with Calico… and everything works fine

frednotet

1.11.6 if I well remember (> 1.11 anyway since I integrate Spotinst and it needs 1.11)

frednotet

thanks for your help, even if I reset everything…

frednotet

I can reproduce actually… My cluster was working fine after a fresh installation… I edit the instancegroup to add more nodes and then I had to rolling-update the cluster

frednotet

the new master comes up; the old is terminated… but the new ones has a /var/lib/kubelet/kubeconfig set on 127.0.0.1 instead of the API

frednotet
kops rolling-update cluster k8s.stg.**********.io --state=s3://***********-stg-kops-state --yes                  
NAME			STATUS		NEEDUPDATE	READY	MIN	MAX	NODES
master-eu-west-1a	NeedsUpdate	1		0	1	1	1
master-eu-west-1b	NeedsUpdate	1		0	1	1	1
master-eu-west-1c	NeedsUpdate	1		0	1	1	1
nodes			NeedsUpdate	5		0	5	20	5
I0225 23<i class="em em-04"></i>28.528274   63403 instancegroups.go:165] Draining the node: "ip-10-62-103-158.eu-west-1.compute.internal".
node/ip-10-62-103-158.eu-west-1.compute.internal cordoned
node/ip-10-62-103-158.eu-west-1.compute.internal cordoned
WARNING: Ignoring DaemonSet-managed pods: calico-node-4ql85
pod/calico-kube-controllers-77bb8588fc-qcb4h evicted
pod/dns-controller-5dc57b7c99-dtw8j evicted
I0225 23<i class="em em-04"></i>42.275404   63403 instancegroups.go:358] Waiting for 1m30s for pods to stabilize after draining.
I0225 23<i class="em em-06"></i>12.280987   63403 instancegroups.go:185] deleting node "ip-10-62-103-158.eu-west-1.compute.internal" from kubernetes
I0225 23<i class="em em-06"></i>12.340897   63403 instancegroups.go:299] Stopping instance "i-07f15ebb7078aec08", node "ip-10-62-103-158.eu-west-1.compute.internal", in group "<http://master-eu-west-1c.masters.k8s.stg.musimap.io>" (this may take a while).
I0225 23<i class="em em-06"></i>15.287836   63403 instancegroups.go:198] waiting for 5m0s after terminating instance
I0225 23<i class="em em-11"></i>15.299756   63403 instancegroups.go:209] Validating the cluster.
I0225 23<i class="em em-11"></i>17.347229   63403 instancegroups.go:273] Cluster did not pass validation, will try again in "30s" until duration "5m0s" expires: machine "i-0567076920fedf435" has not yet joined cluster.
I0225 23<i class="em em-11"></i>48.468847   63403 instancegroups.go:273] Cluster did not pass validation, will try again in "30s" until duration "5m0s" expires: machine "i-0567076920fedf435" has not yet joined cluster.
I0225 23<i class="em em-12"></i>23.592726   63403 instancegroups.go:273] Cluster did not pass validation, will try again in "30s" until duration "5m0s" expires: machine "i-0567076920fedf435" has not yet joined cluster.
I0225 23<i class="em em-12"></i>48.538343   63403 instancegroups.go:273] Cluster did not pass validation, will try again in "30s" until duration "5m0s" expires: machine "i-0567076920fedf435" has not yet joined cluster.
I0225 23<i class="em em-13"></i>18.516763   63403 instancegroups.go:273] Cluster did not pass validation, will try again in "30s" until duration "5m0s" expires: machine "i-0567076920fedf435" has not yet joined cluster.
I0225 23<i class="em em-13"></i>48.512016   63403 instancegroups.go:273] Cluster did not pass validation, will try again in "30s" until duration "5m0s" expires: machine "i-0567076920fedf435" has not yet joined cluster.
I0225 23<i class="em em-14"></i>18.697398   63403 instancegroups.go:273] Cluster did not pass validation, will try again in "30s" until duration "5m0s" expires: machine "i-0567076920fedf435" has not yet joined cluster.
I0225 23<i class="em em-14"></i>48.490544   63403 instancegroups.go:273] Cluster did not pass validation, will try again in "30s" until duration "5m0s" expires: machine "i-0567076920fedf435" has not yet joined cluster.
I0225 23<i class="em em-15"></i>18.539400   63403 instancegroups.go:273] Cluster did not pass validation, will try again in "30s" until duration "5m0s" expires: machine "i-0567076920fedf435" has not yet joined cluster.
I0225 23<i class="em em-15"></i>48.672146   63403 instancegroups.go:273] Cluster did not pass validation, will try again in "30s" until duration "5m0s" expires: master "ip-10-62-103-6.eu-west-1.compute.internal" is not ready.
E0225 23<i class="em em-16"></i>17.352484   63403 instancegroups.go:214] Cluster did not validate within 5m0s

master not healthy after update, stopping rolling-update: "error validating cluster after removing a node: cluster did not validate within a duration of \"5m0s\""

are you saying that you are changing the number of nodes and it brings you new masters ?

2019-02-21

mpogrebnyak

hello, does anyone know, how can i limit inbound traffic using AWS EKS nodes?

joshmyers

Limit inbound according to?

1
joshmyers

Close your security groups

for helm, do you guys do multiple helm installs for dependent helm packages or do you nest them in your helm package for the application being deployed?

Erik Osterman

I avoid chart dependencies and use mostly helmfiles; makes it easier to swap out pieces and target individual services for upgrades

helm/charts

Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

and im curious how I should use it because it sets the namespace to be the namespace of the helm release but what if I don’t necessarily want to do that? Should I just modify the helm package files after I fetch them or is it bad practice

Erik Osterman

are you passing --namespace?

i wanted to avoid passing –namespace

2019-02-19

do you guys have an example using alb-ingress-controller with istio?

Erik Osterman

not together

2019-02-18

Anyone using Vault instead of Kiam, I’m new to k8s, and wondering what advantages&drawbacks are over using vault like this.

joshmyers

For AWS authentication? You have to manage Vault for a start

joshmyers

Vault could allow more flexibility than Kiam

Figured the kiam server needs to be managed as well, was hoping for it to be more elegant like the ecs-agent in that respect.

joshmyers

Yeah, you need to manage that too, agents and server

joshmyers

Has proved interesting in the past but I think mostly OK now

joshmyers

Vault does a lot more than Kiam though

joshmyers

How much do you want those other features?

I think Vault was chosen for the application secrets, so the logical step here would be adding the iam sessions

joshmyers

kiam is strictly around AWS services

joshmyers

If already using Vault, I’d stick with it over Kiam for IAM stuff

joshmyers

if not, kiam maybe a lower hanging fruit

thanks Josh!

joshmyers

IMO anyway, others will have other views

for sure, no worries

( Still liking ECS even more, knowing all this )

joshmyers

Nope ^^ , but if you are already running it and have gone through that pain…

joshmyers

If you are AWS, SSM and Kiam may get you what you want easier

but I guess what vault can also do, is probably combining GCP with AWS, for the ones thinking about that ..

joshmyers

Sure….

joshmyers

but I don’t know of many folks actually doing that

joshmyers

Multi provider is hard.

joshmyers

Vendor lock in is a thing

joshmyers

It’s all a tradeoff

joshmyers

I also don’t really care about being locked into AWS

me neither, they keep adding new stuff, and it works.

1
Erik Osterman
banzaicloud/bank-vaults

A Vault swiss-army knife: A K8s operator. Go client with automatic token renewal, Kubernetes support, dynamic secrets, multiple unseal options and more. A CLI tool to init, unseal and configure Vau…

1
Erik Osterman

saw that the other day

Erik Osterman

looks interesting and is related

joshmyers

Ah nice

joshmyers
UKHomeOffice/vault-sidekick

Vault sidekick. Contribute to UKHomeOffice/vault-sidekick development by creating an account on GitHub.

joshmyers

bank-vaults looks fuller featured

joshmyers

Certainly more complex than Kiam to manage

Erik Osterman

haha yea

Erik Osterman

seriously

Erik Osterman

what I’d like to see (and there probably exists), is something that implements the AWS IAM metadata proxy pattern of kube2iam, kiam but uses vault as the mediator

Erik Osterman

then uses the <http://iam.amazonaws.com/role> annotation just like kube2iam and kiam

Erik Osterman

that way the interface is interchangable

joshmyers

Annotations is a super nice way to drive those things in k8s

Erik Osterman
gardener/machine-controller-manager

Declarative way of managing machines for Kubernetes cluster - gardener/machine-controller-manager

Erik Osterman

Erik Osterman

looks sweet

Erik Osterman

apparently 100% open source

2019-02-15

Erik Osterman
open-policy-agent/gatekeeper

Gatekeeper - Policy Controller for Kubernetes. Contribute to open-policy-agent/gatekeeper development by creating an account on GitHub.

What container registry do u guys use

johncblandii

Just stood up JFrog. We’re actively moving there.

ECR is the current option we use.

johncblandii

You?

Erik Osterman

Are you also using other parts of Artifactory?

johncblandii

As in Xray? If so, about to. As in other registries, definitely will be using it for npm and potentially some maven/etc packages.

we use quay, but im getting very frustrated with their support cause I havent been able to upgrade our plan for more private repos

how is ECR @johncblandii

johncblandii

ECR is ok but can be a pain. you do 1 registry per image (can tag separately) so you don’t say “mydockerreg/image:tag” to reference multiple tags. You create a registry per image and reference the whole thing like: [registryid].dkr.ecr.[region].<http<i class="em em-//amazonaws.com/[image]"></i>[tag]\|amazonaws.com/[image]:[tag]>. Up to the [tag] part is locked in as the image URI.

I guess you could get fancy with a generic image name and customize per tag for the rest but layers would prob be an issue at that point.

johncblandii

but it is decent. it definitely wouldn’t be something I’d recommend for someone with a lot of images

thanks @johncblandii

would you guys say if we were to use Istio for traffic management, we could just stay with classic AWS ELBs?

Erik Osterman

Yes

Erik Osterman

I’m still not jazzed on ALBs + k8s

Erik Osterman

current implementation creates one ALB per Ingress

Erik Osterman

also, enabling NLBs on classic ELBs is trivial

Erik Osterman
  annotations:
    # by default the type is elb (classic load balancer).
    <http<i class="em em-//service.beta.kubernetes.io/aws-load-balancer-type\|service.beta.kubernetes.io/aws-load-balancer-type>"></i> nlb
Erik Osterman

the downside with ELB classic is you lose the client IP

Erik Osterman

this can be hacked with Proxy Protocol

Erik Osterman

but nginx-ingress doesn’t report the target port with Proxy Protocol correctly, so you don’t know if the user is using TLS or not

sarkis

do ALBs still take forever to create?

Erik Osterman

Yea they slow the create too

2019-02-14

sarkis

what instance sizes are your master/worker nodes @

sarkis

i was reading abut some issues with t3, m5, c5 or basically the new hypervisor (nitro) instances having this problem

i am using r5 instances, @sarkis and checked they are supported

@sarkis can you link where you were reading that?

sarkis
Pods stuck in ContainerCreating due to CNI Failing to Assing IP to Container Until aws-node is deleted · Issue #59 · aws/amazon-vpc-cni-k8s

On a node that is only 3 days old all containers scheduled to be created on this node get stuck in ContainerCreating. This is on an m4.large node. The AWS console shows that it has the maximum numb…

sarkis

multiple reports of t3, m5, r5 ^ which are all the new nitro instances

oo thanks, looks like its happening as much as 3 days ago. I guess i will revert to r4 instances

sarkis

nw! curious were you also seeing these issues? and doubly curious if it fixes the problem

Erik Osterman
05:20:40 AM

@Erik Osterman set the channel purpose: Archive: https://archive.sweetops.com/kubernetes/

2019-02-13

Erik Osterman

@ haven’t had to do that

Erik Osterman

though I have had to do other things related to networking in kops and it’s always led to that I destroy/recreate =(

Erik Osterman

Erik Osterman

@Ryan
Have you ran <https://github.com/mumoshu/aws-secret-operator>

Because for the life of me I can’t get it to create secrets <https://github.com/mumoshu/aws-secret-operator/issues/1>
Is my issuse as well .. just curious if you ran into this

Erik Osterman

@mumoshu

Ryan
11:09:04 PM

@Ryan has joined the channel

have you guys used envoy?

thoughts on it?

Erik Osterman

we have a basic example……

Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman

with istio (envoy sidecar injection)

Erik Osterman

TL;DR: was impressed how it works and want to do more with it

Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

i dont really need service mesh/service discovery

is it worth it just for proxying/traffic mgmt

Erik Osterman

yea

Erik Osterman

traffic mgmt / shapping is what i like

Erik Osterman

circuit breakers, rate limiting, auth, etc

whats shapping?

Erik Osterman

how the traffic flows across deployments (canary releases)

ahh

sorry im not super familiar with istio, is it recommended to run envoy w/istio?

i haven’t used it yet, but i like the promise of standardized request logging also

can i just run envoy as my proxy layer?

Erik Osterman

so istio is a way to manage envoy sidecars

Erik Osterman

linkerd does the same thing

Erik Osterman

and there are other ways too

ah so i deploy istio and it deploys envoy sidecars for me in my pods

Erik Osterman

yup

so i currently use traefik as my reverse proxy

deployed as daemon set (pod on each node)

is envoy considered an optimization?

Erik Osterman

basically isitio helps you deploy envoy on k8s

Erik Osterman

i like traefik too, but we haven’t used it in the same context

Erik Osterman

not sure if the feature set overlaps

have you guys used istio with EKS?

not sure if its outdated, but if you look under prereqs it doesn’t mention EKS

Erik Osterman

no

Erik Osterman

@johncblandii might also have done some research into that

Getting Started with Istio on Amazon EKS | Amazon Web Services

Service Meshes enable service-to-service communication in a secure, reliable, and observable way. In this multi-part blog series, Matt Turner, founding engineer at Tetrate, will explain the concept of a Service Mesh, shows how Istio can be installed as a Service Mesh on a Kubernetes cluster running on AWS using Amazon EKS, and then explain some […]

sweet

Erik Osterman

ohhh

Erik Osterman

i misread EKS (!= ECS)

yeah no, eks

after using k8s, no point in using ecs

Erik Osterman

lol

Erik Osterman

yes

johncblandii

I didn’t actually use Istio. I started to mess with it but hadn’t. We are using EKS and ECS (Fargate), though.

Does anyone faced CoreDNS pods are getting stuck at “ContainerCreating” issue?

Erik Osterman

What do you see when you describe pod?

kubelet, ip-10-225-0-236.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “2c2fa70a9231264ea9e67bd058126b67fee7409691c74165590a75bfecf29d1f” network for pod “coredns-7bcbfc4774-kxqmd”: NetworkPlugin cni failed to set up pod “coredns-7bcbfc4774-kxqmd_kube-system” network: add cmd: failed to assign an IP address to container

something like that

cni plugin version is 1.2.1

i have checked, its not related to EC2 instance or networking or IP addresses in subnet

Erik Osterman

Haven’t had that, but that error looks to be a pretty good hint

my subnet has lot of free IP ‘s and instance has only 3 ENI’s used and it can attach up to 10

2019-02-12

hi all, wondering how can we retain the NATIP when recreating a cluster using kops.

there is an open issue https://github.com/kubernetes/kops/issues/3182 but couldn’t find a better solution

Re-using existing elastic IPs for NAT gateways created by kops · Issue #3182 · kubernetes/kops

We currently have a kops cluster with a private topology. If we need to re-create this cluster, the elastic IPs associated with the NAT gateways are deleted, and new EIPs are allocated when the rep…

all solutions are more about deleting the cluster manually

2019-02-10

dryack
09:06:50 AM

@dryack has joined the channel

2019-02-08

nutellinoit

Hi everyone, there is a project that manage EKS workers scale in using lifecycle hooks and lambda?

Erik Osterman

That is what the cluster autoscaler is used for

Erik Osterman

In other words, using a lambda to scale the cluster node pools could work, but it’s not the prescribed way in Kubernetes

Erik Osterman
kubernetes/autoscaler

Autoscaling components for Kubernetes. Contribute to kubernetes/autoscaler development by creating an account on GitHub.

nutellinoit

Thank you Erik

nutellinoit

but i need only to manage the scale in, when a node is removed by asg

nutellinoit

i’m writing a new lambda that does kubectl drain on the node via SNS topic

joshmyers

Doesn’t the autoscaler do scale in too?

1
nutellinoit

i’m using plain asg with eks

joshmyers

plain asg’s as opposed to?

joshmyers
Cluster Autoscaler in Amazon EKS – Alejandro Millan Frias – Medium

Cluster Autoscaler automatically adjusts the number of nodes in a Kubernetes cluster when there are insufficient capacity errors to launch…

2019-02-07

joshmyers
Sysdig | Enable Kubernetes Pod Security Policy with kube-psp-advisor

How to enable Kubernetes Pod Security policy using kube-psp-advisor to address the practical challenges of building a security policy on Kubernetes.

Erik Osterman

maybe a good learning tool

2

@Erik Osterman are you guys catching nodes that are going to have issues ahead of time?

i had a k8s node yesterday that spiked to 100% CPU randomly that had to be cordon & drained

Erik Osterman
kubernetes/node-problem-detector

This is a place for various problem detectors running on the Kubernetes nodes. - kubernetes/node-problem-detector

Erik Osterman

@btai this look good?

interesting

i will try it out

the daemon.log was showing some interesting stuff

on that node that started having issues

Erik Osterman
kubernetes/node-problem-detector

This is a place for various problem detectors running on the Kubernetes nodes. - kubernetes/node-problem-detector

Erik Osterman

If you can generate a check, you can do a custom plugin like this

whats a custom plugin?

Erik Osterman

See example

Erik Osterman

Basically as simple as writing a a script that exits non zero

ah i see

Erik Osterman
danisla/terraform-operator

Kubernetes custom controller for operating terraform - danisla/terraform-operator

Erik Osterman
danisla/terraform-operator

Kubernetes custom controller for operating terraform - danisla/terraform-operator

Erik Osterman
Why the fuck are we templating yaml?

I was at cfgmgmtcamp 2019 in Ghent, and did a talk which I think was well received about the need for some Kubernetes configuration management as well as the…

2019-02-06

do you guys blue/green your k8s clusters when you want to upgrade or do you utilize rolling updates?

aknysh

with kops we usually do rolling updates https://docs.cloudposse.com/geodesic/kops/upgrade-cluster/

You dont manage the cluster with terra right?

aknysh

no

aknysh

with TF we create other resources like kops backend etc.

yeah, but I was curious if you also did kops > terraf > atlantis or similar

aknysh
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” invocations for provisioning reference architectures - cloudposse/terraform-root-modules

aknysh
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” invocations for provisioning reference architectures - cloudposse/terraform-root-modules

aknysh

no, we just provision the resources above with TF, but the cluster using kops commands from a template https://github.com/cloudposse/geodesic/blob/master/rootfs/templates/kops/default.yaml

cloudposse/geodesic

Geodesic is the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! https://slack.cloudposse.com/ - clou…

thanks

I guess you run kops commands out of band? not in CI

aknysh

yea, from geodesic

slow isnt it?

aknysh

yea, takes some time

this is more of a terraform question, but if i had my k8s cluster deployed in its own VPC and I had the database in a seperate VPC. (they are provisioned seperately because I blue/green my k8s clusters when I want to upgrade) If I were to VPC peer, is it possible to not have to upgrade the security group of the database?

basically allow full access to the db if there is a vpc peering connection?

aknysh

when you upgrade the cluster, is it still the same VPC?

nope

new k8s cluster, new vpc

aknysh

can you make two of them in advance and just add the two SGs to the database’s SG?

yes

i can do that

that would require an extra step but i think thats the best approach

1. spin up new k8s cluster/VPC
2. update database terraform with new SG
3. cutover
4. spin down old k8s cluster
5. update database terraform remove old SG 

actually @aknysh, if i provide the db security group to my cluster terraform I could use this

resource "aws_security_group_rule" "allow_all" {
  type            = "ingress"
  from_port       = 0
  to_port         = 65535
  protocol        = "tcp"
  cidr_blocks     = ["0.0.0.0/0"]
  prefix_list_ids = ["pl-12c4e678"]

  security_group_id = "sg-123456"
}

that would automatically do step 2 & 5 for me during cluster spin up and spin down

aknysh

hmm… what about ingress rules for the db SG? (you need to update them as well)

aknysh

when you create a new VPC and VPC peering, you can update the db SG with new ingress rules (unless you always have just the two VPCs and they never change, in which case you can add the SGs to the db ingress just once)

aknysh

or, if you create the two VPCs with the same CIDRs and they never change, you can add the CIDRs to the db SG (after peering, the db will see those CIDRs)

I cant create two vpcs with the same cidr because its in the same account

that aws_security_group_rule will update the db SG with the new vpc_id to allow ingress

aknysh

by the same I meant they could be different for the two VPCs, but they never change so you know the CIDRs in advance

ah yeah

that could work, but risk the chances someone spins up a different service using the same unused CIDR

(theres only 2 of us at my company that works on this stuff so very unlikely)

aknysh

yes

aknysh

so it’s better to just update the db SG with the new rule after you spin a new VPC

yep

Erik Osterman
Open Sourcing our Kubernetes Tools

At Tumblr, we are avid fans of Kubernetes. We have been using Kubernetes for all manner of workloads, like critical-path web requests handling for http://tumblr.com, background task executions like sending…

how are you guys monitoring your kubernetes nodes?

Erik Osterman

Prometheus & grafana

2019-02-05

2019-02-01

nutellinoit

Hi everyone, which is the best way to manage kubernetes deployments using terraform? We are using atlantis to CI/CD infrastructure

nutellinoit

There is the terraform kubernetes provider, but i don’t know if is good for production use

Erik Osterman

Personal opinion is that terraform is not a tool well suited for deployments on top of Kubernetes because it is only really good at creating and destroying resources. But updating resources less so.

3
nutellinoit

fyi, I took the road with helm charts + terraform helm provider

3
Erik Osterman

3
Erik Osterman

3
Erik Osterman

the helm provider is okay

3
Erik Osterman

in our experience, we couldn’t do half of what we do with helmfiles

3
Erik Osterman

terraform template files don’t support conditionals

3
Erik Osterman

so writing flexible values via terraform is difficult

3
Erik Osterman

our use-case is slightly different since we need to support multiple companies/organizations, which leads to more conditionals

3
nutellinoit

atm I’m using helm charts to differentiate between prod, qa, dev stage

3
nutellinoit

it’s so good applying changes with the helm provider, I was afraid it had a lot of bugs being still at version 0.x

3

2019-01-31

Erik Osterman
iJanki/kubecron

Utilities to manage kubernetes cronjobs. Run a CronJob manually for test purposes. Suspend/unsuspend a CronJob - iJanki/kubecron

Erik Osterman

@Daren

2019-01-30

Erik Osterman
Make CrashLoopBackoff timing tuneable, or add mechanism to exempt some exits · Issue #57291 · kubernetes/kubernetes

Is this a BUG REPORT or FEATURE REQUEST?: Feature request /kind feature What happened: As part of a development workflow, I intentionally killed a container in a pod with restartPolicy: Always. The…

Erik Osterman

Would have assumed the threshold off a CrashLoopBackoff be configurable

Erik Osterman

I am working on a demo where we deliberably kill pods

Erik Osterman