#kops

Discussions related to kops for kubernetes Archive: https://archive.sweetops.com/kops/

2019-08-28

Sharanya

Did anyone Come across NPM memory Issues ?

2019-07-25

Just have tested upgrading Works well

2019-07-24

Thank you for explanation. I assumed that. Now that is not a problem.

Guys, if I made the manual changes in the http://terraform.tf file which I exported from kops, will kops upgrade procedure work properly? Or will this procedure discard my changes and apply a configuration that stored in kops? E.g. I use one NAT gw for 3 AZ (other two NAT gws I have removed from http://terraform.tf manually)

Will revert the kops upgrade procedure two nat gws back? Right?

Erik Osterman

I don’t have enough context

Erik Osterman

the <http://terraform.tf> file? …that could be anything

Erik Osterman

Oh, sorry, I ment http://kubernetes.tf :)

2019-07-23

Hi everyone! I have the new question again I am trying to set up k8s cluster using kops. I know that official documentation says that I should provide the user with Iam permissions

AmazonEC2FullAccess
AmazonRoute53FullAccess
AmazonS3FullAccess
IAMFullAccess
AmazonVPCFullAccess

But these permissions include FullAccess. This is very insecure. Do you have any the minimal rule sets that exclude FullAccess?

I would like to know how do you, guys, make this setup Because, some time, we need to set up k8s cluster into the customer account and customer’s admins are afraid the FullAccess policies

Erik Osterman

You’re not giving this level of access to kops

Erik Osterman

you are giving this level of access to the person or process responsible for provisioning kops

Erik Osterman

the user will need CRUD for EC2, ELBs, EBS, EIP, S3, VPC, and at that point, they are basically admins

2019-07-17

Fernanda Martins

I also…looking at KOPS configured with Public DNS but… I also wonder if thats good given that it creates records in etcd…. do you anything about this?

2019-07-15

Fernanda Martins

Hello All, I was reading Kubernetes Security Best Practices and it mentions the practice to use private topology with private VPC. Does anyone here uses a public website of sorts on top of Kops private topology? How is that working out?

Erik Osterman

@Fernanda Martins exactly, so best practice is to run all the masters and nodes on a private topology, but then use an Ingress to expose a service

Erik Osterman

so a service will sit on a private “cluster ip”, and the (public) ingress will send traffic to that service.

Erik Osterman

technically, an ingress can be public or private. in your case, you’d want a public ingress.

Fernanda Martins

That what essentially KOPS does because I see some private subnets and public ones tied with Load Balancer…

1
Fernanda Martins

But I wonder if the public ones are configured in the best way…

Erik Osterman

There’s never a single best way; I guess it depends on the organization. For our use-case, we stick with the kops strategy.

2019-07-03

We are about to explore upgrading kops and k8s to 1.12.x from 1.11.x

I have done the upgrades many times with Kops, not within the scope of geodesic

Anyone else done so yet?

kubernetes/kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management - kubernetes/kops

it does sound a lot safer to spin up a new cluster

yup

100%

Erik Osterman

We are looking at upgrading from Kops 1.11 to 1.12. The upgrade instructions mention that it is a disruptive upgrade without going into details of how much. Is there anyone who has gone through it and can share their experience? cc @Jeremy Grodberg

Erik Osterman

Long thread below that

thanks, I went over that

honestly at this point im thinking I will build the capability to do full cluster backup and restore

try the update path as described in kops

Erik Osterman

probably better to rebuild if you can

if it goes tits up then re-roll the version we have

and restore

since the migration of data from one cluster to another requires downtime as is

feels like a better route

will need some planning and testing as we several prod workloads across 3 regions

will be fun though

2019-06-20

Are people using kops for GCP or Azure? or are people using Kubespray for more multi platform

last i checked (late 2018) kops didnt support azure, but AKS has been plenty good.

Erik Osterman

I wouldn’t use kops even for GCP

however azure postgres (and azure as a whole) have not had great uptime imo

Erik Osterman

I feel like the best option is to use the best tool for the platform. using any kind of generalized tool will likely not give you all the extra jazz provided by the platform.

AKS has been chugging along though but if you have other dependencies within azure

Erik Osterman

e.g. on google, I’d prefer to operate GKE over GCP+Kubernetes

same, AKS is great cause it feels to me fully managed (as opposed to EKS which is highly configurable and even the generic case takes more effort to spin up)

2019-06-19

Tim and I are going to start pushing PR’s up to you

I think time might already have one in for fixing your race condition with multi vpc peering

Erik Osterman

wow, that would be cool!

Daren

We are looking at upgrading from Kops 1.11 to 1.12. The upgrade instructions mention that it is a disruptive upgrade without going into details of how much. Is there anyone who has gone through it and can share their experience? cc @Jeremy Grodberg

Erik Osterman

@btai @ @Jan

btai
07:38:50 PM

@btai has joined the channel

07:38:50 PM

@ has joined the channel

unfortunately still on 1.11.9 for our kops clusters

also im not sure if i will run into this issue even when i do upgrade to 1.12 as my clusters are ephemeral (i would spin up a new 1.12 cluster and deploy/cutover to it)

Technically there is no usable upgrade path from etcd2 to etcd3 that supports HA scenarios, but kops has enabled it using etcd-manager. Nonetheless, this remains a higher-risk upgrade than most other kubernetes upgrades - you are strongly recommended to plan accordingly: back up critical data, schedule the upgrade during a maintenance window, think about how you could recover onto a new cluster, try it on non-production clusters first.

it almost sounds to me that spinning up a new cluster is prob the safest way forward, but im trying to imagine the way you guys are terraforming the cluster/env might make it hard to do that type of blue/green cutover?

Daren

blue/green is difficult for us right now. We are already on etcd 3, but no TLS or etcd-manager. We also use calico

Erik Osterman

Can you use route53 to route traffic to both cluster?

Erik Osterman

that would give you a fall back plan

Erik Osterman

or use external CDN (e.g. cloudflare) with multiple origins

Daren

We still use VPC peering to bridge kops and our backend vpc

Erik Osterman

oh, but you can peer the VPC to both k8s clusters

Erik Osterman

so you create a new kops vpc

Daren

Yes, I said “difficult” not impossible

Erik Osterman

haha, true

Erik Osterman

though this could be a good capability to support

Erik Osterman

even for future upgrades

Daren

yes

Erik Osterman

at the rate k8s moves, this won’t be the last breaking change

thats exactly what we do

kops cluster in its own vpc, peered to our database vpc, new cluster comes up will also vpc peer into db.

Daren

How are you provisioning the kops vpc peering connection?

i would suggest using terraform. cloudposse has an example thats pretty good

aknysh
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

aknysh
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

allows for quick route53 cutover/ you can also do a weighted cutover via route 53 and you have a pretty fast rollback strategy (point route53 back to old cluster)

Jeremy Grodberg

Discussion of some of the options for upgrading etcd (none great): https://gravitational.com/blog/kubernetes-and-offline-etcd-upgrades/

The Horrors of Upgrading Etcd Beneath Kubernetes

Proud new Kubernetes cluster owners are often lulled into a false sense of operational confidence by its consensus database’s glorious simplicity. In this Q&A, we dig into the challenges of in-place upgrades of etcd beneath autonomous Kubernetes clusters running within air-gapped environments.

Jeremy Grodberg

Step-by-step instructions for upgrading kops cluster by replacing it. Probably best for 1.11 to 1.12 upgrade. (I’ve never tried it. I have not had to upgrade a cluster from 1.11 to 1.12 yet.) https://www.bluematador.com/blog/upgrading-your-aws-kubernetes-cluster-by-replacing-it

Upgrading Your AWS Kubernetes Cluster By Replacing It

How to use kops to quickly spin up a production-ready Kubernetes cluster to replace your old cluster in AWS.

2019-06-12

is any one using the <http://dns.alpha.kubernetes.io/internal> annotation on nodes with success?

Erik Osterman

Not sure about that annotation

Erik Osterman

What does it do?

Adds a dns record with the internal (vpc ip) of the nodes

No way to filter nodes though

Not the solution im after, have a few other ideas

Erik Osterman

What is the solution you are after?

A maintained a record listing the vpc ips of all instance in a set instance group

Cassandra dedicated nodes

Exploring a similar solution using external-dns and host port

2019-02-24

Erik Osterman

has anyone looked into using “App Mesh” with kops?

Erik Osterman
awslabs/aws-app-mesh-examples

AWS App Mesh is a service mesh that you can use with your microservices to manage service to service communication. - awslabs/aws-app-mesh-examples

2019-02-22

um

t3 medium and large

and c5 large

Erik Osterman

ok, are you on kops 1.11?

2019-02-19

2019-02-18

Erik Osterman

Anyone have success launching t3.* instances with kops? https://github.com/kubernetes/kops/issues/5755

Launching `t3.medium` EC2 instance failed · Issue #5755 · kubernetes/kops
  1. What kops version are you running? The command kops version, will display this information. Version 1.10.0 (git-782ff1358) This is branch release-1.10 with #5681 cherry-picked 2. What Kubernetes…

i wanted to try, saw that support was added kops 1.11, but i got stuck figuring out how to upgrade kops in geodesic without forking packages

Launching `t3.medium` EC2 instance failed · Issue #5755 · kubernetes/kops
  1. What kops version are you running? The command kops version, will display this information. Version 1.10.0 (git-782ff1358) This is branch release-1.10 with #5681 cherry-picked 2. What Kubernetes…

we use t3

kops version
Version 1.10.0

2019-02-14

Erik Osterman
05:20:15 AM

@Erik Osterman set the channel purpose: Discussions related to kops for kubernetes Archive: https://archive.sweetops.com/kops/

2019-02-05

joshmyers

@Tim / @Jan How are you getting your kubecfg into place, that supports the aws-iam-authenticator ?

Yes we are using the cloudposse AWS IAM authenticator config map

joshmyers

Nice

The changes in the cluster config I have already seen on your side but it was removed

Will look into it tomorrow

joshmyers

How are you writing the kubecfg ?

Just adding the authentication webhook endpoint and installed a hook which pulls certs and stuff like that from s3

I have seen a commit on the cloudposse side doing exactly the same but it was removed ^^

Right now I don’t have the cluster config here. But I can provide it to you tomorrow

joshmyers

Cool, just wondering as kops export kubecfg doesn’t contain the users block that uses aws-iam-authenticator so wondering how you are doing it

Got it. This is what I was referring to: https://github.com/cloudposse/geodesic/pull/345/files

[kops/template] Update `aws-iam-authenticator` by aknysh · Pull Request #345 · cloudposse/geodesic

what [kops/template] Update aws-iam-authenticator settings why Kubernetes 1.10 and newer has aws-iam-authenticator installed by default, no need to add scripts to install it from S3 references …

Not sure how this does “fix iam authenticator” But that’s the config you need. Additional creating certificates and the config map

It says kubernetes comes with it by default but we were not able to get it working without these changes

kubernetes/kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management - kubernetes/kops

Erik Osterman

I love #variant

Erik Osterman

check this out

Erik Osterman
[kopsctl] add commands to facilitate management of cluster by osterman · Pull Request #378 · cloudposse/geodesic

what Add commands to easily rotate a kops cluster&#39;s ssh keys Add command to easily connect to a kops cluster Add command to see a kops plan why This are routine operations that are complicat…

2019-02-04

joshmyers

@Jan Any gotchas with implementing aws-iam-authenticator ? did you use https://github.com/cloudposse/terraform-aws-kops-iam-authenticator-config ?

cloudposse/terraform-aws-kops-iam-authenticator-config

Terraform module to create and apply a Kubernetes ConfigMap for aws-iam-authenticator to be used with Kops to map IAM principals to Kubernetes users - cloudposse/terraform-aws-kops-iam-authentica…

heya

joshmyers
cloudposse/geodesic

Geodesic is the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! https://slack.cloudposse.com/ - clou…

um shew let me think

I dont think so

let me see what we did

in the middle of a refactor, please remind me later

joshmyers

np

@Tim ball is on your court

Tim
08:23:21 PM

@Tim has joined the channel

2019-01-29

so…. who wants to talk me through whats going on here?

module "kops_state_backend" {
  source           = "git:<i class="em em-<https"></i>//github.com/cloudposse/terraform-aws-kops-state-backend.git?ref=tags/0.1.5>"
...
  cluster_name     = "${var.region}"
  region           = "${var.region}"
...
}
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” invocations for provisioning reference architectures - cloudposse/terraform-root-modules

aknysh

so in the example we use cluster name as the DNS zone subdomain name, e.g. if the domain for the nevironment is <http://prod.example.com>, then the module will create a subdomain us-west-2 and you’ll access your cluster at <http://us-west-2.prod.example.com>

aknysh

instead of the region, you can use any name that suits your needs, e.g. <http://kops.prod.example.com>

2019-01-18

yo

when runnign k8s via kops in aws

what creates the KubernetesAdmin role referenced here?

cloudposse/terraform-aws-kops-iam-authenticator-config

Terraform module to create and apply a Kubernetes ConfigMap for aws-iam-authenticator to be used with Kops to map IAM principals to Kubernetes users - cloudposse/terraform-aws-kops-iam-authentica…

kubernetes-sigs/aws-iam-authenticator

A tool to use AWS IAM credentials to authenticate to a Kubernetes cluster - kubernetes-sigs/aws-iam-authenticator

here

Erik Osterman

might have to wait for @aknysh

are they prerequisites or do you create them automatically?

Erik Osterman
kubernetes-sigs/aws-iam-authenticator

A tool to use AWS IAM credentials to authenticate to a Kubernetes cluster - kubernetes-sigs/aws-iam-authenticator

Erik Osterman

I don’t have first-hand experience setting this up yet

Erik Osterman

so I can just google =P

hahah

yea all good

will figure it out

cloudposse/terraform-aws-kops-iam-authenticator-config

Terraform module to create and apply a Kubernetes ConfigMap for aws-iam-authenticator to be used with Kops to map IAM principals to Kubernetes users - cloudposse/terraform-aws-kops-iam-authentica…

so liek this is the 2nd half of what I would have expected

the 1st part would have been the creation of the iam roles

will extend and submit

inf act its not needed

2019-01-17

Ajay Tripathy
04:28:26 PM

@Ajay Tripathy has joined the channel

2019-01-15

Are you using rbac by default?

erg wrong channel I think

Erik Osterman

in our kops manifest, we have it disabled by default

Erik Osterman

but there’s a boolean to enable it

Erik Osterman
cloudposse/geodesic

Geodesic is the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! https://slack.cloudposse.com/ - clou…

yea

Have it enabled

2019-01-09

bazbremner
11:49:06 AM

@bazbremner has joined the channel

06:41:56 AM

@ has joined the channel

2019-01-08

kubernetes/node-problem-detector

This is a place for various problem detectors running on the Kubernetes nodes. - kubernetes/node-problem-detector

Erik Osterman

Terraform module to create and apply a Kubernetes ConfigMap for aws-iam-authenticator to be used with Kops to map IAM principals to Kubernetes users

2019-01-07

2019-01-04

cloudposse/prod.cloudposse.co

Example Terraform/Kubernetes Reference Infrastructure for Cloud Posse Production Organization in AWS - cloudposse/prod.cloudposse.co

Populate chamber secrets for kops project (make sure to change the keys and values to reflect your environment; add new secrets as needed)
Erik Osterman

we’ve automated this now!

Erik Osterman

so these directions are out of date

Erik Osterman

sec

Erik Osterman
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” invocations for provisioning reference architectures - cloudposse/terraform-root-modules

Erik Osterman

We are writing all the settings to SSM

Erik Osterman

then you can just run

Erik Osterman
chamber exec kops -- bash -l
Erik Osterman

and run kops ... commands as normal

Erik Osterman

i can show you a demo

please!

where do I find that list? in aws/kops-aws-platform/chamber-kops.sh?

joshmyers

@Jan good question

busy trying to stand up the root kops as is

then modify the a fork of the module for targeting existing vpc’s

Erik Osterman
Document kops setup by osterman · Pull Request #339 · cloudposse/geodesic

what Move KOPS_* envs out of Dockerfile (prevents deploying multiple clusters) Thoroughly document how to use .envrc with our kops strategy why Our previous strategy of defining all ENVs in the …

Erik Osterman

this should be merged by EOW (sunday)

Erik Osterman

they still don’t take into account some of the latest changes.

2018-12-20

Erik Osterman

Yea Codefresh has an API and cli

So you could then run a k8s job that calls the codefresh api

To run pipelines on a schedule or based on some. Event

Sorry a k8s job

Erik Osterman

hah, haven’t thought that far ahead for terraform+codefresh

Erik Osterman

but yes…

Erik Osterman

if using kops, it would not be recommended to run the upgrade from within a job of on the same cluster

Erik Osterman

if using kube-aws, then I believe that would be possible based on what I learned from @mumoshu

Erik Osterman

…since it uses cfn

Interesting, would need explore it more

Just a brain fart while sitting next to the pool having a beer

1
Erik Osterman

lol

Erik Osterman

Kids are asleep

2018-12-17

Daren
05:22:53 AM

@Daren has joined the channel

Daren

kops is finally getting ready to release 1.11 https://kubernetes.slack.com/archives/C3QUFP0QM/p1545085988524900

Shane

now to release 1.12…

Daren

baby steps….

2018-12-13

Jan
09:33:04 AM

@Jan has joined the channel

12:42:29 PM

@ has joined the channel

2018-12-12

Erik Osterman
08:23:37 PM

@Erik Osterman has joined the channel

Erik Osterman
08:23:37 PM

@Erik Osterman set the channel purpose: Discussions related to kops for kubernetes

Max Moon
08:23:37 PM

@Max Moon has joined the channel

Adam
08:23:38 PM

@Adam has joined the channel

joshmyers
08:23:38 PM

@joshmyers has joined the channel

aknysh
08:23:45 PM

@aknysh has joined the channel

Erik Osterman

Wish we had this

Erik Osterman
rolling-update very slow · Issue #5989 · kubernetes/kops

As I understand it the current behaviour for rolling-update is: for node in stale_nodes: drain(node) validate_stable() delete(node) With the ASG taking care of spawning the new nodes. This is very …

Pablo Costa
08:24:28 PM

@Pablo Costa has joined the channel

Erik Osterman
kubernetes/kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management - kubernetes/kops

Erik Osterman

too bad no option for increasing the number of parallel nodes to be drained+cordoned+terminated

mumoshu
08:27:02 PM

@mumoshu has joined the channel

Shane
08:27:08 PM

@Shane has joined the channel

Andrii
08:27:32 PM

@Andrii has joined the channel

08:46:56 PM

@ has joined the channel

Erik Osterman
kubernetes/kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management - kubernetes/kops

Erik Osterman

neat - short lived worker node certs

09:19:58 PM

@ has joined the channel

sarkis
09:39:20 PM

@sarkis has joined the channel

OScar
09:56:00 PM

@OScar has joined the channel

Shane

I have considered a few times improving how kops updates clusters, but I need to care so few and far apart I have been lazy.

Erik Osterman

yea mostly agree

Erik Osterman

also, if we start doing these more often and unattended with #atlantis or #codefresh , I think I’ll care less

Shane

How are you liking codefresh?

Erik Osterman

I love it. Most fun I’ve had writing pipelines

Erik Osterman

one thing we could do in codefresh is schedule a pipeline to run every X weeks

Erik Osterman

that would be sweet for auto upgrading clusters

Has it got an api?

http://Jobs.in k8s, have the cluster run the pipeline that runs the cluster #meta

joppa27
01:06:10 AM

@joppa27 has joined the channel

davidvasandani
01:12:27 AM

@davidvasandani has joined the channel

    keyboard_arrow_up