#kubernetes (2019-07)

kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2019-07-02

davidvasandani avatar
davidvasandani


Sanic is an omni-tool which lets you build, deploy, and manage Kubernetes clusters.

btai avatar

reminds me of #geodesic

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Looks like a go cli tool that can be added to geodesic

2019-07-03

jose.amengual avatar
jose.amengual

anyone running a container with 200GB+ memory ?

1
jose.amengual avatar
jose.amengual

bad idea ?

leonardo.miranda avatar
leonardo.miranda

just curious, why do you need that size of container?

jose.amengual avatar
jose.amengual

is a in memory key value store

1
jose.amengual avatar
jose.amengual

somebody mentioned a tool for setting ENV variables for docker with support for different environments ?

jose.amengual avatar
jose.amengual

with some template support

jose.amengual avatar
jose.amengual

for the life of me I can’t recall the name….

tomv avatar

dockerize?

jose.amengual avatar
jose.amengual

it was something like systemenv ro something with env

ankur avatar

env consule?

jose.amengual avatar
jose.amengual

nop

tolstikov avatar
tolstikov
mumoshu/variant

Wrap up your bash scripts into a modern CLI today. Graduate to a full-blown golang app tomorrow. - mumoshu/variant

jose.amengual avatar
jose.amengual

I think so…

2019-07-04

2019-07-06

2019-07-08

2019-07-09

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
more examples of config files · Issue #508 · weaveworks/eksctl

Here is an example which should be documented, it uses only pre-existing IAM and VPC resources: apiVersion: eksctl.io/v1alpha4 kind: ClusterConfig metadata: name: test-cluster-c-1 region: eu-north-…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Anyone know if it’s possible to pass settings in the config to run commands on startup (like with kops)?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…in eksctl

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(@mumoshu)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. in kops, we can do:

  hooks:
  # Mitigate CVE-2019-5736
  - before:
    - docker.service
    manifest: |
      Type=oneshot
      ExecStart=/usr/bin/chattr +i /usr/bin/docker-runc
mumoshu avatar
mumoshu

There’s no easy way to add systemd units like that in eksctl. Technically there’re two options though -

  1. use preBootstrapCommands to write systemd units(https://github.com/weaveworks/eksctl/blob/cf5e078273d8d0d8fa802ae704f038d9c56ad8d7/pkg/apis/eksctl.io/v1alpha5/types.go#L457) or

  2. deploy a privileged daemonset that mount host volumes and writes unit files(https://github.com/mumoshu/kube-node-init)

weaveworks/eksctl

a CLI for Amazon EKS. Contribute to weaveworks/eksctl development by creating an account on GitHub.

mumoshu/kube-node-init

Kubernetes daemonset for node initial configuration. Currently for modifying files and systemd services on eksctl nodes without changing userdata - mumoshu/kube-node-init

i5okie avatar

Hi, has anyone tried the bitnami kubeprod.io stack? (https://github.com/bitnami/kube-prod-runtime)

bitnami/kube-prod-runtime

A standard infrastructure environment for Kubernetes - bitnami/kube-prod-runtime

2019-07-10

Alex Co avatar
Alex Co

hi, anyone here knows about use boolean value with k8s ?

Alex Co avatar
Alex Co

i’m trying to read the value from Environment variable, and i realize that when setting the env var inside a pod, it’s always format it as a string

Alex Co avatar
Alex Co

for example MY_VAR=abc will be MY_VAR=‘abc’ inside a pod

Alex Co avatar
Alex Co

if any program is expecting the boolean type, it will throw error

Alex Co avatar
Alex Co

is there a way to solve this ?

Alex Co avatar
Alex Co

thanks

MiLk avatar

environment variables are always strings

MiLk avatar

it needs to be handled by your program or whatever library you are using

Alex Co avatar
Alex Co

thanks @MiLk

2019-07-11

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
[EKS]: Next Generation AWS VPC CNI Plugin · Issue #398 · aws/containers-roadmap

We are working on the next version of the Kubernetes networking plugin for AWS. We've gotten a lot of feedback around the need for adding Kubenet and support for other CNI plugins in EKS. This …

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Higher container density is coming

2019-07-14

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
ibuildthecloud/k3v

Virtual Kubernetes. Contribute to ibuildthecloud/k3v development by creating an account on GitHub.

2019-07-17

jose.amengual avatar
jose.amengual

Hi, I have an idea about how I’m going to implement config templating for our containers and I will like some feedback :

  • Secrets and non secrets will be stored in aws Parameter Store+KMS
  • Chamber will be used to update/create secrets
  • Path IAM roles will be created for every environment /dev/secrets /prod/secrets etc
  • ECS Task per environment will have access to the /env path that have secrets and not secrets ( no shared configs)
  • Dev will use chamber to set ENV variable to run their local containers
  • confd might be use to create the config templates on Docker build
jose.amengual avatar
jose.amengual

most Jenkins will be doing the initial push and CodeDeploy will do the rest for Blue Green

jose.amengual avatar
jose.amengual

any comments ?

2019-07-18

timduhenchanter avatar
timduhenchanter

Anyone familiar enough with Kubernetes API to maybe know a way to get the resourceVersion of the parent for a given pod? Trying to ignore events in the watch if the change was initiated by its parent (I am not the controller). This is related to a ReplicaSet

2019-07-19

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Draft | Easy Kubernetes Development

Draft is a tool for developers to create cloud-native applications on Kubernetes.

btai avatar

saw this at kubecon

btai avatar

my gripes wit this is the feedback loop for rebuilding an image is slow

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what were your thoughts?

btai avatar

so during development esp. if you save alot assuming youre doing some web dev you’d typically want to refresh your browser right after you save

btai avatar

and see the changes right away

btai avatar

even during their demo at kubecon there was a bit of waiting

btai avatar

for the demo image (prob really small) to rebuild

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think developers mostly want live reloading

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. what you get with telepresence

btai avatar

yep

btai avatar

most prob use docker compose to spin up dependencies

btai avatar

and run their app locally

btai avatar

i think itd be awesome if you could have that type of live reloading but your app is hosted on a k8s cluster in the cloud

btai avatar

each developer would have their own namespace

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yep - that’s what we’re working on

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but haven’t yet tackled telepresence

btai avatar

are you able to do live reloading comparable to running it locally

btai avatar

cause thats a gamebreaker i think for most devs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Home - Telepresence

Telepresence: a local development environment for a remote Kubernetes cluster

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so with telepresence you run “the” service locally

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but all your backing services run in k8s (e.g. in a developer namespace)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

telepresence is like a reverse proxy. it sits in k8s and any requests it gets it sends back to the service on your laptop

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so it’s like teleporting your local service into the cluster

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

plus your local service can access everything running in k8s (e.g. database or other backing services)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

since it runs on your local laptop, you get all the benefits

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

easier debugging, attaching debuggers, live reloads, etc

btai avatar

ah yeah

btai avatar

but i guess other than the fact that dependent services hosted in cloud

btai avatar

is there any other benefit than running those in a docker compose

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you’re testing your services in an environment that’s closer to staging/prod

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if you have 30-40 microservices as part of your stack, good luck doing that on your laptop

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if you need large datasets for development, can’t do that easily locally

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

maybe it’s nicer to have all the data stay in AWS from a security perspective

btai avatar

truee

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

multiple developers can be working on pieces of the project at the same time

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and using a shared environment

btai avatar

so you host the db remotely as well

btai avatar

assume youre working on an API

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

its easier for others to QA changes in a public environment (e.g. if your laptop is offline, no one can review)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, usually run db as a container for these env

btai avatar

nice

btai avatar

yeah the large dataset hosted remotely one is actually really useful

btai avatar

allowing multiple devs on same project

cabrinha avatar
cabrinha

Has anyone here been able to get Istio installed to EKS? I’m trying to get it installed with my worker nodes all residing in private subnets and I’m running into weird issues.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Vidhi Virmani I think did

cabrinha avatar
cabrinha

Most of the examples I come across online assume you’re in public subnets with ELBs and security groups open to the world

cabrinha avatar
cabrinha

@Vidhi Virmani if you’ve installed Istio on EKS in private subnets, ping me!

2019-07-22

pericdaniel avatar
pericdaniel

Is there a way to have helm chart create external resources?(RDS/Elasticache)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
awslabs/aws-service-operator

AWS Service Operator allows you to create AWS resources using kubectl. - awslabs/aws-service-operator

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
awslabs/aws-servicebroker

AWS Service Broker. Contribute to awslabs/aws-servicebroker development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then use the Helm raw chart to provision

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The CRDs for RDS

pericdaniel avatar
pericdaniel

Thank you!

pericdaniel avatar
pericdaniel

What are people using as an oauth2 provider to login to their apps like k8 dashboard?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

keycloak

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  • gatekeeper proxies
jose.amengual avatar
jose.amengual

keycloak too but not for k8 but for pretty much anything

2019-07-23

cabrinha avatar
cabrinha

I’m interested to hear how people are using CD in Kubernetes. Is anyone doing Canary deployments on their EKS cluster? How?

Alex Siegman avatar
Alex Siegman

i can’t answer your question, but is that commander keen as your profile pic?

2
Alex Siegman avatar
Alex Siegman

I’d imagine a lot of people are using a service mesh like istio to manage the networking side of the canary deployment, but how they wrap all that in CD I’ve got no experience with

James D. Bohrman avatar
James D. Bohrman

Rio by Rancher. Check it out.

James D. Bohrman avatar
James D. Bohrman
Rio

The MicroPaaS for Kubernetes

2019-07-25

i5okie avatar

last year spent a few months troubleshooting and improving apps running in ElasticBeanstalk. with help from guys here (thank you). This year I’m deleting those ElasticBeanstalk stacks one by one after moving the apps to K8s. #lifeofadevopsengineer

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha too true. Thanks @i5okie for the update.

2019-07-29

jober avatar

I am looking at starting to use Kubernetes for the first time for a small system that I would eventually grow. Would you suggest using Kubernetes directly or using AWS EKS?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I would give eksctl a shot

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Probably the most turn key way to get up and running with EKS for a small project.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(we still use kops)

jober avatar

i was playing around with kops over the weekend

jober avatar

worked really well, just unsure at this point how to manage everything

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

kops is easy to get up and running.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the challenge with kubernetes is updates between major releases can be tricky

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. 1.11 -> 1.12 upgraded to etcd3 and there was no automated way to easily upgrade

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

while on EKS, those kinds of upgrade challenges are handled by the platform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, i believe the upgrade from 1.14 to 1.15 is also that way

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
1
jober avatar

Thanks that is really good to know. I’ll give EKS a go. Much appreciated

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we also have TF modules for EKS

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-eks-workers

Terraform module to provision an AWS AutoScaling Group, IAM Role, and Security Group for EKS Workers - cloudposse/terraform-aws-eks-workers

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

jober avatar

Awesome

jober avatar

Thanks so much!

jober avatar

Just found this as well, seems like a good beginner resource https://eksworkshop.com/introduction/

EKSworkshop.comattachment image

Amazon EKS Workshop

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, I saw that - looks amazing

2019-07-30

sohel2020 avatar
sohel2020

is it possible to create a EKS cluster in my Existing VPC?

1
Steven avatar

yes

sohel2020 avatar
sohel2020
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

jober avatar
module "eks_cluster" {
  source                  = "git::<https://github.com/cloudposse/terraform-aws-eks-cluster.git?ref=master>"
  namespace               = "eg"
  stage                   = "testing"
  name                    = "cluster"
  tags                    = "${var.tags}"
  vpc_id                  = "<YOUR VPC ID>"
  subnet_ids              = ["<YOUR PUBLIC SUBNET ID'S>"]

  # `workers_security_group_count` is needed to prevent `count can't be computed` errors
  workers_security_group_ids   = ["${module.eks_workers.security_group_id}"]
  workers_security_group_count = 1
}

2019-07-31

jafow avatar

I’m looking for a minimally-hacky way to restart running pods in order pickup a change in config data

party_parrot1
jafow avatar

here’s the use case: pods bootup and source some envrionment vars from SSM

jafow avatar

now I change a config value in SSM and would like pods to pick up that value

jafow avatar


I’m looking for a minimally-hacky way
I’m okay with some amount of hack tbh

jafow avatar

1 idea is to delete pods 1 by 1. They get restarted and when they run they fetch the data fresh from SSM and viola, config data is up to date

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
stakater/Reloader

A Kubernetes controller to watch changes in ConfigMap and Secrets and then restart pods for Deployment, StatefulSet, DaemonSet and DeploymentConfig – [✩Star] if you&#39;re using it! - stakater/Relo…

jafow avatar

cool thanks @Erik Osterman (Cloud Posse) I’ve heard of it and will check it out.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it was very easy to setup

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

endofcake avatar
endofcake

I wonder if there’s a way to hook it into Prometheus somehow, so it would only restart a pod if the app overall is healthy. A cursory look at this suggests I’d have to roll my own

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

assuming health is only determined by prometheus…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

however, services should have a healthcheck endpoint

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

per the Reloader README.md, it says:
then perform a rolling upgrade

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the only way to do a rolling upgrade is to wait for new pods to become healthy before moving on

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thus if the new secrets cause problems, that should cause the rollout to hang

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then the prometheus alerts for a pod crash loop should fire on the unhealthy pod

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
stakater/Reloader

A Kubernetes controller to watch changes in ConfigMap and Secrets and then restart pods for Deployment, StatefulSet, DaemonSet and DeploymentConfig – [✩Star] if you&#39;re using it! - stakater/Relo…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

to me, it looks like what they do is update an environment variable which causes k8s to do the rolling update

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thus all rolling update semantics are handled by k8s.

endofcake avatar
endofcake

Yeah, this gets more complicated with distributed applications such as Kafka, hence the need for external monitoring

    keyboard_arrow_up