2018
Dec

#kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2018-12-20

davidvasandani
Kubernetes Now Does Self-Hosting with Kubeadm - The New Stack

Kubernetes can now set up its own clusters. The most recent release of the Kubernetes container orchestration software, earlier this month, came with the ability to bootstrap its own deployments, thanks to the general availability release of kubeadm. Kubeadm can boot up a Kubernetes cluster through a single command. The cluster’s setup is defined by best-practices …

2018-12-19

pericdaniel

anyone take the k8 exam on here?

davidvasandani

Which one?

pericdaniel

cka

Erik Osterman

I had a call with one of the guys at KubeCost - they are looking for beta testers

Erik Osterman

If you missed the post, check this out

Erik Osterman
Effectively Managing Kubernetes with Cost Monitoring

This is the first in a series of posts for managing Kubernetes costs. Article shows how to quickly setup monitoring for basic cost metrics.

Erik Osterman

With kubecost you can see how much things running on kubernetes are costing you backed out to your AWS spend

Erik Osterman

know the cost of any labels

Erik Osterman

know how much a namespace is costing

Erik Osterman

etc.

Erik Osterman

let me know if your interested

2018-12-16

Erik Osterman
mumoshu/aws-secret-operator

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager. - mumoshu/aws-secret-operator

Daren

Interesting.

mumoshu/aws-secret-operator

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager. - mumoshu/aws-secret-operator

Erik Osterman

@mumoshu does a great write up of all the tradeoffs with both helm-secrets and chamber

Erik Osterman

while it sucks to jump ship to aws secrets manager, it might be worth it

Daren

I’m not comfortable with secrets at present because they are in the clear. I attended a kubecon session on security, they talked about the encryption providers: https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/

1
Daren

Just need an AWS kms implementation to pop up

Daren

I’m less worried about the ci ability to decrypt than he is

Erik Osterman

i like the separation of concerns

Erik Osterman

but agree that the KMS provider you mention looks great

Erik Osterman

hadn’t heard of it

Daren

I could only find a poc by awslabs for kms

Erik Osterman

@Daren

Erik Osterman

this looks like the best solution so far

mumoshu
06:02:47 AM

@mumoshu has joined the channel

Erik Osterman
mumoshu/falco-operator

Kubernetes operator for Sysdig Falco that allows developers to manage rules for detecting intruders and backdoors - mumoshu/falco-operator

Erik Osterman
06:26:57 AM
mumoshu/falco-operator

Kubernetes operator for Sysdig Falco that allows developers to manage rules for detecting intruders and backdoors - mumoshu/falco-operator

Erik Osterman

that’s pretty sweet

Erik Osterman

like hightower said a couple weeks ago, “kubectl is the new ssh”

2018-12-14

2018-12-13

patrickleet

Erik Osterman

@Daren how are you deploying Argo?

Daren

Yes

Erik Osterman

I saw it looked like it was 3 charts

Erik Osterman

Do you deploy all of them?

Daren

Lukasz manages it, invite him in and I’m sure he will share

Daren

He has worked closely with them and contributed back

Daren

@wookasz ^

wookasz
12:29:37 AM

@wookasz has joined the channel

Erik Osterman

hey @wookasz! i’m curious about deploying argo for one of our clients

wookasz

We do deploy it with helm using codefresh. It’s not helmfile (yet) so we have a separate overrides file and separate codefresh step for each environment

Erik Osterman

Are you using the official chart(s)?

wookasz

almost, we actually have a fork which adds ability to forgo deploying the UI container, we weren’t using the UI and the image had some critical vulnerabilities so we decided to get rid of it for the time being, just haven’t gotten around to contributing that change back

Erik Osterman

It seems like it’s a few pieces now… a CI service and CD services and something else (forget what… maybe UI?)

Erik Osterman

each one has their own chart

Erik Osterman
argoproj/argo-helm

Contribute to argoproj/argo-helm development by creating an account on GitHub.

Erik Osterman

are you installing all 3?

wookasz

we only use the argo one

wookasz

though argo-events is something i want to look into the future, it combines argo workflows with the ability to trigger them based on kafka messages, http webhooks, and cron schedules

wookasz

we currently deploy cronjobs to trigger argo workflows so argo-events would be a way to simplify that a bit

Erik Osterman

ok, fancy

Erik Osterman

ohhhhhhh

Erik Osterman

interesting. so out of the box argo doesn’t have the scheduling

wookasz

it does not

wookasz

and argo-events didn’t exist when we stared using argo

Erik Osterman

this is where you use argo-events or CronJob resources that curl a webhook or something?

wookasz

the cronjobs schedule a simple container which has the argo cli installed and just call argo submit [parameters]

Erik Osterman

wookasz

the biggest pain with that approach is that we have to build a container for each workflow since the workflow spec needs to be available to argo submit

Erik Osterman

hrmmmm yea, I can see that being a bit of a pain.

Erik Osterman

@daveyu shared this with me: https://brigade.sh/

Brigade | Event-driven scripting for Kubernetes.

Brigade is a tool for running scriptable automated tasks in the cloud. Brigade runs as part of a Kubernetes cluster.

Erik Osterman

Was this available when you guys did your research?

Erik Osterman

not sure if it’s a comparable solution.

wookasz

for some reason brigade sounds familiar, but i can’t remember to a year and a half ago

wookasz

the other issue we came across is that workflow resources are never deleted

wookasz

which caused issues with etcd and kube state metrics

wookasz

so we run a job to clean them up periodically

wookasz

the argo team is working on adding a ttl to workflows though

wookasz

out of curiosity, what’s the use case you’re looking at argo for? if you can share

Erik Osterman

yes - so one of our clients needs to do some scraping and ETL

Erik Osterman

would like a tool to manage and visualize the process

Erik Osterman

(without cough jenkins)

wookasz

Cool seems like it would be a pretty good fit, especially if they’re scraping from multiple sources and hoping to re-use pieces of the pipeline. If they’re loading into S3 or a warehouse that has the ability to copy from files in S3 then they have the added benefit that argo does a really nice job of making writing to S3 as simple as configuring which output files you want written there.

Erik Osterman

was wondering if you could point me in the right direction

Erik Osterman

(are you using a helmfile by anychance?)

Daren

Yes

sarkis

This makes me really worried to touch EKS for anything close to mission critical https://github.com/awsdocs/amazon-eks-user-guide/issues/17

How to Upgrade EKS Worker Nodes and EKS Cluster? · Issue #17 · awsdocs/amazon-eks-user-guide

I was unable to find any documentation on how to upgrade worker nodes for a new Kubernetes version or because of security issues. How will this work with EKS? The other information I could not find…

aknysh
Interactively Upgrade EKS Worker Node · Issue #57 · aws/containers-roadmap

Upgrading EKS worker nodes to the latest version (or to a specific version) should be as easy as clicking a button in the management console, or a single AWS CLI command. For comparison, ECS offers…

Erik Osterman
03:03:39 AM
4
sarkis

Yeah this was the post specifically got me . We’ve been talking about attaching an EKS cluster/vpc to prod here and moving things like Jenkins slaves and Prometheus as a first step but that is just scary.

patrickleet

yea + NLBs are unstable

patrickleet
[EKS] [NLB stability]: Fixes for kube-controller not available on EKS yet · Issue #62 · aws/containers-roadmap

Tell us about your request What do you want us to build? I want network load balancers to work on EKS without breaking Security Groups when nodes change, and I want that to happen before 1.13 which…

patrickleet

always fun knowing in the back of your head the cluster might completely stop working at any [email protected]

1
sarkis

This all got me thinking kops still the way to go for AWS at this time :(. Maybe give it a year or so and revisit eks

sarkis

doh - missed the part about “users reported on kops as well” thanks for sharing that @patrickleet and great find..

patrickleet

didn’t find it on purpose - my NLB stopped working! luckily someone in kubernetes slack had gone through the effort of figuring out why

1
patrickleet

you can use kops without NLB also

1

2018-12-12

sarkis

ksonnet looks really interesting, trying to uncover what advantages this has over go templates

pericdaniel

what is the standard people are using to deploy kubernetes across the board? KOPS or somthing?

aknysh

we use mostly kops since it provides a complete solution with in place cluster updates/upgrades

aknysh

but we also have EKS modules

aknysh

EKS currently lacks some features and requires blue/green deployment with ASGs to perform updates/upgrades

pericdaniel

yea just trying to get some more hands on experience with it!

Erik Osterman

Yea, so there’s no one tool that I would recommend

Erik Osterman

using one tool for all clouds, means you only get the lowest-common-denominator

Erik Osterman

many of the clouds offer some features to differentiate themselves

Erik Osterman

so for google, I’d recommend GKE all the way and using gcloud on the command line

1
Erik Osterman

vs, on AWS I think kops is still the best option

aknysh
Amazon Web Services reveals a public road map for its cloud container services

In what it is calling an experiment, Amazon Web Services shared its near-term road map for features and services built around containers Tuesday, a surprising move from a company that is notoriously…

Erik Osterman

wow

Erik Osterman
aws/containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS). - aws/containers-roadmap

2
2
Erik Osterman

And using github to publish the roadmap

Erik Osterman

that’s forward

Erik Osterman

Erik Osterman

I created a #kops channel for specific chatter around that.

loren
Making Cluster Updates Easy with Amazon EKS | Amazon Web Services

Kubernetes is rapidly evolving, with frequent feature releases, functionality updates, and bug fixes. Additionally, AWS periodically changes the way it configures Amazon Elastic Container Service for Kubernetes (Amazon EKS) to improve performance, support bug fixes, and enable new functionality. Previously, moving to a new Kubernetes version required you to re-create your cluster and migrate your […]

Erik Osterman

I was all… YAAAAAAY

Erik Osterman

and then AWWWWWW

Erik Osterman
02:39:20 AM
1
loren

Oops, swing and a miss!

2018-12-11

Erik Osterman

yea, that’s neat that it works together with helm

Erik Osterman

looks like it introduces a new primitive to manage the process, so it should work well with other CI/CD systems (e.g. #codefresh)

Erik Osterman

@dustinvb

pericdaniel

anyone read this?

davidvasandani

The article? Yeah it looks great. Can’t wait to try it out.

pericdaniel

llol sorry i meant the book i posted below!

pericdaniel

the article looks awesome too tho!

Erik Osterman

haha

sarkis

i haven’t read this one but i the quality of the “in action” series… go in action taught me everything i know about go

pericdaniel
Kubernetes in Action

Authoritative and exhaustive. In a hands-on style, the author teaches how to manage the complete lifecycle of any distributed and scalable application.

1
aknysh
Everything that was announced at KubeCon + CloudNativeCon

KubeCon + CloudNativeCon 2018 is being held this week in Seattle. Here is everything that was announced related to Kubernetes and microservices.

2018-12-07

pericdaniel

what a nightmare

pericdaniel

I’m sure theres people out there not realizing their clusters exposed

Erik Osterman

Created a #kubecon channel for those going. Would be create if we could have some members connect.

2018-12-06

Erik Osterman
Kubernetes being hijacked worldwide

Kubernetes, a container orchestration system used by many companies worldwide, is a type of service we have been monitoring lately as we see issues like CVE-2018-1002105 appear. Another reason for our interest in this service is because we have seen increasing numbers being detected of Kubernetes being exposed to the

1

2018-12-05

Erik Osterman
07:30:40 PM
Erik Osterman
Current recommended version of k8s is insecure, CVE-2018-1002105 · Issue #6151 · kubernetes/kops

k8s just announced a major vulnerability in versions prior to 1.10.11, the recommended version from kops is currently 1.10.6. kubernetes/kubernetes#71411

aknysh
Kubernetes 1.13: Simplified Cluster Management with Kubeadm, Container Storage Interface (CSI), and CoreDNS as Default DNS are Now Generally Available

Author: The 1.13 Release Team We’re pleased to announce the delivery of Kubernetes 1.13, our fourth and final release of 2018! Kubernetes 1.13 has been one of the shortest releases to date at 10 weeks. This release continues to focus on stability and extensibility of Kubernetes with three major features graduating to general availability this cycle in the areas of Storage and Cluster Lifecycle. Notable features graduating in this release include: simplified cluster management with kubeadm, Container Storage Interface (CSI), and CoreDNS as the default DNS.

Erik Osterman

https://news.ycombinator.com/item?id=18612571 gravitational guys have reproduced the kubernetes exploit

Erik Osterman
Google Integrates Istio Service Mesh into Kubernetes Service - The New Stack

Istio, the open source service mesh that helps provide traffic management, observability, and security to microservices and distributed applications, is taking another step forward this week, as Google announces that it will be coming to Google Kubernetes Engine (GKE) next month in the form of a one-click integration. Offered initially in beta, the integration will …

sarkis

Any feels on EKS yet?

Erik Osterman

We’re still not doing much with it

Erik Osterman

Don’t know if I shared this, but I put together some helm stats a few weeks ago - https://github.com/sstarcher/helm-exporter

patrickleet

Erik Osterman

hey @patrickleet! just connected you with @davidvasandani in #terraform

Erik Osterman
Simplifying Kubernetes with Docker Compose and Friends - Docker Blog

Today we’re happy to announce we’re open sourcing our support for using Docker Compose on Kubernetes. We’ve had this capability in Docker Enterprise for a little while but as of today you will be able to use this on any Kubernetes cluster you choose. Why do I need Compose if I already have Kubernetes? The Kubernetes API is really quite large. There are more than 50 first-class objects in the latest release, from Pods and Deployments to ValidatingWebhookConfiguration and ResourceQuota. This can lead to a verbosity in configuration, which then needs to be managed by you, the developer. Let’s look at a concrete example of that. The Sock Shop is the canonical example of a microservices application. It consists of multiple services using different technologies and backends, all packaged up as Docker images. It also provides example configurations using different tools, including both Continue reading…

2

2018-12-03

Erik Osterman

More details

Erik Osterman
Kubernetes' first major security hole discovered | ZDNet

There’s now an invisible way to hack into the popular cloud container orchestration system Kubernetes.

2
Erik Osterman

@here upgrade Kubernetes ASAP. remote root capabilities are believed possible.

sarkis

2018
Dec
    keyboard_arrow_up