#kubernetes (2018-12)

kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2018-12-03

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

More details

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Kubernetes' first major security hole discovered | ZDNetattachment image

There’s now an invisible way to hack into the popular cloud container orchestration system Kubernetes.

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@here upgrade Kubernetes ASAP. remote root capabilities are believed possible.

2018-12-05

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
07:30:40 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Current recommended version of k8s is insecure, CVE-2018-1002105 · Issue #6151 · kubernetes/kops

k8s just announced a major vulnerability in versions prior to 1.10.11, the recommended version from kops is currently 1.10.6. kubernetes/kubernetes#71411

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Kubernetes 1.13: Simplified Cluster Management with Kubeadm, Container Storage Interface (CSI), and CoreDNS as Default DNS are Now Generally Available

Author: The 1.13 Release Team We’re pleased to announce the delivery of Kubernetes 1.13, our fourth and final release of 2018! Kubernetes 1.13 has been one of the shortest releases to date at 10 weeks. This release continues to focus on stability and extensibility of Kubernetes with three major features graduating to general availability this cycle in the areas of Storage and Cluster Lifecycle. Notable features graduating in this release include: simplified cluster management with kubeadm, Container Storage Interface (CSI), and CoreDNS as the default DNS.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

https://news.ycombinator.com/item?id=18612571 gravitational guys have reproduced the kubernetes exploit

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Google Integrates Istio Service Mesh into Kubernetes Service - The New Stackattachment image

Istio, the open source service mesh that helps provide traffic management, observability, and security to microservices and distributed applications, is taking another step forward this week, as Google announces that it will be coming to Google Kubernetes Engine (GKE) next month in the form of a one-click integration. Offered initially in beta, the integration will …

sarkis avatar

Any feels on EKS yet?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re still not doing much with it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Don’t know if I shared this, but I put together some helm stats a few weeks ago - https://github.com/sstarcher/helm-exporter

patrickleet avatar
patrickleet

wave

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hey @patrickleet! just connected you with @davidvasandani in #terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Simplifying Kubernetes with Docker Compose and Friends - Docker Blogattachment image

Today we’re happy to announce we’re open sourcing our support for using Docker Compose on Kubernetes. We’ve had this capability in Docker Enterprise for a little while but as of today you will be able to use this on any Kubernetes cluster you choose. Why do I need Compose if I already have Kubernetes? The Kubernetes API is really quite large. There are more than 50 first-class objects in the latest release, from Pods and Deployments to ValidatingWebhookConfiguration and ResourceQuota. This can lead to a verbosity in configuration, which then needs to be managed by you, the developer. Let’s look at a concrete example of that. The Sock Shop is the canonical example of a microservices application. It consists of multiple services using different technologies and backends, all packaged up as Docker images. It also provides example configurations using different tools, including both Continue reading…

2

2018-12-06

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Kubernetes being hijacked worldwide

Kubernetes, a container orchestration system used by many companies worldwide, is a type of service we have been monitoring lately as we see issues like CVE-2018-1002105 appear. Another reason for our interest in this service is because we have seen increasing numbers being detected of Kubernetes being exposed to the

1

2018-12-07

pericdaniel avatar
pericdaniel

what a nightmare

pericdaniel avatar
pericdaniel

I’m sure theres people out there not realizing their clusters exposed

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Created a #kubecon channel for those going. Would be create if we could have some members connect.

2018-12-11

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, that’s neat that it works together with helm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

looks like it introduces a new primitive to manage the process, so it should work well with other CI/CD systems (e.g. #codefresh)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@dustinvb

pericdaniel avatar
pericdaniel

anyone read this?

davidvasandani avatar
davidvasandani

The article? Yeah it looks great. Can’t wait to try it out.

pericdaniel avatar
pericdaniel

llol sorry i meant the book i posted below!

pericdaniel avatar
pericdaniel

the article looks awesome too tho!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha

sarkis avatar

i haven’t read this one but i the quality of the “in action” series… go in action taught me everything i know about go

pericdaniel avatar
pericdaniel
Kubernetes in Actionattachment image

Authoritative and exhaustive. In a hands-on style, the author teaches how to manage the complete lifecycle of any distributed and scalable application.

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Everything that was announced at KubeCon + CloudNativeConattachment image

KubeCon + CloudNativeCon 2018 is being held this week in Seattle. Here is everything that was announced related to Kubernetes and microservices.

2018-12-12

sarkis avatar

ksonnet looks really interesting, trying to uncover what advantages this has over go templates

pericdaniel avatar
pericdaniel

what is the standard people are using to deploy kubernetes across the board? KOPS or somthing?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we use mostly kops since it provides a complete solution with in place cluster updates/upgrades

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

but we also have EKS modules

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

EKS currently lacks some features and requires blue/green deployment with ASGs to perform updates/upgrades

pericdaniel avatar
pericdaniel

yea just trying to get some more hands on experience with it!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, so there’s no one tool that I would recommend

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

using one tool for all clouds, means you only get the lowest-common-denominator

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

many of the clouds offer some features to differentiate themselves

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so for google, I’d recommend GKE all the way and using gcloud on the command line

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

vs, on AWS I think kops is still the best option

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Amazon Web Services reveals a public road map for its cloud container servicesattachment image

In what it is calling an experiment, Amazon Web Services shared its near-term road map for features and services built around containers Tuesday, a surprising move from a company that is notoriously…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

wow

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
aws/containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS). - aws/containers-roadmap

fb-wow2
2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

And using github to publish the roadmap

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s forward

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I created a #kops channel for specific chatter around that.

loren avatar
Making Cluster Updates Easy with Amazon EKS | Amazon Web Servicesattachment image

Kubernetes is rapidly evolving, with frequent feature releases, functionality updates, and bug fixes. Additionally, AWS periodically changes the way it configures Amazon Elastic Container Service for Kubernetes (Amazon EKS) to improve performance, support bug fixes, and enable new functionality. Previously, moving to a new Kubernetes version required you to re-create your cluster and migrate your […]

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I was all… YAAAAAAY

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and then AWWWWWW

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
02:39:20 AM
1
loren avatar

Oops, swing and a miss!

2018-12-13

patrickleet avatar
patrickleet

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren how are you deploying Argo?

Daren avatar

Yes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I saw it looked like it was 3 charts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Do you deploy all of them?

Daren avatar

Lukasz manages it, invite him in and I’m sure he will share

Daren avatar

He has worked closely with them and contributed back

Daren avatar

@wookasz ^

wookasz avatar
wookasz
12:29:37 AM

@wookasz has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hey @wookasz! i’m curious about deploying argo for one of our clients

wookasz avatar
wookasz

We do deploy it with helm using codefresh. It’s not helmfile (yet) so we have a separate overrides file and separate codefresh step for each environment

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Are you using the official chart(s)?

wookasz avatar
wookasz

almost, we actually have a fork which adds ability to forgo deploying the UI container, we weren’t using the UI and the image had some critical vulnerabilities so we decided to get rid of it for the time being, just haven’t gotten around to contributing that change back

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It seems like it’s a few pieces now… a CI service and CD services and something else (forget what… maybe UI?)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

each one has their own chart

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
argoproj/argo-helm

Contribute to argoproj/argo-helm development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

are you installing all 3?

wookasz avatar
wookasz

we only use the argo one

wookasz avatar
wookasz

though argo-events is something i want to look into the future, it combines argo workflows with the ability to trigger them based on kafka messages, http webhooks, and cron schedules

wookasz avatar
wookasz

we currently deploy cronjobs to trigger argo workflows so argo-events would be a way to simplify that a bit

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, fancy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ohhhhhhh

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

interesting. so out of the box argo doesn’t have the scheduling

wookasz avatar
wookasz

it does not

wookasz avatar
wookasz

and argo-events didn’t exist when we stared using argo

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is where you use argo-events or CronJob resources that curl a webhook or something?

wookasz avatar
wookasz

the cronjobs schedule a simple container which has the argo cli installed and just call argo submit [parameters]

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

wookasz avatar
wookasz

the biggest pain with that approach is that we have to build a container for each workflow since the workflow spec needs to be available to argo submit

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrmmmm yea, I can see that being a bit of a pain.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@daveyu shared this with me: https://brigade.sh/

Brigade | Event-driven scripting for Kubernetes.

Brigade is a tool for running scriptable automated tasks in the cloud. Brigade runs as part of a Kubernetes cluster.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Was this available when you guys did your research?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not sure if it’s a comparable solution.

wookasz avatar
wookasz

for some reason brigade sounds familiar, but i can’t remember to a year and a half ago

wookasz avatar
wookasz

the other issue we came across is that workflow resources are never deleted

wookasz avatar
wookasz

which caused issues with etcd and kube state metrics

wookasz avatar
wookasz

so we run a job to clean them up periodically

wookasz avatar
wookasz

the argo team is working on adding a ttl to workflows though

wookasz avatar
wookasz

out of curiosity, what’s the use case you’re looking at argo for? if you can share

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes - so one of our clients needs to do some scraping and ETL

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

would like a tool to manage and visualize the process

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(without cough jenkins)

wookasz avatar
wookasz

Cool seems like it would be a pretty good fit, especially if they’re scraping from multiple sources and hoping to re-use pieces of the pipeline. If they’re loading into S3 or a warehouse that has the ability to copy from files in S3 then they have the added benefit that argo does a really nice job of making writing to S3 as simple as configuring which output files you want written there.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

was wondering if you could point me in the right direction

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(are you using a helmfile by anychance?)

Daren avatar

Yes

sarkis avatar

This makes me really worried to touch EKS for anything close to mission critical https://github.com/awsdocs/amazon-eks-user-guide/issues/17

How to Upgrade EKS Worker Nodes and EKS Cluster? · Issue #17 · awsdocs/amazon-eks-user-guide

I was unable to find any documentation on how to upgrade worker nodes for a new Kubernetes version or because of security issues. How will this work with EKS? The other information I could not find…

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Interactively Upgrade EKS Worker Node · Issue #57 · aws/containers-roadmap

Upgrading EKS worker nodes to the latest version (or to a specific version) should be as easy as clicking a button in the management console, or a single AWS CLI command. For comparison, ECS offers…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
03:03:39 AM
4
sarkis avatar

Yeah this was the post specifically got me . We’ve been talking about attaching an EKS cluster/vpc to prod here and moving things like Jenkins slaves and Prometheus as a first step but that is just scary.

patrickleet avatar
patrickleet

yea + NLBs are unstable

patrickleet avatar
patrickleet
[EKS] [NLB stability]: Fixes for kube-controller not available on EKS yet · Issue #62 · aws/containers-roadmap

Tell us about your request What do you want us to build? I want network load balancers to work on EKS without breaking Security Groups when nodes change, and I want that to happen before 1.13 which…

patrickleet avatar
patrickleet

always fun knowing in the back of your head the cluster might completely stop working at any time@

1
sarkis avatar

This all got me thinking kops still the way to go for AWS at this time :(. Maybe give it a year or so and revisit eks

sarkis avatar

doh - missed the part about “users reported on kops as well” thanks for sharing that @patrickleet and great find..

patrickleet avatar
patrickleet

didn’t find it on purpose - my NLB stopped working! luckily someone in kubernetes slack had gone through the effort of figuring out why

1
patrickleet avatar
patrickleet

you can use kops without NLB also

1

2018-12-14

2018-12-16

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mumoshu/aws-secret-operator

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager. - mumoshu/aws-secret-operator

Daren avatar

Interesting.

mumoshu/aws-secret-operator

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager. - mumoshu/aws-secret-operator

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mumoshu does a great write up of all the tradeoffs with both helm-secrets and chamber

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

while it sucks to jump ship to aws secrets manager, it might be worth it

Daren avatar

I’m not comfortable with secrets at present because they are in the clear. I attended a kubecon session on security, they talked about the encryption providers: https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/

1
Daren avatar

Just need an AWS kms implementation to pop up

Daren avatar

I’m less worried about the ci ability to decrypt than he is

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i like the separation of concerns

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but agree that the KMS provider you mention looks great

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hadn’t heard of it

Daren avatar

I could only find a poc by awslabs for kms

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this looks like the best solution so far

mumoshu avatar
mumoshu
06:02:47 AM

@mumoshu has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mumoshu/falco-operator

Kubernetes operator for Sysdig Falco that allows developers to manage rules for detecting intruders and backdoors - mumoshu/falco-operator

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:26:57 AM
mumoshu/falco-operator

Kubernetes operator for Sysdig Falco that allows developers to manage rules for detecting intruders and backdoors - mumoshu/falco-operator

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s pretty sweet

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

like hightower said a couple weeks ago, “kubectl is the new ssh”

2018-12-19

pericdaniel avatar
pericdaniel

anyone take the k8 exam on here?

davidvasandani avatar
davidvasandani

Which one?

pericdaniel avatar
pericdaniel

cka

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I had a call with one of the guys at KubeCost - they are looking for beta testers

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you missed the post, check this out

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Effectively Managing Kubernetes with Cost Monitoringattachment image

This is the first in a series of posts for managing Kubernetes costs. Article shows how to quickly setup monitoring for basic cost metrics.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

With kubecost you can see how much things running on kubernetes are costing you backed out to your AWS spend

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

know the cost of any labels

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

know how much a namespace is costing

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

etc.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

let me know if your interested

2018-12-20

davidvasandani avatar
davidvasandani
Kubernetes Now Does Self-Hosting with Kubeadm - The New Stackattachment image

Kubernetes can now set up its own clusters. The most recent release of the Kubernetes container orchestration software, earlier this month, came with the ability to bootstrap its own deployments, thanks to the general availability release of kubeadm. Kubeadm can boot up a Kubernetes cluster through a single command. The cluster’s setup is defined by best-practices …

    keyboard_arrow_up