#kubernetes (2018-12)
Archive: https://archive.sweetops.com/kubernetes/
2018-12-03

K8s API server cve https://groups.google.com/forum/m/#!topic/kubernetes-announce/GVllWCg6L88


More details


There’s now an invisible way to hack into the popular cloud container orchestration system Kubernetes.

@here upgrade Kubernetes ASAP. remote root capabilities are believed possible.

2018-12-05


k8s just announced a major vulnerability in versions prior to 1.10.11, the recommended version from kops is currently 1.10.6. kubernetes/kubernetes#71411

Author: The 1.13 Release Team We’re pleased to announce the delivery of Kubernetes 1.13, our fourth and final release of 2018! Kubernetes 1.13 has been one of the shortest releases to date at 10 weeks. This release continues to focus on stability and extensibility of Kubernetes with three major features graduating to general availability this cycle in the areas of Storage and Cluster Lifecycle. Notable features graduating in this release include: simplified cluster management with kubeadm, Container Storage Interface (CSI), and CoreDNS as the default DNS.

https://news.ycombinator.com/item?id=18612571 gravitational guys have reproduced the kubernetes exploit


Istio, the open source service mesh that helps provide traffic management, observability, and security to microservices and distributed applications, is taking another step forward this week, as Google announces that it will be coming to Google Kubernetes Engine (GKE) next month in the form of a one-click integration. Offered initially in beta, the integration will …

Any feels on EKS yet?

We’re still not doing much with it

Don’t know if I shared this, but I put together some helm stats a few weeks ago - https://github.com/sstarcher/helm-exporter


hey @patrickleet! just connected you with @davidvasandani in #terraform


Today we’re happy to announce we’re open sourcing our support for using Docker Compose on Kubernetes. We’ve had this capability in Docker Enterprise for a little while but as of today you will be able to use this on any Kubernetes cluster you choose. Why do I need Compose if I already have Kubernetes? The Kubernetes API is really quite large. There are more than 50 first-class objects in the latest release, from Pods and Deployments to ValidatingWebhookConfiguration and ResourceQuota. This can lead to a verbosity in configuration, which then needs to be managed by you, the developer. Let’s look at a concrete example of that. The Sock Shop is the canonical example of a microservices application. It consists of multiple services using different technologies and backends, all packaged up as Docker images. It also provides example configurations using different tools, including both Continue reading…
2018-12-06

Kubernetes, a container orchestration system used by many companies worldwide, is a type of service we have been monitoring lately as we see issues like CVE-2018-1002105 appear. Another reason for our interest in this service is because we have seen increasing numbers being detected of Kubernetes being exposed to the
2018-12-07

what a nightmare

I’m sure theres people out there not realizing their clusters exposed

Created a #kubecon channel for those going. Would be create if we could have some members connect.
2018-12-11


Customizable multi-cluster rollouts for Kubernetes

yea, that’s neat that it works together with helm

looks like it introduces a new primitive to manage the process, so it should work well with other CI/CD systems (e.g. #codefresh)

@dustinvb

anyone read this?

The article? Yeah it looks great. Can’t wait to try it out.

llol sorry i meant the book i posted below!

the article looks awesome too tho!

haha

i haven’t read this one but i the quality of the “in action” series… go in action taught me everything i know about go


Authoritative and exhaustive. In a hands-on style, the author teaches how to manage the complete lifecycle of any distributed and scalable application.


KubeCon + CloudNativeCon 2018 is being held this week in Seattle. Here is everything that was announced related to Kubernetes and microservices.
2018-12-12

ksonnet looks really interesting, trying to uncover what advantages this has over go templates

what is the standard people are using to deploy kubernetes across the board? KOPS or somthing?

we use mostly kops
since it provides a complete solution with in place cluster updates/upgrades

but we also have EKS modules

EKS currently lacks some features and requires blue/green deployment with ASGs to perform updates/upgrades

yea just trying to get some more hands on experience with it!

Yea, so there’s no one tool that I would recommend

using one tool for all clouds, means you only get the lowest-common-denominator

many of the clouds offer some features to differentiate themselves

so for google, I’d recommend GKE all the way and using gcloud
on the command line

vs, on AWS I think kops
is still the best option


In what it is calling an experiment, Amazon Web Services shared its near-term road map for features and services built around containers Tuesday, a surprising move from a company that is notoriously…

wow

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS). - aws/containers-roadmap


And using github to publish the roadmap

that’s forward


I created a #kops channel for specific chatter around that.

Speaking of cluster upgrades… https://aws.amazon.com/blogs/compute/making-cluster-updates-easy-with-amazon-eks/

Kubernetes is rapidly evolving, with frequent feature releases, functionality updates, and bug fixes. Additionally, AWS periodically changes the way it configures Amazon Elastic Container Service for Kubernetes (Amazon EKS) to improve performance, support bug fixes, and enable new functionality. Previously, moving to a new Kubernetes version required you to re-create your cluster and migrate your […]

I was all… YAAAAAAY

and then AWWWWWW


Oops, swing and a miss!
2018-12-13


@Daren how are you deploying Argo?

Yes

I saw it looked like it was 3 charts

Do you deploy all of them?

Lukasz manages it, invite him in and I’m sure he will share

He has worked closely with them and contributed back

@wookasz ^

@wookasz has joined the channel

hey @wookasz! i’m curious about deploying argo for one of our clients

We do deploy it with helm
using codefresh. It’s not helmfile
(yet) so we have a separate overrides file and separate codefresh step for each environment

Are you using the official chart(s)?

almost, we actually have a fork which adds ability to forgo deploying the UI container, we weren’t using the UI and the image had some critical vulnerabilities so we decided to get rid of it for the time being, just haven’t gotten around to contributing that change back

It seems like it’s a few pieces now… a CI service and CD services and something else (forget what… maybe UI?)

each one has their own chart

Contribute to argoproj/argo-helm development by creating an account on GitHub.

are you installing all 3?

we only use the argo
one

though argo-events
is something i want to look into the future, it combines argo workflows with the ability to trigger them based on kafka messages, http webhooks, and cron schedules

we currently deploy cronjobs to trigger argo workflows so argo-events
would be a way to simplify that a bit

ok, fancy

ohhhhhhh

interesting. so out of the box argo
doesn’t have the scheduling

it does not

and argo-events
didn’t exist when we stared using argo

this is where you use argo-events
or CronJob
resources that curl a webhook or something?

the cronjobs schedule a simple container which has the argo
cli installed and just call argo submit [parameters]


the biggest pain with that approach is that we have to build a container for each workflow since the workflow
spec needs to be available to argo submit

hrmmmm yea, I can see that being a bit of a pain.

@daveyu shared this with me: https://brigade.sh/
Brigade is a tool for running scriptable automated tasks in the cloud. Brigade runs as part of a Kubernetes cluster.

Was this available when you guys did your research?

not sure if it’s a comparable solution.

for some reason brigade sounds familiar, but i can’t remember to a year and a half ago

the other issue we came across is that workflow resources are never deleted

which caused issues with etcd and kube state metrics

so we run a job to clean them up periodically

the argo team is working on adding a ttl to workflows though

out of curiosity, what’s the use case you’re looking at argo
for? if you can share

yes - so one of our clients needs to do some scraping and ETL

would like a tool to manage and visualize the process

(without cough jenkins)

Cool seems like it would be a pretty good fit, especially if they’re scraping from multiple sources and hoping to re-use pieces of the pipeline. If they’re loading into S3 or a warehouse that has the ability to copy from files in S3 then they have the added benefit that argo does a really nice job of making writing to S3 as simple as configuring which output files you want written there.

was wondering if you could point me in the right direction

(are you using a helmfile
by anychance?)

Yes

This makes me really worried to touch EKS for anything close to mission critical https://github.com/awsdocs/amazon-eks-user-guide/issues/17
I was unable to find any documentation on how to upgrade worker nodes for a new Kubernetes version or because of security issues. How will this work with EKS? The other information I could not find…

they are working on that https://github.com/aws/containers-roadmap/issues/57
Upgrading EKS worker nodes to the latest version (or to a specific version) should be as easy as clicking a button in the management console, or a single AWS CLI command. For comparison, ECS offers…


Yeah this was the post specifically got me . We’ve been talking about attaching an EKS cluster/vpc to prod here and moving things like Jenkins slaves and Prometheus as a first step but that is just scary.

yea + NLBs are unstable

Tell us about your request What do you want us to build? I want network load balancers to work on EKS without breaking Security Groups when nodes change, and I want that to happen before 1.13 which…

always fun knowing in the back of your head the cluster might completely stop working at any time@

This all got me thinking kops still the way to go for AWS at this time :(. Maybe give it a year or so and revisit eks

doh - missed the part about “users reported on kops as well” thanks for sharing that @patrickleet and great find..

didn’t find it on purpose - my NLB stopped working! luckily someone in kubernetes slack had gone through the effort of figuring out why

2018-12-14
2018-12-16

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager. - mumoshu/aws-secret-operator

Interesting.
A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager. - mumoshu/aws-secret-operator

@mumoshu does a great write up of all the tradeoffs with both helm-secrets
and chamber

while it sucks to jump ship to aws secrets manager, it might be worth it

I’m not comfortable with secrets at present because they are in the clear. I attended a kubecon session on security, they talked about the encryption providers: https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/

Just need an AWS kms implementation to pop up

I’m less worried about the ci ability to decrypt than he is

i like the separation of concerns

but agree that the KMS provider you mention looks great

hadn’t heard of it

I could only find a poc by awslabs for kms

@Daren

this looks like the best solution so far

@mumoshu has joined the channel

Kubernetes operator for Sysdig Falco that allows developers to manage rules for detecting intruders and backdoors - mumoshu/falco-operator

Kubernetes operator for Sysdig Falco that allows developers to manage rules for detecting intruders and backdoors - mumoshu/falco-operator

that’s pretty sweet

like hightower said a couple weeks ago, “kubectl is the new ssh”
2018-12-17

2018-12-19

anyone take the k8 exam on here?

Which one?

cka

I had a call with one of the guys at KubeCost - they are looking for beta testers

If you missed the post, check this out


This is the first in a series of posts for managing Kubernetes costs. Article shows how to quickly setup monitoring for basic cost metrics.

With kubecost you can see how much things running on kubernetes are costing you backed out to your AWS spend

know the cost of any labels

know how much a namespace is costing

etc.

let me know if your interested
2018-12-20


Kubernetes can now set up its own clusters. The most recent release of the Kubernetes container orchestration software, earlier this month, came with the ability to bootstrap its own deployments, thanks to the general availability release of kubeadm. Kubeadm can boot up a Kubernetes cluster through a single command. The cluster’s setup is defined by best-practices …