#kubernetes (2018-09)

kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2018-09-28

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@pericdaniel are you referring to EKS?

Max Moon avatar
Max Moon

Does anyone else follow this blog? https://blog.jessfraz.com/ tons of good stuff regarding containers, security, and k8s

Jessie Frazelle's Blog: Home

The blog of Jessie Frazelle, hacker of all things Linux & Containers

Max Moon avatar
Max Moon

Same person is responsible for: https://github.com/genuinetools

genuinetools

We are the home of quality software with a focus on simplicity, usability, security and minimalism. No bullshit, no politics, all genuine. - genuinetools

Max Moon avatar
Max Moon

again, lots of really good/cool/interesting tools there too

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

thanks for the links @Max Moon

Max Moon avatar
Max Moon

np

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

She’s awesome! I have been following her for years on twitter

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

She runs Coreos on the desktop

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Has containerized everything

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
jessfraz/dockerfiles

Various Dockerfiles I use on the desktop and on servers. - jessfraz/dockerfiles

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

On an unrelated note…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
gofunct/cloudnative-engineer

Contribute to gofunct/cloudnative-engineer development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Certified Kubernetes Administrator Exam Prep

2018-09-27

pericdaniel avatar
pericdaniel

i saw the eks tf files that were created

pericdaniel avatar
pericdaniel

do we have anything for aws-config

pericdaniel avatar
pericdaniel

aws-auth

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

the eks project is being tested now and the modules will be updated to the latest version this week

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

what are you doing with aws-config?

pericdaniel avatar
pericdaniel

for aws-auth

pericdaniel avatar
pericdaniel

i can authorize specific users to be able to make changes to the clusters and deploy environments

2018-09-26

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
heptiolabs/eventrouter

A simple introspective kubernetes service that forwards events to a specified sink. - heptiolabs/eventrouter

Daren avatar
Daren

:–1:

heptiolabs/eventrouter

A simple introspective kubernetes service that forwards events to a specified sink. - heptiolabs/eventrouter

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(this was the project I was thinking of… came across it today by accident looking at heptio projects)

2018-09-23

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
kubernetes/node-problem-detector

This is a place for various problem detectors running on the Kubernetes nodes. - kubernetes/node-problem-detector

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


node-problem-detector aims to make various node problems visible to the upstream layers in cluster management stack. It is a daemon which runs on each node, detects node problems and reports them to apiserver. node-problem-detector can either run as a DaemonSet or run standalone. Now it is running as a Kubernetes Addon enabled by default in the GCE cluster.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Works with draino

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
negz/draino

Automatically cordon and drain Kubernetes nodes based on node conditions - negz/draino

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


Draino automatically drains Kubernetes nodes based on labels and node conditions. Nodes that match all of the supplied labels and any of the supplied node conditions will be cordoned immediately and drained after a configurable drain-buffer time.

2018-09-21

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Provision AWS Services Through Kubernetes Using the AWS Service Broker | Amazon Web Services attachment image

使用 AWS Service Broker 通过 Kubernetes 配置 AWS 服务 There’s no doubt that containers have changed how we build projects. One of the guiding principles of a containerized workflow approach has been to give back control to the developer, allowing them to choose their dependencies and how to consume them – most importantly, when they […]

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

“And that’s all folks” - wasn’t that easy? :P

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Joking aside - pretty cool. Basically let’s you provision AWS backing services from within Kubernetes

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

H/T @fdrescher

fdrescher avatar
fdrescher
03:15:20 PM

@fdrescher has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’ve released our EKS terraform modules for Kubernetes this week.

Welcome feedback

cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

cloudposse/terraform-aws-eks-workers

Terraform module to provision an AWS AutoScaling Group, IAM Role, and Security Group for EKS Workers - cloudposse/terraform-aws-eks-workers

rms1000watt avatar
rms1000watt

@stobiewankenobi ^^^

stobiewankenobi avatar
stobiewankenobi
07:27:11 PM

@stobiewankenobi has joined the channel

rms1000watt avatar
rms1000watt

it looks clean!

rms1000watt avatar
rms1000watt

@Erik Osterman (Cloud Posse) should post on Reddit if you don’t already

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks @rms1000watt yes - plan to do this soon

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think @Andriy Knysh (Cloud Posse) is doing some more testing today

Jeremy avatar
Jeremy

Hi there, I was wondering whether the prometheus-to-cloudwatch solution can be adapted to scape metrics from the metrics server instead of kube-state-metrics.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

ohh, the module was created just as an experiment, tested a little bit, and then forgotten (meaning not supported anymore b/c there many more official solutions to do prometeus-to-cloudwatch)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Erik Osterman (Cloud Posse) can explain the whole situation

Jeremy avatar
Jeremy

thanks @Andriy Knysh (Cloud Posse). Can you direct me to where I can find those other solutions?

Jeremy avatar
Jeremy

I saw an exporter for exporting CW metrics to prometheus but not the other way around.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
prometheus/cloudwatch_exporter

Metrics exporter for Amazon AWS CloudWatch. Contribute to prometheus/cloudwatch_exporter development by creating an account on GitHub.

Initial experiences with the Prometheus monitoring system attachment image

I’ve been looking for a while for a monitoring system written in Go, self-contained and easy to deploy. I think I finally found what I was…

Jeremy avatar
Jeremy

So far, your solution was the only solution I found for exporting prometheus metrics to CW.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Erik Osterman (Cloud Posse) ^

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

this should be done in a more official way by using Prometheus Operator

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

this tool to do it was already mentioned https://github.com/operator-framework

Operator Framework

The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. - Operator Framework

Jeremy avatar
Jeremy

That makes sense.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Operator will allow a much better integration with Prometheus

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(but I agree, our tool is simpler )

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

but @Jeremy, what you asked (to scape metrics from the metrics server instead of kube-state-metrics) could be done by installing kube-prometheus and then scraping it, no?

Jeremy avatar
Jeremy

yeah, i guess that’s what I’m asking. do i simply need to change the url in the values.yaml?

Jeremy avatar
Jeremy

(I haven’t had time to look at the code you’re using to scape the metrics yet)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
kubernetes/kube-state-metrics

Add-on agent to generate and expose cluster-level metrics. - kubernetes/kube-state-metrics

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)


Additionally, some monitoring systems such as Prometheus do not use Heapster(metrics-server) for metric collection at all and instead implement their own, but Prometheus can scrape metrics from heapster itself to alert on Heapster(metrics-server)’s health. Having kube-state-metrics as a separate project enables access to these metrics from those monitoring systems

Jeremy avatar
Jeremy

i’ll have a look. I’m interested in surfacing the metrics described in this blog https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-4-the-kubernetes-api-server-72f1e1210770 to CW.

A Deep Dive into Kubernetes Metrics — Part 4: The Kubernetes API Server attachment image

This is Part 4 of a multi-part series about all the metrics you can gather from your Kubernetes cluster.

Jeremy avatar
Jeremy

i believe i should be able to get these without running the metrics server.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yea, it’s a very convoluted topic

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so I think you need to install Prometheus (via kube-prometheus https://github.com/coreos/prometheus-operator/tree/master/helm/kube-prometheus for example ) and then will be able to scrape it using a scraping tool

coreos/prometheus-operator

Prometheus Operator creates/configures/manages Prometheus clusters atop Kubernetes - coreos/prometheus-operator

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

in our collection of helmfiles, we have examples on how to do it https://github.com/cloudposse/helmfiles/tree/master/helmfile.d

cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy (Cloud Posse) we’ve added better formatting for prometheus alerts. See this PR by @Igor Rodionov https://github.com/cloudposse/helmfiles/pull/48

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)
09:37:53 PM

@Jeremy (Cloud Posse) has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, deploying Grafana dashboards with configmaps: https://github.com/cloudposse/helmfiles/pull/18

[grafana] Use sidecar for datasources and dashboards by osterman · Pull Request #18 · cloudposse/helmfiles

what Update to use sidecar pattern Provide integration with kube-prometheus (collecting metrics / import grafana dashboards) Collect metrics for nginx ingress and display them Fix Portal Fix nginx…

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

I think it’s a different Jeremy :)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy (Cloud Posse) is with PopChest <— using our older versions of kube-prometheus

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

But, @Erik Osterman (Cloud Posse) There is also @Jeremy Cowan, who I think you meant to be referring to.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ahk

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I see.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

well, welcome to #kubernetes

2018-09-19

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for context, we’re moving this discussion here: https://sweetops.slack.com/archives/CB2PXUHLL/p1537386287000100

I’ve literally never had this issue before, docker image built on my local machine gets uploaded to ECR and when i deploy that image, it comes across corrupt with the exact same configuration and it makes 0 sense. It appears my image being uploaded is corrupt and i’ve been troubleshooting for hours

Max Moon avatar
Max Moon

I’d be curious to know, are you getting that composer error from the log from kubectl logs <pod name> or elsewhere?

Matthew avatar
Matthew

Getting it from the container that is being build

Matthew avatar
Matthew

built*

Matthew avatar
Matthew

image: report-portal:develop built locally works 100% of the time. Deployed using this, somehow libraries are lost and dropped.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what happens if you pull and run that ECR image locally?

Matthew avatar
Matthew

When i build the ECR image locally, same error is produced

Matthew avatar
Matthew

But when I build the image that is being uploaded, it works smoothly

Matthew avatar
Matthew

I have 0 idea why my ECR image would be corrupt but the one building that image works

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Matthew please explain it again step-by-step for people to be able to help you, something like this…

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. I have a Dockerfile (show it here) which I build locally and then start the container and it works locally
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Then I push the already built image to ECR manually
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. When the image gets deployed from the ECR repo to Kubernetes, it throws errors
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

explain what works and what does not, where you build it and how

2018-09-18

Max Moon avatar
Max Moon

curious if anyone has given creating an Operator themselves a go: https://github.com/operator-framework

Operator Framework

The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. - Operator Framework

Max Moon avatar
Max Moon

chances are you are (either knowingly or unknowingly) already using one or many in your cluster

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re using all the prometheus operators in our latest rollouts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I would love to see a Terraform operator

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haven’t yet considered taking the plunge to write one

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

is there something in particular you want to build?

Max Moon avatar
Max Moon

nothing concrete yet, have been trying to think of some ideas

2018-09-17

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
heptio/ark

Heptio Ark is a utility for managing disaster recovery, specifically for your Kubernetes cluster resources and persistent volumes. Brought to you by Heptio. - heptio/ark

Max Moon avatar
Max Moon

love ark

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

have you used it?

Max Moon avatar
Max Moon

yes indeedy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

was it easy to get up and running?

Max Moon avatar
Max Moon

oh yeah

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

are you guys using it now?

Max Moon avatar
Max Moon

not yet

Max Moon avatar
Max Moon

it was the backup tool of choice at my last company, used it on every single cluster, took me a morning to put in place

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
helm/charts

Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

Max Moon avatar
Max Moon

whenever i had issues with it, the main guy, andy, was very quick to help

Max Moon avatar
Max Moon

they have a channel on the main k8s slack

2018-09-14

rms1000watt avatar
rms1000watt

Really liking cloudposse/charts/monochart

1
rms1000watt avatar
rms1000watt
added extra conditions so stubs are not required in values.yaml by rms1000watt · Pull Request #171 · cloudposse/charts

This allows values.yaml to be simplified like: replicaCount: 1 image: repository: rms1000watt/dummy-golang-project tag: latest pullPolicy: IfNotPresent deployment: enabled: true service: …

2018-09-10

rohit.verma avatar
rohit.verma

can someone point me towards changing timezone on kops created instances

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
kubernetes/kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management - kubernetes/kops

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Add support for timezones / user data · Issue #1794 · kubernetes/kops

We have a need to create our kops nodes in PST rather than UTC. It would be helpful if kops either had an option to set the instance timezone via the ig config, or if ec2 user data could be passed …

2018-09-07

rms1000watt avatar
rms1000watt

to helm or not to helm… (as a n00b).. that is the question

Max Moon avatar
Max Moon

im a fan of helm, but i would suggest not using it until you’re extra comfy with kubectl and writing manifests yourself

:--1:1
Max Moon avatar
Max Moon

it can feel like a bit of a black box, even if you wrote the charts yourself

rms1000watt avatar
rms1000watt

Ah, interesting

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we recently did an engagememnt with caltech students in a research lab

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what worked well for them was to first write all the resources by hand to get comfy with it - the way @Max Moon describes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then write the charts later

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/charts

The “Cloud Posse” Distribution of Kubernetes Applications - cloudposse/charts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

which is our declarative helm chart. this means it will work for the most common use-cases and you won’t need to write a custom chart. you just define all the settings in values.yaml

rms1000watt avatar
rms1000watt

Going to start with this and see how far it’ll take us

rms1000watt avatar
rms1000watt

copy pasta ftw

Max Moon avatar
Max Moon

the above is what i recommend as well, every company i’ve used helm at, i’ve used the same approach

rms1000watt avatar
rms1000watt

That’s awesome–I appreciate the direction

Max Moon avatar
Max Moon

np!

Max Moon avatar
Max Moon

also think about your CICD tool(s) of choice

Max Moon avatar
Max Moon

for instance, last i used it, spinnaker was pretty opinionated and wanted control over deployments and deployment management, so we couldnt use helm

Max Moon avatar
Max Moon

that might have changed, but just something to keep in mind

Max Moon avatar
Max Moon

pretty much everything else should play along fine

rms1000watt avatar
rms1000watt

gotcha–yeah totally

rms1000watt avatar
rms1000watt

kind of doing some skunkwork at the moment

rms1000watt avatar
rms1000watt

Just want to get looking at the right stuff in the right way (that has worked for others)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mcuadros/terraform-provider-helm

Terraform Helm provider. Contribute to mcuadros/terraform-provider-helm development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

terraform alternative for helmfile

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hadn’t seen that before, but looks to be identical in interface to chamber

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
segmentio/chamber

CLI for managing secrets. Contribute to segmentio/chamber development by creating an account on GitHub.

:--1:1

2018-09-06

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@michal.matyjek have you guys tried https://github.com/hypnoglow/helm-s3

hypnoglow/helm-s3

Helm plugin that allows to use AWS S3 as a [private] chart repository. - hypnoglow/helm-s3

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

?

michal.matyjek avatar
michal.matyjek

we have not, the built-in codefresh one seems to be good enough for now

:--1:1

2018-09-03

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cmattoon/aws-ssm

Populates Kubernetes Secrets from AWS Parameter Store - cmattoon/aws-ssm

endofcake avatar
endofcake

Have you seen this project, Erik?

https://github.com/glassechidna/pstore

I liked how it transparently fetches the secrets when the container runs in AWS, but still allows you to set them directly when you work with docker-compose locally.

glassechidna/pstore

Environment variable-based AWS Parameter Store command shim - glassechidna/pstore

2018-09-02

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
clems4ever/authelia

Authentication server providing SSO, 2FA and ACLs for web apps. - clems4ever/authelia

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Good alternative for bitly oauth2 proxy?

    keyboard_arrow_up