#kubernetes (2018-09)

kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2018-09-28

Erik Osterman avatar
Erik Osterman

@pericdaniel are you referring to EKS?

Max Moon avatar
Max Moon

Does anyone else follow this blog? https://blog.jessfraz.com/ tons of good stuff regarding containers, security, and k8s

Jessie Frazelle's Blog: Home

The blog of Jessie Frazelle, hacker of all things Linux & Containers

Max Moon avatar
Max Moon

Same person is responsible for: https://github.com/genuinetools

genuinetools

We are the home of quality software with a focus on simplicity, usability, security and minimalism. No bullshit, no politics, all genuine. - genuinetools

Max Moon avatar
Max Moon

again, lots of really good/cool/interesting tools there too

aknysh avatar
aknysh

thanks for the links @Max Moon

Max Moon avatar
Max Moon

np

Erik Osterman avatar
Erik Osterman

She’s awesome! I have been following her for years on twitter

Erik Osterman avatar
Erik Osterman

She runs Coreos on the desktop

Erik Osterman avatar
Erik Osterman

Has containerized everything

Erik Osterman avatar
Erik Osterman
jessfraz/dockerfiles

Various Dockerfiles I use on the desktop and on servers. - jessfraz/dockerfiles

Erik Osterman avatar
Erik Osterman

On an unrelated note…

Erik Osterman avatar
Erik Osterman
gofunct/cloudnative-engineer

Contribute to gofunct/cloudnative-engineer development by creating an account on GitHub.

Erik Osterman avatar
Erik Osterman

Certified Kubernetes Administrator Exam Prep

2018-09-27

pericdaniel avatar
pericdaniel

i saw the eks tf files that were created

pericdaniel avatar
pericdaniel

do we have anything for aws-config

pericdaniel avatar
pericdaniel

aws-auth

aknysh avatar
aknysh

the eks project is being tested now and the modules will be updated to the latest version this week

aknysh avatar
aknysh

what are you doing with aws-config?

pericdaniel avatar
pericdaniel

for aws-auth

pericdaniel avatar
pericdaniel

i can authorize specific users to be able to make changes to the clusters and deploy environments

2018-09-26

Erik Osterman avatar
Erik Osterman
heptiolabs/eventrouter

A simple introspective kubernetes service that forwards events to a specified sink. - heptiolabs/eventrouter

Daren avatar
Daren

:–1:

heptiolabs/eventrouter

A simple introspective kubernetes service that forwards events to a specified sink. - heptiolabs/eventrouter

Erik Osterman avatar
Erik Osterman

(this was the project I was thinking of… came across it today by accident looking at heptio projects)

2018-09-23

Erik Osterman avatar
Erik Osterman
kubernetes/node-problem-detector

This is a place for various problem detectors running on the Kubernetes nodes. - kubernetes/node-problem-detector

Erik Osterman avatar
Erik Osterman


node-problem-detector aims to make various node problems visible to the upstream layers in cluster management stack. It is a daemon which runs on each node, detects node problems and reports them to apiserver. node-problem-detector can either run as a DaemonSet or run standalone. Now it is running as a Kubernetes Addon enabled by default in the GCE cluster.

Erik Osterman avatar
Erik Osterman

Works with draino

Erik Osterman avatar
Erik Osterman
negz/draino

Automatically cordon and drain Kubernetes nodes based on node conditions - negz/draino

:--1:1
Erik Osterman avatar
Erik Osterman


Draino automatically drains Kubernetes nodes based on labels and node conditions. Nodes that match all of the supplied labels and any of the supplied node conditions will be cordoned immediately and drained after a configurable drain-buffer time.

2018-09-21

Erik Osterman avatar
Erik Osterman
Provision AWS Services Through Kubernetes Using the AWS Service Broker | Amazon Web Services attachment image

使用 AWS Service Broker 通过 Kubernetes 配置 AWS 服务 There’s no doubt that containers have changed how we build projects. One of the guiding principles of a containerized workflow approach has been to give back control to the developer, allowing them to choose their dependencies and how to consume them – most importantly, when they […]

Erik Osterman avatar
Erik Osterman

“And that’s all folks” - wasn’t that easy? :P

Erik Osterman avatar
Erik Osterman

Joking aside - pretty cool. Basically let’s you provision AWS backing services from within Kubernetes

:--1:1
Erik Osterman avatar
Erik Osterman

H/T @fdrescher

fdrescher avatar
fdrescher
03:15:20 PM

@fdrescher has joined the channel

Erik Osterman avatar
Erik Osterman

We’ve released our EKS terraform modules for Kubernetes this week.

Welcome feedback

cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

cloudposse/terraform-aws-eks-workers

Terraform module to provision an AWS AutoScaling Group, IAM Role, and Security Group for EKS Workers - cloudposse/terraform-aws-eks-workers

rms1000watt avatar
rms1000watt

@stobiewankenobi ^^^

stobiewankenobi avatar
stobiewankenobi
07:27:11 PM

@stobiewankenobi has joined the channel

rms1000watt avatar
rms1000watt

it looks clean!

rms1000watt avatar
rms1000watt

@Erik Osterman should post on Reddit if you don’t already

Erik Osterman avatar
Erik Osterman

Thanks @rms1000watt yes - plan to do this soon

:--1:1
Erik Osterman avatar
Erik Osterman

I think @aknysh is doing some more testing today

Jeremy avatar
Jeremy

Hi there, I was wondering whether the prometheus-to-cloudwatch solution can be adapted to scape metrics from the metrics server instead of kube-state-metrics.

aknysh avatar
aknysh

ohh, the module was created just as an experiment, tested a little bit, and then forgotten (meaning not supported anymore b/c there many more official solutions to do prometeus-to-cloudwatch)

aknysh avatar
aknysh

@Erik Osterman can explain the whole situation

Jeremy avatar
Jeremy

thanks @aknysh. Can you direct me to where I can find those other solutions?

Jeremy avatar
Jeremy

I saw an exporter for exporting CW metrics to prometheus but not the other way around.

aknysh avatar
aknysh
prometheus/cloudwatch_exporter

Metrics exporter for Amazon AWS CloudWatch. Contribute to prometheus/cloudwatch_exporter development by creating an account on GitHub.

Initial experiences with the Prometheus monitoring system attachment image

I’ve been looking for a while for a monitoring system written in Go, self-contained and easy to deploy. I think I finally found what I was…

Jeremy avatar
Jeremy

So far, your solution was the only solution I found for exporting prometheus metrics to CW.

aknysh avatar
aknysh

@Erik Osterman ^

aknysh avatar
aknysh

this should be done in a more official way by using Prometheus Operator

aknysh avatar
aknysh

this tool to do it was already mentioned https://github.com/operator-framework

Operator Framework

The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. - Operator Framework

Jeremy avatar
Jeremy

That makes sense.

aknysh avatar
aknysh

Operator will allow a much better integration with Prometheus

aknysh avatar
aknysh

(but I agree, our tool is simpler )

aknysh avatar
aknysh

but @Jeremy, what you asked (to scape metrics from the metrics server instead of kube-state-metrics) could be done by installing kube-prometheus and then scraping it, no?

Jeremy avatar
Jeremy

yeah, i guess that’s what I’m asking. do i simply need to change the url in the values.yaml?

Jeremy avatar
Jeremy

(I haven’t had time to look at the code you’re using to scape the metrics yet)

aknysh avatar
aknysh
kubernetes/kube-state-metrics

Add-on agent to generate and expose cluster-level metrics. - kubernetes/kube-state-metrics

aknysh avatar
aknysh


Additionally, some monitoring systems such as Prometheus do not use Heapster(metrics-server) for metric collection at all and instead implement their own, but Prometheus can scrape metrics from heapster itself to alert on Heapster(metrics-server)’s health. Having kube-state-metrics as a separate project enables access to these metrics from those monitoring systems

Jeremy avatar
Jeremy

i’ll have a look. I’m interested in surfacing the metrics described in this blog https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-4-the-kubernetes-api-server-72f1e1210770 to CW.

A Deep Dive into Kubernetes Metrics — Part 4: The Kubernetes API Server attachment image

This is Part 4 of a multi-part series about all the metrics you can gather from your Kubernetes cluster.

Jeremy avatar
Jeremy

i believe i should be able to get these without running the metrics server.

aknysh avatar
aknysh

yea, it’s a very convoluted topic

aknysh avatar
aknysh

so I think you need to install Prometheus (via kube-prometheus https://github.com/coreos/prometheus-operator/tree/master/helm/kube-prometheus for example ) and then will be able to scrape it using a scraping tool

coreos/prometheus-operator

Prometheus Operator creates/configures/manages Prometheus clusters atop Kubernetes - coreos/prometheus-operator

aknysh avatar
aknysh

in our collection of helmfiles, we have examples on how to do it https://github.com/cloudposse/helmfiles/tree/master/helmfile.d

cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Erik Osterman avatar
Erik Osterman

@Jeremy Grodberg we’ve added better formatting for prometheus alerts. See this PR by @Igor Rodionov https://github.com/cloudposse/helmfiles/pull/48

Jeremy Grodberg avatar
Jeremy Grodberg
09:37:53 PM

@Jeremy Grodberg has joined the channel

Erik Osterman avatar
Erik Osterman

Also, deploying Grafana dashboards with configmaps: https://github.com/cloudposse/helmfiles/pull/18

[grafana] Use sidecar for datasources and dashboards by osterman · Pull Request #18 · cloudposse/helmfiles

what Update to use sidecar pattern Provide integration with kube-prometheus (collecting metrics / import grafana dashboards) Collect metrics for nginx ingress and display them Fix Portal Fix nginx…

aknysh avatar
aknysh

I think it’s a different Jeremy :)

Erik Osterman avatar
Erik Osterman

@Jeremy Grodberg is with PopChest <— using our older versions of kube-prometheus

Jeremy Grodberg avatar
Jeremy Grodberg

But, @Erik Osterman There is also @Jeremy Cowan, who I think you meant to be referring to.

Erik Osterman avatar
Erik Osterman

ahk

Erik Osterman avatar
Erik Osterman

I see.

Erik Osterman avatar
Erik Osterman

well, welcome to #kubernetes

2018-09-19

Erik Osterman avatar
Erik Osterman

for context, we’re moving this discussion here: https://sweetops.slack.com/archives/CB2PXUHLL/p1537386287000100

I’ve literally never had this issue before, docker image built on my local machine gets uploaded to ECR and when i deploy that image, it comes across corrupt with the exact same configuration and it makes 0 sense. It appears my image being uploaded is corrupt and i’ve been troubleshooting for hours

Max Moon avatar
Max Moon

I’d be curious to know, are you getting that composer error from the log from kubectl logs <pod name> or elsewhere?

Matthew avatar
Matthew

Getting it from the container that is being build

Matthew avatar
Matthew

built*

Matthew avatar
Matthew

image: report-portal:develop built locally works 100% of the time. Deployed using this, somehow libraries are lost and dropped.

Erik Osterman avatar
Erik Osterman

what happens if you pull and run that ECR image locally?

Matthew avatar
Matthew

When i build the ECR image locally, same error is produced

Matthew avatar
Matthew

But when I build the image that is being uploaded, it works smoothly

Matthew avatar
Matthew

I have 0 idea why my ECR image would be corrupt but the one building that image works

aknysh avatar
aknysh

@Matthew please explain it again step-by-step for people to be able to help you, something like this…

aknysh avatar
aknysh
  1. I have a Dockerfile (show it here) which I build locally and then start the container and it works locally
aknysh avatar
aknysh
  1. Then I push the already built image to ECR manually
aknysh avatar
aknysh
  1. When the image gets deployed from the ECR repo to Kubernetes, it throws errors
aknysh avatar
aknysh

explain what works and what does not, where you build it and how

2018-09-18

Max Moon avatar
Max Moon

curious if anyone has given creating an Operator themselves a go: https://github.com/operator-framework

Operator Framework

The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. - Operator Framework

Max Moon avatar
Max Moon

chances are you are (either knowingly or unknowingly) already using one or many in your cluster

Erik Osterman avatar
Erik Osterman

we’re using all the prometheus operators in our latest rollouts

Erik Osterman avatar
Erik Osterman

I would love to see a Terraform operator

Erik Osterman avatar
Erik Osterman

haven’t yet considered taking the plunge to write one

Erik Osterman avatar
Erik Osterman

is there something in particular you want to build?

Max Moon avatar
Max Moon

nothing concrete yet, have been trying to think of some ideas

2018-09-17

Erik Osterman avatar
Erik Osterman
heptio/ark

Heptio Ark is a utility for managing disaster recovery, specifically for your Kubernetes cluster resources and persistent volumes. Brought to you by Heptio. - heptio/ark

Max Moon avatar
Max Moon

love ark

Erik Osterman avatar
Erik Osterman

have you used it?

Max Moon avatar
Max Moon

yes indeedy

Erik Osterman avatar
Erik Osterman

was it easy to get up and running?

Max Moon avatar
Max Moon

oh yeah

Erik Osterman avatar
Erik Osterman

are you guys using it now?

Max Moon avatar
Max Moon

not yet

Max Moon avatar
Max Moon

it was the backup tool of choice at my last company, used it on every single cluster, took me a morning to put in place

Erik Osterman avatar
Erik Osterman
helm/charts

Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

Max Moon avatar
Max Moon

whenever i had issues with it, the main guy, andy, was very quick to help

Max Moon avatar
Max Moon

they have a channel on the main k8s slack

2018-09-14

rms1000watt avatar
rms1000watt

Really liking cloudposse/charts/monochart

1
rms1000watt avatar
rms1000watt
added extra conditions so stubs are not required in values.yaml by rms1000watt · Pull Request #171 · cloudposse/charts

This allows values.yaml to be simplified like: replicaCount: 1 image: repository: rms1000watt/dummy-golang-project tag: latest pullPolicy: IfNotPresent deployment: enabled: true service: …

2018-09-10

rohit.verma avatar
rohit.verma

can someone point me towards changing timezone on kops created instances

aknysh avatar
aknysh
kubernetes/kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management - kubernetes/kops

aknysh avatar
aknysh
Add support for timezones / user data · Issue #1794 · kubernetes/kops

We have a need to create our kops nodes in PST rather than UTC. It would be helpful if kops either had an option to set the instance timezone via the ig config, or if ec2 user data could be passed …

2018-09-07

rms1000watt avatar
rms1000watt

to helm or not to helm… (as a n00b).. that is the question

Max Moon avatar
Max Moon

im a fan of helm, but i would suggest not using it until you’re extra comfy with kubectl and writing manifests yourself

:--1:1
Max Moon avatar
Max Moon

it can feel like a bit of a black box, even if you wrote the charts yourself

rms1000watt avatar
rms1000watt

Ah, interesting

Erik Osterman avatar
Erik Osterman

so we recently did an engagememnt with caltech students in a research lab

Erik Osterman avatar
Erik Osterman

what worked well for them was to first write all the resources by hand to get comfy with it - the way @Max Moon describes

Erik Osterman avatar
Erik Osterman

then write the charts later

Erik Osterman avatar
Erik Osterman
cloudposse/charts

The “Cloud Posse” Distribution of Kubernetes Applications - cloudposse/charts

Erik Osterman avatar
Erik Osterman

which is our declarative helm chart. this means it will work for the most common use-cases and you won’t need to write a custom chart. you just define all the settings in values.yaml

rms1000watt avatar
rms1000watt

Going to start with this and see how far it’ll take us

rms1000watt avatar
rms1000watt

copy pasta ftw

Max Moon avatar
Max Moon

the above is what i recommend as well, every company i’ve used helm at, i’ve used the same approach

rms1000watt avatar
rms1000watt

That’s awesome–I appreciate the direction

Max Moon avatar
Max Moon

np!

Max Moon avatar
Max Moon

also think about your CICD tool(s) of choice

Max Moon avatar
Max Moon

for instance, last i used it, spinnaker was pretty opinionated and wanted control over deployments and deployment management, so we couldnt use helm

Max Moon avatar
Max Moon

that might have changed, but just something to keep in mind

Max Moon avatar
Max Moon

pretty much everything else should play along fine

rms1000watt avatar
rms1000watt

gotcha–yeah totally

rms1000watt avatar
rms1000watt

kind of doing some skunkwork at the moment

rms1000watt avatar
rms1000watt

Just want to get looking at the right stuff in the right way (that has worked for others)

Erik Osterman avatar
Erik Osterman
mcuadros/terraform-provider-helm

Terraform Helm provider. Contribute to mcuadros/terraform-provider-helm development by creating an account on GitHub.

Erik Osterman avatar
Erik Osterman

terraform alternative for helmfile

Erik Osterman avatar
Erik Osterman

hadn’t seen that before, but looks to be identical in interface to chamber

Erik Osterman avatar
Erik Osterman
segmentio/chamber

CLI for managing secrets. Contribute to segmentio/chamber development by creating an account on GitHub.

:--1:1

2018-09-06

Erik Osterman avatar
Erik Osterman

@michal.matyjek have you guys tried https://github.com/hypnoglow/helm-s3

hypnoglow/helm-s3

Helm plugin that allows to use AWS S3 as a [private] chart repository. - hypnoglow/helm-s3

Erik Osterman avatar
Erik Osterman

?

michal.matyjek avatar
michal.matyjek

we have not, the built-in codefresh one seems to be good enough for now

:--1:1

2018-09-03

Erik Osterman avatar
Erik Osterman
cmattoon/aws-ssm

Populates Kubernetes Secrets from AWS Parameter Store - cmattoon/aws-ssm

endofcake avatar
endofcake

Have you seen this project, Erik?

https://github.com/glassechidna/pstore

I liked how it transparently fetches the secrets when the container runs in AWS, but still allows you to set them directly when you work with docker-compose locally.

glassechidna/pstore

Environment variable-based AWS Parameter Store command shim - glassechidna/pstore

2018-09-02

Erik Osterman avatar
Erik Osterman
clems4ever/authelia

Authentication server providing SSO, 2FA and ACLs for web apps. - clems4ever/authelia

Erik Osterman avatar
Erik Osterman

Good alternative for bitly oauth2 proxy?

    keyboard_arrow_up