#kubernetes (2019-10)

kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2019-10-29

Barani avatar
Barani

Hi I need a help on creating configmap. resource “kubernetes_config_map” “env” { metadata { name =”tf-${var.project}-${var.component}-env” namespace = “${var.namespace}”

labels = { app = “tf-${var.project}-${var.component}” } }

data = { MINIO_ACCESS_KEY=”minio” MINIO_SECRET_KEY=”minio123” }

}

In the above I want to declare the values of data as variable and change it as per environment

.I am not able to declare it as string. Can someone please assist

variable “env_values” { type = string

} env_values = “MINIO_ACCESS_KEY="minio" \nMINIO_SECRET_KEY="minio123"”

I tried many possible combination but nothing works I tried using a file to declare all env variables and it worked but Minio is not picking the username in that way

Barani avatar
Barani

Kindly give a suggestion

2019-10-28

2019-10-25

Jord avatar

Thanks @Erik Osterman for the invite

Erik Osterman avatar
Erik Osterman

Welcome @Jord!

Erik Osterman avatar
Erik Osterman

Hey everyone! @Jord has a really neat product for learning kubernetes.

Erik Osterman avatar
Erik Osterman

Clearly a lot of thought has gone into this.

Erik Osterman avatar
Erik Osterman
Magic Sandbox | Next-gen Kubernetes Training

Magic Sandbox is a hands-on learning platform for engineers, by engineers. Immersive Kubernetes training on real infrastructure where engineering teams learn from hands-on Kubernetes training on real infra.

Jord avatar

Thanks for the shout out - if you have any Qs just DM me or mail me at [email protected]

Hasan avatar
Hasan

I like MSB

2019-10-23

Chris Fowles avatar
Chris Fowles

does anyone have an elegant solution to applying the stupid eks aws-auth config map via terraform without using a public endpoint on eks (and without being inside the vpc)? - i’m pretty sure this is pretty much technically impossible

Erik Osterman avatar
Erik Osterman

using atlantis running in the vpc (or peer vpc), you can accomplish it.

Erik Osterman avatar
Erik Osterman

we run atlantis inside of ECS fargate for this reason

Erik Osterman avatar
Erik Osterman

but if the requirement is to apply it without being inside and without being outside, maybe look into aws ssm agent?

Chris Fowles avatar
Chris Fowles

yeh - it’s a frustrating requirement in that i want to be able to stand up the environment and hook up roles so that things within that environment can manage itself and connect everything - but i can’t set up access to the cluster without being able to connect to the cluster. it would be nice if eks could bootstrap the rbac config on cluster creation or you could pass through a cluster admin role arn rather than just granting system:master to the user that created the cluster

Chris Fowles avatar
Chris Fowles

hopefully that’s on the roadmap somewhere

2019-10-22

Taras avatar
Taras

Hi guys,

Have just installed AWS EKS + autoscaller. All seem to be good except autoscaler failing with the error as follow:

E1021 18:40:49.320402       1 aws_manager.go:148] Failed to regenerate ASG cache: cannot autodiscover ASGs: RequestError: send request failed
caused by: Post <https://autoscaling.eu-west-2.amazonaws.com/>: dial tcp: i/o timeout
F1021 18:40:49.320431       1 aws_cloud_provider.go:330] Failed to create AWS Manager: cannot autodiscover ASGs: RequestError: send request failed
caused by: Post <https://autoscaling.eu-west-2.amazonaws.com/>: dial tcp: i/o timeout

Not sure why it can’t reach internal AWS’s API service. Autoscaller has been successfully installed using helm. Hence there is connectivity on the worker node. Any advices of what else shall I check?

Taras avatar
Taras

ok. Resolved. dnsPolicy changed to Default and that is it.

Taras avatar
Taras

Now another issue is that new nodes can’t attach to the cluster:

27s         Warning   ScaleUpTimedOut     configmap/cluster-autoscaler-status                               Nodes added to group londynek-02019102113431054090000000e failed to register within 5m5.36167321s
Taras avatar
Taras

Ok. Resolved. Some subnets I put workers could not communicate to EKS cluster.

2019-10-18

Brandon Shutter avatar
Brandon Shutter
Brandon Shutter avatar
Brandon Shutter

Just deployed k8s via the k8s-workers module, everything is working great. Being able to add iam users and roles via terraform is amazing.

Brandon Shutter avatar
Brandon Shutter

Attempting to deploy a gitlab helm chart results in

Error creating load balancer (will retry): failed to ensure load balancer for service default/gitlab-nginx-ingress-controller: could not find any suitable subnets for creating the ELB
Brandon Shutter avatar
Brandon Shutter

I used CloudPosse’s VPC, Subnets, EKS, local.tag and EKS Workers modules

Brandon Shutter avatar
Brandon Shutter

I figured it out

4
Brandon Shutter avatar
Brandon Shutter

I needed to add the var.tags to the subnet module

aknysh avatar
aknysh

@Brandon Shutter thanks! Have you looked at this working example https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/main.tf

cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

aknysh avatar
aknysh
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

aknysh avatar
aknysh

I believe you are talking about these tags https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/main.tf#L19 (shared is required by EKS)

cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

2019-10-16

Chris Fowles avatar
Chris Fowles

interested in thoughts - my thoughts are it sounds like it’s trying to separate dev and ops which i do not like

Erik Osterman avatar
Erik Osterman
Announcing Cloud Native Application Bundle (CNAB) - Docker Blog attachment image

Learn from Docker experts to simplify and advance your app development and management with Docker. Stay up to date on Docker events and new version announcements!

Erik Osterman avatar
Erik Osterman
CNAB: a spec for packaging distributed apps. attachment image

Cloud Native Application Bundles facilitate the bundling, installing and managing of container-native apps — and their coupled services.

Chris Fowles avatar
Chris Fowles

sort of - it seems more like a way to implement an abstraction layer between teams of dev/ops/infra teams. cnab feels like more of a packaging tool kit to me, where this feels more like enterprise service catalogish kind of stuff (insert hand-waving)

Chris Fowles avatar
Chris Fowles

while i understand the pain that’s driving the need, i’m not sure i’d like to deal with an environment where that was required

Chris Fowles avatar
Chris Fowles

i’m also a little sick of abstractions over the kube apis that just look like the kube apis

1

2019-10-12

Erik Osterman avatar
Erik Osterman
kubernetes/kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management - kubernetes/kops

Erik Osterman avatar
Erik Osterman

this is what I was referring to

Erik Osterman avatar
Erik Osterman

then there are some other terraform modules (not by us) that leverage this (i think)

2019-10-11

AG avatar

@Erik Osterman Thanks

AG avatar

I’m trying to pass encrypted values to secrets and use them as variables, will that work?

AG avatar
{{ (tpl (.Files.Glob “configs/*“).AsSecrets . ) indent 2 }}
jarv avatar

Hey all, trying to set up kops in a new environment set up with the reference-architectures repo, so right now trying to run kops-aws-platform (https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops-aws-platform) and it seems it expects IAM roles like masters.us-west-2.testing.ryanjarv.sh and nodes.us-west-2.testing.ryanjarv.sh to be set up. Wondering if there is some step I missed that handles that.

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

Erik Osterman avatar
Erik Osterman

those are provisioned by kops

jarv avatar

Ok thanks will look into that. It did run ok but might need a more recent version or something.

jarv avatar

Think I got it figured out, missed the extra steps here before. (https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops)

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

Erik Osterman avatar
Erik Osterman

just so there’s no confusion we’re not using the terraform mode for kops

Erik Osterman avatar
Erik Osterman

there are some other modules out there by others that do that

Erik Osterman avatar
Erik Osterman

our module is for setting up the aws integration points that kops expects.

jarv avatar

Terraform mode? Suppose I don’t know much to much about managing kops/k8s. Is that just managing individual pods with terraform? k8s in general still gets set up with the kops-aws-platform module right?

Edit: ok nvm seems the cluster itself is set up with kops.

2019-10-10

Austin Cawley-Edwards avatar
Austin Cawley-Edwards

Awesome, thank you both!

sarkis avatar
sarkis

cross posting from #security because it is relevant here: https://sweetops.slack.com/archives/CBXSAR45B/p1570720099000200

Kubernetes 'Billion Laughs' Vulnerability Is No Laughing Matter - The New Stack attachment image

A new vulnerability has been discovered within the Kubernetes API. This flaw is centered around the parsing of YAML manifests by the Kubernetes API server. During this process the API server is open to potential Denial of Service (DoS) attacks. The issue (CVE-2019-11253 — which has yet to have any details fleshed out on the page) has been labeled a ‘Billion Laughs’ attack because it targets the parsers to carry out the attack.

Michael Cram avatar
Michael Cram

This is why you always use a bastion host and isolate your cluster from everyone.

Kubernetes 'Billion Laughs' Vulnerability Is No Laughing Matter - The New Stack attachment image

A new vulnerability has been discovered within the Kubernetes API. This flaw is centered around the parsing of YAML manifests by the Kubernetes API server. During this process the API server is open to potential Denial of Service (DoS) attacks. The issue (CVE-2019-11253 — which has yet to have any details fleshed out on the page) has been labeled a ‘Billion Laughs’ attack because it targets the parsers to carry out the attack.

AG avatar

how to encrypt passwords in helm values.yaml, any good documents is appreciated. Thanks

Erik Osterman avatar
Erik Osterman

I assume you’re referring to helm’s values.yaml

AG avatar

right

AG avatar

I used helm secrets to make sure passwords are hidden when pushed to code repositories

AG avatar

I was not sure about helm get values

AG avatar

can you please let me know other startegies

AG avatar

@Erik Osterman ^

Erik Osterman avatar
Erik Osterman

@AG there’s the helm-secrets plugin that tries to address this

:--1:1
Erik Osterman avatar
Erik Osterman

but secrets will still be clear-text in the if you run helm get values

Erik Osterman avatar
Erik Osterman

(which is why you just can’t pass any secrets via helm that you truly care about)

Erik Osterman avatar
Erik Osterman

instead, the better pattern is to assume the secrets have been installed some other way…. basically assume the resource already exists and don’t provision with helm

Erik Osterman avatar
Erik Osterman

then when you install the chart release, it will block until that secret exists.

Erik Osterman avatar
Erik Osterman

there are a few strategies for populating secrets

Erik Osterman avatar
Erik Osterman

basically, you want to decouple the lifecycle of secrets with the lifecycle of helm releases

2019-10-09

Austin Cawley-Edwards avatar
Austin Cawley-Edwards

Hey all, not sure if this belongs in this channel so please let me know if it’s not the place, but I just opened up a neat feature PR for the cloudposse/prometheus-to-cloudwatch app - if anyone uses that and has some time to give some feedback I would really appreciate it, thanks! https://github.com/cloudposse/prometheus-to-cloudwatch/pull/28

feat: add ability to exclude dimensions per-metric by austince · Pull Request #28 · cloudposse/prometheus-to-cloudwatch

Closes #27 This feature allows users to exclude a set of dimensions from metrics. It should be easy enough to add a dimensions whitelist as well, which seems to be in the style of this application,…

:--1:3
Erik Osterman avatar
Erik Osterman

@aknysh will review

Erik Osterman avatar
Erik Osterman

@Austin Cawley-Edwards thanks for the contribution

2019-10-07

rms1000watt avatar
rms1000watt

@PePe @Cameron Boulton I love you

5
rms1000watt avatar
rms1000watt

like.. project is done already..

1
rms1000watt avatar
rms1000watt

global accelerator is amazing

1
1
1
Cameron Boulton avatar
Cameron Boulton

Yea, 80% of infra solutions are like this: people fall back on what they know and build these Rube Goldberg machines that have already been solved.

:100:3
PePe avatar

I’m glad it worked for you

:100:1

2019-10-05

Erik Osterman avatar
Erik Osterman

This should be possible today using simple nginx ingress with the right annotations

rms1000watt avatar
rms1000watt

it’s not available on k8s 1.14 which is the highest eks version

Erik Osterman avatar
Erik Osterman

service.beta.kubernetes.io/aws-load-balancer-type: nlb service.beta.kubernetes.io/aws-load-balancer-eip-allocations: “eipalloc-07e3afcd4b7b5d644,eipalloc-0d9cb0154be5ab55d,eipalloc-0e4e5ec3df81aa3ea”

Erik Osterman avatar
Erik Osterman

Ah so need to run a newer version of k8s not supported by eks

Erik Osterman avatar
Erik Osterman
EIP allocation for NLB Nginx-ingress · Issue #81421 · kubernetes/kubernetes

The issue points to the reported closed issue here : #63959 I tested this but its not working correctly and ingress is not respecting the annotations : I have hard time getting this working with NL…

maarten avatar
maarten
Using static IP addresses for Application Load Balancers | Amazon Web Services attachment image

Introduction In August 2016, Elastic Load Balancing launched Application Load Balancer (ALB), which enable many layer 7 features for your HTTP traffic. People use Application Load Balancers because they scale automatically to adapt to changes in your traffic. This makes planning for growth easy, but it has a side effect of changing the IP addresses […]

rms1000watt avatar
rms1000watt

I referenced this one initially. It is an option i’m considering

Using static IP addresses for Application Load Balancers | Amazon Web Services attachment image

Introduction In August 2016, Elastic Load Balancing launched Application Load Balancer (ALB), which enable many layer 7 features for your HTTP traffic. People use Application Load Balancers because they scale automatically to adapt to changes in your traffic. This makes planning for growth easy, but it has a side effect of changing the IP addresses […]

rms1000watt avatar
rms1000watt

it’s pretty gnarly, but definitely last resort

rms1000watt avatar
rms1000watt

I appreciate you sharing this

PePe avatar

it is so much easier to use global accelerator

:--1:1
rms1000watt avatar
rms1000watt

thank you, I’m taking a look. I haven’t heard of it before

2019-10-04

rms1000watt avatar
rms1000watt

I got a tricky one for you peeps.. At a high level, I need a static IP (Elastic IP) in front of a k8s service or ing.

aws-alb-ingress-controller doesn’t help since ALBs can’t use EIPs out of the box.. (yes, you can put an NLB in front of it.. and have a lambda function keep the NLB target group up to date the ALB IPs.. https://aws.amazon.com/blogs/networking-and-content-delivery/using-static-ip-addresses-for-application-load-balancers/)

Using nlb annotations in a svc is feature poor even with the latest version of EKS (k8s 1.14) and doesn’t properly attach EIPs to the NLB.

What else should I look at? Things that sound nice but I’ve never touched before (CRDs, Operators, etc..) could maybe help.. or not? What do you think?

roth.andy avatar
roth.andy

Does it have to be an IP? Can it be a domain name? nginx-ingress controller works really well. Set up a domain in Route53 and use nginx-ingress controller, so your service is myservice.example.com, or whatever you want it to be.

rms1000watt avatar
rms1000watt

Yeah, IP. Someone needs to whitelist our IP for an integration.

Cameron Boulton avatar
Cameron Boulton

For inbound traffic @Ryan? As in the integration is going to PUSH to your IP?

rms1000watt avatar
rms1000watt

@Cameron Boulton exactly

Cameron Boulton avatar
Cameron Boulton

Huh. I agree with Pepe: Global Accelerator is probably your best bet.

rms1000watt avatar
rms1000watt

Was going to just stand up NLB -> ECS (with Traefik) -> ALB (DNS)

1
1
1
1
1
rms1000watt avatar
rms1000watt

interesting

rms1000watt avatar
rms1000watt

lemme take a look at that.. haven’t heard of it

rms1000watt avatar
rms1000watt

Alternatively.. I can use terraform to stand up an NLB + EIPs.. then use a lambda function or some code somewhere to constantly update the NLB target group with the results from kubectl get nodes

2019-10-03

2019-10-02

sohel2020 avatar
sohel2020

Does sweetops has any terraform module to create Kubernetes cluster using kops?

davidvasandani avatar
davidvasandani

No they use kops from the cli to provision kubernetes.

:--1:1
Alex Siegman avatar
Alex Siegman

That’s true, however they still set up a lot of dependent resources with terraform. See:

https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops

and

https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops-aws-platform

and there’s other modules in that same repo to assist kops with some stuff.

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

Alex Siegman avatar
Alex Siegman

but correct, no automation of kops itself

Erik Osterman avatar
Erik Osterman

Ya we haven’t automated kops because what kops does it does better than terraform

Erik Osterman avatar
Erik Osterman

It’s purpose built for managing the lifecycle of the cluster with the business logic of how to do updates. Terraform is more like a bulldozer.

2019-10-01

ruan.arcega avatar
ruan.arcega

i am using terraform-aws-elasticsearch module in my stack and im loved it

ruan.arcega avatar
ruan.arcega

from cloudposse repository congratulations to those involved!!

Erik Osterman avatar
Erik Osterman

Awesome! We use that one all the time

Erik Osterman avatar
Erik Osterman

It’s great with fluentd and k8s

ruan.arcega avatar
ruan.arcega

yeah, so, i got some trouble, when kibana record the CNAME on route53, the path /_plugin/kibanamust not be part of the record.

there is a issue for it to fix: https://github.com/cloudposse/terraform-aws-elasticsearch/issues/14

kibana_hostname contains invalid records · Issue #14 · cloudposse/terraform-aws-elasticsearch

When dns_zone_id is supplied, the module attempts to create a CNAME Route53 record for the domain&#39;s Kibana endpoints. These endpoints look like &quot;xxx.<region>.es.amazonaws.com/_plugin

ruan.arcega avatar
ruan.arcega

must be just [vpc-sb-shared-elasticsearch-6m6ftgtu6n74l3dh3drw3vwmvq.us-east-1.es.amazonaws.com](http://vpc-sb-shared-elasticsearch-6m6ftgtu6n74l3dh3drw3vwmvq.us-east-1.es.amazonaws.com)

Erik Osterman avatar
Erik Osterman

@aknysh this looks like a bug

Erik Osterman avatar
Erik Osterman

that’s odd though since we deploy this regularly

aknysh avatar
aknysh

this is a feature

aknysh avatar
aknysh
cloudposse/terraform-aws-elasticsearch

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash. - cloudposse/terraform-aws-elasticsearch

aknysh avatar
aknysh

and

aknysh avatar
aknysh
cloudposse/terraform-aws-elasticsearch

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash. - cloudposse/terraform-aws-elasticsearch

aknysh avatar
aknysh

use the same domain name [testing.cloudposse.co](http://testing.cloudposse.co)

aknysh avatar
aknysh
TestExamplesComplete 2019-07-28T22:37:01Z command.go:121: domain_hostname = [es-test.testing.cloudposse.co](http://es-test.testing.cloudposse.co)                                                        
TestExamplesComplete 2019-07-28T22:37:01Z command.go:121: kibana_hostname = [kibana-es-test.testing.cloudposse.co](http://kibana-es-test.testing.cloudposse.co)                                                 
aknysh avatar
aknysh

we don’t add /_plugin/kibana to it

aknysh avatar
aknysh

we add it in the helmfiles

aknysh avatar
aknysh

one of those could be removed since they point to the same thing

aknysh avatar
aknysh

[es-test.testing.cloudposse.co](http://es-test.testing.cloudposse.co) is the ES domain endpoint

Erik Osterman avatar
Erik Osterman

right, but I think @ruan.arcega is saying the cname was created automatically with the /_plugin/kibana which is wrong

:--1:1
Erik Osterman avatar
Erik Osterman
aknysh avatar
aknysh

[es-test.testing.cloudposse.co](http://es-test.testing.cloudposse.co) /_plugin/kibana would be the Kibana URL

Erik Osterman avatar
Erik Osterman

right, but look at his screenshot from route53

aknysh avatar
aknysh

i see it. Maybe something is changed already in AWS. We deployed it last time a few months ago

Erik Osterman avatar
Erik Osterman
Erik Osterman avatar
Erik Osterman
Erik Osterman avatar
Erik Osterman

so our DNS is pointing to the wrong output

Erik Osterman avatar
Erik Osterman

should it be using domain_name

Erik Osterman avatar
Erik Osterman

?

aknysh avatar
aknysh

domain_name is not URL

aknysh avatar
aknysh

it’s just the name of ES domain

aknysh avatar
aknysh

we have

[vpc-xxx-xxxxx-elasticsearch-xxxx.eu-west-2.es.amazonaws.com/_plugin/kibana/](http://vpc-xxx-xxxxx-elasticsearch-xxxx.eu-west-2.es.amazonaws.com/_plugin/kibana/)

as CNAME and it’s working

aknysh avatar
aknysh

(I mean AWS accepted the record before and accepting it now)

Erik Osterman avatar
Erik Osterman

yea, so it’s accepting the record

Erik Osterman avatar
Erik Osterman

but the record is still garbage

Erik Osterman avatar
Erik Osterman

/ is invalid in DNS

:--1:1
aknysh avatar
aknysh
Type	Domain Name	Canonical Name	TTL
CNAME	[kibana-elasticsearch.eu-west-2.xxx.xxx.io>	<http://vpc-xxx-xxx-elasticsearch-xxxx.eu-west-2.es.amazonaws.com/_plugin/kibana/|vpc-xxx-xxx-elasticsearch-xxxx.eu-west-2.es.amazonaws.com/_plugin/kibana/](http://kibana-elasticsearch.eu-west-2.xxx.xxx.io)
aknysh avatar
aknysh

resolution works too

aknysh avatar
aknysh

but I agree since those are the same, one could be removed

    keyboard_arrow_up