#kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2019-10-29

Barani

Hi I need a help on creating configmap. resource “kubernetes_config_map” “env” { metadata { name =”tf-${var.project}-${var.component}-env” namespace = “${var.namespace}”

labels = { app = “tf-${var.project}-${var.component}” } }

data = { MINIO_ACCESS_KEY=”minio” MINIO_SECRET_KEY=”minio123” }

}

In the above I want to declare the values of data as variable and change it as per environment

.I am not able to declare it as string. Can someone please assist

variable “env_values” { type = string

} env_values = “MINIO_ACCESS_KEY="minio" \nMINIO_SECRET_KEY="minio123"”

I tried many possible combination but nothing works I tried using a file to declare all env variables and it worked but Minio is not picking the username in that way

Barani

Kindly give a suggestion

2019-10-28

2019-10-25

Thanks @Erik Osterman for the invite

Erik Osterman

Welcome @Jord!

Erik Osterman

Hey everyone! @Jord has a really neat product for learning kubernetes.

Erik Osterman

Clearly a lot of thought has gone into this.

Erik Osterman
Magic Sandbox | Next-gen Kubernetes Training

Magic Sandbox is a hands-on learning platform for engineers, by engineers. Immersive Kubernetes training on real infrastructure where engineering teams learn from hands-on Kubernetes training on real infra.

Thanks for the shout out - if you have any Qs just DM me or mail me at [email protected]

Hasan

I like MSB

2019-10-23

Chris Fowles

does anyone have an elegant solution to applying the stupid eks aws-auth config map via terraform without using a public endpoint on eks (and without being inside the vpc)? - i’m pretty sure this is pretty much technically impossible

Erik Osterman

using atlantis running in the vpc (or peer vpc), you can accomplish it.

Erik Osterman

we run atlantis inside of ECS fargate for this reason

Erik Osterman

but if the requirement is to apply it without being inside and without being outside, maybe look into aws ssm agent?

Chris Fowles

yeh - it’s a frustrating requirement in that i want to be able to stand up the environment and hook up roles so that things within that environment can manage itself and connect everything - but i can’t set up access to the cluster without being able to connect to the cluster. it would be nice if eks could bootstrap the rbac config on cluster creation or you could pass through a cluster admin role arn rather than just granting system:master to the user that created the cluster

Chris Fowles

hopefully that’s on the roadmap somewhere

2019-10-22

Taras

Hi guys,

Have just installed AWS EKS + autoscaller. All seem to be good except autoscaler failing with the error as follow:

E1021 18:40:49.320402       1 aws_manager.go:148] Failed to regenerate ASG cache: cannot autodiscover ASGs: RequestError: send request failed
caused by: Post <https://autoscaling.eu-west-2.amazonaws.com/>: dial tcp: i/o timeout
F1021 18:40:49.320431       1 aws_cloud_provider.go:330] Failed to create AWS Manager: cannot autodiscover ASGs: RequestError: send request failed
caused by: Post <https://autoscaling.eu-west-2.amazonaws.com/>: dial tcp: i/o timeout

Not sure why it can’t reach internal AWS’s API service. Autoscaller has been successfully installed using helm. Hence there is connectivity on the worker node. Any advices of what else shall I check?

Taras

ok. Resolved. dnsPolicy changed to Default and that is it.

Taras

Now another issue is that new nodes can’t attach to the cluster:

27s         Warning   ScaleUpTimedOut     configmap/cluster-autoscaler-status                               Nodes added to group londynek-02019102113431054090000000e failed to register within 5m5.36167321s
Taras

Ok. Resolved. Some subnets I put workers could not communicate to EKS cluster.

2019-10-18

Brandon Shutter
Brandon Shutter

Just deployed k8s via the k8s-workers module, everything is working great. Being able to add iam users and roles via terraform is amazing.

Brandon Shutter

Attempting to deploy a gitlab helm chart results in

Error creating load balancer (will retry): failed to ensure load balancer for service default/gitlab-nginx-ingress-controller: could not find any suitable subnets for creating the ELB
Brandon Shutter

I used CloudPosse’s VPC, Subnets, EKS, local.tag and EKS Workers modules

Brandon Shutter

I figured it out

4
Brandon Shutter

I needed to add the var.tags to the subnet module

aknysh

@Brandon Shutter thanks! Have you looked at this working example https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/main.tf

cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

aknysh
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

aknysh

I believe you are talking about these tags https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/main.tf#L19 (shared is required by EKS)

cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

2019-10-16

Chris Fowles

interested in thoughts - my thoughts are it sounds like it’s trying to separate dev and ops which i do not like

Erik Osterman
Announcing Cloud Native Application Bundle (CNAB) - Docker Blog

Learn from Docker experts to simplify and advance your app development and management with Docker. Stay up to date on Docker events and new version announcements!

Erik Osterman
CNAB: a spec for packaging distributed apps.

Cloud Native Application Bundles facilitate the bundling, installing and managing of container-native apps — and their coupled services.

Chris Fowles

sort of - it seems more like a way to implement an abstraction layer between teams of dev/ops/infra teams. cnab feels like more of a packaging tool kit to me, where this feels more like enterprise service catalogish kind of stuff (insert hand-waving)

Chris Fowles

while i understand the pain that’s driving the need, i’m not sure i’d like to deal with an environment where that was required

Chris Fowles

i’m also a little sick of abstractions over the kube apis that just look like the kube apis

1

2019-10-12

Erik Osterman
kubernetes/kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management - kubernetes/kops

Erik Osterman

this is what I was referring to

Erik Osterman

then there are some other terraform modules (not by us) that leverage this (i think)

2019-10-11

@Erik Osterman Thanks

I’m trying to pass encrypted values to secrets and use them as variables, will that work?

{{ (tpl (.Files.Glob “configs/*“).AsSecrets . ) indent 2 }}

Hey all, trying to set up kops in a new environment set up with the reference-architectures repo, so right now trying to run kops-aws-platform (https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops-aws-platform) and it seems it expects IAM roles like masters.us-west-2.testing.ryanjarv.sh and nodes.us-west-2.testing.ryanjarv.sh to be set up. Wondering if there is some step I missed that handles that.

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

Erik Osterman

those are provisioned by kops

Ok thanks will look into that. It did run ok but might need a more recent version or something.

Think I got it figured out, missed the extra steps here before. (https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops)

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

Erik Osterman

just so there’s no confusion we’re not using the terraform mode for kops

Erik Osterman

there are some other modules out there by others that do that

Erik Osterman

our module is for setting up the aws integration points that kops expects.

Terraform mode? Suppose I don’t know much to much about managing kops/k8s. Is that just managing individual pods with terraform? k8s in general still gets set up with the kops-aws-platform module right?

Edit: ok nvm seems the cluster itself is set up with kops.

2019-10-10

Austin Cawley-Edwards

Awesome, thank you both!

sarkis

cross posting from #security because it is relevant here: https://sweetops.slack.com/archives/CBXSAR45B/p1570720099000200

Kubernetes 'Billion Laughs' Vulnerability Is No Laughing Matter - The New Stack

A new vulnerability has been discovered within the Kubernetes API. This flaw is centered around the parsing of YAML manifests by the Kubernetes API server. During this process the API server is open to potential Denial of Service (DoS) attacks. The issue (CVE-2019-11253 — which has yet to have any details fleshed out on the page) has been labeled a ‘Billion Laughs’ attack because it targets the parsers to carry out the attack.

Michael Cram

This is why you always use a bastion host and isolate your cluster from everyone.

Kubernetes 'Billion Laughs' Vulnerability Is No Laughing Matter - The New Stack

A new vulnerability has been discovered within the Kubernetes API. This flaw is centered around the parsing of YAML manifests by the Kubernetes API server. During this process the API server is open to potential Denial of Service (DoS) attacks. The issue (CVE-2019-11253 — which has yet to have any details fleshed out on the page) has been labeled a ‘Billion Laughs’ attack because it targets the parsers to carry out the attack.

how to encrypt passwords in helm values.yaml, any good documents is appreciated. Thanks

Erik Osterman

I assume you’re referring to helm’s values.yaml

right

I used helm secrets to make sure passwords are hidden when pushed to code repositories

I was not sure about helm get values

can you please let me know other startegies

@Erik Osterman ^

Erik Osterman

@AG there’s the helm-secrets plugin that tries to address this

1
Erik Osterman

but secrets will still be clear-text in the if you run helm get values

Erik Osterman

(which is why you just can’t pass any secrets via helm that you truly care about)

Erik Osterman

instead, the better pattern is to assume the secrets have been installed some other way…. basically assume the resource already exists and don’t provision with helm

Erik Osterman

then when you install the chart release, it will block until that secret exists.

Erik Osterman

there are a few strategies for populating secrets

Erik Osterman

basically, you want to decouple the lifecycle of secrets with the lifecycle of helm releases

2019-10-09

Austin Cawley-Edwards

Hey all, not sure if this belongs in this channel so please let me know if it’s not the place, but I just opened up a neat feature PR for the cloudposse/prometheus-to-cloudwatch app - if anyone uses that and has some time to give some feedback I would really appreciate it, thanks! https://github.com/cloudposse/prometheus-to-cloudwatch/pull/28

feat: add ability to exclude dimensions per-metric by austince · Pull Request #28 · cloudposse/prometheus-to-cloudwatch

Closes #27 This feature allows users to exclude a set of dimensions from metrics. It should be easy enough to add a dimensions whitelist as well, which seems to be in the style of this application,…

3
Erik Osterman

@aknysh will review

Erik Osterman

@Austin Cawley-Edwards thanks for the contribution

2019-10-07

rms1000watt

@PePe @Cameron Boulton I love you

5
rms1000watt

like.. project is done already..

1
rms1000watt

global accelerator is amazing

1
1
1
Cameron Boulton

Yea, 80% of infra solutions are like this: people fall back on what they know and build these Rube Goldberg machines that have already been solved.

3

I’m glad it worked for you

1

2019-10-05

Erik Osterman

This should be possible today using simple nginx ingress with the right annotations

rms1000watt

it’s not available on k8s 1.14 which is the highest eks version

Erik Osterman

<http nlb <http “eipalloc-07e3afcd4b7b5d644,eipalloc-0d9cb0154be5ab55d,eipalloc-0e4e5ec3df81aa3ea”

Erik Osterman

Ah so need to run a newer version of k8s not supported by eks

Erik Osterman
EIP allocation for NLB Nginx-ingress · Issue #81421 · kubernetes/kubernetes

The issue points to the reported closed issue here : #63959 I tested this but its not working correctly and ingress is not respecting the annotations : I have hard time getting this working with NL…

maarten
Using static IP addresses for Application Load Balancers | Amazon Web Services

Introduction In August 2016, Elastic Load Balancing launched Application Load Balancer (ALB), which enable many layer 7 features for your HTTP traffic. People use Application Load Balancers because they scale automatically to adapt to changes in your traffic. This makes planning for growth easy, but it has a side effect of changing the IP addresses […]

rms1000watt

I referenced this one initially. It is an option i’m considering

Using static IP addresses for Application Load Balancers | Amazon Web Services

Introduction In August 2016, Elastic Load Balancing launched Application Load Balancer (ALB), which enable many layer 7 features for your HTTP traffic. People use Application Load Balancers because they scale automatically to adapt to changes in your traffic. This makes planning for growth easy, but it has a side effect of changing the IP addresses […]

rms1000watt

it’s pretty gnarly, but definitely last resort

rms1000watt

I appreciate you sharing this

it is so much easier to use global accelerator

1
rms1000watt

thank you, I’m taking a look. I haven’t heard of it before

2019-10-04

rms1000watt

I got a tricky one for you peeps.. At a high level, I need a static IP (Elastic IP) in front of a k8s service or ing.

aws-alb-ingress-controller doesn’t help since ALBs can’t use EIPs out of the box.. (yes, you can put an NLB in front of it.. and have a lambda function keep the NLB target group up to date the ALB IPs.. https://aws.amazon.com/blogs/networking-and-content-delivery/using-static-ip-addresses-for-application-load-balancers/)

Using nlb annotations in a svc is feature poor even with the latest version of EKS (k8s 1.14) and doesn’t properly attach EIPs to the NLB.

What else should I look at? Things that sound nice but I’ve never touched before (CRDs, Operators, etc..) could maybe help.. or not? What do you think?

roth.andy

Does it have to be an IP? Can it be a domain name? nginx-ingress controller works really well. Set up a domain in Route53 and use nginx-ingress controller, so your service is myservice.example.com, or whatever you want it to be.

rms1000watt

Yeah, IP. Someone needs to whitelist our IP for an integration.

Cameron Boulton

For inbound traffic @Ryan? As in the integration is going to PUSH to your IP?

rms1000watt

@Cameron Boulton exactly

Cameron Boulton

Huh. I agree with Pepe: Global Accelerator is probably your best bet.

rms1000watt

Was going to just stand up NLB -> ECS (with Traefik) -> ALB (DNS)

1
1
1
1
1
rms1000watt

interesting

rms1000watt

lemme take a look at that.. haven’t heard of it

rms1000watt

Alternatively.. I can use terraform to stand up an NLB + EIPs.. then use a lambda function or some code somewhere to constantly update the NLB target group with the results from kubectl get nodes

2019-10-03

2019-10-02

sohel2020

Does sweetops has any terraform module to create Kubernetes cluster using kops?

davidvasandani

No they use kops from the cli to provision kubernetes.

1
Alex Siegman

That’s true, however they still set up a lot of dependent resources with terraform. See:

https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops

and

https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops-aws-platform

and there’s other modules in that same repo to assist kops with some stuff.

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

Alex Siegman

but correct, no automation of kops itself

Erik Osterman

Ya we haven’t automated kops because what kops does it does better than terraform

Erik Osterman

It’s purpose built for managing the lifecycle of the cluster with the business logic of how to do updates. Terraform is more like a bulldozer.

2019-10-01

ruan.arcega

i am using terraform-aws-elasticsearch module in my stack and im loved it

ruan.arcega

from cloudposse repository congratulations to those involved!!

Erik Osterman

Awesome! We use that one all the time

Erik Osterman

It’s great with fluentd and k8s

ruan.arcega

yeah, so, i got some trouble, when kibana record the CNAME on route53, the path /_plugin/kibanamust not be part of the record.

there is a issue for it to fix: https://github.com/cloudposse/terraform-aws-elasticsearch/issues/14

kibana_hostname contains invalid records · Issue #14 · cloudposse/terraform-aws-elasticsearch

When dns_zone_id is supplied, the module attempts to create a CNAME Route53 record for the domain&#39;s Kibana endpoints. These endpoints look like &quot;xxx.<region>.es.amazonaws.com/_plugin

ruan.arcega

must be just [vpc-sb-shared-elasticsearch-6m6ftgtu6n74l3dh3drw3vwmvq.us-east-1.es.amazonaws.com](http://vpc-sb-shared-elasticsearch-6m6ftgtu6n74l3dh3drw3vwmvq.us-east-1.es.amazonaws.com)

Erik Osterman

@aknysh this looks like a bug

Erik Osterman

that’s odd though since we deploy this regularly

aknysh

this is a feature

aknysh
cloudposse/terraform-aws-elasticsearch

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash. - cloudposse/terraform-aws-elasticsearch

aknysh

and

aknysh
cloudposse/terraform-aws-elasticsearch

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash. - cloudposse/terraform-aws-elasticsearch

aknysh

use the same domain name [testing.cloudposse.co](http://testing.cloudposse.co)

aknysh
TestExamplesComplete 2019-07-28T22:37:01Z command.go:121: domain_hostname = [es-test.testing.cloudposse.co](http://es-test.testing.cloudposse.co)                                                        
TestExamplesComplete 2019-07-28T22:37:01Z command.go:121: kibana_hostname = [kibana-es-test.testing.cloudposse.co](http://kibana-es-test.testing.cloudposse.co)                                                 
aknysh

we don’t add /_plugin/kibana to it

aknysh

we add it in the helmfiles

aknysh

one of those could be removed since they point to the same thing

aknysh

[es-test.testing.cloudposse.co](http://es-test.testing.cloudposse.co) is the ES domain endpoint

Erik Osterman

right, but I think @ruan.arcega is saying the cname was created automatically with the /_plugin/kibana which is wrong

1
Erik Osterman
aknysh

[es-test.testing.cloudposse.co](http://es-test.testing.cloudposse.co) /_plugin/kibana would be the Kibana URL

Erik Osterman

right, but look at his screenshot from route53

aknysh

i see it. Maybe something is changed already in AWS. We deployed it last time a few months ago

Erik Osterman
Erik Osterman
Erik Osterman

so our DNS is pointing to the wrong output

Erik Osterman

should it be using domain_name

Erik Osterman

?

aknysh

domain_name is not URL

aknysh

it’s just the name of ES domain

aknysh

we have

[vpc-xxx-xxxxx-elasticsearch-xxxx.eu-west-2.es.amazonaws.com/_plugin/kibana/](http://vpc-xxx-xxxxx-elasticsearch-xxxx.eu-west-2.es.amazonaws.com/_plugin/kibana/)

as CNAME and it’s working

aknysh

(I mean AWS accepted the record before and accepting it now)

Erik Osterman

yea, so it’s accepting the record

Erik Osterman

but the record is still garbage

Erik Osterman

/ is invalid in DNS

1
aknysh
Type	Domain Name	Canonical Name	TTL
CNAME	[kibana-elasticsearch.eu-west-2.xxx.xxx.io>	<http://vpc-xxx-xxx-elasticsearch-xxxx.eu-west-2.es.amazonaws.com/_plugin/kibana/|vpc-xxx-xxx-elasticsearch-xxxx.eu-west-2.es.amazonaws.com/_plugin/kibana/](http://kibana-elasticsearch.eu-west-2.xxx.xxx.io)
aknysh

resolution works too

aknysh

but I agree since those are the same, one could be removed

    keyboard_arrow_up