#office-hours (2020-04)

Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cpco.io/slack-office-hours

Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cpco.io/slack-office-hours

2020-04-08

tomkinson avatar
tomkinson

Should we post questions here if we can’t join with video?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

btw, video totally optional

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You can ask it here, but it will be easier if you’re ont he call so it can be interactive.

tomkinson avatar
tomkinson

Just jumping in car. Post question in 2 secs

tomkinson avatar
tomkinson

In meantime as I type you might like http://getcommandeer.com/

Commandeer - The Desktop Cloud IDE attachment image

Commandeer is a Desktop App used to manage your cloud resources. It enables you to manage, AWS Dynamo, SNS, S3, SQS, SNS, IAM and much mroe all from the comfort of your desktop.

Todd Lyons avatar
Todd Lyons

A group of SRE’s in our company wrote this as a templating system useful for getting things out of vault and into helm deploys: https://github.com/PremiereGlobal/stim Goal is simplify Jenkinsfiles that are using Jenkins credentials to construct and run helm commands.

PremiereGlobal/stim

Speeding up development with glue that brings tools together - PremiereGlobal/stim

tomkinson avatar
tomkinson

So we are trying to deploy a HA RabbitMQ deployment using Rancher, K8s and Docker.

We cannot open the port for RMQ websockets (or at very least it will not connect) port 15674.

The load balancer is landing on the proper port but will not fwd to the proper port. It does not detect anything on the other end. It is not HTTP and what is interesting is that the admin section of RMQ is HTTP, and it’s port works.

It is TCP and we use WebSTOMP client library.

We see they port is open on the instance but the ingress LB will not connect. We tried to strip it down still no luck. Basically a 503.

We tried different ingress’ like HAProxy, Traefik, as well, logged in from console, but it didnt work. Even following the RMQ instructions very closely. Seems like it is the ingress config because it doesn’t work with any of them. So it doesn’t seem like this is a NGINX issue to me.

Seems like a configuration issue.

Any ideas? Thanks guys.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Tune into our recording later today to hear our answers.

tomkinson avatar
tomkinson

Digital Ocean atm

tomkinson avatar
tomkinson

Well even if we remove the LB from the discussion, lets just say the issue is the ingress

tomkinson avatar
tomkinson

It is TCP btw

tomkinson avatar
tomkinson

we are using the default NGINX ingress that comes with Rancher

tomkinson avatar
tomkinson

the ingress binds fine to the rancher admin port

tomkinson avatar
tomkinson

it does not work for binding to the rabbit mq webstomp port

tomkinson avatar
tomkinson

WebSTOMP = Simple Text Over Messaging Protocol over TCP

tomkinson avatar
tomkinson

ahhh

tomkinson avatar
tomkinson

what LB did you use?

tomkinson avatar
tomkinson

to summarize we cannot get ws:// or wss:// protocols to work at all this is an ingress config issue IMO

but which config and how to configure - no idea. Have tried everything

Pierre Humberdroz avatar
Pierre Humberdroz

I will get back to you after this. I might have an idea.

tomkinson avatar
tomkinson

Awesome, sorry to overwhelm just thought it was me asking questions. REALLY appreciate that discussion

Pierre Humberdroz avatar
Pierre Humberdroz

How did you deploy rabbitmq?

tomkinson avatar
tomkinson

thank you

tomkinson avatar
tomkinson

using the RMQ chat @Pierre Humberdroz

Pierre Humberdroz avatar
Pierre Humberdroz

via helm install?

tomkinson avatar
tomkinson

Correct

tomkinson avatar
tomkinson

Using HELM

tomkinson avatar
tomkinson

Like this

Pierre Humberdroz avatar
Pierre Humberdroz

so I use the normal rabbitmq chart which works well in HA I have to look in my notes why I decided for that instead of the other one.

tomkinson avatar
tomkinson

Thanks. Yes I would love to know why.

tomkinson avatar
tomkinson

*chart

tomkinson avatar
tomkinson

Have you guys ever used Fanout.io? Enterprise version of Pushpin.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
summerwind/actions-runner-controller

Kubernetes controller for GitHub Actions self-hosted runnners - summerwind/actions-runner-controller

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is how to run github actions (runners) onprem under kubernetes

tomkinson avatar
tomkinson

I ask because with all this RabbitMQ config issues we always had a plan to wait for Kafka to remove the Zookeeper requirement, and then considered Fanout reverse proxy just as good to use as a fanout broker

tomkinson avatar
tomkinson

haha really?

tomkinson avatar
tomkinson

The confluent guys seemed to communicate it was in process and that was late 2019

tomkinson avatar
tomkinson

You guys ever use FoundationDB? We are setting up a JanusGraph on Cassandra and then are considering instead Foundation

tomkinson avatar
tomkinson

tomkinson avatar
tomkinson

Have played quite a bit with Neo4j

tomkinson avatar
tomkinson

It is great, heavy lock in and expensive. We are going with JanusGraph. AWS and IBM using JG (based off old Titan)

tomkinson avatar
tomkinson

It is a GDB

tomkinson avatar
tomkinson

Plus Gremlin Query Lang is great

tomkinson avatar
tomkinson

Well heck we cant get the RMQ stomp PORT open HA!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/slack-notifier

Command line utility to send messages with attachments to Slack channels via Incoming Webhooks - cloudposse/slack-notifier

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is if you want to send slack notifications @Mikael Fridh

tomkinson avatar
tomkinson

Have you guys tried Rancher K3’s?

Pierre Humberdroz avatar
Pierre Humberdroz

only k3sup from alex

tomkinson avatar
tomkinson

Have you ever looked at Begin?

Pierre Humberdroz avatar
Pierre Humberdroz

Just a little snippet that I learned to love. :slightly_smiling_face: Makes helm list -A so much nicer to look at: yq w -i .deployment/$PROJECT_NAME/Chart.yaml appVersion $DOCKER_IMAGE_TAG

tomkinson avatar
tomkinson

serverless stuff https://begin.com/

Begin attachment image

Begin is a ridiculously quick platform for building modern web apps, sites, & APIs. Get started for free, no credit card required.

Pierre Humberdroz avatar
Pierre Humberdroz
Yq

yq is a lightweight and portable command-line YAML processor

Pierre Humberdroz avatar
Pierre Humberdroz
jq -n --arg channel "$SLACK_CHANNEL" --arg text "New version of $CI_PROJECT_TITLE online $DOCKER_IMAGE_TAG" '{"channel":$channel, username: "Changelog", "text": $text, "icon_emoji": ":bulb:"}'
tomkinson avatar
tomkinson

trying to learn about k3’s more see if they cover our K8’s use

tomkinson avatar
tomkinson

Has anyone ever hit a Lambda limit?

tomkinson avatar
tomkinson

You guys are awesome, thanks so much!!

Pierre Humberdroz avatar
Pierre Humberdroz

yes.. I hate lambdas it gives to many constraints on the developer

tomkinson avatar
tomkinson

Ya we are worried about it

Zachary Loeber avatar
Zachary Loeber

I want to join @Pierre Humberdroz, how do I get the secret hand-shake for that one?

Pierre Humberdroz avatar
Pierre Humberdroz

What is your slack handle in the kubernetes slack @Zachary Loeber?

Zoom avatar
Zoom
08:17:34 PM

New Zoom Recording from our Office Hours session on 2020-04-08 is now available.

2020-04-07

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Btw, here’s that chat bot (saas) for aws cloudwatch I wanted to share last week but couldn’t think of the name

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
marbot - AWS Monitoring from Slack

Configure monitoring for Amazon Web Services: CloudWatch, EC2, RDS, EB, Lambda, and more. Receive and manage alerts via Slack. Solve incidents as a team.

btai avatar

@Erik Osterman (Cloud Posse) have you tested it?

marbot - AWS Monitoring from Slack

Configure monitoring for Amazon Web Services: CloudWatch, EC2, RDS, EB, Lambda, and more. Receive and manage alerts via Slack. Solve incidents as a team.

Zoom avatar
Zoom
05:07:07 AM

New Zoom Recording from our Office Hours session on 2020-04-01 is now available.

tomkinson avatar
tomkinson

Is this once a week?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, next one tomorrow

:--1:1
tomkinson avatar
tomkinson

Hey Erik thanks so much that was a lot of fun. Sorry I couldn’t join live. Will there be a URL I can share?

tomkinson avatar
tomkinson

Ah I see it awesome tnx

tomkinson avatar
tomkinson

We have an Nginx ingress port question using websockets. Would this be an appropriate place to ask during the Office?

tomkinson avatar
tomkinson

Thanks Erik. What time or it’s an all day kind of thing

tomkinson avatar
tomkinson

Ah 11:30 am PST got it

1
tomkinson avatar
tomkinson

Tnx

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

See you there!

:100:1

2020-04-04

2020-04-02

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@David Scott were you asking this week in office hours about how to run Github Action’s Runners inside of containers, when the actions themselves run containers?

David Scott avatar
David Scott

Hey @Erik Osterman (Cloud Posse), thanks for following up! I’m currently working with terraform github actions. Running them in a hosted runner in EKS fails to mount the code from the Checkout step into the terraform-github-actions container due to docker-in-docker volume mounting issues:

The docker run command tries to volume-mount the content from the Checkout step into the terraform-github-actions container. Because the docker socket is a volume mount from the EKS worker node, it ends up trying to mount -v "/home/github-runner/_work/_temp/_github_home":"/github/home" from the EKS node, not from the github-runner pod where the git Checkout happened, and terraform can’t find any code when it runs.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Sooooo the reason I reached out is @mumoshu just shared he’s working on a solution for Kubernetes

mumoshu avatar
mumoshu
02:21:04 AM

@mumoshu has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
summerwind/actions-runner-controller

Kubernetes controller for GitHub Actions self-hosted runnners - summerwind/actions-runner-controller

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m not exactly sure it will address your problems, but it was started as a way of addressing how to run GitHub Action’s Runner in a containerized environment (#kubernetes )

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@David Scott you were also running EKS, right?

David Scott avatar
David Scott

Yes! This shows a lot of potential for me. I’ll poke around and see if it can handle things like adding the role annotation to the ServiceAccount, and a few other edge cases.

:--1:1
David Scott avatar
David Scott

Thank you for showing me this.

Blaise Pabon avatar
Blaise Pabon
Weren’t we just talking about this? <https://www.youtube.com/watch?v=Kx110kqoHo0 https://www.youtube.com/watch?v=Kx110kqoHo0>

2020-04-01

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@ welcome to #office-hours

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

any questions you’d like to have answered on our call today?

androogle avatar
androogle
StackStorm - StackStorm

StackStorm connects all your apps, services, and workflows. Why StackStorm? Get Started Open source and trusted by the enterprise Robust Automation Engine From simple if/then rules to complicated workflows, StackStorm lets you automate DevOps your way. See More Features… Integrates with your Existing Infrastructure No need to change your existing processes or workflows, StackStorm connects…

androogle avatar
androogle
StackStorm - StackStorm

StackStorm connects all your apps, services, and workflows. Why StackStorm? Get Started Open source and trusted by the enterprise Robust Automation Engine From simple if/then rules to complicated workflows, StackStorm lets you automate DevOps your way. See More Features… Integrates with your Existing Infrastructure No need to change your existing processes or workflows, StackStorm connects…

Zachary Loeber avatar
Zachary Loeber

I was going to look into this one, have you used it yet?

androogle avatar
androogle

not yet, I’ve really wanted to, just haven’t had an opportunity yet

roth.andy avatar
roth.andy
Getting Started With Hubot

Hubot is your friendly robot sidekick. Install him in your company to dramatically improve employee efficiency.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mumoshu/terraform-provider-helmfile

Deploy Helmfile releases from Terraform. Contribute to mumoshu/terraform-provider-helmfile development by creating an account on GitHub.

Adam Blackwell avatar
Adam Blackwell
lensapp/lens

Lens - The Kubernetes IDE. Contribute to lensapp/lens development by creating an account on GitHub.

:--1:1
Blaise Pabon avatar
Blaise Pabon

This looks supercool, particularly for me because I have trouble remembering the relationships between components in k8s.

lensapp/lens

Lens - The Kubernetes IDE. Contribute to lensapp/lens development by creating an account on GitHub.

Adam Blackwell avatar
Adam Blackwell

Agreed, ArgoCD is great for that as well:

roth.andy avatar
roth.andy
rancher/vm

Package and Run Virtual Machines as Docker Containers - rancher/vm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

\# Configure the Terraform (Enterprise) Provider
provider "tfe" {
  hostname = "${var.hostname}"
  token    = "${var.token}"
  version  = "~> 0.15.0"
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

anyone seen an open source implementation of this api?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

love this one

Ken Y.y avatar
Ken Y.y

Thanks for a great session!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks @!

    keyboard_arrow_up