#office-hours (2020-04)

Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cpco.io/slack-office-hours

Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cpco.io/slack-office-hours

2020-04-29

Zoom avatar
Zoom
06:27:06 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:27:30 PM

Bre Gielissen has joined Public “Office Hours”

Zoom avatar
Zoom
06:27:52 PM

Edward Wizelman has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@here public #office-hours starting now! join us to talk shop zoom https://zoom.us/j/508587304

Zoom avatar
Zoom
06:28:33 PM

Jay Simoni has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:34 PM

Pedro Galvão has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:34 PM

Mario Feliz has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:00 PM

Neil Gealy has joined Public “Office Hours”

roth.andy avatar
roth.andy

Should be able to join in a bit. Can do some show-and-tell with https://github.com/RothAndrew/istio-practice/blob/master/eks/README.md if people are interested

The HTTPS section in particular feels really useful

RothAndrew/istio-practice

Repo to collect the things I do to practice with Istio - RothAndrew/istio-practice

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Nice we can go over this

Zoom avatar
Zoom
06:30:39 PM

Mathieu Frenette has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:40 PM

Gautam Sidhu has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:54 PM

Mike Martin has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:32 PM

CHONAN TSAI has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:39 PM

Pierre Humberdroz has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:40 PM

Cesar Sanchez has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:24 PM

Andy Roth has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:38 PM

hari babu has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:12 PM

Sheldon Hull has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:40 PM

Andrea Bolandrina has joined Public “Office Hours”

Zoom avatar
Zoom
06:37:16 PM

Sri P has joined Public “Office Hours”

Zoom avatar
Zoom
06:38:02 PM

Adam Blackwell has joined Public “Office Hours”

roth.andy avatar
roth.andy
KubeDB by AppsCode attachment image

KubeDB by AppsCode simplifies and automates routine database tasks such as provisioning, patching, backup, recovery, failure detection, and repair for various popular databases on private and public clouds

Zoom avatar
Zoom
06:44:15 PM

Andrew Elkins has joined Public “Office Hours”

Zoom avatar
Zoom
06:45:18 PM

Raghu has joined Public “Office Hours”

Zoom avatar
Zoom
06:46:18 PM
Zoom avatar
Zoom
06:46:52 PM

Sheldon Hull has left Public “Office Hours”

Zoom avatar
Zoom
06:50:07 PM

Neil Gealy has left Public “Office Hours”

Zoom avatar
Zoom
06:50:40 PM

Andrew Elkins has left Public “Office Hours”

Adam Blackwell avatar
Adam Blackwell

https://github.com/RothAndrew/istio-practice/blob/master/eks/README.md#https I’ll definitely run through these docs since I’ve been meaning to play with Istio more.

RothAndrew/istio-practice

Repo to collect the things I do to practice with Istio - RothAndrew/istio-practice

Zoom avatar
Zoom
06:54:35 PM

Blaise Pabon has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@roth.andy flagger is what I am thinking about for managing the business logic of canaries

Zoom avatar
Zoom
06:55:40 PM

Gautam Sidhu has left Public “Office Hours”

Adam Blackwell avatar
Adam Blackwell
New Relic: How To Monitor The Istio Service Mesh with New Relic attachment image

Use this open source adapter to send telemetry data from the Istio service mesh to New Relic, without the need of an agent.

Pierre Humberdroz avatar
Pierre Humberdroz

https://github.com/opendistro-for-elasticsearch/community/issues/25#issuecomment-522084664 Btw super awesome way for getting tracing working with elastic apm and jaeger

APM UI Missing · Issue #25 · opendistro-for-elasticsearch/community

As i understand APM agents should connect to elastic open distro but there is no UI in Kibana as today to visualize application traces and performance as in elasticsearch [1] https://discuss.opendi

Adam Blackwell avatar
Adam Blackwell

Cool! We also published some of the videos from observe 20/20 which taught me a lot about tracing: https://www.youtube.com/channel/UC3UV2PEUA9NvUOxTkXuEhuw

Observe2020 Conference

https://observe2020.io Observe 20/20 is a one-day VIRTUAL technology conference on April 6th, 2020 focused on empowering DevOps practitioners with solutions …

:--1:1
Zoom avatar
Zoom
07:01:50 PM

CHONAN TSAI has left Public “Office Hours”

Zoom avatar
Zoom
07:03:54 PM

raghu aderapalli has joined Public “Office Hours”

Zoom avatar
Zoom
07:04:13 PM

Adam Blackwell has left Public “Office Hours”

Pierre Humberdroz avatar
Pierre Humberdroz
elastic/apm-server

APM Server. Contribute to elastic/apm-server development by creating an account on GitHub.

Zoom avatar
Zoom
07:07:17 PM

Cesar Sanchez has left Public “Office Hours”

Pierre Humberdroz avatar
Pierre Humberdroz
Rate Limits
Last updated: Mar 5, 2020 See all Documentation Let’s Encrypt provides rate limits to ensure fair usage by as many people as possible. We believe these rate limits are high enough to work for most people by default. We’ve also designed them so renewing a certificate almost never hits a rate limit, and so that large organizations can gradually increase the number of certificates they can issue without requiring intervention from Let’s Encrypt.
Blaise Pabon avatar
Blaise Pabon

Is there a tutorial or example of using cert-manager as the local CA ? That would be handy for my home lab….

roth.andy avatar
roth.andy
CA

The CA issuer represents a Certificate Authority whereby its certificate and private key are stored inside the cluster as a Kubernetes Secret, and will be used to sign incoming certificate requests. This internal CA certificate can then be used to trust resulting signed certificates. This issuer type is typically used in a Public Key Infrastructure (PKI) setup to secure your infrastructure components to establish mTLS or otherwise provide a means to issue certificates where you also own the private key.

Pierre Humberdroz avatar
Pierre Humberdroz
Current Sponsors and Donors

We’d like to thank all of our sponsors and donors for making Let’s Encrypt possible. If you or your organization would like to sponsor or donate to Let’s Encrypt please click here. Platinum Gold Silver

Zoom avatar
Zoom
07:16:50 PM
roth.andy avatar
roth.andy
Pierre Humberdroz avatar
Pierre Humberdroz
Zoom avatar
Zoom
07:22:21 PM

raghu aderapalli has left Public “Office Hours”

Zoom avatar
Zoom
07:27:33 PM

Blaise Pabon has left Public “Office Hours”

Zoom avatar
Zoom
07:27:42 PM

Jay Simoni has left Public “Office Hours”

Zoom avatar
Zoom
07:27:42 PM

Mario Feliz has left Public “Office Hours”

Zoom avatar
Zoom
07:27:43 PM

Andy Roth has left Public “Office Hours”

Zoom avatar
Zoom
07:27:44 PM

Pierre Humberdroz has left Public “Office Hours”

Zoom avatar
Zoom
07:27:46 PM

Mathieu Frenette has left Public “Office Hours”

Zoom avatar
Zoom
07:27:54 PM

Mike Martin has left Public “Office Hours”

Zoom avatar
Zoom
07:27:55 PM

Bre Gielissen has left Public “Office Hours”

Zoom avatar
Zoom
07:28:07 PM

Erik Osterman (Cloud Posse) has left Public “Office Hours”

Zoom avatar
Zoom
07:28:08 PM

Andrea Bolandrina has left Public “Office Hours”

Zoom avatar
Zoom
07:28:08 PM

Edward Wizelman has left Public “Office Hours”

Zoom avatar
Zoom
07:28:10 PM

hari babu has left Public “Office Hours”

Zoom avatar
Zoom
07:28:10 PM

Pedro Galvão has left Public “Office Hours”

Zoom avatar
Zoom
08:15:27 PM

New Zoom Recording from our Office Hours session on 2020-04-29 is now available.

2020-04-27

2020-04-26

chonan tsai avatar
chonan tsai

Got an interesting case here. One of our internal applications grew organically over time. as we approached 30 internal users, things was beginning to slow. We can’t do a LB with many smaller instances due to the fact we have a couple of places using databases table as queue (this would result in some race condition - no locking mechanism). So what we ended up doing is separate all the async & recurring tasks and have them run on another server. We did this by setting up an elaborate CICD where the master repo would sync to a secondary repo (98% code is the same), then updates the primary instance and then updates the secondary async-task instance. This is providing user some relieve on the UX. But, the overall architecture is hard to maintain from a devops point of view. What would you do from here on to clean this up? A) Clean up application logic so that there is no reliance on using db as queue. Using SQS to implement the queue then implement LB. B) Instead of using a secondary repo, just keep everything in the master repo but have different set of config for the secondary instance. C) All of above

What tools would you recommend for managing all this?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what is your DB backend? Postgres, MySQL, etc…?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If postgres, I think you can improve the queue handling to support concurrency without introducing any new technologies.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g.

DELETE FROM queue
WHERE itemid = (
  SELECT itemid
  FROM queue
  ORDER BY itemid
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

See SKIP LOCKED here: https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/

What is SKIP LOCKED for in PostgreSQL 9.5? - 2ndQuadrant | PostgreSQL
PostgreSQL 9.5 introduces a new SKIP LOCKED option to SELECT … FOR [KEY] UPDATE SHARE. It’s used in the same place as NOWAIT and, like NOWAIT, affects behaviour when the tuple is locked by another transaction. The main utility of SKIP LOCKED is for building simple, reliable and efficient concurrent work queues. Most work queue implementations …
Martin Tooming avatar
Martin Tooming

I would recommend option C SQS is a very nice tool! Reminding just in case that Standard queues have “at least once delivery”, so you need to have logic on the consumer side to prevent discrepancies. https://aws.amazon.com/sqs/faqs/

randomy avatar
randomy

One server only handling 30 users seems not great, I’d look into why it can’t handle more. You probably don’t need any extra technologies as most DBs have the ability to do transactions, locking, makeshift queues, etc. If it is actually hitting resource limits on the server, then scale horizontally (usually preferable for resilience) or vertically (usually easier). SQS is great but you probably don’t need it.

chonan tsai avatar
chonan tsai

@Erik Osterman (Cloud Posse) it is MYSQL

chonan tsai avatar
chonan tsai

@randomy Agreed. 30 users not great. We put in newrelic to see. Turns out celery, django do consume way too much memory. And, we do have a couple of bad Django ORM queries that are way too resource intensive.

chonan tsai avatar
chonan tsai

@Martin Tooming starting to use SQS. Agreed. Very nice tool. Better than having to deal with redis/celery all in the same instance as my application.

chonan tsai avatar
chonan tsai

Currently I have a docker compose file that spins up, django, celeryworker, celerybeat, redis all in the same instance. Should I try to separate them onto different instance?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@ what version of mysql?

chonan tsai avatar
chonan tsai

@Erik Osterman (Cloud Posse) 5.6

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Okay, yep, then pretty much SOL without upgrading DB or moving to something like SQS.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


redis all in the same instance

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I thought celery uses redis as the queue system, not mysql

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(in which case spinning up elasticache redis should help)

chonan tsai avatar
chonan tsai

we were using database as queue in the beginning, then added celery but didn’t exactly remove the old implementation.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Aha, gotcha - so straddling the old and the new

chonan tsai avatar
chonan tsai

talking about mountain of debt… paying back every penny plus interests

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


Amazon RDS currently supports the major version upgrades from MySQL version 5.5 to version 5.6, from MySQL version 5.6 to version 5.7, and from MySQL version 5.7 to version 8.0.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
MySQL on Amazon RDS - Amazon Relational Database Service

Create DB instances and DB snapshots, read replicas, point-in-time restores and automated or manual backups using MySQL on Amazon RDS.

2020-04-22

Zachary Loeber avatar
Zachary Loeber

Maybe worth mentioning on office hours today, two items (of several) that made the kubernetes podcast “news of the week’ (https://kubernetespodcast.com/episode/100-community-redux/). One is Pluto from Fairwinds. They always seem to produce quality tooling, Pluto is used to discover depreciated apiVersions in Kubernetes . Another is Magicpak, an interesting tool for building minimal docker images without static linking.

Kubernetes Podcast from Google: Episode 100 - Kubernetes Community Redux, with Paris Pittman attachment image

To celebrate our 100th episode we welcome back our first ever guest, Paris Pittman, open source program manager at Google Cloud and member of the Kubernetes steering committee - among many other roles. Paris looks at how the community has changed and how it has stayed the same, and how other projects are able to adopt learnings from Kubernetes.

2
Blaise Pabon avatar
Blaise Pabon

This may be a FAQ,,, What’s the most sane way to spin up fresh mySQL/MariaDB instances in containers….. the images in Docker hub don’t have the right address binding to be accessible from the host OS and it was a P I T A !!

Zoom avatar
Zoom
06:28:39 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:42 PM

zloeber has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:47 PM

Mateusz Sobczak has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:53 PM

Jawwad Yunus has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:55 PM

Sahil has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:59 PM

Adam Crews has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:49 PM

Neil Gealy has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:53 PM

Pierre Humberdroz has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:55 PM

Ronak Patel has joined Public “Office Hours”

Blaise Pabon avatar
Blaise Pabon

I have a little show and tell today.

Blaise Pabon avatar
Blaise Pabon
blaisep/hiring-engineers

Technical Exercise. Contribute to blaisep/hiring-engineers development by creating an account on GitHub.

Zoom avatar
Zoom
06:30:32 PM

hari babu has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:45 PM

Paul Barros has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:45 PM

Raghu has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:04 PM

Blaise Pabon has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:09 PM

Andrew Elkins has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:31 PM

Mike Martin has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:33 PM

Christian Roy has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:37 PM

Sheldon Hull has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:09 PM

CHONAN TSAI has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:52 PM

Bob Chen has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:56 PM

Harry M has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:12 PM

Mateusz Sobczak has left Public “Office Hours”

Zoom avatar
Zoom
06:35:35 PM

Adam Watson has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:10 PM

Matt Gowie has joined Public “Office Hours”

Zoom avatar
Zoom
06:37:03 PM

Adedayo Akinpelu has joined Public “Office Hours”

Zoom avatar
Zoom
06:38:55 PM

James Wade has joined Public “Office Hours”

Zoom avatar
Zoom
06:39:00 PM

Omer Sen has joined Public “Office Hours”

Zoom avatar
Zoom
06:39:22 PM

James Wade has left Public “Office Hours”

Zoom avatar
Zoom
06:41:09 PM

Andrew Roth has joined Public “Office Hours”

Zoom avatar
Zoom
06:44:42 PM

Marc Tamsky has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Have proposal to allow tooling/project/module that is general and not company specific to be published on github publicly by engineers. I blog regularly and have been advocating for this to be openly supported and finally have progress. I would like a policy that promotes trust but also gives the company that is used to everything being locked down a policy to assure what type of material is permitted and some basic log of this.

a prebuilt policy out there would really help. Having trouble finding one.  I’m not looking to publish under company github account, just share tooling/automation oriented work. 

The essence of what I’d summarize: 

Assumed • No secrets… ever • No logins/references to company • No business logic • No database schema for business application Allowed Types of Code Automation, infrastructure-as-code (minus any specific company configurations), developer tooling (like helper scripts and apps that just help workflow and save time) contributions to open source to improve tooling we leverage Blogging on general technology concepts learned (excluding anything of course sensitive or giving knowledge to competitors on projects or anything else)

Process • run history scan for secrets to validate repo history • validate license and disclaimer applied • log with manager and register in central log/workitem tracking to have record of it

… in progress… hoping for a jump start. Again this is all very much “got a build-harness thing here” or “terraform module to contribute on managing github repos” etc…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
zricethezav/gitleaks

Scan git repos for secrets using regex and entropy - zricethezav/gitleaks

Pierre Humberdroz avatar
Pierre Humberdroz
pivotal/LicenseFinder

Find licenses for your project’s dependencies. Contribute to pivotal/LicenseFinder development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
UKHomeOffice/repo-security-scanner

CLI tool that finds secrets accidentally committed to a git repo, eg passwords, private keys - UKHomeOffice/repo-security-scanner

Pierre Humberdroz avatar
Pierre Humberdroz
helm-notifier/service-template

Contribute to helm-notifier/service-template development by creating an account on GitHub.

Zoom avatar
Zoom
06:53:56 PM

Adam Watson has left Public “Office Hours”

Zoom avatar
Zoom
06:53:57 PM

Libert Schmidt has joined Public “Office Hours”

Zoom avatar
Zoom
06:54:17 PM
Zoom avatar
Zoom
06:54:32 PM

Andrew Elkins has left Public “Office Hours”

Pierre Humberdroz avatar
Pierre Humberdroz

Google’s Open Sourcing Guidelines: https://opensource.google/docs/creating/

Blaise Pabon avatar
Blaise Pabon
zalando-incubator/kopf

A Python framework to write Kubernetes operators in just few lines of code. - zalando-incubator/kopf

Zoom avatar
Zoom
06:54:44 PM

Adam Crews has left Public “Office Hours”

Zoom avatar
Zoom
06:59:03 PM

Marc Tamsky has left Public “Office Hours”

Zoom avatar
Zoom
06:59:05 PM

CHONAN TSAI has left Public “Office Hours”

Zoom avatar
Zoom
06:59:52 PM

Matt Gowie has left Public “Office Hours”

omerfsen avatar
omerfsen

Is there a way to disable some certain fields not to written on terraform state file. For example i put my secrets as encrypted variables on my terraform.tfvars but i can see it unencrypted in state file

raghu avatar
raghu

What if your tf repo shared with devs. It would be problem with security. I dont prefer secrets in tf vars

omerfsen avatar
omerfsen

They are not clear text

omerfsen avatar
omerfsen

They are encrpyted via KMS and show as encrypted

Zoom avatar
Zoom
07:02:39 PM

Chris Topinka has joined Public “Office Hours”

Zoom avatar
Zoom
07:03:09 PM

Mikael Fridh has joined Public “Office Hours”

omerfsen avatar
omerfsen

But they can be seen clear text on s3 Bucket(which is also encrypted with KMS but visible on AWS console etc)

omerfsen avatar
omerfsen

I read them and decrpt them on the fly

Blaise Pabon avatar
Blaise Pabon

Personal appeal I’m (in a rush) working on an interview exercise and need some help UN-phucking a docker-compose: https://github.com/blaisep/hiring-engineers/blob/feature/blaisep/zero2datadog/docker/docker-compose.yaml if someone can do me a solid and help me figure out why I can’t connect to mysql from the host os, you’ll have a friend in Santa Cruz….

blaisep/hiring-engineers

Technical Exercise. Contribute to blaisep/hiring-engineers development by creating an account on GitHub.

Zoom avatar
Zoom
07:06:29 PM

Paul Barros has left Public “Office Hours”

Zoom avatar
Zoom
07:08:08 PM

hari babu has left Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
variantdev/mod

Missing package manager for any task runners and build tools e.g. make and variant - variantdev/mod

Pierre Humberdroz avatar
Pierre Humberdroz

my ci cd list

Zoom avatar
Zoom
07:10:02 PM

Adedayo Akinpelu has left Public “Office Hours”

Zoom avatar
Zoom
07:12:41 PM

Paul Barros has joined Public “Office Hours”

Blaise Pabon avatar
Blaise Pabon
tektoncd/pipeline

A K8s-native Pipeline resource. Contribute to tektoncd/pipeline development by creating an account on GitHub.

Zoom avatar
Zoom
07:14:12 PM
Zoom avatar
Zoom
07:14:49 PM

Ronak Patel has left Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/reference-architectures

Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

Zoom avatar
Zoom
07:22:34 PM

Libert Schmidt has left Public “Office Hours”

Zoom avatar
Zoom
07:22:48 PM

Sheldon Hull has left Public “Office Hours”

Zoom avatar
Zoom
07:24:50 PM

Neil Gealy has left Public “Office Hours”

Zoom avatar
Zoom
07:27:29 PM

Bob Chen has left Public “Office Hours”

Zoom avatar
Zoom
07:30:36 PM

Christian Roy has left Public “Office Hours”

Zoom avatar
Zoom
07:30:59 PM

Pierre Humberdroz has left Public “Office Hours”

Zoom avatar
Zoom
07:30:59 PM

Harry M has left Public “Office Hours”

Zoom avatar
Zoom
07:31:03 PM

Paul Barros has left Public “Office Hours”

Zoom avatar
Zoom
07:31:03 PM

Andrew Roth has left Public “Office Hours”

Zoom avatar
Zoom
07:31:04 PM

Chris Topinka has left Public “Office Hours”

Zoom avatar
Zoom
07:31:05 PM

Mike Martin has left Public “Office Hours”

Zoom avatar
Zoom
07:31:08 PM

Jawwad Yunus has left Public “Office Hours”

Zoom avatar
Zoom
07:31:11 PM

zloeber has left Public “Office Hours”

Zoom avatar
Zoom
07:31:12 PM

Mikael Fridh has left Public “Office Hours”

Zoom avatar
Zoom
07:31:12 PM

Omer Sen has left Public “Office Hours”

Zoom avatar
Zoom
07:31:13 PM

Blaise Pabon has left Public “Office Hours”

Zoom avatar
Zoom
07:31:13 PM

Erik Osterman (Cloud Posse) has left Public “Office Hours”

Zoom avatar
Zoom
08:26:36 PM

New Zoom Recording from our Office Hours session on 2020-04-22 is now available.

2020-04-21

Pratap avatar
Pratap

Ho to all, can anyone help to to fetch the alerting details or data from Elasticsearch and send it to otrs ticketing tool using terraform. I read the aws kinesis using terraform but a confusion is still there. So I am asking here. If anyone worked with on this plz help me.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@ join us tomorrow on the Zoom call and we’ll try to provide some guidance on how to accomplish this.

2020-04-17

2020-04-15

SlackBot avatar
SlackBot
07:12:12 PM

This message was deleted.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
elastic/apm-server

APM Server. Contribute to elastic/apm-server development by creating an account on GitHub.

Pierre Humberdroz avatar
Pierre Humberdroz
elastic/apm-server

APM Server. Contribute to elastic/apm-server development by creating an account on GitHub.

Pierre Humberdroz avatar
Pierre Humberdroz
Pierre Humberdroz avatar
Pierre Humberdroz
Pierre Humberdroz avatar
Pierre Humberdroz
Pierre Humberdroz avatar
Pierre Humberdroz

@sheldonh

Pierre Humberdroz avatar
Pierre Humberdroz

let me know if you want to see something else

Pierre Humberdroz avatar
Pierre Humberdroz

The distribution part I found is most amazing.. I can go to the highest chunk ( all the way to the right) and can check with the spans below what it spend the most time on

Pierre Humberdroz avatar
Pierre Humberdroz

similar to stack driver trace

Pierre Humberdroz avatar
Pierre Humberdroz

@Zachary Loeber @roth.andy maybe also interesting for you two.. Elastic’s APM

sheldonh avatar
sheldonh

I spun up Elastic cloud service and connected uptime and all. It’s not an easy start, but pretty powerful.

I think since we drive so much from logs (like sumologic) if we had a combined metrics + log that would be great. The catch is figuring out how deep with .net APM and all we want to go.

sheldonh avatar
sheldonh

This is a cool breakdown on the performance. Will have to explore more. I’m in a mix stack environment. A lot of these tools focus on web tech we don’t use.

I have some stuff with ASP.NET CORE built on .NET framework for example. makes it tough. Have to figure out what to support and what not to support

Pierre Humberdroz avatar
Pierre Humberdroz

For sure but you could use the elastic cluster for logs as well you would just to have to setup either fluentd or logstash

sheldonh avatar
sheldonh

i would use the hosted version. It’s all included

sheldonh avatar
sheldonh

So I love this concept. Cancelling influxdb for now and going to flip over. I can’t figure out the pricing for logs though. It can’t really by 30gb a day = 96 a month. That seems crazy cheap.

Ok… I guess another question. New to Kibana/ELK.

• Would this be a solid replacement for grafana now with uptime and beats?

• Would I be better served looking at another vendor like Logz.io which has been out longer and likely tons of templates and all? I can’t even find yet on Elastic how to import a dashboard template someone created.

sheldonh avatar
sheldonh

Logz seems crazy expensive on log costs, while Elastic … i can’t be reading right now cheap it seems.

The web gui is a bit slow to load, but seems promising

Pierre Humberdroz avatar
Pierre Humberdroz

I will send you a price for year in private

Pierre Humberdroz avatar
Pierre Humberdroz

I stuck with prometheus-operator for kubernetes metrics because it is just super simple to set up.

Pierre Humberdroz avatar
Pierre Humberdroz

We can also screenshare a bit on my ELK stack.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
:--1:1
Pierre Humberdroz avatar
Pierre Humberdroz

@roth.andy you mentioned that you would not use gitlab’s container registry why is that?

roth.andy avatar
roth.andy

It has lacked critical features like lifecycle policies and overall has just been a pain to work with. I will admit I am using an older version of GitLab however. I do see that the latest version has added lifecycle policies.

Pierre Humberdroz avatar
Pierre Humberdroz

okay cool! good to know b/c I wanted to go with it now for work stuff..

bradym avatar
bradym

We’re using gitlab.com and the container registry is working well for us. As Andrew mentioned they recently added tag expiration policies for new repos, they’re working on releasing it to all repos in an upcoming version.

bradym avatar
bradym

I’ve never self-hosted gitlab, so I can’t comment on anything from that angle.

Pierre Humberdroz avatar
Pierre Humberdroz

Yea we are currently on self hosted gitlab and I am looking to move away from it because it is the best move for us right now.

bradym avatar
bradym

Yeah, the fewer things I have to manage myself, the better.

Zoom avatar
Zoom
10:16:50 PM

New Zoom Recording from our Office Hours session on 2020-04-15 is now available.

2020-04-08

tomkinson avatar
tomkinson

Should we post questions here if we can’t join with video?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

btw, video totally optional

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You can ask it here, but it will be easier if you’re ont he call so it can be interactive.

tomkinson avatar
tomkinson

Just jumping in car. Post question in 2 secs

tomkinson avatar
tomkinson

In meantime as I type you might like http://getcommandeer.com/

Commandeer - The Desktop Cloud IDE attachment image

Commandeer is a Desktop App used to manage your cloud resources. It enables you to manage, AWS Dynamo, SNS, S3, SQS, SNS, IAM and much mroe all from the comfort of your desktop.

Todd Lyons avatar
Todd Lyons

A group of SRE’s in our company wrote this as a templating system useful for getting things out of vault and into helm deploys: https://github.com/PremiereGlobal/stim Goal is simplify Jenkinsfiles that are using Jenkins credentials to construct and run helm commands.

PremiereGlobal/stim

Speeding up development with glue that brings tools together - PremiereGlobal/stim

tomkinson avatar
tomkinson

So we are trying to deploy a HA RabbitMQ deployment using Rancher, K8s and Docker.

We cannot open the port for RMQ websockets (or at very least it will not connect) port 15674.

The load balancer is landing on the proper port but will not fwd to the proper port. It does not detect anything on the other end. It is not HTTP and what is interesting is that the admin section of RMQ is HTTP, and it’s port works.

It is TCP and we use WebSTOMP client library.

We see they port is open on the instance but the ingress LB will not connect. We tried to strip it down still no luck. Basically a 503.

We tried different ingress’ like HAProxy, Traefik, as well, logged in from console, but it didnt work. Even following the RMQ instructions very closely. Seems like it is the ingress config because it doesn’t work with any of them. So it doesn’t seem like this is a NGINX issue to me.

Seems like a configuration issue.

Any ideas? Thanks guys.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Tune into our recording later today to hear our answers.

tomkinson avatar
tomkinson

Digital Ocean atm

tomkinson avatar
tomkinson

Well even if we remove the LB from the discussion, lets just say the issue is the ingress

tomkinson avatar
tomkinson

It is TCP btw

tomkinson avatar
tomkinson

we are using the default NGINX ingress that comes with Rancher

tomkinson avatar
tomkinson

the ingress binds fine to the rancher admin port

tomkinson avatar
tomkinson

it does not work for binding to the rabbit mq webstomp port

tomkinson avatar
tomkinson

WebSTOMP = Simple Text Over Messaging Protocol over TCP

tomkinson avatar
tomkinson

ahhh

tomkinson avatar
tomkinson

what LB did you use?

tomkinson avatar
tomkinson

to summarize we cannot get ws:// or wss:// protocols to work at all this is an ingress config issue IMO

but which config and how to configure - no idea. Have tried everything

Pierre Humberdroz avatar
Pierre Humberdroz

I will get back to you after this. I might have an idea.

tomkinson avatar
tomkinson

Awesome, sorry to overwhelm just thought it was me asking questions. REALLY appreciate that discussion

Pierre Humberdroz avatar
Pierre Humberdroz

How did you deploy rabbitmq?

tomkinson avatar
tomkinson

thank you

tomkinson avatar
tomkinson

using the RMQ chat @Pierre Humberdroz

Pierre Humberdroz avatar
Pierre Humberdroz

via helm install?

tomkinson avatar
tomkinson

Correct

tomkinson avatar
tomkinson

Using HELM

tomkinson avatar
tomkinson

Like this

Pierre Humberdroz avatar
Pierre Humberdroz

so I use the normal rabbitmq chart which works well in HA I have to look in my notes why I decided for that instead of the other one.

tomkinson avatar
tomkinson

Thanks. Yes I would love to know why.

tomkinson avatar
tomkinson

*chart

tomkinson avatar
tomkinson

Have you guys ever used Fanout.io? Enterprise version of Pushpin.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
summerwind/actions-runner-controller

Kubernetes controller for GitHub Actions self-hosted runnners - summerwind/actions-runner-controller

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is how to run github actions (runners) onprem under kubernetes

tomkinson avatar
tomkinson

I ask because with all this RabbitMQ config issues we always had a plan to wait for Kafka to remove the Zookeeper requirement, and then considered Fanout reverse proxy just as good to use as a fanout broker

tomkinson avatar
tomkinson

haha really?

tomkinson avatar
tomkinson

The confluent guys seemed to communicate it was in process and that was late 2019

tomkinson avatar
tomkinson

You guys ever use FoundationDB? We are setting up a JanusGraph on Cassandra and then are considering instead Foundation

tomkinson avatar
tomkinson

tomkinson avatar
tomkinson

Have played quite a bit with Neo4j

tomkinson avatar
tomkinson

It is great, heavy lock in and expensive. We are going with JanusGraph. AWS and IBM using JG (based off old Titan)

tomkinson avatar
tomkinson

It is a GDB

tomkinson avatar
tomkinson

Plus Gremlin Query Lang is great

tomkinson avatar
tomkinson

Well heck we cant get the RMQ stomp PORT open HA!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/slack-notifier

Command line utility to send messages with attachments to Slack channels via Incoming Webhooks - cloudposse/slack-notifier

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is if you want to send slack notifications @Mikael Fridh

tomkinson avatar
tomkinson

Have you guys tried Rancher K3’s?

Pierre Humberdroz avatar
Pierre Humberdroz

only k3sup from alex

tomkinson avatar
tomkinson

Have you ever looked at Begin?

Pierre Humberdroz avatar
Pierre Humberdroz

Just a little snippet that I learned to love. :slightly_smiling_face: Makes helm list -A so much nicer to look at: yq w -i .deployment/$PROJECT_NAME/Chart.yaml appVersion $DOCKER_IMAGE_TAG

tomkinson avatar
tomkinson

serverless stuff https://begin.com/

Begin attachment image

Begin is a ridiculously quick platform for building modern web apps, sites, & APIs. Get started for free, no credit card required.

Pierre Humberdroz avatar
Pierre Humberdroz
Yq

yq is a lightweight and portable command-line YAML processor

Pierre Humberdroz avatar
Pierre Humberdroz
jq -n --arg channel "$SLACK_CHANNEL" --arg text "New version of $CI_PROJECT_TITLE online $DOCKER_IMAGE_TAG" '{"channel":$channel, username: "Changelog", "text": $text, "icon_emoji": ":bulb:"}'
tomkinson avatar
tomkinson

trying to learn about k3’s more see if they cover our K8’s use

tomkinson avatar
tomkinson

Has anyone ever hit a Lambda limit?

tomkinson avatar
tomkinson

You guys are awesome, thanks so much!!

Pierre Humberdroz avatar
Pierre Humberdroz

yes.. I hate lambdas it gives to many constraints on the developer

:100:1
tomkinson avatar
tomkinson

Ya we are worried about it

Zachary Loeber avatar
Zachary Loeber

I want to join @Pierre Humberdroz, how do I get the secret hand-shake for that one?

Pierre Humberdroz avatar
Pierre Humberdroz

What is your slack handle in the kubernetes slack @Zachary Loeber?

Zoom avatar
Zoom
08:17:34 PM

New Zoom Recording from our Office Hours session on 2020-04-08 is now available.

2020-04-07

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Btw, here’s that chat bot (saas) for aws cloudwatch I wanted to share last week but couldn’t think of the name

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
marbot - AWS Monitoring from Slack

Configure monitoring for Amazon Web Services: CloudWatch, EC2, RDS, EB, Lambda, and more. Receive and manage alerts via Slack. Solve incidents as a team.

btai avatar

@Erik Osterman (Cloud Posse) have you tested it?

marbot - AWS Monitoring from Slack

Configure monitoring for Amazon Web Services: CloudWatch, EC2, RDS, EB, Lambda, and more. Receive and manage alerts via Slack. Solve incidents as a team.

Zoom avatar
Zoom
05:07:07 AM

New Zoom Recording from our Office Hours session on 2020-04-01 is now available.

tomkinson avatar
tomkinson

Is this once a week?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, next one tomorrow

:--1:1
tomkinson avatar
tomkinson

Hey Erik thanks so much that was a lot of fun. Sorry I couldn’t join live. Will there be a URL I can share?

tomkinson avatar
tomkinson

Ah I see it awesome tnx

tomkinson avatar
tomkinson

We have an Nginx ingress port question using websockets. Would this be an appropriate place to ask during the Office?

tomkinson avatar
tomkinson

Thanks Erik. What time or it’s an all day kind of thing

tomkinson avatar
tomkinson

Ah 11:30 am PST got it

1
tomkinson avatar
tomkinson

Tnx

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

See you there!

:100:1

2020-04-04

2020-04-02

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@David Scott were you asking this week in office hours about how to run Github Action’s Runners inside of containers, when the actions themselves run containers?

David Scott avatar
David Scott

Hey @Erik Osterman (Cloud Posse), thanks for following up! I’m currently working with terraform github actions. Running them in a hosted runner in EKS fails to mount the code from the Checkout step into the terraform-github-actions container due to docker-in-docker volume mounting issues:

The docker run command tries to volume-mount the content from the Checkout step into the terraform-github-actions container. Because the docker socket is a volume mount from the EKS worker node, it ends up trying to mount -v "/home/github-runner/_work/_temp/_github_home":"/github/home" from the EKS node, not from the github-runner pod where the git Checkout happened, and terraform can’t find any code when it runs.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Sooooo the reason I reached out is @mumoshu just shared he’s working on a solution for Kubernetes

mumoshu avatar
mumoshu
02:21:04 AM

@mumoshu has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
summerwind/actions-runner-controller

Kubernetes controller for GitHub Actions self-hosted runnners - summerwind/actions-runner-controller

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m not exactly sure it will address your problems, but it was started as a way of addressing how to run GitHub Action’s Runner in a containerized environment (#kubernetes )

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@David Scott you were also running EKS, right?

David Scott avatar
David Scott

Yes! This shows a lot of potential for me. I’ll poke around and see if it can handle things like adding the role annotation to the ServiceAccount, and a few other edge cases.

:--1:1
David Scott avatar
David Scott

Thank you for showing me this.

Blaise Pabon avatar
Blaise Pabon
Weren’t we just talking about this? <https://www.youtube.com/watch?v=Kx110kqoHo0 https://www.youtube.com/watch?v=Kx110kqoHo0>

2020-04-01

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@ welcome to #office-hours

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

any questions you’d like to have answered on our call today?

androogle avatar
androogle
StackStorm - StackStorm

StackStorm connects all your apps, services, and workflows. Why StackStorm? Get Started Open source and trusted by the enterprise Robust Automation Engine From simple if/then rules to complicated workflows, StackStorm lets you automate DevOps your way. See More Features… Integrates with your Existing Infrastructure No need to change your existing processes or workflows, StackStorm connects…

androogle avatar
androogle
StackStorm - StackStorm

StackStorm connects all your apps, services, and workflows. Why StackStorm? Get Started Open source and trusted by the enterprise Robust Automation Engine From simple if/then rules to complicated workflows, StackStorm lets you automate DevOps your way. See More Features… Integrates with your Existing Infrastructure No need to change your existing processes or workflows, StackStorm connects…

Zachary Loeber avatar
Zachary Loeber

I was going to look into this one, have you used it yet?

androogle avatar
androogle

not yet, I’ve really wanted to, just haven’t had an opportunity yet

roth.andy avatar
roth.andy
Getting Started With Hubot

Hubot is your friendly robot sidekick. Install him in your company to dramatically improve employee efficiency.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mumoshu/terraform-provider-helmfile

Deploy Helmfile releases from Terraform. Contribute to mumoshu/terraform-provider-helmfile development by creating an account on GitHub.

Adam Blackwell avatar
Adam Blackwell
lensapp/lens

Lens - The Kubernetes IDE. Contribute to lensapp/lens development by creating an account on GitHub.

:--1:1
Blaise Pabon avatar
Blaise Pabon

This looks supercool, particularly for me because I have trouble remembering the relationships between components in k8s.

lensapp/lens

Lens - The Kubernetes IDE. Contribute to lensapp/lens development by creating an account on GitHub.

Adam Blackwell avatar
Adam Blackwell

Agreed, ArgoCD is great for that as well:

roth.andy avatar
roth.andy
rancher/vm

Package and Run Virtual Machines as Docker Containers - rancher/vm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

\# Configure the Terraform (Enterprise) Provider
provider "tfe" {
  hostname = "${var.hostname}"
  token    = "${var.token}"
  version  = "~> 0.15.0"
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

anyone seen an open source implementation of this api?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

love this one

Ken Y.y avatar
Ken Y.y

Thanks for a great session!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks @!

    keyboard_arrow_up