#office-hours (2020-04)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2020-04-01
any questions you’d like to have answered on our call today?
I was going to look into this one, have you used it yet?
not yet, I’ve really wanted to, just haven’t had an opportunity yet
This looks supercool, particularly for me because I have trouble remembering the relationships between components in k8s.
Agreed, ArgoCD is great for that as well:
# Configure the Terraform (Enterprise) Provider
provider "tfe" {
hostname = "${var.hostname}"
token = "${var.token}"
version = "~> 0.15.0"
}
anyone seen an open source implementation of this api?
love this one
Thanks for a great session!
Thanks @Ken Y.y!
2020-04-02
@David Scott were you asking this week in office hours about how to run Github Action’s Runners inside of containers, when the actions themselves run containers?
Hey @Erik Osterman (Cloud Posse), thanks for following up! I’m currently working with terraform github actions. Running them in a hosted runner in EKS fails to mount the code from the Checkout step into the terraform-github-actions container due to docker-in-docker volume mounting issues:
The docker run command tries to volume-mount the content from the Checkout step into the terraform-github-actions container. Because the docker socket is a volume mount from the EKS worker node, it ends up trying to mount -v "/home/github-runner/_work/_temp/_github_home":"/github/home" from the EKS node, not from the github-runner pod where the git Checkout happened, and terraform can’t find any code when it runs.
Sooooo the reason I reached out is @mumoshu just shared he’s working on a solution for Kubernetes
@mumoshu has joined the channel
I’m not exactly sure it will address your problems, but it was started as a way of addressing how to run GitHub Action’s Runner in a containerized environment (#kubernetes )
@David Scott you were also running EKS, right?
Yes! This shows a lot of potential for me. I’ll poke around and see if it can handle things like adding the role annotation to the ServiceAccount, and a few other edge cases.
Thank you for showing me this.
Weren’t we just talking about this? https://www.youtube.com/watch?v=Kx110kqoHo0
2020-04-04
2020-04-07
Btw, here’s that chat bot (saas) for aws cloudwatch I wanted to share last week but couldn’t think of the name
@Erik Osterman (Cloud Posse) have you tested it?
New Zoom Recording from our Office Hours session on 2020-04-01 is now available.
Is this once a week?
Yes, next one tomorrow
Hey Erik thanks so much that was a lot of fun. Sorry I couldn’t join live. Will there be a URL I can share?
Ah I see it awesome tnx
We have an Nginx ingress port question using websockets. Would this be an appropriate place to ask during the Office?
Thanks Erik. What time or it’s an all day kind of thing
Tnx
2020-04-08
Should we post questions here if we can’t join with video?
btw, video totally optional
You can ask it here, but it will be easier if you’re ont he call so it can be interactive.
Just jumping in car. Post question in 2 secs
In meantime as I type you might like http://getcommandeer.com/
A group of SRE’s in our company wrote this as a templating system useful for getting things out of vault and into helm deploys: https://github.com/PremiereGlobal/stim Goal is simplify Jenkinsfiles that are using Jenkins credentials to construct and run helm commands.
So we are trying to deploy a HA RabbitMQ deployment using Rancher, K8s and Docker.
We cannot open the port for RMQ websockets (or at very least it will not connect) port 15674.
The load balancer is landing on the proper port but will not fwd to the proper port. It does not detect anything on the other end. It is not HTTP and what is interesting is that the admin section of RMQ is HTTP, and it’s port works.
It is TCP and we use WebSTOMP client library.
We see they port is open on the instance but the ingress LB will not connect. We tried to strip it down still no luck. Basically a 503.
We tried different ingress’ like HAProxy, Traefik, as well, logged in from console, but it didnt work. Even following the RMQ instructions very closely. Seems like it is the ingress config because it doesn’t work with any of them. So it doesn’t seem like this is a NGINX issue to me.
Seems like a configuration issue.
Any ideas? Thanks guys.
Tune into our recording later today to hear our answers.
Digital Ocean atm
Well even if we remove the LB from the discussion, lets just say the issue is the ingress
It is TCP btw
we are using the default NGINX ingress that comes with Rancher
the ingress binds fine to the rancher admin port
it does not work for binding to the rabbit mq webstomp port
WebSTOMP = Simple Text Over Messaging Protocol over TCP
ahhh
what LB did you use?
to summarize we cannot get ws:// or wss:// protocols to work at all this is an ingress config issue IMO
but which config and how to configure - no idea. Have tried everything
I will get back to you after this. I might have an idea.
Awesome, sorry to overwhelm just thought it was me asking questions. REALLY appreciate that discussion
How did you deploy rabbitmq?
thank you
using the RMQ chat @Pierre Humberdroz
via helm install?
Correct
Using HELM
Like this
so I use the normal rabbitmq chart which works well in HA I have to look in my notes why I decided for that instead of the other one.
Thanks. Yes I would love to know why.
*chart
Have you guys ever used Fanout.io? Enterprise version of Pushpin.
this is how to run github actions (runners) onprem under kubernetes
I ask because with all this RabbitMQ config issues we always had a plan to wait for Kafka to remove the Zookeeper requirement, and then considered Fanout reverse proxy just as good to use as a fanout broker
haha really?
The confluent guys seemed to communicate it was in process and that was late 2019
You guys ever use FoundationDB? We are setting up a JanusGraph on Cassandra and then are considering instead Foundation
Have played quite a bit with Neo4j
It is great, heavy lock in and expensive. We are going with JanusGraph. AWS and IBM using JG (based off old Titan)
It is a GDB
Plus Gremlin Query Lang is great
Well heck we cant get the RMQ stomp PORT open HA!
this is if you want to send slack notifications @mfridh
Have you guys tried Rancher K3’s?
only k3sup from alex
Have you ever looked at Begin?
Just a little snippet that I learned to love. :slightly_smiling_face: Makes helm list -A so much nicer to look at:
yq w -i .deployment/$PROJECT_NAME/Chart.yaml appVersion $DOCKER_IMAGE_TAG
jq -n --arg channel "$SLACK_CHANNEL" --arg text "New version of $CI_PROJECT_TITLE online $DOCKER_IMAGE_TAG" '{"channel":$channel, username: "Changelog", "text": $text, "icon_emoji": ":bulb:"}'
trying to learn about k3’s more see if they cover our K8’s use
Has anyone ever hit a Lambda limit?
You guys are awesome, thanks so much!!
Ya we are worried about it
I want to join @Pierre Humberdroz, how do I get the secret hand-shake for that one?
What is your slack handle in the kubernetes slack @Zachary Loeber?
New Zoom Recording from our Office Hours session on 2020-04-08 is now available.
2020-04-15
This message was deleted.
@sheldonh
let me know if you want to see something else
The distribution part I found is most amazing.. I can go to the highest chunk ( all the way to the right) and can check with the spans below what it spend the most time on
similar to stack driver trace
@Zachary Loeber @roth.andy maybe also interesting for you two.. Elastic’s APM
I spun up Elastic cloud service and connected uptime and all. It’s not an easy start, but pretty powerful.
I think since we drive so much from logs (like sumologic) if we had a combined metrics + log that would be great. The catch is figuring out how deep with .net APM and all we want to go.
This is a cool breakdown on the performance. Will have to explore more. I’m in a mix stack environment. A lot of these tools focus on web tech we don’t use.
I have some stuff with ASP.NET CORE built on .NET framework for example. makes it tough. Have to figure out what to support and what not to support
For sure but you could use the elastic cluster for logs as well you would just to have to setup either fluentd or logstash
i would use the hosted version. It’s all included
So I love this concept. Cancelling influxdb for now and going to flip over. I can’t figure out the pricing for logs though. It can’t really by 30gb a day = 96 a month. That seems crazy cheap.
Ok… I guess another question. New to Kibana/ELK.
• Would this be a solid replacement for grafana now with uptime and beats?
• Would I be better served looking at another vendor like Logz.io which has been out longer and likely tons of templates and all? I can’t even find yet on Elastic how to import a dashboard template someone created.
Logz seems crazy expensive on log costs, while Elastic … i can’t be reading right now cheap it seems.
The web gui is a bit slow to load, but seems promising
I will send you a price for year in private
I stuck with prometheus-operator for kubernetes metrics because it is just super simple to set up.
We can also screenshare a bit on my ELK stack.
Links from Todays Office Hours: Highlights from the last week: https://github.blog/2020-04-14-github-is-now-free-for-teams/ https://about.gitlab.com/blog/2020/03/30/new-features-to-core/ https://slack.engineering/deploys-at-slack-cd0d28c61701 https://zacharyloeber.com/2020/04/devops-patterns/ https://lethain.com//build-vs-buy/ Related to discussions: https://sweetops.slack.com/archives/CBW0HJDS8/p1586886308064000:
@roth.andy you mentioned that you would not use gitlab’s container registry why is that?
It has lacked critical features like lifecycle policies and overall has just been a pain to work with. I will admit I am using an older version of GitLab however. I do see that the latest version has added lifecycle policies.
okay cool! good to know b/c I wanted to go with it now for work stuff..
We’re using gitlab.com and the container registry is working well for us. As Andrew mentioned they recently added tag expiration policies for new repos, they’re working on releasing it to all repos in an upcoming version.
I’ve never self-hosted gitlab, so I can’t comment on anything from that angle.
Yea we are currently on self hosted gitlab and I am looking to move away from it because it is the best move for us right now.
Yeah, the fewer things I have to manage myself, the better.
New Zoom Recording from our Office Hours session on 2020-04-15 is now available.
2020-04-17
2020-04-21
Ho to all, can anyone help to to fetch the alerting details or data from Elasticsearch and send it to otrs ticketing tool using terraform. I read the aws kinesis using terraform but a confusion is still there. So I am asking here. If anyone worked with on this plz help me.
@Pratap join us tomorrow on the Zoom call and we’ll try to provide some guidance on how to accomplish this.
2020-04-22
Maybe worth mentioning on office hours today, two items (of several) that made the kubernetes podcast “news of the week’ (https://kubernetespodcast.com/episode/100-community-redux/). One is Pluto from Fairwinds. They always seem to produce quality tooling, Pluto is used to discover depreciated apiVersions in Kubernetes . Another is Magicpak, an interesting tool for building minimal docker images without static linking.
This may be a FAQ,,, What’s the most sane way to spin up fresh mySQL/MariaDB instances in containers….. the images in Docker hub don’t have the right address binding to be accessible from the host OS and it was a P I T A !!
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
zloeber has joined Public “Office Hours”
Mateusz Sobczak has joined Public “Office Hours”
Jawwad Yunus has joined Public “Office Hours”
Sahil has joined Public “Office Hours”
Adam Crews has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Pierre Humberdroz has joined Public “Office Hours”
Ronak Patel has joined Public “Office Hours”
I have a little show and tell today.
I’m taking this: https://github.com/blaisep/hiring-engineers/blob/feature/blaisep/README.md and I am producing this: https://zero2datadog.readthedocs.io/en/latest/index.html
hari babu has joined Public “Office Hours”
Paul Barros has joined Public “Office Hours”
Raghu has joined Public “Office Hours”
Blaise Pabon has joined Public “Office Hours”
Andrew Elkins has joined Public “Office Hours”
Mike Martin has joined Public “Office Hours”
Christian Roy has joined Public “Office Hours”
Sheldon Hull has joined Public “Office Hours”
CHONAN TSAI has joined Public “Office Hours”
Bob Chen has joined Public “Office Hours”
Harry M has joined Public “Office Hours”
Mateusz Sobczak has left Public “Office Hours”
Adam Watson has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Adedayo Akinpelu has joined Public “Office Hours”
James Wade has joined Public “Office Hours”
Omer Sen has joined Public “Office Hours”
James Wade has left Public “Office Hours”
Andrew Roth has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
https://github.com/pivotal/LicenseFinder <- for closed source
for Js I wrote this little test: https://github.com/helm-notifier/service-template/blob/master/test/license-check.js
Adam Watson has left Public “Office Hours”
Libert Schmidt has joined Public “Office Hours”
Raghu has left Public “Office Hours”
Andrew Elkins has left Public “Office Hours”
Google’s Open Sourcing Guidelines: https://opensource.google/docs/creating/
Contributing guidelines: https://github.com/zalando-incubator/kopf/blob/master/docs/contributing.rst#sign-your-code
Adam Crews has left Public “Office Hours”
Marc Tamsky has left Public “Office Hours”
CHONAN TSAI has left Public “Office Hours”
Matt Gowie has left Public “Office Hours”
Is there a way to disable some certain fields not to written on terraform state file. For example i put my secrets as encrypted variables on my terraform.tfvars but i can see it unencrypted in state file
What if your tf repo shared with devs. It would be problem with security. I dont prefer secrets in tf vars
They are not clear text
They are encrpyted via KMS and show as encrypted
Chris Topinka has joined Public “Office Hours”
Mikael Fridh has joined Public “Office Hours”
But they can be seen clear text on s3 Bucket(which is also encrypted with KMS but visible on AWS console etc)
I read them and decrpt them on the fly
Personal appeal I’m (in a rush) working on an interview exercise and need some help UN-phucking a docker-compose: https://github.com/blaisep/hiring-engineers/blob/feature/blaisep/zero2datadog/docker/docker-compose.yaml if someone can do me a solid and help me figure out why I can’t connect to mysql from the host os, you’ll have a friend in Santa Cruz….
Paul Barros has left Public “Office Hours”
hari babu has left Public “Office Hours”
my ci cd list
Adedayo Akinpelu has left Public “Office Hours”
Paul Barros has joined Public “Office Hours”
Are there any opinions about Tekton https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md
Sahil has left Public “Office Hours”
Ronak Patel has left Public “Office Hours”
Libert Schmidt has left Public “Office Hours”
Sheldon Hull has left Public “Office Hours”
Neil Gealy has left Public “Office Hours”
Bob Chen has left Public “Office Hours”
Christian Roy has left Public “Office Hours”
Pierre Humberdroz has left Public “Office Hours”
Harry M has left Public “Office Hours”
Paul Barros has left Public “Office Hours”
Andrew Roth has left Public “Office Hours”
Chris Topinka has left Public “Office Hours”
Mike Martin has left Public “Office Hours”
Jawwad Yunus has left Public “Office Hours”
zloeber has left Public “Office Hours”
Mikael Fridh has left Public “Office Hours”
Omer Sen has left Public “Office Hours”
Blaise Pabon has left Public “Office Hours”
Erik Osterman (Cloud Posse) has left Public “Office Hours”
New Zoom Recording from our Office Hours session on 2020-04-22 is now available.
2020-04-26
Got an interesting case here. One of our internal applications grew organically over time. as we approached 30 internal users, things was beginning to slow. We can’t do a LB with many smaller instances due to the fact we have a couple of places using databases table as queue (this would result in some race condition - no locking mechanism). So what we ended up doing is separate all the async & recurring tasks and have them run on another server. We did this by setting up an elaborate CICD where the master repo would sync to a secondary repo (98% code is the same), then updates the primary instance and then updates the secondary async-task instance. This is providing user some relieve on the UX. But, the overall architecture is hard to maintain from a devops point of view. What would you do from here on to clean this up? A) Clean up application logic so that there is no reliance on using db as queue. Using SQS to implement the queue then implement LB. B) Instead of using a secondary repo, just keep everything in the master repo but have different set of config for the secondary instance. C) All of above
What tools would you recommend for managing all this?
what is your DB backend? Postgres, MySQL, etc…?
If postgres, I think you can improve the queue handling to support concurrency without introducing any new technologies.
e.g.
DELETE FROM queue
WHERE itemid = (
SELECT itemid
FROM queue
ORDER BY itemid
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING *;
See SKIP LOCKED here: https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/
I would recommend option C SQS is a very nice tool! Reminding just in case that Standard queues have “at least once delivery”, so you need to have logic on the consumer side to prevent discrepancies. https://aws.amazon.com/sqs/faqs/
One server only handling 30 users seems not great, I’d look into why it can’t handle more. You probably don’t need any extra technologies as most DBs have the ability to do transactions, locking, makeshift queues, etc. If it is actually hitting resource limits on the server, then scale horizontally (usually preferable for resilience) or vertically (usually easier). SQS is great but you probably don’t need it.
@Erik Osterman (Cloud Posse) it is MYSQL
@randomy Agreed. 30 users not great. We put in newrelic to see. Turns out celery, django do consume way too much memory. And, we do have a couple of bad Django ORM queries that are way too resource intensive.
@Martin Tooming starting to use SQS. Agreed. Very nice tool. Better than having to deal with redis/celery all in the same instance as my application.
Currently I have a docker compose file that spins up, django, celeryworker, celerybeat, redis all in the same instance. Should I try to separate them onto different instance?
@chonan tsai what version of mysql?
@Erik Osterman (Cloud Posse) 5.6
Okay, yep, then pretty much SOL without upgrading DB or moving to something like SQS.
redis all in the same instance
I thought celery uses redis as the queue system, not mysql
(in which case spinning up elasticache redis should help)
we were using database as queue in the beginning, then added celery but didn’t exactly remove the old implementation.
Aha, gotcha - so straddling the old and the new
Amazon RDS currently supports the major version upgrades from MySQL version 5.5 to version 5.6, from MySQL version 5.6 to version 5.7, and from MySQL version 5.7 to version 8.0.
@Erik Osterman (Cloud Posse) @chonan tsai not sure if this would help. I may misunderstand, but when first reading i thought you could do something similar to how liquibase does locking on database migrations.
it has 2 tables database_changelog(id INT, AUTHOR varchar,executed BOOLEAN, ....) (in your case this would be your table you are using as a queue??) and database_changeloglock(id INT, locked BIT(1), LOCKGRANTED datetime, LOCKEDBY varchar)
So for a database migration if i have 2 instances of the same app, which are starting at the same time and trying to run migrations, one of the instances gets there first and sets LOCKED=1. The instance with the lock then runs the migrations and if a migration is successful it can set EXECUTED=1 in the database_changelog table since it has the lock. Once its done all the migrations it just sets LOCKED=0. Then the next instance that is waiting for the lock can grab it
you would probably need to implement some application logic in that case though
@casey interesting. Our use case isn’t just specific to database migration. It is just for day-to-day queue task consumption.
For liquibase, does it work with mysql 5.6?
yeah, i understand. I was just giving an example
and yeah im using liquibase on 5.6
ok good. will look into this.
for you it could be something like a task_queue table and a task_queue_lock table. You could then potentially only allow workers to grab tasks from task_queue table if they have the lock
but just fyi, liquibase is a database migration tool for java projects. I just thought the way they do locking could be an example of how to do it in mysql 5.6
@casey we are a python shop so this may not work… But thank you for bringing this to my attention. I will still look into it. May be helpful in the future.
@chonan tsai it has nothing to do with liquibase. Was just pointing out how they lock tables on mysql 5.6
2020-04-27
2020-04-29
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Bre Gielissen has joined Public “Office Hours”
Edward Wizelman has joined Public “Office Hours”
@here public #office-hours starting now! join us to talk shop https://zoom.us/j/508587304
Jay Simoni has joined Public “Office Hours”
Pedro Galvão has joined Public “Office Hours”
Mario Feliz has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Should be able to join in a bit. Can do some show-and-tell with https://github.com/RothAndrew/istio-practice/blob/master/eks/README.md if people are interested
The HTTPS section in particular feels really useful
Nice we can go over this
Mathieu Frenette has joined Public “Office Hours”
Gautam Sidhu has joined Public “Office Hours”
Mike Martin has joined Public “Office Hours”
CHONAN TSAI has joined Public “Office Hours”
Pierre Humberdroz has joined Public “Office Hours”
Cesar Sanchez has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
hari babu has joined Public “Office Hours”
Sheldon Hull has joined Public “Office Hours”
Andrea Bolandrina has joined Public “Office Hours”
Sri P has joined Public “Office Hours”
Adam Blackwell has joined Public “Office Hours”
Andrew Elkins has joined Public “Office Hours”
Raghu has joined Public “Office Hours”
Raghu has left Public “Office Hours”
Sheldon Hull has left Public “Office Hours”
Neil Gealy has left Public “Office Hours”
Andrew Elkins has left Public “Office Hours”
https://github.com/RothAndrew/istio-practice/blob/master/eks/README.md#https I’ll definitely run through these docs since I’ve been meaning to play with Istio more.
Blaise Pabon has joined Public “Office Hours”
@roth.andy flagger is what I am thinking about for managing the business logic of canaries
Gautam Sidhu has left Public “Office Hours”
I have to drop off at 3, but https://blog.newrelic.com/product-news/how-to-monitor-istio-service-mesh/
https://github.com/opendistro-for-elasticsearch/community/issues/25#issuecomment-522084664 Btw super awesome way for getting tracing working with elastic apm and jaeger
Cool! We also published some of the videos from observe 20/20 which taught me a lot about tracing: https://www.youtube.com/channel/UC3UV2PEUA9NvUOxTkXuEhuw
CHONAN TSAI has left Public “Office Hours”
raghu aderapalli has joined Public “Office Hours”
Adam Blackwell has left Public “Office Hours”
Cesar Sanchez has left Public “Office Hours”
Is there a tutorial or example of using cert-manager as the local CA ? That would be handy for my home lab….
I believe it is this: https://cert-manager.io/docs/configuration/ca/
Sri P has left Public “Office Hours”
raghu aderapalli has left Public “Office Hours”
Blaise Pabon has left Public “Office Hours”
Jay Simoni has left Public “Office Hours”
Mario Feliz has left Public “Office Hours”
Andy Roth has left Public “Office Hours”
Pierre Humberdroz has left Public “Office Hours”
Mathieu Frenette has left Public “Office Hours”
Mike Martin has left Public “Office Hours”
Bre Gielissen has left Public “Office Hours”
Erik Osterman (Cloud Posse) has left Public “Office Hours”
Andrea Bolandrina has left Public “Office Hours”
Edward Wizelman has left Public “Office Hours”
hari babu has left Public “Office Hours”
Pedro Galvão has left Public “Office Hours”
New Zoom Recording from our Office Hours session on 2020-04-29 is now available.