#office-hours (2020-05)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2020-05-04
Here’s a great question for office hours this week: https://sweetops.slack.com/archives/CQCDCLA1M/p1588441766212200
@HS will you join us today on office hours? ~20 minutes from now
Yes
2020-05-06
So I am curious if this is just something that would be useful to me. But would a tool where you can aggregate events from multiple sources onto a single timeline be useful for y’all?
This is just a quick mock up of something I have been thinking about.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Andrew Roth has joined Public “Office Hours”
zloeber has joined Public “Office Hours”
Adam Watson has joined Public “Office Hours”
Adedayo Akinpelu has joined Public “Office Hours”
Edward Wizelman has joined Public “Office Hours”
Pierre Humberdroz has joined Public “Office Hours”
Marcin Branski has joined Public “Office Hours”
David Scott has joined Public “Office Hours”
Darrin Rentschler has joined Public “Office Hours”
Darrin Rentschler has left Public “Office Hours”
kiran k has joined Public “Office Hours”
Szymon Matuszewski has joined Public “Office Hours”
Brian Tai has joined Public “Office Hours”
Sri P has joined Public “Office Hours”
Gemini Agaloos has joined Public “Office Hours”
Jose Netto has joined Public “Office Hours”
Jordan Levington has joined Public “Office Hours”
Jordan Levington has left Public “Office Hours”
Omer Sen has joined Public “Office Hours”
Libert Schmidt has joined Public “Office Hours”
Andrew Elkins has joined Public “Office Hours”
Miranda Pearson has joined Public “Office Hours”
Mikael Fridh has joined Public “Office Hours”
Lewis Jenkins has joined Public “Office Hours”
Geoff Weinhold has joined Public “Office Hours”
derp, need to rotate those…
it is fine.
Charlie Mathews has joined Public “Office Hours”
Loki is great for the way it can integrate nicely with prometheus metrics due to the labels - you can basically match them up… But honestly, what has me going right now is https://github.com/flant/loghouse + https://github.com/ClickHouse/ClickHouse . Not fully tested it yet though.
Just look at this statement… ClickHouse works 100-1000x faster than traditional approaches. Most vendors and projects will have bold claims… but they are rarely this confident .
asadana has joined Public “Office Hours”
Olivier Chaine has joined Public “Office Hours”
devops handbook? Really like this login example. Makes a lot of sense.
yep
Miranda Pearson has left Public “Office Hours”
Marc Boudreau has joined Public “Office Hours”
thanks @roth.andy
Omer Sen has left Public “Office Hours”
Brian Tai has left Public “Office Hours”
PSA
in 1.16 /extenionsons/v1beta1 api going away for certain resources
https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html#1-16-prequisites
Specifically these resources are removed:
DaemonSet, Deployment, StatefulSet, ReplicaSet, NetworkPolicy, PodSecurityPolicy
Other things, like Ingress, can still be used with extensions/v1beta1
The most important thing in regards to this: Check the apiVersion: on your kube-proxy DaemonSet. AWS’s instructions for updating kube-proxy only have you update the image tag, so it is likely that it will be using the deprecated API.
Adedayo Akinpelu has left Public “Office Hours”
Charlie Mathews has left Public “Office Hours”
Adam Watson has left Public “Office Hours”
David Scott has left Public “Office Hours”
zloeber has left Public “Office Hours”
Pierre Humberdroz has left Public “Office Hours”
Andrew Roth has left Public “Office Hours”
Geoff Weinhold has left Public “Office Hours”
asadana has left Public “Office Hours”
Marcin Branski has left Public “Office Hours”
Szymon Matuszewski has left Public “Office Hours”
Andrew Elkins has left Public “Office Hours”
Edward Wizelman has left Public “Office Hours”
Marc Boudreau has left Public “Office Hours”
Mikael Fridh has left Public “Office Hours”
Libert Schmidt has left Public “Office Hours”
Olivier Chaine has left Public “Office Hours”
Jose Netto has left Public “Office Hours”
Erik Osterman (Cloud Posse) has left Public “Office Hours”
Sri P has left Public “Office Hours”
Gemini Agaloos has left Public “Office Hours”
kiran k has left Public “Office Hours”
Lewis Jenkins has left Public “Office Hours”
~Crap - I think I forgot to hit record. Zoom UI changed and through me off.~still processing
New Zoom Recording from our Office Hours session on 2020-05-06 is now available.
2020-05-08
New Zoom Recording from our Office Hours session on 2020-04-29 is now available.
2020-05-10
2020-05-11
2020-05-12
2020-05-13
Question for today’s office hour: What are the best practices for version numbering of multiple resources (apps, charts, docker images, etc) stored in the same monorepo? Using git tags (ie: v1.0.1) to track semantic versions seems awkward, because all resources would share the same version “counter”. Using the short commit hash seems more appropriate, but it is not allowed as chart version, which expects a semantic version in the form 1.0.1. I’m currently considering using something like 1.0.1589390493, where the last number is the number of seconds since UNIX epoch, calculated using the commit’s timestamp:
$(date "+%s" -d "$(git show -s --format=%ci)")
The major and minor versions could be stored in a file in the base dir of each resource. Any other suggestions?
Thanks @Mathieu Frenette! We’ll discuss
@here please share your questions for today’s office hours!
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Marc Tanne has joined Public “Office Hours”
rohit g has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Andrew Roth has joined Public “Office Hours”
zloeber has joined Public “Office Hours”
Omer Sen has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Andrew Elkins has joined Public “Office Hours”
Andrea Bolandrina has joined Public “Office Hours”
David Scott has joined Public “Office Hours”
Robert Horrox has joined Public “Office Hours”
Brian Tai has joined Public “Office Hours”
Mathieu Frenette has joined Public “Office Hours”
Vitali Bystritski has joined Public “Office Hours”
CHONAN TSAI has joined Public “Office Hours”
1.0.0+eh12345
Gemini Agaloos has joined Public “Office Hours”
As an alternate, check this: https://kislyuk.github.io/yq/
The syntax for this is much more similar to jq, which helps.
We ultimately selected the one by mikefarah because it’s in go and distributes binary releases
Adedayo Akinpelu has joined Public “Office Hours”
What do you use to authenticate users against aws eks? I want to use external authentication mechanism aside from aws-auth configmap. Maybe use my AD users/groups?
Jboss keycloak
Okta replacer
Steve Boardwell has joined Public “Office Hours”
Edward Wizelman has joined Public “Office Hours”
Question on video stream transcoding from h.265/HEVC to h.264
soyer has joined Public “Office Hours”
Another question. I want to limit pod/deployment to only accept ingress(incoming) traffic from Aws Alb Ingress only not any other pod/deployments on SAME k8s namespace. Networkpolicy is limited namespaces, labels but seperating namespaces requires seperate alb ingress AND seperate external-dns (as both of them works on one Namespace only) what do u suggest?
Muhammed Soyer has joined Public “Office Hours”
Since ingress is not a deployment…
Instead of istio
Not yet ;)
Or solo meshctl
Anyone heard about BackStage by Spotify ? It is a developer portal to provision applications etc .. I heard about it recently, just started evaluating.. https://backstage.io/
https://thenewstack.io/how-the-u-s-air-force-deployed-kubernetes-and-istio-on-an-f-16-in-45-days/
USAF Chief Software Officer likes to say “If Kubernetes is good enough for missiles and F-16s then it is good enough for you”
Is their any example repos with variant2 running terraform modules
We haven’t published our solution yet
Ok then I’ll start winging it
Happy to give you a walk through sometime if you want to see what it does
That would be great
New Zoom Recording from our Office Hours session on 2020-05-13 is now available.
@chonan tsai perfect time! got this in my inbox today
@Erik Osterman (Cloud Posse) Fantastic - thanks for sharing today. This is gonna be useful.
PSA:
2020-05-14
@omerfsen check out https://github.com/sighupio/permission-manager
Permission-manager does NOT work with EKS
(@Zachary Loeber just linked me to the github org and was looking through the repos when I foudn this)
Hi
let me check
thank you
Hey folks, I just got a job offer from Very Good Security… I found out about them right here, thanks to @Erik Osterman (Cloud Posse)
That’s great @Blaise Pabon!
Is it a developer advocacy role?
Well, the title is Solution architect and it’s the sort of early stage “pre-and-post sales support while preparing the path to excellence” that is my specialty. …Honestly, in this market, I don’t feel comfortable turning my nose up an a paycheck from good people, doing something that I’m not ashamed of.
I think there’s a lot of potential. I didn’t meet a single person who gave me the “works on my machine… it’s above my pay grade” attitude.
woot!
2020-05-15
2020-05-20
I might not be here still want to leave something for later today.
So I was affected by quay incident quite heavily I had a test running in my development cluster which drains a node every hour and adds a new one to have a rotation and short lived nodes. Well since quay was not up some of my pods were not able to be scheduled (image pull back) and this caused quite a bit of headache.. But I am happy that I learned from it. I might have to cache/reupload to my own registry.
So sometimes it might be worth to not touch a running system.
This makes for a good argument in favor of self-hosted container registry infrastructure….
With the Windows additions of great Mac/Linux-y things like
• A MacOS-style “spotlight” feature
• WSL2 Are we finally getting to the point where I don’t have to dual-boot Linux on my windows machine to do serious development in a containerized/kubernetes world?
does docker still run in a VM on windows? Or can it run easily in WSL and you connect locally (in windows) somehow with the docker client?
does docker still run in a VM on windows?
On my machine, yes. I haven’t tried WSL2 yet though
every few months I try to develop on my windows desktop at home and just get mad at it heh. every time i try though, more things work. i usually end up back on osx though
I’m looking forward to the GUI support. WSL 2 + Docker Desktop + VS Code is getting pretty good but there are still too many times where I need to run a web browser in Linux because of AWS credentials, VPNs, SSH tunnels, etc. I tried an X server but ran into issues with dbus and things kept crashing after a while, so proper GUI support will be good.
Hyper-V with a “quick create” Ubuntu image is pretty decent. No need to dual boot. But I’ve only tried this on my machine with 32 GB of memory…
Hyper-V with a “quick create” Ubuntu image is pretty decent
Do you have a link you can point me to with some docs on this? I’ve messed around a little with Hyper-V and VirtualBox but my experience was very underwhelming, even with 32GB of memory
@randomy
I don’t know any docs in particular but this sums it up https://www.thomasmaurer.ch/2019/06/how-to-create-an-ubuntu-vm-on-windows-10/
I’m using the Ubuntu 18 version. I think I tried a newer one a while back but ran into issues. I use i3 which may or may not improve performance. And finally, I have a little AutoHotKey script that positions the VM window properly because Hyper-V annoyingly doesn’t let you maximize a VM without going into full-screen mode.
This podcast was a decent summary of the latest with WSL2 and Docker - https://hanselminutes.com/736/making-docker-lovely-for-developers-with-simon-ferquel
What was the driving reason why you chose the approach of bundling Terraform infra configuration files within a Geodesic image (and are you still using that approach?), versus treating them as two distinct entities (that could still be versioned side-by-side within the same repo and used together in a pipeline)?
We are moving our CI/CD pipelines from Jenkins X (which is 100% Gitops driven) to Codefresh, where we have the ease of use of shared configs and secrets that we can manage easily via the UI and inject into our pipelines as environment variables. However we realize that such configs and secrets are not version-controlled and may sometimes be tightly coupled with the pipelines as they evolve. If for some reason we need to rollback our pipelines, all related configurations will not follow accordingly. What are your thoughts and experience about such external configurations that escape the Gitops domain?
we’re using codefresh… and it’s so unreliable we’ve decided to do anything but that now
wow! we have a lot of interesting talking points today. excited
Just another heads up, we’ve had to enable passwords on the Zoom calls (zoom forcing our hand on this). The password is sweetops if you’re prompted for it…
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Marc Tanne has joined Public “Office Hours”
Robert Horrox has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
Mukul Garg has joined Public “Office Hours”
zloeber has joined Public “Office Hours”
Adam Crown has joined Public “Office Hours”
Jie Chen has joined Public “Office Hours”
Mathieu Frenette has joined Public “Office Hours”
Adrian Todorov has joined Public “Office Hours”
Marcin Branski has joined Public “Office Hours”
Dale-Kurt Murray has joined Public “Office Hours”
eddie.wizelman has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Alex Siegman has joined Public “Office Hours”
Pierre has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Pierre has joined Public “Office Hours”
cho has joined Public “Office Hours”
Brian Tai has joined Public “Office Hours”
Andrew Elkins has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Prasanna Pawar has joined Public “Office Hours”
sri has joined Public “Office Hours”
rohit g has joined Public “Office Hours”
José Netto has joined Public “Office Hours”
sri has joined Public “Office Hours”
Life is full of contradictions…. Trump becomes president… and now we even have the NPM guys doing dependency management!
Sorry I couldnt make it today, I miss you guys. Homeschooling in the time of covid….
we had an absolutely amazing office hours today! recording will be posted in a few hours.
New Zoom Recording from our Office Hours session on 2020-05-20 is now available.
Btw, we’re also uploading all our office hours to youtube!
if you haven’t already subscribed to our channel, it would be a big help! we need to reach 100 subscribers to claim our URL.
2020-05-21
Thanks for bringing up my questions! I didn’t have a chance to join the call, but it was a pleasure to watch.
Regarding unanswered question on Tech Radar. I can elaborate a bit and probably it can be a topic for the next session.
What ThoughtWorks says now is that:
We recommend caution in managing stateful systems via container orchestration platforms such as Kubernetes. Some databases are not built with native support for orchestration — they don’t expect a scheduler to kill and relocate them to a different host. Building a highly available service on top of such databases is not trivial, and we still recommend running them on bare metal hosts or a virtual machine (VM) rather than to force-fit them into a container orchestration platform
Kinda a warning. And I know some cases when clients want to do such a move no matter what (I mean to migrate databases to K8s). There might be plenty of databases especially when it is a single tenant app and a bunch of microservices around. What’s your experience with managing databases (Postgres, MariaDB) in K8s given a current state of tooling? Managed services vs operators vs helm charts?
We’ll discuss this some more today
Thanks for bringing up my question again! It’s not a perfect time for me to participate, but I’ll do my best to join in the future.
suggested topic: (via @Zachary Loeber) https://sweetops.slack.com/archives/CBW699XE0/p1590086472292600
2020-05-26
Heads up! next wednesday on June 3rd (not this week), we’ll have a guest speaker to answer any/all your questions on Cloud Formation.
22020-05-27
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
It’s asking me to enter a password, but don’t see one in the calendar invite
rohit g has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
Marc Tanne has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Ianculov Vucomir has joined Public “Office Hours”
Robert Horrox has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Walter Sosa has joined Public “Office Hours”
Hilal Jaffan has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Alex Siegman has joined Public “Office Hours”
Haroon Rasheed has joined Public “Office Hours”
Daniel Blue has joined Public “Office Hours”
Please I need help joining the meeting, The meeting link is requesting password
Same for me
sweetops
thanks
Worked - thank you!
Adedayo Akinpelu has joined Public “Office Hours”
Michael Martin has joined Public “Office Hours”
zloeber has joined Public “Office Hours”
Andrea Bolandrina has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Michael Martin has joined Public “Office Hours”
eddie.wizelman has joined Public “Office Hours”
Jeremy Schuller has joined Public “Office Hours”
Mike Marseglia has joined Public “Office Hours”
PePe Amengual has joined Public “Office Hours”
Jie Chen has joined Public “Office Hours”
Vijay Ravi has joined Public “Office Hours”
Jeremy Schuller has joined Public “Office Hours”
Brian Tai has joined Public “Office Hours”
gemini has joined Public “Office Hours”
Jeremy Schuller has joined Public “Office Hours”
Juan Soto has joined Public “Office Hours”
- ThoughtWorks now recommends caution in managing stateful systems via container orchestration platforms such as Kubernetes.
- https://www.thoughtworks.com/radar/techniques/managing-stateful-systems-via-container-orchestration
- Our retort: https://cloudposse.com/devops/should-you-run-stateful-systems-via-container-orchestration/
We are using Stash from AppsCode. So far so good, but I’ve heard complains about KubeDB support
Michael Martin has joined Public “Office Hours”
@Erik Osterman (Cloud Posse) didn’t you have a “things to think about when choosing a ci/cd platform” article/spreadhseet/list/whatnot you shared a while back?
@Alex Siegman - we do - let me dig that up. Thanks for reminding me.
Might help folks form their own ideas about choosing codefresh vs jenkins vs whatever
@roth.andy FYI, unless you pay a base licensing cost for “Enterprise” per month/year, which you probably would anyways for self-hosted, codefresh does charge per user as well as per concurrency. I’ve been through that with them before Then again, I make no secret that I’m very anti-per-user pricing.
Yep, Enterprise is all we are looking at
Still bothers me that it’s an upcharge to get unlimited users, when the usage factor is primarily from pipeline runs
Feels very double-dip to me, but the platform does have a lot of positives
Here are some ways to control your AWS costs: https://cloudposse.com/aws-cost-controls/
Huh… the link spotinstance.com forwards to aws.amazon.com.
sure does, that’s weird
I must be linking to the wrong site - I’ll fix
New Zoom Recording from our Office Hours session on 2020-05-27 is now available.
Hello, my co-worker linked the above office hours video where he asked the question about NAT networks. Is it appropriate to ask a question in this channel about the above video?
Sure thing @Benjamin Hudgens
Regarding the conversation about NAT gateways, vs NAT instances, vs IGW’s (approx 45mins in video); why is the NAT gateway or NAT instance meaningfully different than the NAT provided by the IGW? As in, how does one nat provide a different level of security over the other?
The IGW documentation defines the function as NAT’ing; what makes the IGW nat fail an audit, so to speak? (vs the other two methods)
From Docs:
https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html
I was hoping maybe you guys had experience with audits that flagged this and could help me understand why it was flagged?
We literally couldn’t come up with a reason to pay for either NAT solution when IGW’s allow free inbound traffic and zero maintenance? We tried really hard to justify the NAT subnet setup given it’s the classical network config.. Compliance was a concern in our chat, and I’m wondering if you guys have actually seen it flagged (and why).
(I will respond to you)
Somebody feel free to correct me if I’m wrong here: NAT Gateways and NAT Instances don’t allow inbound traffic. They’re a purely outbound mechanism for Private Subnet resources to communicate to the wider internet.
IGW allows inbound and outbound traffic — It’s the VPC’s interface to the internet.
Generally, the thought behind putting services like Databases or Web app in private subnets is so that those resources can’t be accessed by the public internet. They’re only accessible via certain fronting services like load balancers and therefore that makes them more secure against port scanning and what not.
If I’m running a NAT Instance, an IGW would be functionally identical to PORT forwarding back to one of the machines serviced by that particular instance.
I do think you’re correct; I’m not aware of the NGW’s supporting any sense of forwarding.
So; hmm, that’s a good perspective.
We’re really thinking about dropping these and leaning on sec groups given the IGW is more like a PAT instead of a NAT. I’ll bring up your point tomorrow, that’s at least ONE functional difference we didn’t come up with on our own. TY!
One thing that’s worth mentioning again is that if you folks switch to NAT instances + an S3 VPC Endpoint then you’ll still be able to retain the same level of security (not have all your resources in public subnets), while also saving a ton.
It won’t be as cheap as it is to run all your resources in public subnets, but that would definitely be frowned upon by any security audit. At a minimum, I would say there are very few cases where your DB shouldn’t be run in a private subnet.
Perhaps. I’m trying to understand why it’s not better to run our DB in an IGW subnet and simply do not attach an EIP (or public ip at all). We can mount an EIP (or EIN) for maintenance with a very specific sec group. Again, my understanding is the IGW is still a NAT per their docs.
We actually do leverage the VPCE’s in a few of our route tables for S3. Mike didn’t bring up the various other expensive things in our environment. Talking to S3/Dynamo is free over the VPCE’s .. but most other services incur a cost.
We’re challenging our assumptions about the setup; trying to understand why an audit is going to flag an external IP with a deny all sec group, or how that’s different than a nat instance that could employ port forwarding just the same.
Really appreciation the outside thoughts. The one you highlighted above is good, and compliance audits was our other concern. We just couldn’t understand the ‘technical’ reasons it would be flagged. Erik highlighted it a bit on your call; there are “best practices” we all employ, and we’re challenging those a bit to understand why we’re paying extra $$. We were failing to come up with good reasons. I thank you for giving this some thought from an outside perspective!
(Will respond a bit later)
Sure sure! No problem! Appreciate the feedback. Mike has been representing us on your calls. I lead his team and I’ve wanted to join, but I haven’t had the luxury yet. We talk about you guys quite a bit. Pretty sure I skipped the Terraform docs, and just learned terraform reading you guys’ work.
Actually, one request: since this is such a well prepared question, would you mind posting it instead here: https://ask.sweetops.com/
Oh! Sure!
That will allow for an equally constructive response
Absolutely.
Sweet! I will get back to you - thanks for posting
Great question. Typo here with can/can’t: (and NAT Instances) *can* permit inbound traffic *at all*