#office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! :point_right: https://cpco.io/slack-office-hours
2019-10-16
Office hours today?
Yep!
@oscar bump
@here public #office-hours starting now! join us to talk shop 
https://zoom.us/j/508587304
a nodes group per az, equivalent to an eks worker pull i believe
Validating cluster us-east-1.staging.spoton.sh
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
bastions Bastion t3.small 1 1 utility-us-east-1c,utility-us-east-1d,utility-us-east-1a
master-us-east-1a Master t3.medium 1 1 us-east-1a
master-us-east-1c Master t3.medium 1 1 us-east-1c
master-us-east-1d Master t3.medium 1 1 us-east-1d
nodes-us-east-1a Node t3.medium 1 3 us-east-1a
nodes-us-east-1c Node t3.medium 1 3 us-east-1c
nodes-us-east-1d Node t3.medium 1 3 us-east-1d
awesome turn out! thanks everyone for joining and sharing what your working on. hope we answered your questions
make sure to check out the links that were shared
2019-10-15
I have a question for tomorrow’s Office Hours. How to maintain a single source of truth and updating a secrets manager (AWS SM or HashiCorp Vault) while having some audit and using a CI?
Good question!
2019-10-10
2019-10-09
Will there be a an office hours meeting today?
Sorry guys! Had to go to emergency hospital to pick up doggie and totally spaced
No problem at all. Is your dog doing better?
Yes.. thanks! but it will be a few weeks. He had big surgery
Hope he get’s better soon.
thanks @Robert!
Will be back next week, same time and place
2019-10-07
What’s that website that was shared before where you can see what other companies are paying for a SaaS?
We’re looking at terraform cloud
Right now it’s mostly just a newsletter it seems like, and to join (at least when I did a month or two ago) you have to give them a certain number of pricing stories, but I’m interested to see where this goes.
Thanks - a shame it isn’t widely adopted though
2019-10-03
It’s not really needed. It is so you can define the business logic of how into deploy the application.
For example how to do blue green
2019-10-02
@here public #office-hours starting now! join us to talk shop https://zoom.us/j/508587304
Do you know if this is a good PID 1? (killing things that need killed, etc.) Could always start with something like https://github.com/Yelp/dumb-init
2019-09-26
@Jeremy Grodberg @aknysh @Igor Rodionov @ maybe something nice to check out
@Jeremy Grodberg has joined the channel
@Igor Rodionov has joined the channel
@ has joined the channel
2019-09-25
I seriously use k9s every freaking day. It is my favorite tool since brew
The maintainer is super active, nice, reliable, quick to respond, and brilliant
I’ve tried a bunch of times to get him to set up the donation stuff but he doesn’t seem interested in money. I want to buy the man like 10 beers
2019-09-18
@here public #office-hours starting now! join us to talk shop https://zoom.us/j/508587304
What was the issue with using kube2iam?
Every node is responsible for negotiating with AWS apis to get sts tokens
If you restart all your pods or launch a lot of pods you will overload the kube2iam server and AWS apis
Rate limits are account wide
So you can basically DOS attack the AWS apis and they return the favor by blocking you
We have had this happen across multiple accounts and customers before switching to Kiam
Also the security model of kube2iam means the nodes them selves need an admin IAM role
With Kiam only the server needs it and that can be deployed to a dedicated node pool
Coincidentally, my team member just asked about using kube2iam and I recalled that from our office hours.
And it caches the credentials so it’s both much faster and doesn’t DoS AWS
Read all about it here
Found this as well, which was interesting: https://www.bluematador.com/blog/iam-access-in-kubernetes-kube2iam-vs-kiaim
page not found
paywall?
@Erik Osterman Can you share that link to the AWS Service Operator-like operator for Terraform?
thanks
that downside with these operators is surfacing errors
I can see that, another one would be, for example, provisioning an RDS DB takes 30+ minutes sometimes
Is the rest of your deployment going to sit there and wait? I guess it will have to
yea, for that reason we use containers for disposable staging environments
usually prebaked with datasets to speed up delivery
@dalekurt here is how we deploy kiam with cert-manager: https://github.com/cloudposse/helmfiles/tree/master/releases
2019-09-11
joining now!
sorry running late
#office-hours starting now! ask questions, get answers. free for everyone. https://zoom.us/j/508587304
just me today? i don’t have anything, was just going to listen in. no offices for me to be noisy in
ok!
yea, no one yet.
@ saw you registered for office hours
we’re on now.
also helps if i unmute my headphones lol
I just got home
Gimmie a 5 mins
sure, i’ll just hang out
I do have something we could work through if i can steal an office
i need to debug why i’m getting a “too many redirects” on a new service, might be interesting for folks who want to see the workings of the CP stack
nothing sensitive that i’d be afraid to show
ok, sure thing
@Alex Siegman <http<i class="em em-//forecastle.stakater.com/url\|forecastle.stakater.com/url>"></i> 'https://{{- env "KEYCLOAK_INGRESS_HOSTS" -}}/auth/admin/'
2019-09-10
@tamsky To clarify, we have VPC peering between the AWS accounts and AWS<~>GCP we are using VPN
2019-09-04
#office-hours starting now! ask questions, get answers. free for everyone. https://zoom.us/j/508587304
It is done @Erik Osterman?
Mentioned in today’s #office-hours was “VPC peering between GCP and AWS.” I hadn’t heard of this before, and after looking around, I still don’t know if what was meant was actually VPC peering between AWS<>GCP or a VPN solution.
2019-08-28
What’s the zoom session for today?
#office-hours starting now! join us here https://zoom.us/s/508587304
sign up for the beta
Sorry guys, I had to drop off for a bit
Did anyone Come across NPM memory Issues ?
2019-08-23
2019-08-16
2019-08-15
2019-08-14
this is the module I know others have used related to AWS secrets
@nian
resource "aws_db_instance" "example" {
# ...
timeouts {
create = "60m"
delete = "2h"
}
}
Cleanup CloudFront distributions after? https://medium.com/@romaninsh/remove-multiple-cloudfront-distributions-with-aws-cli-86442f7084c6
2019-08-13
Thanks @Erik Osterman that is kind of you. She’s ok, just getting old and seems to have pulled a muscle. I’m looking forward to this week’s office hours. I have been spreading the word around the office.
2019-08-07
Hope everything is ok…
2019-07-24
@here Public #office-hours with cloud posse starting now! https://zoom.us/s/508587304 join if you have any questions or want to listen in.
2019-07-22
2019-07-20
@Blaise Pabon I finally remembered the OS… it’s called NixOS
What was the the name of the sandbox style package manager for red hat?
Oh! I remember nixOS! my old boss Ariya is a huge fan of their package manager Nix
Yep!
OK, so the redhat OS package manager is ostree
for the system image
Do you know how ostree compares to nix? Or is it apples to oranges
and the next-gen package manager for applications is called flatpak.
Sooo… ostree is for the system image
yes, the ostree upstream project is a little behind on docs. This guy has lots of good links for Nix resources:
https://wiki.nikitavoloboev.xyz/package-managers/nix
right and NixOS has a similar approach to system images of NixOS.
rpm-ostree is for RHEL style os
actually the genealogy is more like this:
CoreOS begat ostree (which is mostly found on RedHat systems)
NixOS begat nix (although now nix runs on other OS, including Mac OS!)
IF you’re booting NixOS, then nix can manage your system images.
If you’re booting CoreOS, then ostree will manage your system images.
…and since CoreOS has tight collaboration with Fedora and RHEL, the ecosystem of .rpms is ported over…. whereas there are fewer nixOS binaries and some packages you will have to build from source.
I suspect that NixOS will become the NetBSD of the devops world: bombproof, stable, late to adopt. and CoreOS/OStree will become the RHEL of the Enterprise world.
I agree… don’t see NixOS taking off
Don’t get me wrong, ‘nix is a very kickass solution if you are switiching beteween monolithix development environments (python, Java) where a sdk version can mean a whole different set of dependencies.
2019-07-17
2019-07-10
Public #office-hours starting now! Join me here: https://zoom.us/meeting/register/dd2072a53834b30a7c24e00bf0acd2b8
Have any questions? This is your chance to ask us anything.
missed it today, had a meeting conflict.
2019-06-25
New registration page for upcoming “office hours”
@Erik Osterman set the channel topic: Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cpco.io/slack-office-hours
2019-06-19
Hey @Erik Osterman no room for me today so no mic, but I’m listening in
or i would be but i’m not hearing anything
still not hearing anything?
fixed
was issue on my end
#demo
I have not, no.
Is there one for extracting data from Prometheus?
You recording this btw? I have a couple folks who might be interested in how http://singer.io / ETL stuff works
Nah, pretty straight forward to me. Curious how much extra work it would be to conform to the format for devs building ETL pipelines for our own dashboards.
@Erik Osterman - question; it seems like singer takes care of “extract” via taps, and “load” via targets. That misses the “Transform” part of ETL. does that happen as part of the TAP?
well, it does a form of transform.
it munges the data from one system to the next
now, if you want to do extra munging, that’s when you write your own.
this is a basic example of extracting the data and transforming it JSON. then the JSON is loaded.
holy thread necro batman, 2 months ago
The extract part is also what does the transform
technically, since it’s a pipeline though
you can have an additional transformation step
those are the demos from today
2019-06-12
the “from-module” change in 0.12: https://github.com/hashicorp/terraform/blob/v0.12.0/internal/initwd/from_module.go#L78
Bah was caught up in work today, was unable to join. Any demos that we have a recording of?
No recordings today
Most of the talk was about upgrade path to 0.12
We had a good turn out today!
@Blaise Pabon has joined the channel
2019-06-05
@here office hours starting now: https://zoom.us/j/684901853
For got to share the last office hours from May 22nd. Here’s the recording: https://zoom.us/recording/play/gmEpOhxtlR0X3xAvfgY4-IWEkks_SGZA7IME0eKj1mT5Gf0FDHi4Y1WSOlK9Pbo4?continueMode=true
2019-05-29
i’m dropping off since no one joined today.
Sorry, was really busy today~
2019-05-22
@Tega McKinney has joined the channel
@Tega McKinney brought up a good point that we need to document how to get the outputs for the users created in the reference-architectures
maybe we should use nohup or tee to keep a log of everything that happens
in @Tega McKinney’s case he logged out of his terminal window and lost the history
@Erik Osterman first time ever joining office hours for a project…thanks for the suggestions and discussions around what you all are building. Enjoyed the convo.
Thanks @Tega McKinney! Really love doing these office hours too. I always learn something from them. Will share the link in a little bit.
2019-05-16
15 or 30 min?
Let’s block off 30
We can use less
@Erik Osterman at office hours yesterday you mentioned the root-dns modules that reads state from another aws account using a role that has permissions to do so. i’m trying to set that role up and give it just enough permissions to read the terraform state from s3, but keep getting Access Denied. can you show me the correct permissions for this role? here is the policy i have for it so far that isn’t working:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn<img src="/assets/images/custom_emojis/aws.png" class="em em-aws">s3:::mynamespace-sandbox-terraform-state"
},
{
"Sid": "",
"Effect": "Allow",
"Action": "s3:*",
"Resource": "arn<img src="/assets/images/custom_emojis/aws.png" class="em em-aws">s3:::mynamespace-sandbox-terraform-state/tfstate-backend/terraform.tfstate"
},
{
"Sid": "",
"Effect": "Allow",
"Action": "dynamodb:*",
"Resource": "arn<img src="/assets/images/custom_emojis/aws.png" class="em em-aws">dynamodb<i class="em em-us-east-1"></i>myaccountnum:table/bw-sandbox-terraform-state-lock"
}
]
}
i’m wondering if maybe it has something to do with KMS permissions… but that seems very complex to allow because it has to be allowed from both the role AND the kms side i believe.
i solved this issue btw Erik… i just added s3:* to the tstatebucket with /* wildcard. that’s good enough for now.
2019-05-15
Let me know whenever you can show us teleport in action!
@AgustínGonzalezNicolini @Josh Larsen if you want a demo of Teleport/Keycloak please find a time here:
Here’s the recording of today’s office hours:
2019-05-13
2019-05-09
What day is it normally
every wednesday at 11:30 am
2019-05-08
@ has joined the channel
@cabrinha has joined the channel
yeah, looking at different SDLCs for Infrastructure as Code: https://docs.gitlab.com/ee/workflow/gitlab_flow.html
@AgustínGonzalezNicolini has joined the channel
data "aws_iam_policy_document" "collection_manager_trigger_service_policy_document" {
statement = {
actions = [
"${module.sns.publish_action}"
],
resources = [
"${data.aws_sns_topic.collection_topic.arn}",
"${data.aws_sns_topic.debin_topic.arn}"
]
}
`
output "publish_action" {
value = "sns:Publish"
}
@ash has joined the channel
Nothing atm.
suggestion pull various actions together and use submodules
But thanks for asking!
I’d love a blog / writeup on these layers … I think it’s a great way to organize the foundation for all these other architectural conversations
going to take off but this has been really insightful. hope these can be recorded in the future!
If the host edits the recording perms for the meeting, we could all record from the zoom client. 
thats a good point
@cabrinha I want to record them as well.
I have cloud recording available. We tried it one time, but for certain community members, they didn’t like that since they didn’t feel like they could talk candidly.
I’m not sure there’s an expectation of privacy in such an open venue though, or more appropriately that their should be such an expectation
I suggest for next wee we’ll have two parts: start with demo and recording, q&a. then we can always have a second part that is “off the record” where we disable recordings.
yes, all in all, my hope is this is more open/transparent than a cloak and dagger community
thanks everyone for joining today! that was awesome. that was our largest turnout to date
22019-04-25
@Mohamed.Naseer has joined the channel
2019-04-23
@fernando.alvan has joined the channel
2019-04-22
@dalekurt has joined the channel
2019-04-17
apk add [email protected]
tmate-session --start # \| --stop \| --server
2019-04-10
@jober has joined the channel
terraform init -from-module=....
@raehik has joined the channel
@Josh Larsen has joined the channel
I tuned in late, are these office hours recorded by zoom where I could rewatch it later?
Thanks all, especially @Erik Osterman for hosting and @oscarsullivan_old for the demo
1
1
Sad I couldn’t join in at the beginning, sprint planning ran long
happy to answer Qs on the flow or show again
Thanks so much for the time it was very helpful!
big thanks for that! lot of stuff I can discuss with the team & hopefully get using the newer Geodesic versions (we’re using a ver older than my PR T_T)
My boilerplate http://terraform.tf – saves a lot of repetition across projects and took a long time to build up.
I think the confusion I have in adopting it is what goes in Dockerfile vs what might go in an .envrc and what/how do i split up my terraform in to digestable chunks. I haven’t really dug in in earnest yet, though. It keeps getting bumped in sprint planning and I haven’t had time to just get started outside work hours
Have just added comments to the http://terraform.tf
Dockerfile Vs .envrc/tfvars == Public Vs Private
I have a little debate with myself here https://github.com/osulli/geodesic-getting-started/blob/master/docs/getting-started.md#geodesic-usage-variables-and-terraform
Yeah, but what is local to you - a specific app/service? a whole environment inside an account?
a terraform project
Examples:
Global variables: AWS_REGION, BUCKET_NAME, and STAGE
Local variables: JENKINS_INSTANCE_SIZE, API_NODE_COUNT, and CLUSTER_NAME
Example variables set in the Dockerfile AKA public variables:
ENV TF_VAR_stage="sandbox"
ENV TF_BUCKET_REGION="${AWS_REGION}"
ENV TF_BUCKET="${NAMESPACE}-${TF_VAR_stage}-terraform-state"
ENV TF_DYNAMODB_TABLE="${NAMESPACE}-${TF_VAR_stage}-terraform-state-lock"
Yeah, my issue is, what does that terraform project represent? looks like to you it means a service? Jenkins in that example
Example .envrc AKA ‘local variables’
# Set JENKINS specific variables and export them to Terraform, Atlantis, and env
export JENKINS_INSTANCE_SIZE="t2.large"
export JENKINS_DOMAIN_NAME="<http://jenkins.domain.com>"
Yes I usually break down a TF project into a service
If I did a whole stack it’d be a nightmare IMO
That would just become a monolithic infrastructure project
Yeah, that’s what I’m trying to avoid
It’s like.. a micro-service but infrastructure
that’s how I go about it
or maybe just a service, not a micro-service.. key point is that my project called jenkins only contains jenkins related stuff
As much as I’d love to be able to go “aws, give me a new account, run terraform on it, hey look my new environment is done” I’m not sure that’s as useful as it sounds.
I’m sure I shared a project I use
ah damn not merged and its about a month old
already changed a fair bit
I mean, I’m working my boss, I’m hoping I can just hire Erik and the team to jumpstart the whole process, this becomes a moot-ish point that I don’t have to decide alone
haha thanks @Alex Siegman
On that thread, what’s the best way to have a more formal discussion with y’all about that?
@Alex Siegman let’s jump on a call later this week. Find a time here that works for you: https://calendly.com/cloudposse
2019-04-03
Yeah, no conference rooms available
I dislike open offices~
ah bummer
any questions I can monologue to you?
Yeah, I pitched the idea of hiring y’all to help bootstrap a bunch of projects we don’t have time for
Could you explain how you might go through getting started with us and basic costs
I can hear, just can’t talk
Yeah, we need line cooks
Yeah, I like the aws account arch + kubernetes + release engineering + all that
I built out our roadmap long before I found y’all, and they almost exactly match
kubecost sounds awesome
I hadn’t worked out the details on a lot of stuff, just the general arch
we’re moving towards a services architecture
When you say 8 sprints, is that 8 weeks?
ah, okay, so 16 weeks
Right now we have a bunch of legacy crap, about 8 or so services, that need a bunch of code work to split up
and has a data monolith behind the app/code monolith
and we have about 6 new services
that are all containered / 12factor etc.
You can thank me for that
I pushed hard lol
Then on top of that, we have another company we bought that i need to fold in to this at some point
so you basically have a prod / preprod / unlimited-staging environments clusters?
okay, so, 2 there, plus shared
got it
yeah, 100%
my concern there with unlimited staging though is, then that differs from prod
i don’t quite understand the technical way to set that up, but i love the idea
yeah, mock sqs, eg.
so with unlimited staging, are you still mocking stuff like SQS?
Yeah, slick
So, how would you approach a customer who has an existing production environment?
Would you use that as your root account, and build org out from there?
Yeah, I was looking at doing that completely fresh thing, but the logistics of migrating 8ish years of data boggled my mind
DMS doesn’t work for us, we tried =(
It doesn’t support some postgres features we use
That it is not But downtime is frowned upon
I’m pushing separate down to RDS, all new services are working that way
But, maybe I can make a case for it yet. I’m trying, we just don’t have enough dudes to do the work
I don’t disagree
I understand the value add
I briefly looked at it
I guess lastly, how do you approach helping a dev team learn and adopt new ways to develop services? We’ve got a lot of old school dudes who are still using vagrant boxes and stuff
doggo stream!
Grats!
I have to tap out. Thanks for the time @Erik Osterman; enlightening as always. I’m excited to see updates to the ref archs. Hopefully I can dedicate time to geodesic and getting my stuff off the ground here soon Or talk someone in to spending more money
2019-04-01
I won’t be able to make this Wednesday’s Office Hours
thanks for the heads up
actually, I should update it now to announce to #general
2019-03-29
@Jan has joined the channel
2019-03-27
@Erik Osterman has joined the channel
@oscarsullivan_old has joined the channel
@tamsky has joined the channel
@mmuehlberger has joined the channel
@Alex Siegman has joined the channel
I just learned about YAML anchors a few days ago, shortened up a docker-compose file I had quite nicely
No mic here at the office, just listening in
@Erik Osterman ~does cert manager act as a private CA?~ nevermind, it sounds like it’s really just providing a way for key exchange
@ has joined the channel
@aknysh has joined the channel
Yeah, I’m working with a LE + Traefik + Consul project right now for customer domain hosting
I think I may have DCd
looks like erik froze for me too
did a bomb hit LA? office internet out?
I’m at this same spot on this question. I want to go with k8s, but i need to really evaluate whether that makes the most sense
Yeah I live in a cloudformation world, so slow, so limited
@oscarsullivan_old kubernetes up and running is a great book and does an amazing job at teaching you the primitives in k8s, starting with pods and going all the way through deployments, statefulsets, etc. it’s aging, but the primitives still line up
I’ve used both, Kubernetes the hard way didn’t help me that much. It would probably help more if you were planning to build your own cluster from scratch versus something like GKE or EKS. It did help teach me the various parts of kubernetes that are under the hood, but for running an app, not so much
@Mo has joined the channel
@Erik Osterman what’s your thoughts on building your own K8S cluster with something like kops or kubeadm or whatever versus using EKS?
I know Kubernetes fairly well from a lab standpoint, I want to get it implemented at my new job here though
@Erik Osterman yeah good distinction, thanks for reminding me of that
I really want to just use EKS, but I don’t know enough to know where that will bite me, or if it will.
Will kubeaws generate cloudformation that I can keep committed in my own git repo, so that I can “gitops” all day long?
@oscarsullivan_old no you still need to provision non-kubernetes resources outside of k8s
This looks like a way to let K8S do cloudformation on your behalf, I’d bet as part of a yaml configuration or something, like you would for a pod or replicaset or deployment
More office hours tomorrow?
Thanks @Erik Osterman
we offer private consultative office hours for those who might be interested
It’s a good thing to have in the back of the pocket. I’d want to have a more defined problem to solve for such a service. I’m really enjoying having found this “SweetOps” community though, and if you guys get more clients out of it, all the better
What we realize is that sometimes a company might not even know what problems they might need to be solving. Our job as architects is to shape that process and help you start discussing it. Keep in mind, most companies re-platform at most every 3-5 years. We re-platform 12x a year or more, which offers tremendous accelerated learning. As a outside 3rd party we can help propose things from a different/fresh perspective.
@Alex Siegman Likewise thought the same: “I need a problem to solve before asking for help”… and yet I easily have 10 questions a day, 9 of which I can answer myself.. but what if I had someone else to: 1) bounce more advanced ideas with 2) give their 2 cents on my future plans 3) suggest something entirely new that I never thought of
That’s why we just signed up starting next week
We’re actually having a lot of trouble hiring to build out our devops/central services/whatever team, having a place I could hire might come in handy here soon
Yep! Launch a jobs site is on our roadmap
o wait pre order
it’s been preorder for ever
oh hahaha
says sep 2019
ok so I’ll go for first edition
@Erik Osterman set the channel topic: This is our public “Office Hours” held every Wednesday at 11:30 PST via Zoom (https://zoom.us/j/684901853). It’s open to everyone. It’s a place to ask questions related to our projects or ask general questions related to DevOps & Cloud.
If you’re not dead set on a paper copy, there’s a github repo with it out there… loads right up on a kindle
i read it in like 3 days on my train ride to/from work