#office-hours (2020-08)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2020-08-04
2020-08-05
@here office hours is starting in 30 minutes! Remember to post your questions here.
need to know about future with k3s
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
sahil kamboj has joined Public “Office Hours”
Andrew Roth has joined Public “Office Hours”
Nathaniel Alconcel has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Evgenii Prokofev has joined Public “Office Hours”
Eddie Wizelman has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Tyler Stilwagen has joined Public “Office Hours”
Robert Horrox has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Brian Tai has joined Public “Office Hours”
Marcin Branski has joined Public “Office Hours”
Nigel Kirby has joined Public “Office Hours”
• Terraform AWS 3.0 Provider https://www.hashicorp.com/blog/announcing-v3-0-of-the-terraform-aws-provider/ https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-3-upgrade#region-attribute-is-now-read-only
• GitHub Actions Improve Workflows for Public Repos https://github.blog/2020-08-03-github-actions-improvements-for-fork-and-pull-request-workflows/
• Datadog Operator for Kubernetes (finally!) https://www.fairwinds.com/blog/introducing-astro-managing-monitors-in-a-dynamic-environment-0
• Synthetic Monitoring Agent for Kubernetes https://github.com/Comcast/kuberhealthy
• Best Way to Support AWS Partitions (e.g. GovCloud, China, etc) https://github.com/cloudposse/docs/issues/492 (Example implementation)
Gabriel Tam has joined Public “Office Hours”
Adam Crown has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
rb rb has joined Public “Office Hours”
rb rb has joined Public “Office Hours”
Adam Watson has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Scott Rogers has joined Public “Office Hours”
Andrey Nazarov has joined Public “Office Hours”
Scott Rogers has joined Public “Office Hours”
Adam Blackwell has joined Public “Office Hours”
Babajide Hassan has joined Public “Office Hours”
PePe Amengual has joined Public “Office Hours”
Mailing list vote for k3s to CNCF: https://lists.cncf.io/g/cncf-toc/topic/vote_k3s_for_sandbox/75908946?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,75908946
Andrew Roth has joined Public “Office Hours”
Adam Blackwell has joined Public “Office Hours”
Btw, @Erik Osterman (Cloud Posse) am I getting you right that you stopped using Atlantis?
ya, not really using it in new engagements. pushing towards terraform cloud / enterprise.
We still support atlantis for current customers and have many deployments of atlantis.
Just it doesn’t fit in a nice CI/CD workflow that promotes changes across multiple changes automatically in a pipeline
Do you know how much it might cost if a 200 person mostly opensource software organization were to onboard Terraform Enterprise.
Cool looking OPA solution: https://github.com/fugue/regula
Maybe worth a try to test terraform with python?
https://github.com/GoogleCloudPlatform/terraform-python-testing-helper#example-usage
This example looks really readable comparing to terratest
Zoom chat
(sorry everyone - I tend to not look at the zoom chat during the call)
New Zoom Recording from our Office Hours session on 2020-08-05 is now available.
2020-08-12
@here office hours is starting in 30 minutes! Remember to post your questions here.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Eddie Wizelman has joined Public “Office Hours”
venkata has joined Public “Office Hours”
Andrew Roth has joined Public “Office Hours”
There was a great discussion last time without too many questions, but if we’re looking for topics: it’d be great to chat through the solutions folks are using to solve the problem of disjointed terraform workflows.
For example, I’ve got a project where we’re using RDS with a bunch of databases. The RDS instance is of course in private subnets and access is only grants to particular application SGs and a Bastion SG. We want to use the Postgres Terraform provider to provide bootstrapping of the databases and its extensions, roles, etc. Now the problem is that the Postgres provider can’t connect to RDS without an SSH tunnel through our Bastion instance. So my solution was to carve up our project into multiple terraform projects / directories and then when creating a new workspace / environment the flow is to:
terraform applythe AWS infra in a particular directory — Creates RDS, Bastion, etc etc etc- Create the ssh tunnel to the RDS instance through the Bastion instance now that it’s up
- Go to the postgres terraform directory and then
terraform applythere. This works, but I of course wonder if there is a better way and it’d be great to hear how others are tackling this type of thing!
@Matt Gowie I will take a shame to propose 2 hacky solutions to described problem
- If you are a terragrunt user and would like to use terraform’s native postgres provider, you can use before/after hooks to port-forward ssh port on bastion host to localhost using SSM Session Manager. Obviously this requires your bastion host to be registered with SSM Session Manager (SSM agent running on bastion host + certain IAM permissions attached to instance profile). So before hook starts port-forwarding, tf postgres provider in your module connects to RDS via ssh tunnel and provisions required resources. After hook stops port-forwarding and that’s it.
- You can use Lambda function deployed to your VPC/subnets to provision required resources. I have tf module with such lambda opensources on github. It doesn’t have support for SSL connections and pg extensions yet, but is able to provision databases and roles in RDS instances with pg and mysql engines.
This is a good topic. Will discuss today!
Aha ya’ll didn’t discuss during the last one — Cool. I had to drop early.
so.. what’s the current best common practice for this specific use case?
Wow, this thread is a blast from the past!
I’d say a GitOps approach is the current best solution. E.g. self-hosted GitHub Action Runners that have network connectivity to your database. Run terraform on those runners, then it works out-of-the-box.
@joey ditto what Erik said – Self hosting runners is usually the best option. Another option is to use Tailscale or a similar modern VPN / BeyondCorp tool to have your terraform runner gain access to your network before managing any internal network resources.
Matt Gowie has joined Public “Office Hours”
Victor Fondevilla has joined Public “Office Hours”
Nathaniel Alconcel has joined Public “Office Hours”
Michael Holt has joined Public “Office Hours”
Torsten Trzeciak has joined Public “Office Hours”
Michael Martin has joined Public “Office Hours”
Marcos Soutullo Rodriguez has joined Public “Office Hours”
Robert Jackson has joined Public “Office Hours”
Brian Tai has joined Public “Office Hours”
Gabriel Tam has joined Public “Office Hours”
Geoff Weinhold has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Dan Meyers has joined Public “Office Hours”
Anton Shakh has joined Public “Office Hours”
Adam Blackwell has joined Public “Office Hours”
Adam Blackwell has joined Public “Office Hours”
Adam Watson has joined Public “Office Hours”
Anton Shakh has joined Public “Office Hours”
name: Dependabot-hack
on:
schedule:
# run everyday at 11:00
- cron: '0 11 * * *'
jobs:
Dependabot:
runs-on: ubuntu-latest
steps:
- name: Clone repo
uses: actions/checkout@v2
with:
token: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
- name: Update Terraform modules
uses: patrickjahns/dependabot-terraform-action@v1
with:
github_dependency_token: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
token: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
target_branch: 'master'
https://github.com/dependabot/dependabot-core/pull/1388 for the upstream PR
pepe amengual has joined Public “Office Hours”
How a PR looks
^from some experiments
https://octobox.io for GitHub notification hell
Gabriel Tam has joined Public “Office Hours”
This is reallly cool. I’ve been looking for something like this. Is this well endorsed or being phased out?
IIRC Service Broker is being phased out, and aws-controllers-k8s is in RFC for v2
Hey all - I’m trying to put together a career journey for SRE’s in our company. Does anyone have any examples? Whether public or you’d be willing to share?
Almost like an internal job description…
Might be useful to throw into the mix:
Vicken Simonian has joined Public “Office Hours”
New Zoom Recording from our Office Hours session on 2020-08-12 is now available.
2020-08-13
it’s insecure to set secrets via environment variables, so I’ve read, because they can be accessed by any user on the OS. So, it would make sense to inject secrets at image build time (assuming the use of containers) by the CI runner. Any thoughts on this?
the point you bring up is correct regarding environment variables, but different companies will have different tolerances for this.
secrets should absolutely not be injected at build time as then it’s on the image itself.
We’re going via EnvVars when the code is a legacy one. If not, I’ve teached the devs to use the SDK for retrieving the passwords
I’ve done secret management at runtime, never at build time.
The issue I see with build time secrets other than what Erik has highlighted is being able to make changes to the env var when needed without rebuilding the image.
I’ve been thinking some of this and I’m wondering if secrets held in Kubernetes secrets are really more secure since it’s just encoded. You could also encrypt the secret and put it in k8s secrets but that is cumbersome to make changes. I’m still not sure what best practices are in k8s
2020-08-14
2020-08-15
2020-08-17
Update your nodes just like you update your deployments(rolling update) + custom eks ami using launch templates also now can change instance type within worker node group specs (ie no need to create new node group)
Tag ec2 instances just like you tag worker node groups (finally we will have eks nodes with Name tag on AWS Console)
Still no Spot support But it’s on the way!
Aws wont want us to use spot ;)
2020-08-19
@here office hours is starting in 30 minutes! Remember to post your questions here.
Question : Setting = No IAM authentication, Group limit on IAM is a problem, No SSO and all manually setup in a separated aws account Question: How do you go about managing all users, groups and adding SSO and MFA with assume role using TF or other tool ( AWS Organizations, Control Tower etc)( we have Keycloak and google)
Question for today although I’m not sure I can attend live (are these recorded?): is there a way of allowing an AWS lambda to http a service running in same VPC behind an AWS classic LB that filters on IP addresses? In other words I have a classic LB that I want to configure to allow incoming traffic only from corp network (I have done that part), or from the Lambdas. I’m thinking that it cannot be done robustly (I would have to find WAN IP of the lambdas), instead I need to create internal LB that the Lambda will target. Any insight would be much appreciated!
M Hunter has joined Public “Office Hours”
Jose Netto has joined Public “Office Hours”
Andrew Roth has joined Public “Office Hours”
venkata has joined Public “Office Hours”
Rob Horrox has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Alex Vorona has joined Public “Office Hours”
Anton Shakh has joined Public “Office Hours”
Dan Meyers has joined Public “Office Hours”
Christopher Picht has joined Public “Office Hours”
Anton Shakh has joined Public “Office Hours”
Andrey Nazarov has joined Public “Office Hours”
Anton Shakh has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Gabriel Tam has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Igor Bronovskyi has joined Public “Office Hours”
Pedro Torres has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Kareem Shahin has joined Public “Office Hours”
pepe amengual has joined Public “Office Hours”
Babajide Hassan has joined Public “Office Hours”
Eddie Wizelman has joined Public “Office Hours”
Nigel Kirby has joined Public “Office Hours”
Blaisep has joined Public “Office Hours”
Robert Jackson has joined Public “Office Hours”
Adam Watson has joined Public “Office Hours”
Rahul has joined Public “Office Hours”
Drew Davies has joined Public “Office Hours”
sri has joined Public “Office Hours”
sri has joined Public “Office Hours”
O A has joined Public “Office Hours”
Shoutout to https://github.com/haya14busa/action-bumpr which also does automatic releases I really like that action
Mike Drummond has joined Public “Office Hours”
For automatic comment /test all : https://github.com/peter-evans/create-or-update-comment or Mergify / Pullapprove ( they both have options to post comments when something happens IIRC)
Ayrton Araújo has joined Public “Office Hours”
I’ve heard Google has something similar (or will have). Will try to search for it.
Everybody does it It’s the new hotness. Pulumi and Terraform also have identical projects
i should have asked this during office hours. doh.
https://sweetops.slack.com/archives/CCT1E7JJY/p1597864789200900
Have you asked AWS to increase the limit? They do that for a lot of limits… not sure about the SG one.
nope, 60 is a hard limit
i think we may just be using the wrong tool for the job here. i think there might be a better solution for this than using security groups.
Yeah that or continue using SGs but get a VPN for the office / team.
I just learned about AWS VPN, but it seems insanely expensive if used at scale.
ah i havent looked into that
we have all these office ips that we cannot consolidate cause they are all from our external vpn provider
it would be ideal if they provided us with a single ipcidr block but unfortunately, it’s 19 different ones
You could also have an office SG 1 and an office SG 2 couldn’t you? And then attach both to the resource that you’re looking to provide access?
ya that might be the easiest option
thanks Gowiem for your input!
Maybe reference SG2 from SG1, as SG may be nested
you could do that but when nested SGs you can’t use a SG id to allow connection in the Nested SG
only on the first level SG you can do that
hmm i dont think i have done that before. Let’s say that our office security group is split into 2 security groups. office (same name) and sg2.
then we reference sg2 in sg1 and sg1 will then absorb all of sg2’s rules ?
yes
sg1 <– sg2 but sg2 is port and address only
the limitation I think is on when
sg1 <– sg2 and sg2 have SG ids as sources
that will not work
is basically a single level reference, you can’t cascade to multiple level sgs
ah I see so basically the SG1 that references a SG2 will only get SG2’s rules but they will not include SG2’s reference to SG3’s rules
oh ok that makes sense
exactly
so ok relevant too, AWS just increased my limit from 60 to 100 rules per sg
but but but, now I want to split my rules up into multiple security groups, and keep my current sg that contains all my rules as the parent sg
then this parent sg can reference sg1, sg2, and sg3
SGmain
• SG1
• SG2
• SG3
• etc
sigh…
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html#security-group-rules
Another security group. This allows instances that are associated with the specified security group to access instances associated with this security group. Choosing this option does not add rules from the source security group to this security group.
it does no add them up but it does allow the connection
how the heck we did this back then????? we had like 300 rules
or we attached multiple SGs to the resource?????
ya you probably added multiple security groups
I’m pretty sure this works but you can’t combine sg-ids with ports
i checked with aws support and they confirmed that it won’t work. they pointed me to the docs
but i haven’t tested it.
so SG-pepebullshit can’t have ports and sg-ids?
ahh what they are saying is that if sg-pepebullshit could have a sg-id as source but that source can’t have other sg-ids as source
lol
unfortunately i cant attend today but food for thought for the next one
New Zoom Recording from our Office Hours session on 2020-08-19 is now available.
Kubecon sessions should be available on YouTube in early September they say.
2020-08-20
@loren has joined the channel
2020-08-26
@here office hours is starting in 30 minutes! Remember to post your questions here.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Kareem Shahin has joined Public “Office Hours”
Michael Holt has joined Public “Office Hours”
Eddie Wizelman has joined Public “Office Hours”
@here our devops #office-hours are starting now! join us on zoom to talk shop url//cloudposse.zoom.us/j/508587304) *password: sweetops
Sheldon Hull has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Alex Vorona has joined Public “Office Hours”
Andrey Nazarov has joined Public “Office Hours”
Ian Bartholomew has joined Public “Office Hours”
sri has joined Public “Office Hours”
Scott Rogers has joined Public “Office Hours”
Perumal Varadharajulu has joined Public “Office Hours”
Robert Horrox has joined Public “Office Hours”
Nick James has joined Public “Office Hours”
Drew Davies has joined Public “Office Hours”
nata lie has joined Public “Office Hours”
Another GUI for K8s https://kui.tools/
We’ve used Lens a bit. It was ok, but sloooow
Anton Shakh has joined Public “Office Hours”
Victor Avila has joined Public “Office Hours”
Andrew Elkins has joined Public “Office Hours”
Alex Siegman has joined Public “Office Hours”
Victor Ma has joined Public “Office Hours”
Jawwad Yunus has joined Public “Office Hours”
Wanderley Teixeira has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Robert Horrox has joined Public “Office Hours”
https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-eks-now-supports-udp-load-balancing-with-network-load-balancer/ we get UDP with load balancers in 2020
Christopher Picht has joined Public “Office Hours”
Adedayo Akinpelu has joined Public “Office Hours”
https://blog.intive-fdv.com/a-comparison-between-packer-and-ecs-image-builder/#<i class="em em-~"</i>text=EC2%20Image%20Builder%20is%20relatively,them%20when%20creating%20different%20images>.
Adam Crown has joined Public “Office Hours”
If we have time I want a 101 of your future go based task project framework you are working on. a teaser?
Actually it’s sad to see that there is no future for Kontena. There were cool.
Eric Berg has joined Public “Office Hours”
Afaik Kontena tools were written in Ruby;)
Definitely. The catch is i think the tooling we are talking about is more a “make file” type of framework that is a replacement.
Potentially could run a “build” that runs on windows, mac, linux, etc and full cross platform compatibility with a single binary for example. It’s a different approach rather than lots of other dependencies
For me, the hard part for me to evaluate this will be ensuring all my powershell based tasks don’t require massive reworking. I’m looking forward to learning more on this.
Not Lens though)
Somebody might find this useful https://cuelang.org/
Nice ! Thanks, can I use json schema with cuelang ?
Oh, so maybe go-releaser for your tasks and you’d have a single binary easily for all project. Super interesting.
raphael francis has joined Public “Office Hours”
MT has joined Public “Office Hours”
Great teaser. excited. As long as I can figure out how not to start from scratch and use perhaps some of my powershell 7 stuff, i’m going to try it!
raphael francis has joined Public “Office Hours”
Cool! That’s what I wanted to ask. How do you bootstrap things at Cloud Posse))
look at their build-harness repo. super cool concept
Had to hop off. Couldn’t find the ECS thread about setting env vars but outside of pulling them down from SSM programmatically in the app’s entrypoint you can directly reference them in SSM in the task definition
also checkout cloudposses ecs task module as you can use terraform module inputs to generate the json. I think that’s super cool!
I use InvokeBuild (Cross platform, and like Make on steroids), but the caveat is that you need apt install powershell for example. A precompiled binary provides consistency in multiple environments + also be able to run the same task in a build with no rework.
if this is too early to adopt and you have flexibility for considering powershell (object based vs text based is the main difference) then InvokeBuild is also great
general info on the Visual Studio Code “Task Explorer” i mentioned. Find it super helpful to ease of use. Wasn’t talking about a language server, but making dev experience super smooth to execute the same thing you run in build for example.
This is just a json “runner” built into vscode that calls a command. Would love to see a variant2 runner added in future. Play button to run, grouping of tasks and more.
Keep up the great work
New Zoom Recording from our Office Hours session on 2020-08-26 is now available.
2020-08-27
For the DockerHub new limit thingie: https://twitter.com/pgarbe/status/1298280715575087108
Hello guys, I’m trying to install lens in centos 7.6(64-bit) using snap (https://snapcraft.io/install/kontena-lens/centos) and installation is successful, but when i run kontena-lens it gives below /snap/kontena-lens/110/kontena-lens: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory sudo yum provides libgtk-3.so.0 Last metadata expiration check: 031 ago on Fri 28 Aug 2020 0651 AM UTC. gtk3-3.22.30-3.el8.i686 : GTK+ graphical user interface library Repo : @System Matched from: Provide : libgtk-3.so.0
gtk3-3.22.30-3.el8.i686 : GTK+ graphical user interface library Repo : rhel-8-appstream-rhui-rpms Matched from: Provide : libgtk-3.so.0
sudo yum install gtk3-3.22.30-3.el8.i686 -y
after installing all dependency packages still getting same error any suggestion on this ?