#office-hours (2020-06)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2020-06-03
Maybe this isn’t so much of an office hours question than a request so let me know if its misfiled. Does anyone have any experience with https://github.com/pomerium/pomerium?
just a reminder: today we have a special guest speaker (@Chuck Gehman) who will be answering questions related to Cloud Formation.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Robert Horrox has joined Public “Office Hours”
Marc Tanne has joined Public “Office Hours”
Chuck Gehman has joined Public “Office Hours”
Dan Overholt has joined Public “Office Hours”
Thomas Mundt has joined Public “Office Hours”
Alex Siegman has joined Public “Office Hours”
tamsky has joined Public “Office Hours”
zloeber has joined Public “Office Hours”
Vijay Ravi has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Adam Crown has joined Public “Office Hours”
Stephanie Koerlin has joined Public “Office Hours”
Ben Wart has joined Public “Office Hours”
Marcin Branski has joined Public “Office Hours”
Igor Miltchman has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Brian Tai has joined Public “Office Hours”
Joe Hosteny has joined Public “Office Hours”
Not on topic of cloudformation but if you have time:
I’m creating an EKS cluster with the cloudposse EKS cluster module, and running kubectl apply from the main.tf (using a localfile with localexec). Sometimes the kubectl apply gets run before the kube api server is ready, so I’m wondering what is the recommended way to deal with that: time_sleep resource? a way to indicate dependency so the localexec does not run too early? a way to only run the kubectl exec once API responds? (actually that will work for sure if I write a bash script that loops until kubectl apply works).
Jie Chen has joined Public “Office Hours”
Blaise pabon has joined Public “Office Hours”
Question for Chuck: We have a series of CloudFormation scripts that collectively add up to our stack. Only one contains a series of nested scripts. We would like to be able to combine 20+ cloudformation scripts into one execution to create a new Dev stack and subsequently be able to destroy that when necessary. Is this an appropriate use for Service Catalog or what is the recommended approach?
^^^
Cloudformation Question: I’ve used CF extensively at past roles, but my recollection is that if a resource was not initially created with CF, you couldn’t really manage that resource with CF ever in it’s lifetime. Recreation might be fine for ephemeral stuff, but for data stores and things it’s not always practical. How does CF handle that kind of scenario where you might need to import an existing resource in to a stack?
Ola Ade has joined Public “Office Hours”
Harry Moreno has joined Public “Office Hours”
Ben Wart has joined Public “Office Hours”
Blaise pabon has joined Public “Office Hours”
HariPrasad Venkatanarayana has joined Public “Office Hours”
Alex Vorona has joined Public “Office Hours”
ola server has joined Public “Office Hours”
This message was deleted.
eddie.wizelman has joined Public “Office Hours”
Terrastack uses https://github.com/aws/jsii: “[jsii] allows code in any language to naturally interact with JavaScript classes. It is the technology that enables the AWS Cloud Development Kit to deliver polyglot libraries from a single codebase”
Maybe not first but probably in the first 3 The 2 others are:
• Learn CloudFormation came out in July 2018
• Mastering CloudFormation just came out this month
Question: when I used CloudFormation last summer on one project the most frustrating aspect was when the stack being updated had a mistake then would rollback and even the rollback failed. How do you avoid this?
@Erik Osterman (Cloud Posse) Early in this meeting you talked about the 4 layers that comprises infrastructure, did you have an article that goes over the layers and such?
Unfortunately, no - it’s something I really need to write up, however
You have mentioned it a few times, I for one would like to drink more koolaide please
Eric Berg has joined Public “Office Hours”
@Erik Osterman (Cloud Posse) definitely interested in that ebook AWS CF in Action
DM’d you the code
Peter Sbarski from ACloudGuru wrote the serverless book
This book, AWS Security is by Dylan Shields… 5 chapters in the MEAP preview program, https://www.manning.com/books/aws-security
Thanks again everyone! Thanks Erik!
Thanks @Chuck Gehman! really enjoyed having you on our session today
Also, forgot to post @Chuck Gehman’s book! https://www.manning.com/books/aws-cloudformation-in-action
Discount code podposse20 (40% off) or DM me for 100% off coupon.
Also, just a reminder - if you’re an expert on tools like Flux, Argo, Jenkins on Kubernetes, Open Policy Agent, Pulumi, Serverless, etc - hit me up. I would love to have a deeper conversation around some of these tools and your real-world experiences on one of our upcoming office hours
I’ve been using Jenkins in Kubernetes for almost 2 years now. I’d love to talk about it.
New Zoom Recording from our Office Hours session on 2020-06-03 is now available.
2020-06-05
2020-06-10
Office Hours starting in 15 minutes
Make sure to post your questions
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Kurt O’Connor has joined Public “Office Hours”
@here public #office-hours starting now! join us to talk shop https://cloudposse.zoom.us/j/508587304
Dan Overholt has joined Public “Office Hours”
Vijay Ravi has joined Public “Office Hours”
Robert Horrox has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
David Medinets has joined Public “Office Hours”
HariPrasad Venkatanarayana has joined Public “Office Hours”
eddie.wizelman has joined Public “Office Hours”
Zachary Loeber has joined Public “Office Hours”
Aarat Nathwani has joined Public “Office Hours”
Adam Crown has joined Public “Office Hours”
Adam Crews has joined Public “Office Hours”
i am hearing no sound.
Sound seems to be working fine. Something on your end?
youtube is working fine. I can’t join from the browser. I don’t know what is wrong.
(youtube is just for past recordings; no live streaming to youtube)
@David Medinets were you able to get the audio working? are you using the Zoom client or web UI?
I was using the zoom client on ubuntu. I have not had any audio issues in the past using zoom. I was not able to get audio working.
To follow-up. I ran into this audio issue again. When I connected via the browser, I heard audio.
sri has joined Public “Office Hours”
Omer Sen has joined Public “Office Hours”
Marc Tanne has joined Public “Office Hours”
Alex Siegman has joined Public “Office Hours”
Jie Chen has joined Public “Office Hours”
Latika Wadhwani has joined Public “Office Hours”
Michael Martin has joined Public “Office Hours”
Rahul Muraleedharan has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Bircan Bilici has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Hilal Jaffan has joined Public “Office Hours”
Amin Amos has joined Public “Office Hours”
Jeremy Schuller has joined Public “Office Hours”
PePe Amengual has joined Public “Office Hours”
Andrew Elkins has joined Public “Office Hours”
Thomas Mundt has joined Public “Office Hours”
Brian Tai has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
@Alex Siegman have a link handy to the 2-tier web architecture recommendation?
For connecting internal alb to nlb, lambda functions can be used.
you know, i’ve referenced it a bunch of times and i just cannot find it, they’ve changed how the site is with their new well-architected stuff, i’m looking though
“Before now, you had to choose either the benefits of NLB or the benefits of ALB, but you couldn’t have both together. This blog post shows you how to have your cake and eat it too, by putting an Application Load Balancer behind a Network Load Balancer.”
the picture at the top of this blog is what i was referring to though https://blog.stratus10.com/aws-best-practices-3-tier-infrastructure
but those used to be available directly from amazon with accompanying materials
New Zoom Recording from our Office Hours session on 2020-06-10 is now available.
2020-06-11
2020-06-15
is aws moving away from docker? Aws fargate 1.4(latest version) is using containerd(https://containerd.io) instead of docker as runtime engine.. https://aws.amazon.com/blogs/containers/under-the-hood-fargate-data-plane/
yes it is.
---------- Forwarded message ---------
From: Amazon Web Services, Inc. <[email protected]>
Date: Tue, Jun 9, 2020 at 2:55 AM
Subject: AWS Fargate Platform Version LATEST Flag Update [AWS Account: ]
Hello,
In the coming few months, AWS Fargate will update the LATEST flag to Platform Version (PV) 1.4.0. This means all new Amazon Elastic Container Service (ECS) Tasks or ECS Services that use the Fargate launch type and have the platformVersion field in their Task Definition set to LATEST will automatically resolve to PV 1.4.0. For customers who use Amazon VPC Endpoints along with their ECS tasks running on Fargate, the new platform version has changes that may require customer action. For more information see the FAQs below. If you do not use VPC endpoints for Amazon ECR, AWS Secrets Manager or AWS Systems Manager no action is necessary.
How does this impact me?
Customers who have set up VPC endpoints for Amazon ECR, AWS Secrets manager or AWS Systems Manager need to perform below steps:
1. Add ecr.api to their Amazon ECR VPC endpoint.
2. Ensure the AWS Secrets Manager or Systems Manager AWS VPC endpoint interfaces are added to the VPCs and subnets that are used by ECS services or ECS tasks that run on Fargate.
3. Ensure the security group in the Elastic Network Interface (ENI) associated with the task has rules to allow traffic from the task to VPC endpoints.
Till 1.3 version of aws fargate it was using docker as it’s runtime engine
2020-06-17
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
Adam Crown has joined Public “Office Hours”
@here public #office-hours starting now! join us to talk shop https://cloudposse.zoom.us/j/508587304
Brandon Wilson has joined Public “Office Hours”
Joey Freeland has joined Public “Office Hours”
Alex Vorona has joined Public “Office Hours”
Brian Tai has joined Public “Office Hours”
Marc Tanne has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Robert Horrox has joined Public “Office Hours”
Alex Siegman has joined Public “Office Hours”
sri has joined Public “Office Hours”
Stuart King has joined Public “Office Hours”
Michael Martin has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Rahul Muraleedharan has joined Public “Office Hours”
Adam Watson has joined Public “Office Hours”
Muhammed Soyer has joined Public “Office Hours”
Joe Hosteny has joined Public “Office Hours”
Mythreyee Sammeta has joined Public “Office Hours”
Am I supposed to know the password already?
hah, I actually tried “SweetOps” without having any idea…
Omer Sen has joined Public “Office Hours”
Andrew Elkins has joined Public “Office Hours”
Mythreyee Sammeta has joined Public “Office Hours”
cho has joined Public “Office Hours”
Mikael Fridh has joined Public “Office Hours”
eddie.wizelman has joined Public “Office Hours”
Can use Kensis to make scaling Database driven so that Lambda step functions can perform customized health checks and update the DB, which then can forward the WAL logs (using WAL2JSON plugin) into another Lambda to trigger scaling actions based on the data being updated? https://aws.amazon.com/blogs/database/stream-changes-from-amazon-rds-for-postgresql-using-amazon-kinesis-data-streams-and-aws-lambda/
PePe Amengual has joined Public “Office Hours”
Potential talking point: https://codefresh.io/codefresh-news/announcing-codefresh-runner/
Marcin Branski has joined Public “Office Hours”
not so much a question as much as a statement/comment of “whoa, where have i been” … but i started playing with localstack recently and am setting up a local development environment and ci/cd. pretty impressive stuff.
does anyone know if there are similar things for gcp and azure?
this is the GCP equivalent, to the extent there is one: https://cloud.google.com/sdk/gcloud/reference/beta/emulators
nice! thanks James!
Amin Amos has joined Public “Office Hours”
Anyone using ECR image scanning as part of the CI/CD ?
I have it enabled on some images on push but not synchronously as some sort of a quality gate.
James Huffman has joined Public “Office Hours”
Bircan Bilici has joined Public “Office Hours”
Bastille linux (http://bastille-linux.sourceforge.net/) not active anymore
Cisecurity hardening
Openscap
Michael Martin has joined Public “Office Hours”
Opensource docker image scanning: trivy
From aquasec
Docker cache: My ideal is probably Nginx pull through cache in each cluster, upstreaming to ECR and possibly https://goharbor.io/ for the nifty management of things.
in-cluster Storage - anyone used https://github.com/longhorn/longhorn ?
Something new, AWS lambda can now use EFS .. https://aws.amazon.com/about-aws/whats-new/2020/06/aws-lambda-support-for-amazon-elastic-file-system-now-generally-/
Network latency test: https://www.solarwinds.com/network-performance-monitor/use-cases/network-latency
any folks here with significant experience with DynamoDB?
New Zoom Recording from our Office Hours session on 2020-06-17 is now available.
2020-06-18
2020-06-21
Guys? I am looking for some tool ( nonexistant ) that is able to define projects in git monorepo ( similar to workflows in atlantis ) by fs path for example in yaml, json, etc ( I am using Gitlab CI, btw) Tool should be able:
• to detect changes in projects ( this will help to determine specific project to change in CI pipeline triggered by MR/PR ) via git diff
• run commands/tasks on changed projects
• to do basic conditions, like if multiple projects change detected ; then exit ; fi
• to do retries if error code of command/task != 0
• to timeout if command/taks is taking more than X seconds
• validate projects -> if contains specific files Of course it does not exist, currently I am using bunch of scripts, buts its not so good.
- i am using json file to define projects
- tests and looping are done via jq ( validating json schema )
- everything is bash ( a lot of non standard tools )
- changes are detected with git diff
- a lot of binaries needed, terraform,terragrunt What I found so far:
https://github.com/mumoshu/variant https://github.com/go-task/task , but these are more like only task runners.
Unfortunately monorepos are not ready yet, like if you dont want recursively find all projects, which is very inefficient…
I did some prototype:
defaults:
projects:
single_project_change: true # fail if more than 1 project is changed
validation:
- task: project:validate # run task to validate projects
spec: # schema validation
- id: name
type: string
required: yes
unique: yes
- id: path
type: regex
regex: .*
required: yes
unique: yes
- id: name
type: string
required: no
unique: no
- id: cloud
type: string
oneof:
- aws
- gcp
- azure
required: yes
unique: no
projects:
- name: foo
path: foo/john/doe
cloud: azure
depends_on:
- bar
- name: bar
path: bar/john/doe
cloud: aws
vars:
FOO: BAR
env:
HOME: /home/abc/
tasks:
before:
run:
- task: environment:validate
main:
environemnt:validate:
run:
- sh: command -v terraform
success_message: "ok"
fail_message: "terraform binary not found"
allow_exitcodes:
- 128
project:validate:
run:
- sh: if ! [ -d {{ .project.path }} ]; then printf 'non existant dir' exit 1; fi
- sh: if ! [ -f {{ .project.path }}/main.tf ]; then printf 'no tf file' exit 1; fi
terraform:validate:
run:
- sh: terraform validate {{ .project.path }}
terraform:plan:
run:
- task: login:aws
vars:
- custom: var
- custom2: var2
- sh: echo {{ .task.login:aws.output.stdout.foo }}
- sh: terraform plan {{ .project.path }}
terraform:apply:
run:
- sh: terraform apply {{ .project.path }}
login:aws:
run:
- sh: aws sso login
- sh: aws sts get-caller-identity-o json
output: foo
timeout: 300
retries: 3 # kill & start three times
login:azure:
run:
- sh: az login
- sh: az account status
output: foo
timeout: 300
retries: 3
logout:aws:
run:
- sh: aws logout
allow_fail: true
logout:azure:
run:
- sh: az logout
allow_fail: true
after:
run:
- task: logout:aws
- task: logout:azure
#/bin/sh
/usr/local/bin/myCLI -h
commands:
run_task # run task
detect_changes # show what project was changed
vars:
--auto-detect true
-p PROJECT
--var foo=bar
--env foo=bar
/usr/local/bin/myCLI run_task terraform:validate ( -p PROJECT )
/usr/local/bin/myCLI run_task terraform:plan ( -p PROJECT )
/usr/local/bin/myCLI run_task terraform:apply ( -p PROJECT )
inspired by gitlab-ci.yaml, taskfile, variant, ansible, https://github.com/mbtproject/mbt ..
@mumoshu would be nice to have something like this as variant alternative
btw, recommend starting with variant2 - as variant will probably receive less support. all engineering efforts are going into variant2 a total redesign based on HCL
@muhaha will you be on the call today? (starting in 30 m)
unfortunatelly no
its just a description of “something”, I have hard times with monorepo …
2020-06-24
Anybody have experience building a Unity3D project in CI?
@here - remember to post your questions. office hours starting in 30 minutes
I would like to resurface old question since we have some updates:
we have some async tasks. maybe around 20+ or so. Some of them run at odd hours in the middle of the night and some of them can take up to 20 min to run. I want to get super alerted if something doesn’t run or fail to run. Looking for advice on dashboarding versus alerting. Currently, the team has been trained to keep a close eye on Sentry alerts that comes in thr Slack. We had email alerts from AWS in the past but the team got tuned out.
We are trying a few things to get basic monitoring setup. Looking for general validation.
- The async tasks are running on celery. Set up the APM for Celery then create monitors in DD for that.
- Use DD custom metrics. Basically a version of statsd
- DD Support staff recommended building a lambda function and crawling the log
- Build a custom agent through datadog
I’m interested in discussing best practices around running terraform destroy in CI and any exception handling that may be used. Right now I just have terraform destroy in a after(always)) block. If terraform destroy fails the Jenkins build will fail, but I want to do something more to handle possible failure cases
And the thing about building a Unity3D project in CI
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Robert Horrox has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
Bircan Bilici has joined Public “Office Hours”
David Scott has joined Public “Office Hours”
Terraform currently does not support re-creation of resources if dependent object is changed or re-created. I’m talking about trigger like behaviour, but considering all other resources in general. It’s discussed here https://github.com/hashicorp/terraform/issues/8099 Do you know any other way to handle this in more elegant way, without using taint?
Amin Amos has joined Public “Office Hours”
Thomas Mundt has joined Public “Office Hours”
Michael Holt has joined Public “Office Hours”
cho has joined Public “Office Hours”
David Medinets has joined Public “Office Hours”
Ryan Moore has joined Public “Office Hours”
Brian Tai has joined Public “Office Hours”
sri has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
I have a question about cloudposse bastion and its integration to ping slack when someone connects. Its not working when its used just to directly tunnel through to another server. Also what’s the password for zoom?
Kurt O’Connor has joined Public “Office Hours”
Latika Wadhwani has joined Public “Office Hours”
CHONAN TSAI has joined Public “Office Hours”
CHONAN TSAI has joined Public “Office Hours”
Rahul Muraleedharan has joined Public “Office Hours”
Paul has joined Public “Office Hours”
Andrew Elkins has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Eddie Wizelman has joined Public “Office Hours”
Andrea Bolandrina has joined Public “Office Hours”
Santiago Campuzano has joined Public “Office Hours”
PePe Amengual has joined Public “Office Hours”
Primoz Cankar has joined Public “Office Hours”
I just realized I do have an ansible question. How can I name a resource like “centos-<timestamp>”?
babajide hassan has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
My question is actually a terraform question.
resource "aws_eip" "centos" {
instance = aws_instance.centos.id
vpc = true
tags = {
Name = "centos-<TIMESTAMP>"
}
}
Primoz Cankar has joined Public “Office Hours”
Santiago Campuzano has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
New Zoom Recording from our Office Hours session on 2020-06-24 is now available.
question for next office hours. upgrading a docker image in an ecs service with zero downtime with pokayoke in mind.
original thread from #aws: https://sweetops.slack.com/archives/CCT1E7JJY/p1593017494348900
2020-06-29
Any idea for opensource cloud native p2s vpn (oidc, saml integration would be nice) ?
seems that https://hub.kubeapps.com/charts/cloudposse/openvpn is integrated with github oidc only, i did not find any source code tho, generic oidc would be nice..
What is p2s?
Reasonable pricing, unfortunatelly no oidc support in opensource version .. Thanks