#office-hours (2023-05)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2023-05-03
For discussion: GenAI bundled in with your observability tools….
I say yes…with caveats.
Meet the first generative AI assistant for observability, New Relic Grok.
It’s promising!
Meet the first generative AI assistant for observability, New Relic Grok.
Honeycomb just announced something very similar – https://venturebeat.com/ai/honeycomb-announces-generative-ai-driven-natural-language-querying-for-observability/
Honeycomb’s new Query Assistant capability empowers engineers to ask questions in plain English instead of a query language.
NR’s video is insanely impressive though. They make that look like magic.
Would love to hear anyone with DD + NR experience and their thoughts on the two. Of the people that I know who use NR, they usually are not big fans of it and I’ve seen some folks switch to DD.
i’ve used both, depending on which one was in place with the team i was supporting. These days its all DD. last team was all NR but was looking to make the switch to DD.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
@here office hours is starting in 30 minutes! Remember to post your questions here.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Manoj Bhagwat has joined Public “Office Hours”
Nenna Salinas has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Michael Pursifull has joined Public “Office Hours”
Elisha Bello has joined Public “Office Hours”
Diego Maia has joined Public “Office Hours”
Kris Musard has joined Public “Office Hours”
Ishank Bansal has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Charles Smith has joined Public “Office Hours”
Ian Bartholomew has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Jonathan Eunice has joined Public “Office Hours”
Josh has joined Public “Office Hours”
Wasim Khan has joined Public “Office Hours”
Afolabi Omotoso has joined Public “Office Hours”
Ozzy Al has joined Public “Office Hours”
David Hawthorne has joined Public “Office Hours”
dave lundgren has joined Public “Office Hours”
Abdul Aziz Tetteh has joined Public “Office Hours”
Wasim Khan has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Jeremy / Nuru (Cloud Posse) has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Jose Figueredo has joined Public “Office Hours”
Michael Vasilenko has joined Public “Office Hours”
John Mitchell has joined Public “Office Hours”
Michael Vasilenko has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Sandro Aldave has joined Public “Office Hours”
Adelia R has joined Public “Office Hours”
Adebiyi Adegboye has joined Public “Office Hours”
Adelia R has joined Public “Office Hours”
Adebiyi Adegboye has joined Public “Office Hours”
Adelia R has joined Public “Office Hours”
Adebiyi Adegboye has joined Public “Office Hours”
Wasim Khan has joined Public “Office Hours”
John Mitchell has joined Public “Office Hours”
High availability implementation of AWS NAT instances.
Terraform module to provision a NAT Instance using an Auto Scaling Group and Spot Instance from $1/month
Description
Request to have new service + resources created for AWS Verified Access
Requested Resource(s) and/or Data Source(s)
☐ add resource: aws_verifiedaccess_endpoint ☐ add resource: aws_verifiedaccess_endpoint_policy ☐ #29784 ☐ #29742 ☐ add resource: aws_verifiedaccess_instance_logging_configuration ☐ #29723 ☐ #29781
Potential Terraform Configuration
No response
References
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/operation-list-verified-access.html
Would you like to implement a fix?
Yes
Description
Support for recently announced VPC Lattice
• https://aws.amazon.com/blogs/aws/simplify-service-to-service-connectivity-security-and-monitoring-with-amazon-vpc-lattice-now-generally-available/ • https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonvpclatticeservices.html • https://awscli.amazonaws.com/v2/documentation/api/latest/reference/vpc-lattice/index.html?highlight=lattice
Requested Resource(s) and/or Data Source(s)
☑︎ aws_vpclattice_service
☑︎ aws_vpclattice_service_network
☑︎ aws_vpclattice_service_network_service_association
☑︎ aws_vpclattice_service_network_vpc_association
☑︎ aws_vpclattice_listener
☑︎ aws_vpclattice_listener_rule
☑︎ aws_vpclattice_target_group
☑︎ aws_vpclattice_access_log_subscription
☑︎ aws_vpclattice_auth_policy
☑︎ aws_vpclattice_resource_policy
☑︎ aws_vpclattice_target_group_attachment
Potential Terraform Configuration
TBD
References
• https://aws.amazon.com/blogs/aws/simplify-service-to-service-connectivity-security-and-monitoring-with-amazon-vpc-lattice-now-generally-available/ • https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonvpclatticeservices.html • https://awscli.amazonaws.com/v2/documentation/api/latest/reference/vpc-lattice/index.html?highlight=lattice
Would you like to implement a fix?
None
Cool, TF supports Lattice now
Description
Support for recently announced VPC Lattice
• https://aws.amazon.com/blogs/aws/simplify-service-to-service-connectivity-security-and-monitoring-with-amazon-vpc-lattice-now-generally-available/ • https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonvpclatticeservices.html • https://awscli.amazonaws.com/v2/documentation/api/latest/reference/vpc-lattice/index.html?highlight=lattice
Requested Resource(s) and/or Data Source(s)
☑︎ aws_vpclattice_service
☑︎ aws_vpclattice_service_network
☑︎ aws_vpclattice_service_network_service_association
☑︎ aws_vpclattice_service_network_vpc_association
☑︎ aws_vpclattice_listener
☑︎ aws_vpclattice_listener_rule
☑︎ aws_vpclattice_target_group
☑︎ aws_vpclattice_access_log_subscription
☑︎ aws_vpclattice_auth_policy
☑︎ aws_vpclattice_resource_policy
☑︎ aws_vpclattice_target_group_attachment
Potential Terraform Configuration
TBD
References
• https://aws.amazon.com/blogs/aws/simplify-service-to-service-connectivity-security-and-monitoring-with-amazon-vpc-lattice-now-generally-available/ • https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonvpclatticeservices.html • https://awscli.amazonaws.com/v2/documentation/api/latest/reference/vpc-lattice/index.html?highlight=lattice
Would you like to implement a fix?
None
Links from today’s office hours:
https://github.com/Madh93/tpm https://github.com/paololazzari/terraform-repl https://github.com/cloudposse/bastion https://github.com/hashicorp/terraform-provider-aws/milestone/226 https://aws.amazon.com/about-aws/whats-new/2022/12/aws-compute-optimizer-amazon-ecs-services-aws-fargate/ https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-s3-security-best-practices-buckets-default/ https://aws.amazon.com/about-aws/whats-new/2023/04/aws-verified-access-generally-available/ https://www.hashicorp.com/blog/kubernetes-vault-integration-via-sidecar-agent-injector-vs-csi-provider https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-rds-m7g-r7g-database-instances/ https://aws.amazon.com/blogs/aws/new-set-up-your-aws-notifications-in-one-place/ https://github.com/hashicorp/terraform-provider-aws/issues/29689 https://github.com/hashicorp/terraform-provider-aws/issues/30380 https://github.com/cloudposse/geodesic https://github.com/cloudposse/packages/tree/master/vendor https://aws.amazon.com/verified-access/pricing/ https://docs.aws.amazon.com/chatbot/latest/adminguide/what-is.html https://marbot.io https://cloudonaut.io/ https://cloudonaut.io/ec2-checklist-seven-things-to-do-after-launching-an-instance/ https://aws.amazon.com/marketplace/pp/prodview-sykoblbsdgw2o https://bucketav.com/features/ https://docs.konghq.com/hub/ https://github.com/1debit/alternat https://www.krakend.io/ https://github.com/int128/terraform-aws-nat-instance
2023-05-10
@here office hours is starting in 30 minutes! Remember to post your questions here.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
Dave Gregory has joined Public “Office Hours”
Dan Medeiros has joined Public “Office Hours”
Isa has joined Public “Office Hours”
Kris Musard has joined Public “Office Hours”
dag viggo lokoeen has joined Public “Office Hours”
Nenna Salinas has joined Public “Office Hours”
Jonathan Eunice has joined Public “Office Hours”
dag viggo lokoeen has joined Public “Office Hours”
Ishank Bansal has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
David Hawthorne has joined Public “Office Hours”
Henri Gabriel has joined Public “Office Hours”
Pamela Hita has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Jeremy White has joined Public “Office Hours”
Alex Atkinson has joined Public “Office Hours”
Mike Martin has joined Public “Office Hours”
Paul Marcelin has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Rahul has joined Public “Office Hours”
Michael Pursifull has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Chris Dobbyn has joined Public “Office Hours”
Alex Siegman has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Darren Pham has joined Public “Office Hours”
Dendron is a local-first, Markdown-based, hierarchical note taking tool. It is meant to help you create, organize, and collaborate on knowledge bases of any size.
A privacy-first, open-source platform for knowledge management and collaboration.
Johnmary Odenigbo has joined Public “Office Hours”
Keep It for Mac. Write notes, keep things and find them again.
Hao Wang has joined Public “Office Hours”
I’m having problems with setting incomingCidrs
, using aws-load-balancer-controller
Helm chart…depoyed as a helm_release resource. I’m getting the list of IPs from the Cloudflare provider, which returns lists of CIDR blocks, but I can’t seem to get the escaping or whatever right. I keep getting
│ Error: failed parsing key "inboundCidrs" with value "103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,104.16.0.0/13,104.24.0.0/14,108.162.192.0/18,131.0.72.0/22,141.101.64.0/18,162.158.0.0/15,172.64.0.0/13,173.245.48.0/20,188.114.96.0/20,190.93.240.0/20,197.234.240.0/22,198.41.128.0/17,2400:cb00::/32,2405:8100::/32,2405:b500::/32,2606:4700::/32,2803:f800::/32,2a06:98c0::/29,2c0f:f248::/32", key "0/22" has no value (cannot end with ,)
I tried "\"${join(",", data.cloudflare_ip_ranges.cloudflare.ipv4_cidr_blocks)}\""
, but same problem.
This appears to me to be related to helm’s wrapping long lines, which results in these YAML key issues.
When long, single-line strings (i.e. does not contain “\n”) are marshalled, the yaml emitter seems to force line breaks at 80 characters, but does so without prefixing the string with the multi-line notation (e.g. “>” or “ | ”). This results in unusable yaml output like in the example below. |
Example:
data := yaml.MapSlice{{"test", "abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd"}}
res, _ := yaml.Marshal(data)
fmt.Println(string(res))
Output:
test: abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd
abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd
As a workaround, simply appending “\n” to the end of a string that’s known to be lengthy works fine, although not ideal since the yaml output now exceeds the 80 char width. Mutiline “>” prefixed outputs would be preferable.
Output with trailing \n:
test: |
abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd aabdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd
Appreciate if someone could confirm whether this is an issue or just incorrect usage. If a fix is needed, I’d be happy to work on a PR. Thanks!
Output of helm version
:
Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”}
Output of kubectl version
:
Not relevant.
Cloud Provider/Platform (AKS, GKE, Minikube etc.):
Not relevant.
An example of a template:
apiVersion: v1
kind: ConfigMap
metadata:
labels:
name: my_lovely_configmap
data:
my_key: |-
{{- .Values.first_layer.second.third | default dict | toYaml | trim | nindent 4 }}
Values file (description
value is one long string with spaces):
first_layer:
second:
third:
fourth:
- name: first_array_element
another_layer:
- annotations:
description: asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas
Rendering command:
helm-2.13.1 template -f bug_reproduce_values --execute templates/bug-reproduce.yaml helm-charts/bug-reproduce
Rendering result (description
value is split to several lines.):
---
# Source: bug-reproduce/templates/bug-reproduce.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
name: my_lovely_configmap
data:
my_key: |-
fourth:
- another_layer:
- annotations:
description: asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas
asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas
asdasdasdas asdasdasdas
name: first_array_element
I’ve tried to play with several helm versions - didn’t help much. I also noticed that helm starts to break the line only if there’s whitespace met after a certain character number (~73th ??? :O).
Changing my_key: |-
to others like >
or just |
also doesn’t help. Quoting the value also doesn’t help.
Rendering is broken both when you do helm template
and helm upgrade/install
.
The underlying data type is def a list of strings (array in helm language)
And you’re passing in a single string.
Thanks, guys. That’s pretty much the point that I’ve come to: input format. How do I pass a list of strings into the TF helm_release
set?
I’ve tried wrapping it in escaped quotes, wrapping the values in {}
and []
blocks…nothing seems to work. Either it adds the annotation in a bad format, such as this:
alb.ingress.kubernetes.io/inbound-cidrs: '[888.21.244.0/22 103.22.200.0/23 103.31.4.0/24 104.16.0.0/13]'
which yields this error:
Warning FailedBuildModel 10m (x18 over 21m) ingress Failed build model due to ingress: servicer/servicer-ingress: invalid inbound-cidrs settings on Ingress: servicer/servicer-ingress: invalid CIDR addr │
ess: [888.21.244.0/22 103.22.200.0/23 103.31.4.0/24 104.16.0.0/13]
So, how do i pass lists of strings like this?
I believe that I’m generally getting stopped at the input validation point, not in the larger rendering of the chart. I commented out the use of the value and it’s still erroring.
how about hacking it first just on the command line to get it working by calling helm
If you find the correct value, it’ll be easier to terraform it.
Turned out that you have to escape the commas in the list.
So, to generate the string in TF:
lb_ingress_cidrs = concat(
data.cloudflare_ip_ranges.cloudflare.ipv4_cidr_blocks,
var.additional_lb_ingress_cidrs,
)
joined_cidrs = join("\\,", local.lb_ingress_cidrs)
lb_ingress_cidrs_str = "\"${local.joined_cidrs}\""
And in the chart, it’s simply this:
metadata:
annotations:
alb.ingress.kubernetes.io/inbound-cidrs: {{ .Values.inboundCidrs }}
Hrmm… so odd to me that inside of a quoted sting you would need to escape the commas.
Links from today’s office hours: https://servian.dev/terraform-local-providers-and-registry-mirror-configuration-b963117dfffa https://aws.amazon.com/about-aws/whats-new/2023/05/aws-backup-cross-region-backups-four-regions/ https://techcrunch.com/2023/05/10/aws-open-sources-snapchange-and-cedar-sdk/amp/ https://github.com/cedar-policy/ https://www.bleepingcomputer.com/news/security/github-now-auto-blocks-token-and-api-key-leaks-for-all-repos/amp/ https://www.githubstatus.com/history https://help.evernote.com/hc/en-us/articles/12748274247059-Collaborative-Editing-Overview https://obsidian.md/ https://sweetops.slack.com/archives/CHDR1EWNA/p1683742684063129 https://github.com/develeap/terraform-provider-chatgpt https://janik6n.net/posts/manage-multiple-terraform-projects-in-monorepo/ https://atmos.tools/ https://aws.amazon.com/about-aws/whats-new/2023/05/aws-appsync-graphql-apis-private-api-support/ https://github.blog/changelog/2023-05-10-github-actions-actions-runner-controller-public-beta/ https://www.osohq.com https://logseq.com/ https://wiki.dendron.so/ https://goblin.tools https://reinventedsoftware.com/keepit/ https://twitter.com/slackhq/status/521894442064560128 https://ourtechroom.com/tech/slack-technology-stack/ https://reg.rainfocus.com/flow/github/universe23/cfp/page/cfslandingpage https://github.com/kubernetes-sigs/aws-load-balancer-controller
2023-05-11
As a counterpoint to the seemingly prevailing view (e.g. from the previous week’s OH) away from KRM (Crossplane/Anthos Config Connector), this is interesting to see Spotify blog about their terraform migration.
Their two reasons for ruling out terraform were they wanted it to be fully declarative (config as data, which I guess Cloud Posse does to an extent) and additionally break-glass functionality (able to do a quick e.g. kubectl edit cloudsql
without running a pipeline). They seem to have it working across 3,000 GCP projects, so hopefully they give a talk about the rough edges they encountered.
https://twitter.com/bgrant0607/status/1654870283394891776
https://engineering.atspotify.com/2023/05/fleet-management-at-spotify-part-2-the-path-to-declarative-infrastructure/
And fwiw, Alibaba’s Alipay have brewed something similar internally. https://github.com/KusionStack
Impressive GCP platform automation at scale built by Spotify on top of Config as Data, KRM, GitOps, kpt, Config Connector, and OPA Gatekeeper.
2023-05-12
Just heard about this: https://github.com/flux-subsystem-argo/flamingo not sure when it came out but is anyone using it? It’s looks like it’s the best of both (ArgoCD + FluxCD)
Flux Subsystem for Argo - Landing Repository
We looked into it (specifically @Veronika Gnilitska from my team). There is some non-intuitive stuff going on that made us rethink using it. See https://github.com/flux-subsystem-argo/flamingo/issues/16
Hi!
I’m working on FSA+tf-controller POC now, and have noticed some unexpected behaviour. Will appreciate your help!
- I have deployed ArgoCD Application
infra
with finalizers set, and Kustomization object was reconciled:
[
{
"group":"kustomize.toolkit.fluxcd.io",
"health":{
"message":"ReconciliationSucceeded - Applied revision: poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b",
"status":"Healthy"
},
"kind":"Kustomization",
"name":"infra",
"namespace":"infra",
"status":"Synced",
"version":"v1beta2"
},
{
"group":"source.toolkit.fluxcd.io",
"health":{
"message":"Succeeded - stored artifact for revision 'poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b'",
"status":"Healthy"
},
"kind":"GitRepository",
"name":"infra",
"namespace":"infra",
"status":"Synced",
"version":"v1beta2"
}
]
kubectl -n argocd get app infra -o jsonpath="{.metadata.finalizers}"
["resources-finalizer.argocd.argoproj.io"]
If I delete the Kustomization object, all related resources are removed. But destroying the Application didn’t work out. In the controller logs I see:
level=info msg="Deleting resources" application=infra
level=info msg="Deleting application's resources with Foreground propagation policy" application=infra
level=info msg="Successfully deleted 0 resources" application=infra
So only the Application was deleted.
- Also, if you check the Application resources list, there is a GitRepository what is not declared anywhere - there is no such file in the repo that ArgoCD is looking at. In the configuration I reference to another GitRepository object with is created by another ArgoCD Application, but they both are equal. Is is possible that FSA creates this object?
kubectl get gitrepositories -A
NAMESPACE NAME URL AGE READY STATUS
infra bootstrap <https://github.com/masterpointio/tf-controller-poc> 20h True stored artifact for revision 'poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b'
infra infra <https://github.com/masterpointio/tf-controller-poc> 33m True stored artifact for revision 'poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b'
Thanks in advance!
Cross-posting this from #aws for discussion in this weeks #office-hours (which I may not be able to make, but I will watch the recording if ya’ll discuss this!): https://sweetops.slack.com/archives/CCT1E7JJY/p1683939758963199
Does anyone have strong opinions on how to do AWS Lambda while also managing the infrastructure via Terraform? There are a bunch of options out there, but I’ve never personally seen an implementation that I liked. My team and I are working on how to do this better and are evaluating Serverless framework (CloudFormation ), AWS SAM (has TF support, but doesn’t look great), and classic “build our own”.
Would love to hear someone who has implemented a solution that doesn’t feel disjointed and has strong opinions from real experience!
2023-05-14
2023-05-17
@here office hours is starting in 30 minutes! Remember to post your questions here.
What patterns have you seen for managing configs (such as Helm values) for a large number of repeating services across many clusters.
We currently have a bash script per service with sort-of-templating that generates (hydrates) configs into each cluster/service directory.
Hydrated layout looks something like this:
├── cluster1
│ ├── services
│ │ ├── service1
│ │ │ └── helm-values.yaml
│ │ ├── service2
│ │ │ └── helm-values.yaml
│ │ ├── ...
│ │ └── service128
│ │ │ └── helm-values.yaml
├── cluster2
│ ├── repeat all the services in every cluster
├── ...
├── cluster42
An extra requirement is the templates are fed by infrastructure inputs (mostly from terraform output
).
- And we need to
helm template
the k8s resources without access to the environment. - Our “hack” is to write
terraform output
json to a file in git, like a cache. (And consul has been proposed).
Thanks for hosting. Have to drop for another meeting. My 1st time joining, but will join in the future.
@Sean we wrote atmos exactly for this use-case. See https://atmos.tools
Atmos is a workflow automation tool for DevOps to manage complex configurations with ease. It’s compatible with Terraform and many other tools.
Can show you next office hours
Terraform Cloud’s Free tier now offers new features — including SSO, policy as code, and cloud agents — while new paid offerings update scaling concurrency and more.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Rinchin Shoysoronov has joined Public “Office Hours”
Nenna Salinas has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Emile Fugulin has joined Public “Office Hours”
Hao Wang has joined Public “Office Hours”
David Hawthorne has joined Public “Office Hours”
Joe Caulfield has joined Public “Office Hours”
Jonathan Poczatek has joined Public “Office Hours”
Dave Gregory has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Menekse Tok has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
dag viggo lokoeen has joined Public “Office Hours”
solomon onwuasoanya has joined Public “Office Hours”
Jonathan Eunice has joined Public “Office Hours”
Stevan Arychuk has joined Public “Office Hours”
Paul Marcelin has joined Public “Office Hours”
Isa has joined Public “Office Hours”
Henri Gabriel has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Sean Roberts has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Ian Bartholomew has joined Public “Office Hours”
Brad Curfman has joined Public “Office Hours”
Ozzy Al has joined Public “Office Hours”
Ishank Bansal has joined Public “Office Hours”
Isa has joined Public “Office Hours”
Dariusz Panasiuk has joined Public “Office Hours”
Eduardo Wohlers has joined Public “Office Hours”
Jeremy White has joined Public “Office Hours”
Alex Atkinson has joined Public “Office Hours”
Kris Musard has joined Public “Office Hours”
Amer Zec has joined Public “Office Hours”
Michael Pursifull has joined Public “Office Hours”
Helm-like configuration values loader with support for various sources
Marc Tamsky has joined Public “Office Hours”
Small but mighty update will help its many users – even the unwitting ones
dag viggo lokoeen has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Dave Gregory has joined Public “Office Hours”
Junior Jimenez has joined Public “Office Hours”
Links from today’s office hours:
https://blog.cloudflare.com/r2-super-slurper-ga/ https://www.hashicorp.com/blog/terraform-cloud-updates-plans-with-an-enhanced-free-tier-and-more-flexibility https://news.trendmicro.com/2023/05/13/openai-chatgpt-data-breach/ https://techcrunch.com/2023/05/12/aws-announces-new-version-of-aurora-database-that-strips-out-i-o-costs/amp/ https://github.com/liggitt/audit2rbac https://newsletter.pragmaticengineer.com/p/datadogs-65myear-customer-mystery https://github.com/hashicorp/tfc-workflows-github https://blog.aquasec.com/leveraging-kubernetes-rbac-to-backdoor-clusters https://navendu.me/posts/gateway-and-mesh/ https://itnext.io/k8sgpt-localai-unlock-kubernetes-superpowers-for-free-584790de9b65 https://github.com/flux-subsystem-argo/flamingo https://github.com/AlexNabokikh/tfsort https://www.theregister.com/2023/05/16/alpine_linux_318/ https://martinheinz.dev/blog/92 https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ https://github.com/env0/custom-flows-examples/tree/main/dynamic-backend
Use Super Slurper to quickly, securely, and easily migrate data from S3 to R2.
Terraform Cloud’s Free tier now offers new features — including SSO, policy as code, and cloud agents — while new paid offerings update scaling concurrency and more.
OpenAI Confirms ChatGPT Data Breach: What Happened? How to Protect Yourself?
If you want to auto-detect the specific GH Actions deprecations mentioned today across all your repos, you can do this: https://gist.github.com/AlexAtkinson/b08037e721671ad15fa356d54f6d22e8 Just a quick job… not a pretty pretty pony of a script. Maybe one day.
2023-05-24
@here office hours is starting in 30 minutes! Remember to post your questions here.
Not specifically Terraform, but interested if anyone has recommended platforms for getting security certifications and answering security questionnaires. Ones we know about: OneTrust (formerly known as Tugboat Logic), Vanta, Drata, Secureframe. There seem to be about 73 different options out there, and don’t want to boil the entire ocean, but if anyone has such a platform they love, would appreciate hearing about it.
What’s up with the Datadog metric kubernetes_state.node.age
? It doesn’t seem to report in any logical units, like seconds or minutes, and doesn’t seem well documented, according to The Google. Anyone know how to interpret it?
Finally, we recently had a situation with Karpenter stopping autoscaling (up or down). A bunch of host-xyz
pods were present (to our eyes, a surprising number of those pods). Soon as those were deleted, everything returned to normal / good auto-scaling. Ideas?
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Alex Atkinson has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Dave Gregory has joined Public “Office Hours”
Nenna Salinas has joined Public “Office Hours”
Henri Gabriel has joined Public “Office Hours”
Sean Roberts has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Jonathan Eunice has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Kris Musard has joined Public “Office Hours”
Adebiyi Adegboye has joined Public “Office Hours”
Paul Bullock has joined Public “Office Hours”
Jonathan Eunice has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Dariusz Panasiuk has joined Public “Office Hours”
CHristopher Pieper has joined Public “Office Hours”
Paul Marcelin has joined Public “Office Hours”
This afternoon I’ve been playing with an idea: adding OpenTelemetry to IaC tools (e.g. Terraform here) to get a visual insight into what takes the most time to provision.
Here you can see TF needs to call CreateFunction four times before the IAM role becomes consistent
Hao Wang has joined Public “Office Hours”
We’re only just getting started - more improvements to come later this year! I’m super excited about the kinds of use cases these latency improvements will unblock.
In this Bite we will compare the latency introduced by common messaging services: SQS, SNS, Step Functions, EventBridge, Kinesis, and DynamoDB Streams.
Denis Baryshev has joined Public “Office Hours”
Office Hours Q:
- We use
terraform output
to inform other systems (such as helm values). Is anyone else caching the output somewhere (file in git, S3, …)? a. Side note:terraform init
should support just fetching the state file. i. Instead it requires you to download all providers and modules. For our 1000+ tf roots, that takes a LONG TIME. ii. So I wrote a one-liner that does a simpleaws s3 sync
thenterraform output
of all those tfstates (that doesn’t require an init). Fetched that 1000+ tfstates and printed all outputs in minutes.
Links from today’s office hours:
https://openai.com/blog/introducing-the-chatgpt-app-for-ios https://medium.com/@bobbyrsec/the-dangers-of-googles-zip-tld-5e1e675e59a5 https://nyxt.atlas.engineer/ https://zed.dev/ https://www.pcworld.com/article/1919392/spacetop-is-the-first-laptop-without-a-screen.html https://www.docker.com/blog/welcome-tilt-fixing-the-pains-of-microservice-development-for-kubernetes/ https://blog.visionarycto.com/p/my-20-year-career-is-technical-debt https://github.com/datarootsio/tf-profile/ https://github.com/paololazzari/fuzzy-terraform-rm https://gist.github.com/AlexAtkinson/b08037e721671ad15fa356d54f6d22e8 https://aws.amazon.com/about-aws/whats-new/2023/05/aws-global-accelerator-extends-tcp-termination-ipv6-traffic/ https://aws.amazon.com/about-aws/whats-new/2023/05/improved-end-to-end-latencies-amazon-eventbridge-event-buses/ https://en.wikipedia.org/wiki/Year_2038_problem https://twitter.com/__steele/status/1659822002470014976 https://github.com/gabrie30/ghorg https://twitter.com/nickste/status/1626642619395883008 https://bitesizedserverless.com/bite/serverless-messaging-latency-compared/ https://medium.com/postnl-engineering/improved-eventbridge-latency-opens-up-new-use-cases-at-postnl-910fdf6b5dde https://github.com/DataDog/documentation/blob/master/content/en/integrations/kubernetes_state_core.md https://github.com/aws/karpenter/issues/2021#issuecomment-1485431932 https://developer.hashicorp.com/terraform/language/resources/provisioners/local-exec https://helm.sh/docs/topics/charts_hooks/ https://github.com/argoproj/argo-cd/issues/12060
Sorry for the harshness on consul
at the end there. Every app decision I make these days depends on how well maintained it is regarding CVEs.
Thanks to the SweetOps office hours discussion, kubernetes_state.node.age
confusion resolved:
- The metric indeed measured in seconds. Hat tip: @matt.
- Fargate nodes are indeed nodes. Hat tip: @Vlad Ionescu (he/him) True even if Fargate notes more like the cluster control plane than all other nodes, at least in our configuration. Fargate nodes naturally have much much longer run times , and therefore entirely skew statistics like average and max values.
- Solution: filter out Fargate nodes (e.g.
kubernetes_state.node.age{stage:prod,!node:fargate-*}
) and add formula to/60
or/3600
to move to minutes or hours. Age values now make sense. - Final point: De-provisioned nodes are not nodes, even if they recently were nodes. Possibly obvious…but Datadog’s instantaneous, point-in-time values may not comport with your intuitive feel about cluster/node behavior aggregated over a larger duration. Did not in our case. But with above filters, the numbers now make sense.
Also, average/arithmetic mean is a terrible statistical aggregator, and can get in the way of easily understanding disparate value sizes. Geometric mean better, but AFAIK that not available in Datadog.
2023-05-25
AWS’s Platform Engineering webinar is going on now. They recommend uploading outputs, such as subnet id’s, etc., to Parameter Store for consumption by other modules. So that’s AWS’s official recommendation for abstracting references away from the state files.
And in the same segment, they advise using Hashicorp Vault. It’s on their diagram. Secondarily, kinda as an afterthought they mention AWS Secrets Manager.
Creating parameter store entries and secrets manager secrets as part of the terraform apply is trivial enough. What you can do with other, dependent modules, is pull those values from parameter store and set them up as TF_VAR environment variables ahead of tf apply.
Version 5.0 of the HashiCorp Terraform AWS provider brings improvements to default tags, allowing practitioners to set tags at the provider level.
Does anyone use this? I can’t think of any use case for provider-level tags - But we also have a common module that gives us tags for all the things
Version 5.0 of the HashiCorp Terraform AWS provider brings improvements to default tags, allowing practitioners to set tags at the provider level.
I use the aws provider like this to ensure some required tags are always applied, such as CostCenter. The local module tags allows the addition of extras.
provider "aws" {
region = var.aws_region
default_tags {
tags = merge(
local.global_tags,
local.module_tags
)
}
}
2023-05-26
2023-05-31
@here office hours is starting in 30 minutes! Remember to post your questions here.
Any thoughts or experience with BastionZero? Another of the “we secure your dev and ops connections” contenders (cf Teleport, StrongDM, …)
Any opinions on SigNoz (the open-source version) as an alternative to “build your own” or Datadog? https://signoz.io/ It’s up to 13k stars on GitHub so popularity is clearly growing.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Nenna Salinas has joined Public “Office Hours”
Adebiyi Adegboye has joined Public “Office Hours”
Michael Pursifull has joined Public “Office Hours”
Andy Wortman has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Nick Janjghava has joined Public “Office Hours”
David Hawthorne has joined Public “Office Hours”
Sean Roberts has joined Public “Office Hours”
Soren Jensen has joined Public “Office Hours”
Paul Bullock has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Jonathan Eunice has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Allan Mohr has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Charles Smith has joined Public “Office Hours”
Aaron Cutchin has joined Public “Office Hours”
Jim C has joined Public “Office Hours”
Amer Zec has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Rinchin Shoysoronov has joined Public “Office Hours”
Ozzy Al has joined Public “Office Hours”
Sergei Valevka has joined Public “Office Hours”
Jeremy White has joined Public “Office Hours”
Exec into node via kubectl
Dan Hamilton has joined Public “Office Hours”
Links from today’s office hours:
https://www.theregister.com/2023/05/26/microsoft_azure_linux_container/ https://www.digitaltrends.com/computing/keepass-master-password-plain-text-vulnerability/ https://thehackernews.com/2023/05/severe-flaw-in-google-clouds-cloud-sql.html https://github.com/charmbracelet/mods https://www.reddit.com/r/Terraform/comments/13ru7f9/heads_up_terraform_aws_provider_500/ https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-5-upgrade https://aws.amazon.com/about-aws/whats-new/2023/05/invoice-summary-now-available/ https://blog.brainboard.co/16-best-tools-to-design-your-cicd-engine-%EF%B8%8F-1b3533748a82 https://mkbaio.substack.com/p/please-stop-sending-me-emails-written https://supabase.com/blog/chatgpt-plugins-support-postgres https://www.hashicorp.com/blog/terraform-aws-provider-5-0-adds-updates-to-default-tags https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-eks-eks-distro-kubernetes-version-1-27/ https://aws.amazon.com/about-aws/whats-new/2023/05/aws-appsync-merged-apis-graphql-federation/ https://docs.aws.amazon.com/appsync/latest/devguide/security-authz.html https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-1.27 https://docs.aws.amazon.com/appsync/latest/devguide/WAF-Integration.html https://aws.amazon.com/blogs/mobile/appsync-waf/ https://github.com/bottlerocket-os/bottlerocket/issues/1667 https://www.bastionzero.com/ https://github.com/kvaps/kubectl-node-shell https://cloud.google.com/security/compliance/fips-140-2-validated/ https://learn.microsoft.com/en-us/azure/aks/enable-fips-nodes https://signoz.io/ https://github.com/redpanda-data/redpanda https://newsletter.pragmaticengineer.com/p/the-scoop-47?utm_source=post-email-title&publication_id=458709&post_id=120772763&isFreemail=false&utm_medium=email
LocalAI is cool
There are many examples in it to play with