#office-hours (2023-05)

“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours

Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!

https://cpco.io/slack-office-hours

Meeting password: sweetops

2023-05-03

managedkaos avatar
managedkaos

For discussion: GenAI bundled in with your observability tools….

I say yes…with caveats.

https://newrelic.com/blog/nerdlog/new-relic-grok

Meet New Relic Grok, the first GenAI assistant for observabilityattachment image

Meet the first generative AI assistant for observability, New Relic Grok.

1
jimp avatar

It’s promising!

Meet New Relic Grok, the first GenAI assistant for observabilityattachment image

Meet the first generative AI assistant for observability, New Relic Grok.

Matt Gowie avatar
Matt Gowie
Honeycomb announces generative AI-driven natural language querying for observability attachment image

Honeycomb’s new Query Assistant capability empowers engineers to ask questions in plain English instead of a query language.

Matt Gowie avatar
Matt Gowie

NR’s video is insanely impressive though. They make that look like magic.

Would love to hear anyone with DD + NR experience and their thoughts on the two. Of the people that I know who use NR, they usually are not big fans of it and I’ve seen some folks switch to DD.

1
managedkaos avatar
managedkaos

i’ve used both, depending on which one was in place with the team i was supporting. These days its all DD. last team was all NR but was looking to make the switch to DD.

managedkaos avatar
managedkaos

I guess we can expect a GenAI announcement from DD any day now

2
Zoom avatar
Zoom
05:01:47 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:00:56 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Zoom avatar
Zoom
06:27:29 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:27:57 PM

Jim Park has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:05 PM

Manoj Bhagwat has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:26 PM

Nenna Salinas has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:32 PM

Michael Jenkins has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:35 PM

Michael Pursifull has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:15 PM

Elisha Bello has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:27 PM

Diego Maia has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:56 PM

Kris Musard has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:57 PM

Ishank Bansal has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:58 PM

Vlad Ionescu has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:12 PM

Charles Smith has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:30 PM

Ian Bartholomew has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:36 PM

Isaac M has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:42 PM

Jonathan Eunice has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:52 PM
Zoom avatar
Zoom
06:31:18 PM

Wasim Khan has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:20 PM

Afolabi Omotoso has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:26 PM

Ozzy Al has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:26 PM

David Hawthorne has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:41 PM

dave lundgren has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:10 PM

Abdul Aziz Tetteh has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:25 PM

Wasim Khan has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:28 PM

Tim Gourley has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:53 PM

Jeremy / Nuru (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:59 PM

Matt Calhoun has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:06 PM

Jose Figueredo has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:21 PM

Michael Vasilenko has joined Public “Office Hours”

Zoom avatar
Zoom
06:38:18 PM

John Mitchell has joined Public “Office Hours”

Zoom avatar
Zoom
06:39:24 PM

Michael Vasilenko has joined Public “Office Hours”

Zoom avatar
Zoom
06:40:02 PM

Matt Gowie has joined Public “Office Hours”

Zoom avatar
Zoom
06:42:46 PM

Sandro Aldave has joined Public “Office Hours”

Zoom avatar
Zoom
06:47:52 PM

Adelia R has joined Public “Office Hours”

Zoom avatar
Zoom
06:48:43 PM

Adebiyi Adegboye has joined Public “Office Hours”

Zoom avatar
Zoom
06:48:50 PM

Adelia R has joined Public “Office Hours”

Zoom avatar
Zoom
06:53:37 PM

Adebiyi Adegboye has joined Public “Office Hours”

Zoom avatar
Zoom
06:55:31 PM

Adelia R has joined Public “Office Hours”

Zoom avatar
Zoom
07:00:29 PM

Adebiyi Adegboye has joined Public “Office Hours”

Zoom avatar
Zoom
07:04:58 PM

Wasim Khan has joined Public “Office Hours”

Zoom avatar
Zoom
07:09:18 PM

John Mitchell has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
1debit/alternat

High availability implementation of AWS NAT instances.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
int128/terraform-aws-nat-instance

Terraform module to provision a NAT Instance using an Auto Scaling Group and Spot Instance from $1/month

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
#29689 [New Service]: AWS Verified Access

Description

Request to have new service + resources created for AWS Verified Access

Requested Resource(s) and/or Data Source(s)

☐ add resource: aws_verifiedaccess_endpoint ☐ add resource: aws_verifiedaccess_endpoint_policy ☐ #29784#29742 ☐ add resource: aws_verifiedaccess_instance_logging_configuration ☐ #29723#29781

Potential Terraform Configuration

No response

References

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/operation-list-verified-access.html

Would you like to implement a fix?

Yes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
#30380 [New Service]: VPC Lattice

Description

Support for recently announced VPC Lattice

https://aws.amazon.com/blogs/aws/simplify-service-to-service-connectivity-security-and-monitoring-with-amazon-vpc-lattice-now-generally-available/https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonvpclatticeservices.htmlhttps://awscli.amazonaws.com/v2/documentation/api/latest/reference/vpc-lattice/index.html?highlight=lattice

Requested Resource(s) and/or Data Source(s)

☑︎ aws_vpclattice_service ☑︎ aws_vpclattice_service_network ☑︎ aws_vpclattice_service_network_service_association ☑︎ aws_vpclattice_service_network_vpc_association ☑︎ aws_vpclattice_listener ☑︎ aws_vpclattice_listener_rule ☑︎ aws_vpclattice_target_group ☑︎ aws_vpclattice_access_log_subscription ☑︎ aws_vpclattice_auth_policy ☑︎ aws_vpclattice_resource_policy ☑︎ aws_vpclattice_target_group_attachment

Potential Terraform Configuration

TBD

References

https://aws.amazon.com/blogs/aws/simplify-service-to-service-connectivity-security-and-monitoring-with-amazon-vpc-lattice-now-generally-available/https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonvpclatticeservices.htmlhttps://awscli.amazonaws.com/v2/documentation/api/latest/reference/vpc-lattice/index.html?highlight=lattice

Would you like to implement a fix?

None

Hao Wang avatar
Hao Wang

Cool, TF supports Lattice now

#30380 [New Service]: VPC Lattice

Description

Support for recently announced VPC Lattice

https://aws.amazon.com/blogs/aws/simplify-service-to-service-connectivity-security-and-monitoring-with-amazon-vpc-lattice-now-generally-available/https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonvpclatticeservices.htmlhttps://awscli.amazonaws.com/v2/documentation/api/latest/reference/vpc-lattice/index.html?highlight=lattice

Requested Resource(s) and/or Data Source(s)

☑︎ aws_vpclattice_service ☑︎ aws_vpclattice_service_network ☑︎ aws_vpclattice_service_network_service_association ☑︎ aws_vpclattice_service_network_vpc_association ☑︎ aws_vpclattice_listener ☑︎ aws_vpclattice_listener_rule ☑︎ aws_vpclattice_target_group ☑︎ aws_vpclattice_access_log_subscription ☑︎ aws_vpclattice_auth_policy ☑︎ aws_vpclattice_resource_policy ☑︎ aws_vpclattice_target_group_attachment

Potential Terraform Configuration

TBD

References

https://aws.amazon.com/blogs/aws/simplify-service-to-service-connectivity-security-and-monitoring-with-amazon-vpc-lattice-now-generally-available/https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonvpclatticeservices.htmlhttps://awscli.amazonaws.com/v2/documentation/api/latest/reference/vpc-lattice/index.html?highlight=lattice

Would you like to implement a fix?

None

Nenna avatar
1

2023-05-10

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:01:25 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

Note new releases to fix S3 bucket creation problems:

terraform-aws-s3-bucket v3.1.1 releasedterraform-aws-s3-log-storage v1.3.1 releasedterraform-aws-vpc-flow-logs-s3-bucket v1.0.1 releasedterraform-aws-lb-s3-bucket v0.16.4 released

Zoom avatar
Zoom
06:28:44 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:54 PM

Andy Roth has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:00 PM

Dave Gregory has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:51 PM

Dan Medeiros has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:53 PM
Zoom avatar
Zoom
06:30:11 PM

Kris Musard has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:13 PM

dag viggo lokoeen has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:23 PM

Nenna Salinas has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:36 PM

Jonathan Eunice has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:54 PM

dag viggo lokoeen has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:03 PM

Ishank Bansal has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:14 PM

Vlad Ionescu has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:24 PM

Michael Jenkins has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:35 PM

Matt Calhoun has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:41 PM

David Hawthorne has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:47 PM

Henri Gabriel has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:16 PM

Pamela Hita has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:34 PM

Andrew Vitko has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:37 PM

Jeremy White has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:41 PM

Alex Atkinson has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:04 PM

Mike Martin has joined Public “Office Hours”

Zoom avatar
Zoom
06:34:41 PM

Paul Marcelin has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:32 PM

Isaac M has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:46 PM

Rahul has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:50 PM

Michael Pursifull has joined Public “Office Hours”

Zoom avatar
Zoom
06:37:26 PM

Tim Gourley has joined Public “Office Hours”

Zoom avatar
Zoom
06:38:54 PM

Eric Berg has joined Public “Office Hours”

Zoom avatar
Zoom
06:40:07 PM

Chris Dobbyn has joined Public “Office Hours”

Zoom avatar
Zoom
06:41:47 PM

Alex Siegman has joined Public “Office Hours”

Zoom avatar
Zoom
06:42:34 PM

Matt Gowie has joined Public “Office Hours”

Zoom avatar
Zoom
06:48:55 PM

Darren Pham has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Dendronattachment image

Dendron is a local-first, Markdown-based, hierarchical note taking tool. It is meant to help you create, organize, and collaborate on knowledge bases of any size.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
A privacy-first, open-source knowledge baseattachment image

A privacy-first, open-source platform for knowledge management and collaboration.

Zoom avatar
Zoom
06:57:45 PM

Johnmary Odenigbo has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Keep It for Macattachment image

Keep It for Mac. Write notes, keep things and find them again.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Zoom avatar
Zoom
07:08:33 PM

Hao Wang has joined Public “Office Hours”

Eric Berg avatar
Eric Berg

I’m having problems with setting incomingCidrs , using aws-load-balancer-controller Helm chart…depoyed as a helm_release resource. I’m getting the list of IPs from the Cloudflare provider, which returns lists of CIDR blocks, but I can’t seem to get the escaping or whatever right. I keep getting

│ Error: failed parsing key "inboundCidrs" with value "103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,104.16.0.0/13,104.24.0.0/14,108.162.192.0/18,131.0.72.0/22,141.101.64.0/18,162.158.0.0/15,172.64.0.0/13,173.245.48.0/20,188.114.96.0/20,190.93.240.0/20,197.234.240.0/22,198.41.128.0/17,2400:cb00::/32,2405:8100::/32,2405:b500::/32,2606:4700::/32,2803:f800::/32,2a06:98c0::/29,2c0f:f248::/32", key "0/22" has no value (cannot end with ,)

I tried "\"${join(",", data.cloudflare_ip_ranges.cloudflare.ipv4_cidr_blocks)}\"" , but same problem.

This appears to me to be related to helm’s wrapping long lines, which results in these YAML key issues.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
#166 Marshalling long, single-line strings results in (correct) line breaks
When long, single-line strings (i.e. does not contain “\n”) are marshalled, the yaml emitter seems to force line breaks at 80 characters, but does so without prefixing the string with the multi-line notation (e.g. “>” or “”). This results in unusable yaml output like in the example below.

Example:

data := yaml.MapSlice{{"test", "abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd"}}

res, _ := yaml.Marshal(data)
fmt.Println(string(res))

Output:

test: abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd
  abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd

As a workaround, simply appending “\n” to the end of a string that’s known to be lengthy works fine, although not ideal since the yaml output now exceeds the 80 char width. Mutiline “>” prefixed outputs would be preferable.

Output with trailing \n:

test: |
  abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd aabdasfadsfdasfadsfasd abdasfadsfdasfadsfasd abdasfadsfdasfadsfasd

Appreciate if someone could confirm whether this is an issue or just incorrect usage. If a fix is needed, I’d be happy to work on a PR. Thanks!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
#5568 Helm unexpectedly breaks long lines to several lines during rendering a template

Output of helm version:
Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”}

Output of kubectl version:
Not relevant.

Cloud Provider/Platform (AKS, GKE, Minikube etc.):
Not relevant.

An example of a template:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
  name: my_lovely_configmap
data:
  my_key: |-
    {{- .Values.first_layer.second.third | default dict | toYaml | trim | nindent 4 }}

Values file (description value is one long string with spaces):

first_layer:
  second:
    third:
      fourth:
      - name: first_array_element
        another_layer:
        - annotations:
            description: asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas

Rendering command:
helm-2.13.1 template -f bug_reproduce_values --execute templates/bug-reproduce.yaml helm-charts/bug-reproduce

Rendering result (description value is split to several lines.):

---
# Source: bug-reproduce/templates/bug-reproduce.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
  name: my_lovely_configmap
data:
  my_key: |-
    fourth:
    - another_layer:
      - annotations:
          description: asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas
            asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas asdasdasdas
            asdasdasdas asdasdasdas
      name: first_array_element

I’ve tried to play with several helm versions - didn’t help much. I also noticed that helm starts to break the line only if there’s whitespace met after a certain character number (~73th ??? :O).

Changing my_key: |- to others like > or just | also doesn’t help. Quoting the value also doesn’t help.

Rendering is broken both when you do helm template and helm upgrade/install.

Matt Calhoun avatar
Matt Calhoun

The underlying data type is def a list of strings (array in helm language)

Matt Calhoun avatar
Matt Calhoun

And you’re passing in a single string.

Eric Berg avatar
Eric Berg

Thanks, guys. That’s pretty much the point that I’ve come to: input format. How do I pass a list of strings into the TF helm_release set?

I’ve tried wrapping it in escaped quotes, wrapping the values in {} and [] blocks…nothing seems to work. Either it adds the annotation in a bad format, such as this:

    alb.ingress.kubernetes.io/inbound-cidrs: '[888.21.244.0/22  103.22.200.0/23  103.31.4.0/24  104.16.0.0/13]'

which yields this error:

   Warning  FailedBuildModel  10m (x18 over 21m)  ingress  Failed build model due to ingress: servicer/servicer-ingress: invalid inbound-cidrs settings on Ingress: servicer/servicer-ingress: invalid CIDR addr │
 ess: [888.21.244.0/22  103.22.200.0/23  103.31.4.0/24  104.16.0.0/13]                                                                                                                                           

So, how do i pass lists of strings like this?

Eric Berg avatar
Eric Berg

I believe that I’m generally getting stopped at the input validation point, not in the larger rendering of the chart. I commented out the use of the value and it’s still erroring.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

how about hacking it first just on the command line to get it working by calling helm

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you find the correct value, it’ll be easier to terraform it.

Eric Berg avatar
Eric Berg

Turned out that you have to escape the commas in the list.

So, to generate the string in TF:

  lb_ingress_cidrs = concat(
    data.cloudflare_ip_ranges.cloudflare.ipv4_cidr_blocks,
    var.additional_lb_ingress_cidrs,
  )
  joined_cidrs         = join("\\,", local.lb_ingress_cidrs)
  lb_ingress_cidrs_str = "\"${local.joined_cidrs}\""

And in the chart, it’s simply this:

metadata:
  annotations:
    alb.ingress.kubernetes.io/inbound-cidrs: {{ .Values.inboundCidrs }}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmm… so odd to me that inside of a quoted sting you would need to escape the commas.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks for following up with your answer.

1

2023-05-11

Eamon Keane avatar
Eamon Keane

As a counterpoint to the seemingly prevailing view (e.g. from the previous week’s OH) away from KRM (Crossplane/Anthos Config Connector), this is interesting to see Spotify blog about their terraform migration.

Their two reasons for ruling out terraform were they wanted it to be fully declarative (config as data, which I guess Cloud Posse does to an extent) and additionally break-glass functionality (able to do a quick e.g. kubectl edit cloudsql without running a pipeline). They seem to have it working across 3,000 GCP projects, so hopefully they give a talk about the rough edges they encountered. https://twitter.com/bgrant0607/status/1654870283394891776 https://engineering.atspotify.com/2023/05/fleet-management-at-spotify-part-2-the-path-to-declarative-infrastructure/

And fwiw, Alibaba’s Alipay have brewed something similar internally. https://github.com/KusionStack

Brian Grant on Twitter

Impressive GCP platform automation at scale built by Spotify on top of Config as Data, KRM, GitOps, kpt, Config Connector, and OPA Gatekeeper.

2023-05-12

venkata.mutyala avatar
venkata.mutyala

Just heard about this: https://github.com/flux-subsystem-argo/flamingo not sure when it came out but is anyone using it? It’s looks like it’s the best of both (ArgoCD + FluxCD)

flux-subsystem-argo/flamingo

Flux Subsystem for Argo - Landing Repository

Matt Gowie avatar
Matt Gowie

We looked into it (specifically @Veronika Gnilitska from my team). There is some non-intuitive stuff going on that made us rethink using it. See https://github.com/flux-subsystem-argo/flamingo/issues/16

#16 Objects are not removed when ArgoCD Application is deleted

Hi!

I’m working on FSA+tf-controller POC now, and have noticed some unexpected behaviour. Will appreciate your help!

  1. I have deployed ArgoCD Application infra with finalizers set, and Kustomization object was reconciled:
[
   {
      "group":"kustomize.toolkit.fluxcd.io",
      "health":{
         "message":"ReconciliationSucceeded - Applied revision: poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b",
         "status":"Healthy"
      },
      "kind":"Kustomization",
      "name":"infra",
      "namespace":"infra",
      "status":"Synced",
      "version":"v1beta2"
   },
   {
      "group":"source.toolkit.fluxcd.io",
      "health":{
         "message":"Succeeded - stored artifact for revision 'poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b'",
         "status":"Healthy"
      },
      "kind":"GitRepository",
      "name":"infra",
      "namespace":"infra",
      "status":"Synced",
      "version":"v1beta2"
   }
]
kubectl -n argocd get app infra -o jsonpath="{.metadata.finalizers}"
["resources-finalizer.argocd.argoproj.io"]

If I delete the Kustomization object, all related resources are removed. But destroying the Application didn’t work out. In the controller logs I see:

level=info msg="Deleting resources" application=infra
level=info msg="Deleting application's resources with Foreground propagation policy" application=infra
level=info msg="Successfully deleted 0 resources" application=infra

So only the Application was deleted.

  1. Also, if you check the Application resources list, there is a GitRepository what is not declared anywhere - there is no such file in the repo that ArgoCD is looking at. In the configuration I reference to another GitRepository object with is created by another ArgoCD Application, but they both are equal. Is is possible that FSA creates this object?
kubectl get gitrepositories -A                                      
NAMESPACE   NAME                URL                                                  AGE   READY   STATUS
infra       bootstrap           <https://github.com/masterpointio/tf-controller-poc>   20h   True    stored artifact for revision 'poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b'
infra       infra               <https://github.com/masterpointio/tf-controller-poc>   33m   True    stored artifact for revision 'poc-1/46ca9138db9a958e9251f951f4168a0e21ef396b'

Thanks in advance!

1
1
Matt Gowie avatar
Matt Gowie

Cross-posting this from #aws for discussion in this weeks #office-hours (which I may not be able to make, but I will watch the recording if ya’ll discuss this!): https://sweetops.slack.com/archives/CCT1E7JJY/p1683939758963199

Does anyone have strong opinions on how to do AWS Lambda while also managing the infrastructure via Terraform? There are a bunch of options out there, but I’ve never personally seen an implementation that I liked. My team and I are working on how to do this better and are evaluating Serverless framework (CloudFormation ), AWS SAM (has TF support, but doesn’t look great), and classic “build our own”.

Would love to hear someone who has implemented a solution that doesn’t feel disjointed and has strong opinions from real experience!

1

2023-05-14

2023-05-17

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)

I won’t make it to the call today, sorry

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:01:41 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Sean avatar

What patterns have you seen for managing configs (such as Helm values) for a large number of repeating services across many clusters.

We currently have a bash script per service with sort-of-templating that generates (hydrates) configs into each cluster/service directory.

Hydrated layout looks something like this:

├── cluster1
│   ├── services
│   │   ├── service1
│   │   │   └── helm-values.yaml
│   │   ├── service2
│   │   │   └── helm-values.yaml
│   │   ├── ...
│   │   └── service128
│   │   │   └── helm-values.yaml
├── cluster2
│   ├── repeat all the services in every cluster
├── ...
├── cluster42

An extra requirement is the templates are fed by infrastructure inputs (mostly from terraform output).

  1. And we need to helm template the k8s resources without access to the environment.
  2. Our “hack” is to write terraform output json to a file in git, like a cache. (And consul has been proposed).
Sean avatar

Thanks for hosting. Have to drop for another meeting. My 1st time joining, but will join in the future.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Sean we wrote atmos exactly for this use-case. See https://atmos.tools

Introduction to Atmos | atmos

Atmos is a workflow automation tool for DevOps to manage complex configurations with ease. It’s compatible with Terraform and many other tools.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Can show you next office hours

venkata.mutyala avatar
venkata.mutyala
Terraform Cloud updates plans with an enhanced Free tier and more flexibilityattachment image

Terraform Cloud’s Free tier now offers new features — including SSO, policy as code, and cloud agents — while new paid offerings update scaling concurrency and more.

Zoom avatar
Zoom
06:28:55 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:13 PM

Rinchin Shoysoronov has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:22 PM

Nenna Salinas has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:23 PM

venkata mutyala has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:27 PM

Emile Fugulin has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:50 PM

Hao Wang has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:19 PM

David Hawthorne has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:35 PM

Joe Caulfield has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:39 PM

Jonathan Poczatek has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:41 PM

Dave Gregory has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:52 PM

Michael Jenkins has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:27 PM

Menekse Tok has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:05 PM

Jim Park has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:43 PM

dag viggo lokoeen has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:50 PM

solomon onwuasoanya has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:15 PM

Jonathan Eunice has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:45 PM

Stevan Arychuk has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:52 PM

Paul Marcelin has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:56 PM
Zoom avatar
Zoom
06:33:57 PM

Henri Gabriel has joined Public “Office Hours”

Zoom avatar
Zoom
06:34:27 PM

Isaac M has joined Public “Office Hours”

Zoom avatar
Zoom
06:34:37 PM

Sean Roberts has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:03 PM

Matt Calhoun has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:18 PM

Oliver Schoenborn has joined Public “Office Hours”

Zoom avatar
Zoom
06:39:10 PM

Ian Bartholomew has joined Public “Office Hours”

Zoom avatar
Zoom
06:40:55 PM

Brad Curfman has joined Public “Office Hours”

Zoom avatar
Zoom
06:42:28 PM

Ozzy Al has joined Public “Office Hours”

Zoom avatar
Zoom
06:42:38 PM

Ishank Bansal has joined Public “Office Hours”

Zoom avatar
Zoom
06:42:39 PM
Zoom avatar
Zoom
06:43:12 PM

Dariusz Panasiuk has joined Public “Office Hours”

Zoom avatar
Zoom
06:43:40 PM

Eduardo Wohlers has joined Public “Office Hours”

Zoom avatar
Zoom
06:44:21 PM

Jeremy White has joined Public “Office Hours”

Zoom avatar
Zoom
06:46:10 PM

Alex Atkinson has joined Public “Office Hours”

Zoom avatar
Zoom
06:46:30 PM

Kris Musard has joined Public “Office Hours”

Zoom avatar
Zoom
06:50:36 PM

Amer Zec has joined Public “Office Hours”

Zoom avatar
Zoom
06:52:58 PM

Michael Pursifull has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
helmfile/vals

Helm-like configuration values loader with support for various sources

Zoom avatar
Zoom
06:55:51 PM

Marc Tamsky has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Alpine Linux 3.18 fixes DNS over TCP issueattachment image

Small but mighty update will help its many users – even the unwitting ones

Zoom avatar
Zoom
07:07:53 PM

dag viggo lokoeen has joined Public “Office Hours”

Zoom avatar
Zoom
07:14:22 PM

Oliver Schoenborn has joined Public “Office Hours”

Zoom avatar
Zoom
07:17:14 PM

Dave Gregory has joined Public “Office Hours”

Zoom avatar
Zoom
07:17:30 PM

Junior Jimenez has joined Public “Office Hours”

Nenna avatar
The S3 to R2 Super Slurper is now Generally Availableattachment image

Use Super Slurper to quickly, securely, and easily migrate data from S3 to R2.

Terraform Cloud updates plans with an enhanced Free tier and more flexibilityattachment image

Terraform Cloud’s Free tier now offers new features — including SSO, policy as code, and cloud agents — while new paid offerings update scaling concurrency and more.

OpenAI Confirms ChatGPT Data Breach | Trend Micro Newsattachment image

OpenAI Confirms ChatGPT Data Breach: What Happened? How to Protect Yourself?

Alex Atkinson avatar
Alex Atkinson

If you want to auto-detect the specific GH Actions deprecations mentioned today across all your repos, you can do this: https://gist.github.com/AlexAtkinson/b08037e721671ad15fa356d54f6d22e8 Just a quick job… not a pretty pretty pony of a script. Maybe one day.

2023-05-24

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)

Looks like I might miss today too I am so sorry!

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:01:00 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Jonathan Eunice avatar
Jonathan Eunice

Not specifically Terraform, but interested if anyone has recommended platforms for getting security certifications and answering security questionnaires. Ones we know about: OneTrust (formerly known as Tugboat Logic), Vanta, Drata, Secureframe. There seem to be about 73 different options out there, and don’t want to boil the entire ocean, but if anyone has such a platform they love, would appreciate hearing about it.

1
Jonathan Eunice avatar
Jonathan Eunice

What’s up with the Datadog metric kubernetes_state.node.age? It doesn’t seem to report in any logical units, like seconds or minutes, and doesn’t seem well documented, according to The Google. Anyone know how to interpret it?

1
Jonathan Eunice avatar
Jonathan Eunice

Finally, we recently had a situation with Karpenter stopping autoscaling (up or down). A bunch of host-xyz pods were present (to our eyes, a surprising number of those pods). Soon as those were deleted, everything returned to normal / good auto-scaling. Ideas?

Zoom avatar
Zoom
06:30:45 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:48 PM

Alex Atkinson has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:49 PM

Michael Jenkins has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:52 PM

Dave Gregory has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:03 PM

Nenna Salinas has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:10 PM

Henri Gabriel has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:11 PM

Sean Roberts has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:12 PM

Matt Calhoun has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:17 PM

Vlad Ionescu has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:26 PM

Jonathan Eunice has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:30 PM

Vicken Simonian has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:44 PM

Kris Musard has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:09 PM

Adebiyi Adegboye has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:47 PM

Paul Bullock has joined Public “Office Hours”

Zoom avatar
Zoom
06:48:14 PM

Jonathan Eunice has joined Public “Office Hours”

Zoom avatar
Zoom
06:50:47 PM

Marc Tamsky has joined Public “Office Hours”

Zoom avatar
Zoom
06:55:43 PM

Dariusz Panasiuk has joined Public “Office Hours”

Zoom avatar
Zoom
06:56:04 PM

CHristopher Pieper has joined Public “Office Hours”

Zoom avatar
Zoom
06:56:11 PM

Paul Marcelin has joined Public “Office Hours”

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)
Aidan W Steele on Twitterattachment image

This afternoon I’ve been playing with an idea: adding OpenTelemetry to IaC tools (e.g. Terraform here) to get a visual insight into what takes the most time to provision.

Here you can see TF needs to call CreateFunction four times before the IAM role becomes consistent

Zoom avatar
Zoom
07:00:54 PM

Hao Wang has joined Public “Office Hours”

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)
Nickste on Twitter

We’re only just getting started - more improvements to come later this year! I’m super excited about the kinds of use cases these latency improvements will unblock.

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)
Serverless Messaging: Latency Comparedattachment image

In this Bite we will compare the latency introduced by common messaging services: SQS, SNS, Step Functions, EventBridge, Kinesis, and DynamoDB Streams.

Zoom avatar
Zoom
07:09:19 PM

Denis Baryshev has joined Public “Office Hours”

Sean avatar

Office Hours Q:

  1. We use terraform output to inform other systems (such as helm values). Is anyone else caching the output somewhere (file in git, S3, …)? a. Side note: terraform init should support just fetching the state file. i. Instead it requires you to download all providers and modules. For our 1000+ tf roots, that takes a LONG TIME. ii. So I wrote a one-liner that does a simple aws s3 sync then terraform output of all those tfstates (that doesn’t require an init). Fetched that 1000+ tfstates and printed all outputs in minutes.
Nenna avatar

Links from today’s office hours:

https://openai.com/blog/introducing-the-chatgpt-app-for-ios https://medium.com/@bobbyrsec/the-dangers-of-googles-zip-tld-5e1e675e59a5 https://nyxt.atlas.engineer/ https://zed.dev/ https://www.pcworld.com/article/1919392/spacetop-is-the-first-laptop-without-a-screen.html https://www.docker.com/blog/welcome-tilt-fixing-the-pains-of-microservice-development-for-kubernetes/ https://blog.visionarycto.com/p/my-20-year-career-is-technical-debt https://github.com/datarootsio/tf-profile/ https://github.com/paololazzari/fuzzy-terraform-rm https://gist.github.com/AlexAtkinson/b08037e721671ad15fa356d54f6d22e8 https://aws.amazon.com/about-aws/whats-new/2023/05/aws-global-accelerator-extends-tcp-termination-ipv6-traffic/ https://aws.amazon.com/about-aws/whats-new/2023/05/improved-end-to-end-latencies-amazon-eventbridge-event-buses/ https://en.wikipedia.org/wiki/Year_2038_problem https://twitter.com/__steele/status/1659822002470014976 https://github.com/gabrie30/ghorg https://twitter.com/nickste/status/1626642619395883008 https://bitesizedserverless.com/bite/serverless-messaging-latency-compared/ https://medium.com/postnl-engineering/improved-eventbridge-latency-opens-up-new-use-cases-at-postnl-910fdf6b5dde https://github.com/DataDog/documentation/blob/master/content/en/integrations/kubernetes_state_core.md https://github.com/aws/karpenter/issues/2021#issuecomment-1485431932 https://developer.hashicorp.com/terraform/language/resources/provisioners/local-exec https://helm.sh/docs/topics/charts_hooks/ https://github.com/argoproj/argo-cd/issues/12060

Sean avatar

Sorry for the harshness on consul at the end there. Every app decision I make these days depends on how well maintained it is regarding CVEs.

Jonathan Eunice avatar
Jonathan Eunice

Thanks to the SweetOps office hours discussion, kubernetes_state.node.age confusion resolved:

  1. The metric indeed measured in seconds. Hat tip: @matt.
  2. Fargate nodes are indeed nodes. Hat tip: @Vlad Ionescu (he/him) True even if Fargate notes more like the cluster control plane than all other nodes, at least in our configuration. Fargate nodes naturally have much much longer run times , and therefore entirely skew statistics like average and max values.
  3. Solution: filter out Fargate nodes (e.g. kubernetes_state.node.age{stage:prod,!node:fargate-*} ) and add formula to /60 or /3600 to move to minutes or hours. Age values now make sense.
  4. Final point: De-provisioned nodes are not nodes, even if they recently were nodes. Possibly obvious…but Datadog’s instantaneous, point-in-time values may not comport with your intuitive feel about cluster/node behavior aggregated over a larger duration. Did not in our case. But with above filters, the numbers now make sense.
1
Jonathan Eunice avatar
Jonathan Eunice

Also, average/arithmetic mean is a terrible statistical aggregator, and can get in the way of easily understanding disparate value sizes. Geometric mean better, but AFAIK that not available in Datadog.

2023-05-25

Alex Atkinson avatar
Alex Atkinson

AWS’s Platform Engineering webinar is going on now. They recommend uploading outputs, such as subnet id’s, etc., to Parameter Store for consumption by other modules. So that’s AWS’s official recommendation for abstracting references away from the state files.

And in the same segment, they advise using Hashicorp Vault. It’s on their diagram. Secondarily, kinda as an afterthought they mention AWS Secrets Manager.

1
Alex Atkinson avatar
Alex Atkinson

Creating parameter store entries and secrets manager secrets as part of the terraform apply is trivial enough. What you can do with other, dependent modules, is pull those values from parameter store and set them up as TF_VAR environment variables ahead of tf apply.

venkata.mutyala avatar
venkata.mutyala
Terraform AWS provider 5.0 adds updates to default tagsattachment image

Version 5.0 of the HashiCorp Terraform AWS provider brings improvements to default tags, allowing practitioners to set tags at the provider level.

mrwacky avatar
mrwacky

Does anyone use this? I can’t think of any use case for provider-level tags - But we also have a common module that gives us tags for all the things

Terraform AWS provider 5.0 adds updates to default tagsattachment image

Version 5.0 of the HashiCorp Terraform AWS provider brings improvements to default tags, allowing practitioners to set tags at the provider level.

1
Alex Atkinson avatar
Alex Atkinson

I use the aws provider like this to ensure some required tags are always applied, such as CostCenter. The local module tags allows the addition of extras.

provider "aws" {
  region     = var.aws_region
  default_tags {
    tags = merge(
      local.global_tags,
      local.module_tags
    )
  }
}
Alex Atkinson avatar
Alex Atkinson

Keeps the accountants happy.

1
10002

2023-05-26

2023-05-31

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:00:57 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Jonathan Eunice avatar
Jonathan Eunice

Any thoughts or experience with BastionZero? Another of the “we secure your dev and ops connections” contenders (cf Teleport, StrongDM, …)

Sean avatar

Any opinions on SigNoz (the open-source version) as an alternative to “build your own” or Datadog? https://signoz.io/ It’s up to 13k stars on GitHub so popularity is clearly growing.

Zoom avatar
Zoom
06:27:30 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:27:48 PM

Nenna Salinas has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:05 PM

Adebiyi Adegboye has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:41 PM

Michael Pursifull has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:45 PM

Andy Wortman has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:59 PM

Isaac M has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:22 PM

Vlad Ionescu has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:25 PM

Nick Janjghava has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:08 PM

David Hawthorne has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:26 PM

Sean Roberts has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:29 PM

Soren Jensen has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:51 PM

Paul Bullock has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:06 PM

Marc Tamsky has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:31 PM

Jonathan Eunice has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:12 PM

Michael Jenkins has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:37 PM

Eric Berg has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:38 PM

Allan Mohr has joined Public “Office Hours”

Zoom avatar
Zoom
06:34:50 PM

Matt Gowie has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:27 PM

Matt Calhoun has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:03 PM

Charles Smith has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:07 PM

Aaron Cutchin has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:17 PM

Jim C has joined Public “Office Hours”

Zoom avatar
Zoom
06:37:44 PM

Amer Zec has joined Public “Office Hours”

Zoom avatar
Zoom
06:38:30 PM

Oliver Schoenborn has joined Public “Office Hours”

Zoom avatar
Zoom
06:43:47 PM

Vicken Simonian has joined Public “Office Hours”

Zoom avatar
Zoom
06:44:22 PM

Rinchin Shoysoronov has joined Public “Office Hours”

Zoom avatar
Zoom
06:47:38 PM

Ozzy Al has joined Public “Office Hours”

Zoom avatar
Zoom
06:52:08 PM

Sergei Valevka has joined Public “Office Hours”

Zoom avatar
Zoom
07:09:43 PM

Jeremy White has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
kvaps/kubectl-node-shell

Exec into node via kubectl

Zoom avatar
Zoom
07:20:03 PM

Dan Hamilton has joined Public “Office Hours”

Nenna avatar

Links from today’s office hours:

https://www.theregister.com/2023/05/26/microsoft_azure_linux_container/ https://www.digitaltrends.com/computing/keepass-master-password-plain-text-vulnerability/ https://thehackernews.com/2023/05/severe-flaw-in-google-clouds-cloud-sql.html https://github.com/charmbracelet/mods https://www.reddit.com/r/Terraform/comments/13ru7f9/heads_up_terraform_aws_provider_500/ https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-5-upgrade https://aws.amazon.com/about-aws/whats-new/2023/05/invoice-summary-now-available/ https://blog.brainboard.co/16-best-tools-to-design-your-cicd-engine-%EF%B8%8F-1b3533748a82 https://mkbaio.substack.com/p/please-stop-sending-me-emails-written https://supabase.com/blog/chatgpt-plugins-support-postgres https://www.hashicorp.com/blog/terraform-aws-provider-5-0-adds-updates-to-default-tags https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-eks-eks-distro-kubernetes-version-1-27/ https://aws.amazon.com/about-aws/whats-new/2023/05/aws-appsync-merged-apis-graphql-federation/ https://docs.aws.amazon.com/appsync/latest/devguide/security-authz.html https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-1.27 https://docs.aws.amazon.com/appsync/latest/devguide/WAF-Integration.html https://aws.amazon.com/blogs/mobile/appsync-waf/ https://github.com/bottlerocket-os/bottlerocket/issues/1667 https://www.bastionzero.com/ https://github.com/kvaps/kubectl-node-shell https://cloud.google.com/security/compliance/fips-140-2-validated/ https://learn.microsoft.com/en-us/azure/aks/enable-fips-nodes https://signoz.io/ https://github.com/redpanda-data/redpanda https://newsletter.pragmaticengineer.com/p/the-scoop-47?utm_source=post-email-title&publication_id=458709&post_id=120772763&isFreemail=false&utm_medium=email

1
1
Hao Wang avatar
Hao Wang

LocalAI is cool

Hao Wang avatar
Hao Wang

There are many examples in it to play with

    keyboard_arrow_up