#office-hours (2023-08)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2023-08-01
Might be old news but first i’m seeing it: Replit is getting into app deployment/hosting. makes sense to me!
2023-08-02
This one is also interesting… Digital twins of all parts of the system along with integrated plans and applies… https://www.youtube.com/watch?v=zyEOYl23pd8
cool. might have missed that one.
last one… https://codepal.ai/terraform-writer
and the follow on is the CI/CD writer… https://codepal.ai/cicd-pipeline-writer#
Thanks for the links!
Wow!!
I would take this to mean the LLM has internet capabilities to look up modules
Or maybe the module has been around long enough to be used as a parameter to the LLM
i had pretty good results with chatGPT but say to always use null-label in that new (know something about me) section which seems to make it more useful. It’d be super nice if it the openAI model got trained on stuff like code documentation more frequently.. but alas
In ChatGPT, I wish I could have multiple “about me” because the prompts need to be different, as I wear multiple hats.
It also isn’t smart enough to like figure out when not to use those hints, it seems to wanna jam that knowledge into everything even if it only loosely relates
1
I will not be able to join, but found this an interesting article: https://blog.sicuranext.com/aws-waf-bypass/. I therefore also had a couple of questions, hopefully you can represent the topic:
• What is the general opinion of AWS WAF? Do y’all think it’s a mature product?
• Do you implement AWS WAF for your services or do you think there is a better product for it?
• If you implement AWS WAF, do you implement a lot of custom rules or only (mostly) the AWS Managed Rules?
@here office hours is starting in 30 minutes! Remember to post your questions here.
I ended up trying your suggestion and WAF -> ALB -> NLB (ALB is pointed to the private IPs of the NLB) and it seems to be working as expected. I’m unclear on what the benefit of doing WAF -> NLB -> ALB -> NLB achieves but regardless, I have a working path. Thanks for the help @Matt Calhoun!!
@BATeller Thanks for the idea. I successfully tested WAF -> CloudFront -> NLB
Links from today’s office hours:
https://cybersecuritynews.com/aws-zenbleed-attacks/ https://aws.amazon.com/about-aws/whats-new/2023/07/amazon-route-53-support-14-top-level-domains/ https://blog.sicuranext.com/aws-waf-bypass/ https://replit.com/site/deployments https://www.youtube.com/watch?v=zyEOYl23pd8 https://codepal.ai/terraform-writer https://docs.docker.com/compose/release-notes/#2200 https://nathanpeck.com/rethinking-infrastructure-as-code-from-scratch/ https://atmos.tools/ https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/ https://twitter.com/atoonk/status/1685858408082423808?s=19 https://www.fastly.com/blog/announcing-unified-origin-observability-across-fastly https://www.namecheap.com/domains/handshake-domains/ https://ianix.com/pub/dnssec-outages.html https://aws.amazon.com/security/security-bulletins/AWS-2023-004/ https://tailwindcss.com/docs/utility-first https://aws.amazon.com/security/security-bulletins/AWS-2023-005/ https://www.theregister.com/2023/07/18/us_military_mali_email_typos/ https://www.youtube.com/watch?v=gd5uJ7Nlvvo https://aws.amazon.com/builders-library/implementing-health-checks/ https://github.com/Netflix/Hystrix https://www.supertenant.com/ https://dev.to/aws-builders/aws-alb-with-nginx-ingress-controller-1ofd
2023-08-03
Interesting “think piece” on IaC…. https://nathanpeck.com/rethinking-infrastructure-as-code-from-scratch/
I think it’s definitely a good idea. Marry this to a sophisticated model for deploying and maintaining the state of infrastructure as code (infrastructure has this annoying tendency to change over time!), then I think we’ll be cooking!
I agree with the parts, about Infrastructure under IaC will only get more complex, generic modular abstractions over IaC being unmaintainable and that IaC could learn alot from modern CSS ecosystem. I don’t agree with the solution being typescript based.
I’m curious if Cuelang could provide a good-enough attribute based intermediary to raw TF and k8s manifests without the drawbacks of modules and helm etc
the model that System Initiative is cooking up looks really cool.
2023-08-04
2023-08-07
2023-08-08
2023-08-09
@here office hours is starting in 30 minutes! Remember to post your questions here.
if you have time (this could become quite deep) how do you protect against downfall and ras poisoning when working with multi-tenant environments (some cloud providers might be more prone to this than others, due to bad architechture)
What does “ras” refer to?
Thoughts about using AWS WAF and/or (Advanced) Shield for rate-limiting / DOS protection? WAF rate limiting seems valuable; Advanced Shield seems potentially $$.
Links from today’s office hours:
https://thenewstack.io/bram-moolenaar-author-of-the-open-source-vim-code-editor-has-died/ https://aws.amazon.com/blogs/aws/mountpoint-for-amazon-s3-generally-available-and-ready-for-production-workloads/ https://arstechnica.com/gadgets/2023/08/backblaze-probes-increased-annualized-failure-rate-for-its-240940-hdds/ https://www.warpstream.com/blog/kafka-is-dead-long-live-kafka https://opensourcewatch.beehiiv.com/p/mirantis-unveils-k0smotron-opensource-streamlined-kubernetes-management-project https://github.com/yonahd/kor https://github.com/danswer-ai/danswer https://github.com/28mm/blast-radius https://www.theverge.com/2023/8/9/23824562/slack-redesign-app-dms-activity-later https://venturebeat.com/programming-development/aws-unveils-build-a-new-accelerator-program-for-early-stage-startups-from-around-the-globe/ https://github.com/padok-team/burrito https://github.com/kubernetes/kubernetes https://github.com/Skarlso/crd-bootstrap https://www.humblebundle.com/books/devops-2023-oreilly-books?charity=12390931 https://github.com/Isawan/terrashine https://nrkbeta.no/2023/01/19/the-road-to-nrks-private-terraform-registry/ https://aws.amazon.com/about-aws/whats-new/2023/08/amazon-eks-configure-efs-shared-file-storage/ https://aws.amazon.com/blogs/aws/new-improve-amazon-s3-glacier-flexible-restore-time-by-up-to-85-using-standard-retrieval-tier-and-s3-batch-operations/ https://aws.amazon.com/about-aws/whats-new/2023/08/aws-datasync-copying-data-other-clouds/ https://downfall.page/ https://wired.me/technology/a-trippy-visualization-charts-the-internets-growth-since-1997/ https://aws.amazon.com/snowmobile/ https://www.wired.com/story/apple-new-password-manager-2fa-iphone-ipad/#intcid=_wired-bottom-recirc_ade29f42-cc7a-4779-bf6d-06662126039c_wired-content-attribution-evergreen https://docs.cloudposse.com/reference/best-practices/terraform-best-practices/#docusaurus_skipToContent_fallback https://docs.aws.amazon.com/whitepapers/latest/security-design-of-aws-nitro-system/the-ec2-approach-to-preventing-side-channels.html https://aws.amazon.com/security/security-bulletins/AWS-2023-005/ https://aws.amazon.com/security/security-bulletins/AWS-2023-006/ https://aws.amazon.com/security/security-bulletins/AWS-2023-007/ https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore https://buildbot.net/ https://zuul-ci.org/docs/zuul/3.5.0/index.html https://buck2.build https://medium.com/@taleodor/using-monorepo-do-not-rebuild-unchanged-components-in-ci-c386e7c03426 https://aws.amazon.com/blogs/security/three-most-important-aws-waf-rate-based-rules/
Oh, also if you end up spending more than $3k on AWS WAF it is cheaper to just get Shield Advanced which covers the costs: https://blog.elva-group.com/how-to-save-thousands-of-dollars-on-aws-waf
if you’re spending over $3000 per month on Web ACL and Rule fees, you can effectively cap those costs at $3000 and prevent them from spiraling further as your number of AWS accounts grows by subscribing to AWS Shield and enrolling your resources
Just gonna throw a comment in here, that AWS staff have confirmed to me on calls that EC2 does share cores, that’s how we have stuff like burst unlimited etc (and as the commenter shared cpu steal). There is options to not share cores and have dedicated tenant hardware in EC2 (which sometimes can be a requirement for high security environments) see (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated-instance.html).
AWS Nitro however is an AWS developed hypervisor that has hardware based security offloading that’s designed to help keep tenants isolated from one another, and likely had optimisations of the cores itself to help reduce steal etc - it seems like they also some mitigations against this sidechannel style attacks in the nitro framework itself.
2023-08-10
I wonder if spacelift will be impacted?
It’s technically competing with Terraform cloud. But terraform cloud isn’t an OSS project. However the tool, terraform itself is.
That’s pretty crazy. Maybe there’s enough engineers with env0, Spaceflit, infracost to fork Terraform?
Security patches only until December 2023. Providers still open source and will presumably have breaking changes so fork prob not viable, unless if maybe GCP, Oracle, AWS and Azure chip in a few million per year.
So I’d say they’ll be shutting down.
.png)
I *think* they updated their FAQ:
https://www.hashicorp.com/license-faq
cc: @Erik Osterman (Cloud Posse) @Vlad Ionescu (he/him)
Add pledges here https://github.com/opentffoundation/manifesto/pulls
2023-08-11
here I am getting this error when calling this action, and I am testing with act locally
- name: Tests
uses: cloudposse/github-action-docker-compose-test-run@main
with:
file: ./docker-compose.yml
service: my-app
command: npm test
[Run test suits/build] Success - Main Start stack [Run test suits/build] Run Main Test [Run test suits/build] docker exec cmd=[bash –noprofile –norc -e -o pipefail /var/run/act/workflow/9-composite-3.sh] user= workdir= Creating my-app_run … done /bin/sh: 0: cannot open npm: No such file | ERROR: 2 [Run test suits/build] Failure - Main Test [Run test suits/build] exitcode ‘2’: failure [Run test suits/build] Run Main Stop stacks
2023-08-12
2023-08-14
2023-08-15
https://github.com/ergomake/layerform
I do not know if we talked about this one before
saw this on HN looks interesting
2023-08-16
https://registry.terraform.io/providers/ansible/ansible/latest/docs not sure if this got covered previously but it seems like folks are having a crack at ansible provider for TF. This is kinda interesting to my team as we do a lot of local exec to run ansible to configure machines after creation esp for stuff that you don’t want/cant point in an asg.
you need to put the ansible playbook inside terraform or use git provider to point to repos to fetch collection of playbook
yes instead of local-exec this is also good way to use it
@here office hours is starting in 30 minutes! Remember to post your questions here.
I’ll have to drop early today, but links:
• AWS news ◦ https://aws.amazon.com/blogs/aws/new-amazon-ec2-m7a-general-purpose-instances-powered-by-4th-gen-amd-epyc-processors/ ◦ https://aws.amazon.com/about-aws/whats-new/2023/08/network-load-balancer-supports-security-groups/ ◦ https://aws.amazon.com/about-aws/whats-new/2023/08/aws-fargate-pid-namespace-sharing-kernel-configuration/ and https://aws.amazon.com/blogs/containers/announcing-additional-linux-controls-for-amazon-ecs-tasks-on-aws-fargate/
• Others ◦ System Initiative is now in open beta ▪︎ Launch blog post: https://www.systeminit.com/blog-open-source/ ▪︎ How they’re doing open-source considering waves around: https://www.systeminit.com/open-source/ ▪︎ https://github.com/systeminit/si
Few questions on Atmos & Terraform
- What is the best way to handle global or shared variables across different Atmos components? How do you combat warnings about excess variables when some components only require a subset?
- When using the terraform s3 backend property, how come things are stored as component/stack and not stack/component? Is there ways to change that or is it an important design choice?
Sorry all! will bring back up next wednesday
@Zain Zahran feel free to post this question in atmos
Sounds good - will do, thanks!
Links from today’s office hours:
https://aws.amazon.com/about-aws/whats-new/2023/08/aws-enhanced-startups-featuring-aws-build/ https://github.com/containers/skopeo https://www.hashicorp.com/blog/hashicorp-adopts-business-source-license https://www.hashicorp.com/bsl https://www.hashicorp.com/license-faq https://aws.amazon.com/about-aws/whats-new/2023/08/aws-service-catalog-hashicorp-terraform-cloud/ https://opentf.org/ https://github.com/opentffoundation/manifesto https://spacelift.io/blog/hashicorps-license-change https://aws.amazon.com/about-aws/whats-new/2022/03/aws-proton-terraform-open-source/ https://snyk.io/blog/detect-infrastructure-drift-unmanaged-resources-snyk-iac/ https://github.com/cncf/foundation/issues/617
2023-08-17
2023-08-22
Yes Igor announced this in office hours. He said he was going behind opentf
I missed the last one.
No worries. He mentioned it once and I don’t think I ever saw a formal update outside of that office hours call.
Ha
Well, we need something to take the place of Hashicorps Terraform sooner than later.
I’d argue OpenTF is also going
@Vlad Ionescu (he/him) you attending office hours today? Would be interested to hear what you mean.
I will, but I don’t wanna expand too much on it because it’s… cold and mean.
2023-08-23
Did opentf ever get any answer from hashicorp?
@here office hours is starting in 30 minutes! Remember to post your questions here.
Where to get the link to the office hours zoom session?
never mind I see it in the annoucements
I tried to sign up for the office hours but i get a generic web error “invalid meeting id”
Hi @William Galloway Did you manage to join the office hours yesterday?
Hello, maybe someone can give me some feedback on this question on today’s office hours.
Can atmos projects be managed/deployed by spacelift?
Links from today’s office hours:
https://github.com/opentffoundation/brand-artifacts https://docs.cloudposse.com/components/library/aws/eks/cluster/#changelog https://breadnet.co.uk/google-artifact-registry-virtual/ https://thenewstack.io/kubernetes-1-28-accommodates-the-service-mesh-sudden-outages/ https://www.reddit.com/r/kubernetes/comments/15yzi6a/cncf_announces_graduation_of_kubernetes/?share_id=mkkImgBMjLuYtFtqzv75g&utm_content=2&utm_medium=android_app&utm_name=androidcss&utm_source=share&utm_term=1 https://github.com/patrickchugh/terravision https://www.reddit.com/r/Terraform/comments/15uhlsk/i_added_autocompletion_to_target_you_can_do_it_too/ https://news.ycombinator.com/item?id=37199495 https://aws.amazon.com/about-aws/whats-new/2023/08/aws-dedicated-local-zones/ https://aws.amazon.com/about-aws/whats-new/2023/08/aws-appsync-javascript-all-resolvers-graphql-apis/ https://blogs.oracle.com/cloud-infrastructure/post/offering-a-sovereign-cloud-designed-for-the-european-union https://github.com/hashicorp/terraform/blob/v1.5.6/LICENSE https://www.hashicorp.com/license-faq#security-patch-backporting https://registry.terraform.io/
2023-08-24
OK this one has me very, very interested. I’ve always thought of Excel as the perfect tool to bridge engineering and business tasks. This would make it that much more awesome.
/cdn.vox-cdn.com/uploads/chorus_asset/file/24865497/Y6JVgdE.png)
2023-08-25
2023-08-29
2023-08-30
@here office hours is starting in 30 minutes! Remember to post your questions here.
Links from today’s office hours:
https://fig.io/blog/post/fig-joins-aws https://aws.amazon.com/blogs/containers/amazon-vpc-cni-now-supports-kubernetes-network-policies/ https://opentf.org/announcement https://twitter.com/opentforg/status/1696913055576387599 https://twitter.com/brikis98/status/1696453969118113902 https://www.youtube.com/watch?v=HzBA6FIn_Bo https://github.com/opentffoundation/manifesto https://sweetops.slack.com/archives/CB6GHNLG0/p1693385471811699 https://n8n.io/ https://github.com/cube2222/octosql https://www.theverge.com/2023/8/22/23841167/microsoft-excel-python-integration-support https://jetporch.substack.com/ https://jetporch.substack.com/p/template-module-finished-a-look-inside https://github.com/containers/skopeo https://blogs.vmware.com/management/2023/08/aa-august-release.html https://www.tines.com/ https://nodered.org/ https://humanitec.com/products/score https://oam.dev/
Office hours topic for next week: What are folks using for using internal dev documentation / wikis? A client has their docs split across many areas and we’re looking to help them with this. I started to reach for GitHub’s Wiki, but quickly realized it is no where near as full featured as I would like. Would love to hear of the free + open source options that are easy to use and enable simple, private access for dev teams.
Historically used wordpress, but ghost is alot lighter.
We have started using docusarus for some of our public facing docs. I believe cloudposse is using the same toolchain. So far it seems pretty powerful/flexible. If you have someone on the team that knows javascript it would probably help especially if you use some of their plugins or build your own. It also looks like a major release is coming out with breaking changes so before you lift and shift, you may want to just wait for the latest version: https://github.com/facebook/docusaurus/releases
![https://github.com/opentffoundation/brand-artifacts/blob/main/full/transparent/PNG/[email protected]?raw=true](https://github.com/opentffoundation/brand-artifacts/blob/main/full/transparent/PNG/on-light@2x.png?raw=true){kind=link}
I’m curious about other ideas in this space, this issue has been trouble since I started thinking about internal docs (going back at least 10 years). Never found anything that was everything,
Used StackOverflow for Teams (enterprise product) - which we had to work hard for developers to adopt, and still not completely - getting people to write questions, to edit and to answer - isn’t easy Google Docs - it just becomes a hot mess, although it’s a great document editing tool, and the search is phenomenal Confluence - a bit better than GDocs IMO (it has a better feel IMO for documentation, and automating stuff is slightly easier for dev workflows), my personal method is drafts in Gdocs and then you can actually copy/paste into Confluence without losing any significant formatting. Github Issues/Wiki - personally I love markdown as a format, and this is a great way to document a project, but search sucks, and if you have 600+ repos you are f&%^d Wiki (generic) - this is ugly and not fun, very hard to work with and navigate, if anyone offers - just say no! This problem requires a product that someone thought about for the specific use-case not a general use utility. Slack - no as well, nice for communication, sucks for authoritative docs
My current thought (project in research really) is a portal maybe based around backstage (or similar) - in which we have the documentation (SDKs, articles, etc) along with the service info (production state, deployments and status) - but we didn’t yet start at all (we don’t have a backstage installation yet)
Curious to see what others are doing
2023-08-31
Guys have a question for ArgoCD or actually how to install helm release on different variables for different environments. So I think for ArgoCD we can provide different values.yaml file for each env but..
- so configuration switches are ok so we can use different values.yml for each environment. (or there may be other things which i am not aware hence this message?)
- What about Secrets? How you make sure argocd installs secrets securely or how do you handle secrets on an ArgoCD managed k8s env and installing helm charts requires secrets
I am using external-secrets operator with hashicorp vault. However it doesn’t work well until after you have your cluster up and running. So the developers use it but when I’m deploying argocd and other core cluster components I am managing my secrets outside of vault/external-secrets/kubernetes
on AWS we use Secrets Manager, there is a nice K8S plugin that can inject secrets, probably also an ArgoCD integration - never checked that, but the store is pretty good (as is HashiCorp Vault which we also worked with). What we do is save the paths, and that is the pointer, you can assign different permissions on each “folder” which allowed us to give developers permissions on their secrets, but not the more secretive secrets controlled by SecOps - but managed in the same service (this is supported by both tools).
For cm and secrets change i will Use reloader stakater