#office-hours (2024-02)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2024-02-05
Hi everyone
I am very sad to announce - officially - that Weaveworks will be closing its doors and shutting down commercial operations. Customers and… | 64 comments on LinkedIn |
2024-02-07
I would like to discuss this today: https://github.com/palantir/policy-bot/issues/705
We came across policy-bot
and eagerly deployed it without realizing we couldn’t do what we had hoped to do.
We run a large Open Source GitHub organization with hundreds of repositories. We are a small business. Adding trusted contributors to our enterprise organization as outside collaborators or team members is cost-prohibitive (this is what we currently do).
We would like to instead follow in the footsteps of kubernetes/test-infra
to implement a ChatOps style mechanism to approve PRs for testing and separately approve PRs for merging. Under the hood, this project uses prow.
We would like this solution to play well with GitHub Action triggers
Ideal Workflow
- Untrusted open-source PR is opened against one of our repos from a fork (never a branch in the repo)
- A vetted contributor inspects the PR. If it looks good, they comment
/ok-to-test
policy-bot
receives the payload, and applies its advanced policy controls. Instead of approving the PR, it can be configured to comment on the PR with something like/terratest
(or add a label, such asok-to-test
)- GitHub Actions configured on the PR trigger on comments from the
policy-bot
or labels. Note, non-org members cannot add labels. - Tests pass
policy-bot
awakens and approves the PR (as aCODEOWNER
), so branch-protection rules now pass- A vetted contributor now decides if this PR is mergeable, and then comments
/ok-to-merge
- Something like
bulldozer
then auto-merges the PR.
Limitations
• The major limitation seems that policy-bot
cannot interact with PRs beyond approving them, and we cannot approve a PR without first running tests.
• We cannot run tests automatically on untrusted Pull Requests from open-source forks due to the massive attack surface and the requirement that tests have access to secrets required to validate terraform code.
Alternatives considered
- Implement a native GHA solution. GHA suck for ChatOps. It’s like talking to the Mars Lander. Commands take minutes to receive an acknowledgment.
- Trigger GHA on
policy-bot
status checks. GHAs can only trigger on status checks in the default branch, precluding it’s use on PRs - Write a custom bot using one of the plethora of GH bot frameworks. We would like not to have to maintain a custom solution.
- Deploy prow used by
test-infra
, however, the project feels overwhelming in complexity and doesn’t seem to support GHA (only Jenkins), a requirement for us.
Response received: https://github.com/palantir/policy-bot/issues/705#issuecomment-1932610767
As you discovered, Policy Bot was designed for overall/merge approval workflows and doesn’t support the phased approval (approved for testing vs approved for merge) that your workflow requires. Given the limitations of GitHub Actions and Policy Bot, I don’t think you can achieve what you want using only features that are available today.
That said, here’s an idea that might work with one new Policy Bot feature:
- Implement the feature requested in #387, which is a way for Policy Bot to leave GitHub reviews instead of only posting status checks. Since this is a feature we don’t have a use for internally, I’m not sure my team will have time to implement it, but I think it should be relatively straightforward to add and I’m happy to review a contribution or discuss the implementation.
- For your organization, run two instances of Policy Bot, the “Test” instance and the “Merge” instance
- The “Test” instance looks at a test-specific policy file and leaves an approving review when it is satisfied. You configure the GitHub branch protection and the “Merge” policy to ignore approvals from this app instance. Instead, this approval is the trigger condition for the GitHub Actions workflow.
- The “Merge” instance looks at a merge-specific policy file and approves the PR (either by leaving a review or via status check) when code review is complete and the PR is ready to merge
Running two instances of Policy Bot is awkward, but they can be configured to not conflict and it avoids adding more complicated features.
To achieve what you want with only a single instance, I think we’d have to add significant new features. We don’t really have a concept of sub-policies or policies for different actions (the disapproval
policy is an explicit special case) and I don’t immediately see a place to hook in logic to leave comments or add labels when certain conditions are met.
@here office hours is starting in 30 minutes! Remember to post your questions here.
If there’s time. https://sweetops.slack.com/archives/CBW0HJDS8/p1707330170431539
Here’s one, for you release-engineering types. What’s the approach that most folks take these days for handling db migrations. Laravel, or whatever. Do folks tightly couple the migrations with the application, applying them at the time of app code deploy? I still see this frequently, and it seems aweful. My answer is the same as always – to manage the DB schema/queries/etc., as their own versioned products. This way if you know your app is going to need a new column you can make that change before the app is deployed. As you would do with any dependency. LMK, I’m interested in finding out what others see/do.
Weaveworks shutdown & implications
speedtest.net reaches 50 Billion tests and authors some slightly interesting stats on bandwidth availability over time. https://www.ookla.com/articles/speedtest-50-billion
Two questions:
- External Secrets Operator alternatives? Where our kubernetes applications (high # transactions per min) need to write and read secrets from an external secrets management system. we looked at dapr, eso, and some others, but they only support reading secrets from the source.
- Anyone ever deploy things to kubernetes hosted on vmware infra?
Links from today’s office hours:
https://github.com/hashicorp/terraform-provider-aws/pull/34595#event-11703612954 https://github.com/hashicorp/terraform-provider-aws/milestone/368 https://techcrunch.com/2024/02/05/cloud-native-container-management-platform-weaveworks-shuts-its-doors/amp/ https://github.com/crashappsec/chalk https://forum.defcon.org/node/248360 https://thehackernews.com/2024/02/cloudflare-breach-nation-state-hackers.html?m=1 https://pkl-lang.org/blog/introducing-pkl.html https://cuelang.org/ https://once.com/campfire https://www.linkedin.com/posts/david-heinemeier-hansson-374b18221_introducing-once-activity-7158896455162687488-SZYI/ https://techcrunch.com/2024/02/01/here-is-apples-official-jailbroken-iphone-for-security-researchers/ https://github.com/terraform-docs/terraform-docs/issues/703 https://github.com/ujaved/terraform_infrastructure_rag https://github.com/seal-io/tap https://news.ycombinator.com/item?id=39262650 https://support.pagerduty.com/docs/conference-bridge https://github.com/chalk/chalk https://docs.aws.amazon.com/inspector/latest/user/sbom-export.html https://github.com/anchore/syft https://www.defenseunicorns.com/ https://dodcio.defense.gov/Portals/0/Documents/Library/DoDRefDesignCloudGithub.pdf?ver=zXJ_uO5LfouVaysHo5Ejsw%3D%3D https://github.com/terraform-docs/terraform-docs/pull/749 https://spectrum.ieee.org/gpt-4-calm-down https://twitter.com/AmazonScience/status/1748071766197448891
Sorry. Internet installer guy showed up.
2024-02-08
For next time…. The Container Evolution Kit: https://cekit.io/ Define container images in YAML format which is easy to read and parse! This really makes me believe we are all becoming YAML Engineers
Container image creation tool
YAML rules
i’m not mad at it. i think the more prolific it becomes, the less of an issue it will be to onboard people to a tool/platform.
“Know how to read/write YAML? Good, you can do this”
haha, sounds like they just want to be compatible with atmos.
2024-02-13
Who changed the logo on my slack workspace???????
We need answers. Also, with this new color scheme will there be the ability to purchase SweetOps/CloudPosse t-shirts/swag?
It took me 20m to find SweetOps in the side bar…
HAHaha
Happy valentines day!
2024-02-14
@here office hours is starting in 30 minutes! Remember to post your questions here.
Two questions:
- External Secrets Operator alternatives? Where our kubernetes applications (high # transactions per min) need to write and read secrets from an external secrets management system. we looked at dapr, eso, and some others, but they only support reading secrets from the source.
- Anyone ever deploy things to kubernetes hosted on vmware infra?
Thoughts on https://www.firefly.ai/ and its “Terraform the chaos” and “genuine drift detection” claims
Firefly provides the cloud infrastructure management tools missing from traditional CMDBs
Lack of boot strapping the foundation in IAC
If anyone wanted to continue the discussion about empathy and encouraging devs to learn IaC, feel free to off this. I know that once the zoom call ended, folks might think to look here
Links from today’s office hours:
https://github.com/opentofu/opentofu/pull/1223/ https://github.com/metalbear-co/mirrord https://blog.cloudflare.com/cloudflare-defeats-patent-troll-sable-at-trial https://indico.dns-oarc.net/event/48/contributions/1038/ https://www.infoq.com/news/2024/02/aws-cdk-migrate-ga/ https://github.com/awslabs/llrt https://atmos.tools/reference/terraform-limitations https://atmos.tools/core-concepts/stacks/#best-practices https://atmos.tools/core-concepts/components/#best-practices https://cekit.io/ https://www.ookla.com/articles/speedtest-50-billion https://atmos.tools https://github.com/gitops-bridge-dev/gitops-bridge
2024-02-15
Is this another approach to multi cloud? Looks like a couple tools wrapped into one … A little AWS SDK, some Pulumi, and some Local Stack…
Nitric Open Source Cloud-Native Framework auto-provisions infrastructure for your app from any language for any cloud. Launch now with AWS, GCP and Azure.
2024-02-16
2024-02-20
question for office hours : should errors go into a dedicated slack channel only for errors or should they go into the slack channel the team is already in so that they don’t get ignored?
Good question for today?
2024-02-21
@here office hours is starting in 30 minutes! Remember to post your questions here.
Links from today’s office hours:
https://github.com/weaveworks/weave-gitops-enterprise https://www.ookla.com/articles/speedtest-50-billion https://discuss.httparchive.org/t/warning-14-000-bigquery-charge-in-2-hours/2715/4?utm_source=tldrwebdev https://gitlab.com/gitlab-org/gitlab/-/merge_requests/142547 https://techcrunch.com/2024/02/15/sequoia-open-source-fellowship-developer-funding/amp/ https://saml.to/ https://aws.amazon.com/about-aws/whats-new/2024/02/aws-control-towers-account-factory-terraform-increases-customization/ https://atmos.tools/core-concepts/stacks/#best-practices https://atmos.tools/core-concepts/components/#best-practices https://developer.hashicorp.com/terraform/language/modules#child-modules https://developer.hashicorp.com/terraform/language/modules#the-root-module https://github.com/palantir/policy-bot/issues/705 https://clerk.dev
It’s day it’s about time. And still not enough.
“AWS Free Tier offers free usage each month for AWS services and products. You can use the Free Tier API to programmatically track your free tier usage against the monthly usage limits.”
https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/using-free-tier-api.html
Use the Free Tier API to programmatically get your free tier usage.
2024-02-22
2024-02-26
Was this ever a thing? https://gitness.com/
Unlock superior Git workflows with Gitness. Experience seamless code reviews, robust CI/CD solutions, and up to 4x faster pipeline execution. Begin your journey in just 30 seconds.
We have some great news! Octopus Deploy has acquired Codefresh in a strategic investment to combine forces in creating the best CD and CI platform. Octopus Deploy is a leader in continuous delivery, especially for VMs and Windows with recent additions for Kubernetes while Codefresh has been wholly focused on cloud-native applications. Together, our goal […]
2024-02-28
@Erik Osterman (Cloud Posse) stumbled a week or two ago on Pkl (pronounced pickle) and thought you might be interested in having a look for office hours. Something that might help my team with generating Gitlab-triggered jobs. https://pkl-lang.org/index.html
Oh yep! Brought it up on #office-hours a couple weeks ago
If you kick the tires on it, it would be great to get a report of your experience.
Know a few folks at Apple who are using pkl as a drop in for HCL, basically generating JSON and passing to terraform. They haven’t had to write any HCL in a long time, so they say. Interesting idea. Certainly fills some of the language/DRYness/dyanmic providers etc issues with HCL/Terraform.
At that point, why not use CDK? That’s all its doing.
We created Pkl because we believe that configuration is best expressed in a special-purpose configuration language; a blend between a static configuration format, and a general-purpose programming language.
Because don’t want a full fledged programming language. The horrors I’ve seen devs do “because they can”
Also it is used internally a lot, so consistency++.
@here office hours is starting in 30 minutes! Remember to post your questions here.
Links from today’s office hours:
https://ai-infra.fun/ https://tracebit.com/blog/2024/02/finding-aws-account-id-of-any-s3-bucket/ https://codefresh.io/blog/codefresh-is-joining-octopus-deploy-to-create-the-most-powerful-kubernetes-cd-gitops-ci-and-argo-platform/ https://github.com/DavidGamba/dgtools/tree/master/bt#stacks-a-different-take https://aws.amazon.com/about-aws/whats-new/2024/02/aws-control-tower-apis-register-organizational-units/ https://atmos.tools/cli/configuration/#aliases https://atmos.tools/ https://atmos.tools/cheatsheet https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/using-free-tier-api.html https://www.youtube.com/watch?v=aolI_Rz0ZqY&t=2406s