#office-hours (2022-10)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2022-10-03

Anyone else using Bitbucket for CI/CD? https://bitbucket.org/blog/macos-runners-bitbucket

We are happy to announce that Bitbucket Pipelines now supports macOS self-hosted runners. We have moved from beta to an official…

I been using Bitbucket for pipelines for a while now. What about the runners?

We are happy to announce that Bitbucket Pipelines now supports macOS self-hosted runners. We have moved from beta to an official…

They’ve added support for macOS.
I’ve been using Bitbucket pipelines for a while as well. I think they do a great job and stay on par with pipeline offerings from GitHub and GitLab.
I think Bitbucket is one of the best pipelines that people don’t think to consider. When i saw this post about new runners, I was just curious what other folks in this community might be using it as well.
2022-10-04


In case people are interested I’ll be raising the following topic tomorrow during office hours. I think it’s an interesting one and looking to get people’s feedback on it:
https://sweetops.slack.com/archives/CB6GHNLG0/p1664903787880119
Hey everyone,
Looking to have a bit of a debate on the topic of monitoring as code and whether or not *it actually matters*. More specifically: whether having monitors, dashboards, service level objects and the like actually need to be backed by IaC and within a GitOps workflow.
Many of us have monitoring products like datadog or cloudwatch in which the vast majority of monitors, dashboards, SLOs and the like have been clickops’d. For example at my current shop there are about 350 dashboards and almost none are in IaC and what’s more we don’t really know which ones are critical and which ones can be deleted. And the same goes for monitors and SLOs.
Now imagine that you used Terraformer (or equivalent, if there even is such a thing for Cloudformation) to get all these things into terraform and into all the appropriate repos. And then you even took that a step further and developed a system to do this continuously and also to clean up your monitoring product in the meanwhile, e.g. delete any dashboard not label critical
or something.
My questions to the community are: • so what? All of those clickops’d dashboards are backed up by the CSP or 3rd party; if they have a catastrophic event they’ll probably be able to get them back to you? • and do we really want to be writing dashboards as code? It gets fairly ridiculous. • and as for labeling them and then automating their cleanup: will it be that much of a feng shui or cost improvement? Curious about people’s thoughts regarding this topic because now that I have everything in IaC and a potential solution for automating parity and cleanup I find myself asking, “Who cares?” And of course if there are other reasons for storing monitors, dashboards, SLOs and the like as code please bring those up as I’m always interested in learning how other people are solving problems!
2022-10-05

@Erik Osterman (Cloud Posse) I have a meeting that goes until 12PM PT. I can skip it if I need to because I definitely want to raise the observe-as-code debate, but if you have content until 12PM and I can bring that up after that’d be great. Either way just not sure what you have on the agenda today. Thanks!

Ok, we’ll defer it to 12pm

@Erik Osterman (Cloud Posse) no need, I can miss the other meeting – I’ve thought a lot about this today and yesterday and I want to make sure this gets top treatment so I’ll be there during peak.

Random one: I know the official line is that s3 buckets can’t be moved between accounts, but is that really true even at the “ask AWS support nicely” level? We’ve got a few big enough buckets that, on the face of it, it would cost a few k to copy them from one account to another, which is a blocker for my preferred approach of rebuilding key systems in new accounts to get rid of old cruft. Any stories of non-bank-breaking S3 bucket migrations would be interesting!

@david.gregory_slack I’m not sure if you are aware of the s3p library – but if you do end up copying files I cannot recommend it enough – s3p saved me hours, if not days, on cross-account recursive s3 bucket copying. Pro-tip: spin up a huge ec2 instance or container of some type beforehand, as that will also dramatically help parallelization. Fwiw.
list/copy/sync/compare S3 buckets 5x-50x faster than aws-cli

I should have mentioned: this library is an order of magnitude faster than any other library you will find. At least it was when I used it last year.

heads up, we discussed this today

Thanks, hoped to make it but events. Will catch up.

@here office hours is starting in 30 minutes! Remember to post your questions here.

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Linda Pham (Cloud Posse) has joined Public “Office Hours”

Jose Figueredo has joined Public “Office Hours”

Isaac M has joined Public “Office Hours”

Allan Swanepoel has joined Public “Office Hours”

Jonathan Poczatek has joined Public “Office Hours”

Peter Dada has joined Public “Office Hours”

dag viggo lokoeen has joined Public “Office Hours”

Gabriel Zabal has joined Public “Office Hours”

Michael Jenkins has joined Public “Office Hours”

Matt Calhoun has joined Public “Office Hours”

Guelor Emanuel has joined Public “Office Hours”

Ralf Pieper has joined Public “Office Hours”

Jared Richards has joined Public “Office Hours”

Jonas Steinberg has joined Public “Office Hours”

Zadkiel has joined Public “Office Hours”

Ozzy has joined Public “Office Hours”

I’d like to pose a bit of an involved question on what priority observability as code is.

Marc Tamsky has joined Public “Office Hours”

Harold Sphinx has joined Public “Office Hours”

Charles Smith has joined Public “Office Hours”

sebastian maniak has joined Public “Office Hours”

Adedapo Ajuwon has joined Public “Office Hours”

Vicken Simonian has joined Public “Office Hours”

Vijay Kukreja has joined Public “Office Hours”

Ray Botha has joined Public “Office Hours”

sebastian maniak has joined Public “Office Hours”

Eric Berg has joined Public “Office Hours”

PePe Amengual has joined Public “Office Hours”

Vicken Simonian has joined Public “Office Hours”

Peter Dada has joined Public “Office Hours”

Vijay Kukreja has joined Public “Office Hours”

Thanks everyone for the awesome discussion today

let us know what you end up doing!

I will – I heard you loud and clear on the solution-looking-for-a-problem point, as well as Matt’s “snowflake service” point. I needed to hear those, frankly. Great stuff!!

Thanks, @Jonas Steinberg @matt @Eric Berg @Alanis Swanepoel for the great discussion today.

2022-10-12

@here office hours is starting in 30 minutes! Remember to post your questions here.

Does anybody have experience with AWS EKS using AWS EFS?
I need a place to store/read some data (5-10MB file) very fast and have it available consistently on multiple pods.

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Linda Pham (Cloud Posse) has joined Public “Office Hours”

Isaac M has joined Public “Office Hours”

dag viggo lokoeen has joined Public “Office Hours”

Matt Calhoun has joined Public “Office Hours”

Michael Jenkins has joined Public “Office Hours”

andy miguel has joined Public “Office Hours”

Jared Richards has joined Public “Office Hours”

Amaan Khan has joined Public “Office Hours”

Jose Figueredo has joined Public “Office Hours”

Hao Wang has joined Public “Office Hours”

Oskar Maria Grande has joined Public “Office Hours”

Andrew Hall has joined Public “Office Hours”

Jonathan Poczatek has joined Public “Office Hours”

sebastian maniak has joined Public “Office Hours”

Jim Park has joined Public “Office Hours”

Alexandr Vorona has joined Public “Office Hours”

Shantanu Gole has joined Public “Office Hours”

Marc Tamsky has joined Public “Office Hours”

Oliver Schoenborn has joined Public “Office Hours”


Srivardhan T has joined Public “Office Hours”

Gabriel Zabal has joined Public “Office Hours”

Johnmary Odenigbo has joined Public “Office Hours”

Links from today’s call: https://go.hashicorp.com/index.php/email/emailWebview?md_id=58714 https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ https://humanitec.com/whitepapers/kubernetes-benchmarking-study-2022 https://www.reddit.com/r/Terraform/comments/xxf691/terraform_plugin_framework_is_now_in_public_beta/ https://twitter.com/iamvlaaaaaaad/status/1534489514818686976

2022-10-16

I know there are a number of ways to initialize your vault cluster but personally I am a fan of being able to do things in terraform: https://registry.terraform.io/providers/rickardgranberg/vaultoperator/0.1.6
^^ Sharing in case you folks haven’t heard of it before.
2022-10-19

@Erik Osterman (Cloud Posse) - during some of the office hours calls you point to your internal ?confluence? page where you show how you structure aws accounts in an ldap / ou style

is there any chance that page is publicly accessible?

@here office hours is starting in 30 minutes! Remember to post your questions here.

what is cloudposse?

who are you?


Erik Osterman (Cloud Posse) has joined Public “Office Hours”

John Jarvis has joined Public “Office Hours”

dag viggo lokoeen has joined Public “Office Hours”

Vlad Ionescu has joined Public “Office Hours”

Emile Fugulin has joined Public “Office Hours”

Michael Jenkins has joined Public “Office Hours”

Linda Pham (Cloud Posse) has joined Public “Office Hours”

Jose Figueredo has joined Public “Office Hours”

Matt Calhoun has joined Public “Office Hours”

Ozzy has joined Public “Office Hours”

Isaac M has joined Public “Office Hours”

Matt Gowie has joined Public “Office Hours”

Jonas Frank has joined Public “Office Hours”

Allan Swanepoel has joined Public “Office Hours”

Jonathan Poczatek has joined Public “Office Hours”

RB (Ronak Bhatia) (Cloud Posse) has joined Public “Office Hours”

PePe Amengual has joined Public “Office Hours”

Oliver Schoenborn has joined Public “Office Hours”

Oskar Maria Grande has joined Public “Office Hours”

Isa Aguilar has joined Public “Office Hours”

Joshua Magady has joined Public “Office Hours”

Vicken Simonian has joined Public “Office Hours”

Brian Pauley has joined Public “Office Hours”

Maura Rowell has joined Public “Office Hours”

Hey, popping in and out — kids’ bedtimes — but I’ll check out the recording later. Hoping to take advantage of these more; thanks for doing them!

Jared Richards has joined Public “Office Hours”

Andrew Vitko has joined Public “Office Hours”

Devendra Yadav has joined Public “Office Hours”

PePe Amengual has joined Public “Office Hours”

Andrew Nascimento has joined Public “Office Hours”

Jared Richards has joined Public “Office Hours”

Arthur Dent has joined Public “Office Hours”

Link from today’s session: https://github.com/cloudposse/terraform-aws-ecs-cluster/ https://registry.terraform.io/providers/cloudposse/template/2.2.0?pollNotifications=true https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/ https://github.blog/changelog/2022-10-13-design-improvements-to-github-actions-navigation https://www.docker.com/pricing/october-2022-pricing-change-faq/ https://neon.tech/
Terraform module for provisioning an ECS cluster

Fine-grained personal access tokens offer enhanced security to developers and organization owners, to reduce the risk to your data of compromised tokens.

Design improvements to GitHub Actions navigation

The price increase will allow us to continue to invest in Docker as developers’s #1 most-used, #1 most-loved and #1 most-wanted tool.

2022-10-26

Q: How are people enforcing MFA in AWS? Not using AWS SSO at the moment, just IAM Users and IAM Groups. Have seen a cloudtrail solution that uses a cloudwatch metric filter and alarm which does the trick, but is probably very expensive as you need to use cloudtrail.

Check out the policy in this AWS Doc: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_aws_my-sec-creds-self-manage-mfa-only.html
The relevant bit is here:
{
"Sid": "DenyAllExceptListedIfNoMFA",
"Effect": "Deny",
"NotAction": [
"iam:CreateVirtualMFADevice",
"iam:EnableMFADevice",
"iam:GetUser",
"iam:ListMFADevices",
"iam:ListVirtualMFADevices",
"iam:ResyncMFADevice",
"sts:GetSessionToken"
],
"Resource": "*",
"Condition": {
"BoolIfExists": {"aws:MultiFactorAuthPresent": "false"}
}
}
Use this IAM policy to allow users to manage their MFA device in the AWS Management Console.


``` module “admin_label” { source = “git://github.com/cloudposse/terraform-null-label.git?ref=tags/0.3.3>” namespace = “${var.namespace}” stage = “${var.stage}” name = “${var.admin_name}” delimiter = “${var.delimiter}” attributes = “${var.attributes}” tags = “${var.tags}” }
module “readonly_label” { source = “git://github.com/cloudposse/terraform-null-label.git?ref=tags/0.3.3>” namespace = “${var.namespace}” stage = “${var.stage}” name = “${var.readonly_name}” delimiter = “${var.delimiter}” attributes = “${var.attributes}” tags = “${var.tags}” }
data “aws_caller_identity” “current” {}
data “aws_iam_policy_document” “role_trust” { count = “${local.enabled ? 1 : 0}”
statement { actions = [“sts:AssumeRole”]
principals {
type = "AWS"
identifiers = ["arn<img src="/assets/images/custom_emojis/aws.png" alt="aws" class="em em--custom-icon em-aws">iam:root"]
}
condition {
test = "Bool"
variable = "aws:MultiFactorAuthPresent"
values = ["true"]
} } }
data “aws_iam_policy_document” “manage_mfa” { count = “${local.enabled ? 1 : 0}”
statement { sid = “AllowUsersToCreateEnableResyncTheirOwnVirtualMFADevice”
actions = [
"iam:CreateVirtualMFADevice",
"iam:EnableMFADevice",
"iam:ResyncMFADevice",
]
resources = [
"arn<img src="/assets/images/custom_emojis/aws.png" alt="aws" class="em em--custom-icon em-aws">iam:mfa/&{aws:username}",
"arn<img src="/assets/images/custom_emojis/aws.png" alt="aws" class="em em--custom-icon em-aws">iam:user/&{aws:username}",
] }
statement { sid = “AllowUsersToDeactivateTheirOwnVirtualMFADevice”
actions = [
"iam:DeactivateMFADevice",
]
resources = [
"arn<img src="/assets/images/custom_emojis/aws.png" alt="aws" class="em em--custom-icon em-aws">iam:mfa/&{aws:username}",
"arn<img src="/assets/images/custom_emojis/aws.png" alt="aws" class="em em--custom-icon em-aws">iam:user/&{aws:username}",
]
condition {
test = "Bool"
variable = "aws:MultiFactorAuthPresent"
values = ["true"]
} }
statement { sid = “AllowUsersToDeleteTheirOwnVirtualMFADevice”
actions = [
"iam:DeleteVirtualMFADevice",
]
resources = [
"arn<img src="/assets/images/custom_emojis/aws.png" alt="aws" class="em em--custom-icon em-aws">iam:mfa/&{aws:username}",
"arn<img src="/assets/images/custom_emojis/aws.png" alt="aws" class="em em--custom-icon em-aws">iam:user/&{aws:username}",
]
condition {
test = "Bool"
variable = "aws:MultiFactorAuthPresent"
values = ["true"]
} }
statement { sid = “AllowUsersToListMFADevicesandUsersForConsole”
actions = [
"iam:ListMFADevices",
"iam:ListVirtualMFADevices",
"iam:ListUsers",
]
resources = [
"*",
] } }
data “aws_iam_policy_document” “allow_change_password” { count = “${local.enabled ? 1 : 0}”
statement { actions = [“iam:ChangePassword”]
resources = ["arn<img src="/assets/images/custom_emojis/aws.png" alt="aws" class="em em--custom-icon em-aws">iam:user/&{aws:username}"] }
statement { actions = [“iam:GetAccountPasswordPolicy”] resources = [“*”] }
statement { actions = [“iam:GetLoginProfile”]
resources = ["arn<img src="/assets/images/custom_emojis/aws.png" alt="aws" class="em em--custom-icon em-aws">iam:user/&{aws:username}"]
condition {
test = "Bool"
variable = "aws:MultiFactorAuthPresent"
values = ["true"]
} } }
data “aws_iam_policy_document” “allow_key_management” { statement { actions = [ “iam:DeleteAccessKey”, “iam:GetAccessKeyLastUsed”, “iam:UpdateAccessKey”, “iam:GetUser”, “iam:CreateAccessKey”, “iam:ListAccessKeys”, ]
resources = ["arn<img src="/assets/images/custom_emojis/aws.png" alt="aws" class="em em--custom-icon em-aws">iam:user/&{aws:username}"]
condition {
test = "Bool"
variable = "aws:MultiFactorAuthPresent"
values = ["true"]
} } }
Admin config
locals { enabled = “${var.enabled == “true” ? true : false }” admin_user_names = “${length(var.admin_user_names) > 0 ? true : false}” readonly_user_names = “${length(var.readonly_user_names) > 0 ? true : false}” }
resource “aws_iam_policy” “manage_mfa_admin” { count = “${local.enabled ? 1 : 0}” name = “${module.admin_label.id}-permit-mfa” description = “Allow admin users to manage Virtual MFA Devices” policy = “${join(“”, data.aws_iam_policy_document.manage_mfa.*.json)}” }
resource “aws_iam_policy” “allow_change_password_admin” { count = “${local.enabled ? 1 : 0}” name = “${module.admin_label.id}-permit-change-password” description = “Allow admin users to change password” policy = “${join(“”, data.aws_iam_policy_document.allow_change_password.*.json)}” }
resource “aws_iam_policy” “allow_key_management_admin” { name = “${module.admin_label.id}-allow-key-management” description = “Allow admin users to manage their own access keys” policy = “${data.aws_iam_policy_document.allow_key_management.json}” }
data “aws_iam_policy_document” “assume_role_admin” { count = “${local.enabled ? 1 : 0}”
statement { actions = [“sts:AssumeRole”] resources = [”${join(“”, aws_iam_role.admin.*.arn)}”] } }
resource “aws_iam_policy” “assume_role_admin” { count = “${local.enabled ? 1 : 0}” name = “${module.admin_label.id}-permit-assume-role” description = “Allow assuming admin role” policy = “${join(“”, data.aws_iam_policy_document.assume_role_admin.*.json)}” }
resource “aws_iam_group” “admin” { count = “${local.enabled ? 1 : 0}” name = “${module.admin_label.id}” }
resource “aws_iam_role” “admin” { count = “${local.enabled ? 1 : 0}” name = “${module.admin_label.id}” assume_role_policy = “${join(“”, data.aws_iam_policy_document.role_trust.*.json)}” }
resource “aws_iam_group_policy_attachment” “assume_role_admin” { count = “${local.enabled ? 1 : 0}” group = “${join(“”, aws_iam_group.admin..name)}” policy_arn = “${join(“”, aws_iam_policy.assume_role_admin..arn)}” }
resource “aws_iam_group_policy_attachment” “manage_mfa_admin” { count = “${local.enabled ? 1 : 0}” group = “${join(“”, aws_iam_group.admin..name)}” policy_arn = “${join(“”, aws_iam_policy.manage_mfa_admin..arn)}” }
resource “aws_iam_group_policy_attachment” “allow_chage_password_admin” { count = “${local.enabled ? 1 : 0}” group = “${join(“”, aws_iam_group.admin..name)}” policy_arn = “${join(“”, aws_iam_policy.allow_change_password_admin..arn)}” }
resource “aws_iam_group_policy_attachment” “key_management_admin” { group = “${aws_iam_group.admin.name}” policy_arn = “${aws_iam_policy.allow_key_management_admin.arn}” }
resource “aws_iam_role_policy_attachment” “admin” {
count = “${local.enabled ? 1 : 0}”
role = “${join(“”, aws_iam_role.admin.*.name)}”
policy_arn = “arniam:policy/AdministratorAccess”
}
resource “aws_iam_group_membership” “admin” { count = “${local.enabled && local.admin_user_names ? 1 : 0}” name = “${module.admin_label.id}” group = “${join(“”, aws_iam_group.admin.*.id)}” users = [”${var.admin_user_names}”] }
Readonly config
resource “aws_iam_policy” “manage_mfa_readonly” { count = “${local.enabled ? 1 : 0}” name = “${module.readonly_label.id}-permit-mfa” description = “Allow readonly users to manage Virtual MFA Devices” policy = “${join(“”, data.aws_iam_policy_document.manage_mfa.*.json)}” }
resource “aws_iam_policy” “allow_change_password_readonly” { count = “${local.enabled ? 1 : 0}” name = “${module.readonly_label.id}-permit-change-password” description = “Allow readonly users to change password” policy = “${join(“”, data.aws_iam_policy_document.allow_change_password.*.json)}” }
resource “aws_iam_policy” “allow_key_management_readonly” { name = “${module.readonly_label.id}-permit-manage-keys” description = “Allow readonly users to manage their own access keys” policy = “${data.aws_iam_po…

We are using a product called gravitational teleport, where mfa is added to github, people login via their github profile and only a particular team has access to the aws console itself

Interesting, thanks. We have some non technical users so doing things through github isn’t as ideal unfortunately

They have enterprise plans as well with AD integration

We are using the free version

@here office hours is starting in 30 minutes! Remember to post your questions here.

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Isaac M has joined Public “Office Hours”

Oliver Schoenborn has joined Public “Office Hours”

Vlad Ionescu has joined Public “Office Hours”

Joe Caulfield has joined Public “Office Hours”

Linda Pham (Cloud Posse) has joined Public “Office Hours”

Eric Berg has joined Public “Office Hours”

Ralf Pieper has joined Public “Office Hours”

Oliver Schoenborn has joined Public “Office Hours”

Paul Bullock has joined Public “Office Hours”

Maura Rowell has joined Public “Office Hours”

Jared Richards has joined Public “Office Hours”

Brian Pauley has joined Public “Office Hours”

Jared Richards has joined Public “Office Hours”

sebastian maniak has joined Public “Office Hours”

Joshua Magady has joined Public “Office Hours”

Amaan Khan has joined Public “Office Hours”

Amaan Khan has joined Public “Office Hours”

Jim Park has joined Public “Office Hours”

Matt Gowie has joined Public “Office Hours”

Sean TUrner has joined Public “Office Hours”

Olad Oke has joined Public “Office Hours”

Srivardhan T has joined Public “Office Hours”

Olad Oke has joined Public “Office Hours”

Links from office hours: https://github.com/cloudposse/infra-live/pull/184 https://aws.amazon.com/about-aws/whats-new/2022/10/aws-organizations-console-centrally-manage-primary-contact-information-aws-accounts/ https://aws.amazon.com/about-aws/whats-new/2022/10/aws-batch-supports-amazon-eks/ https://devopsian.net/posts/terraform-data-sources-over-remote-state/ https://brendanthompson.com/posts/2022/10/terraform-for-expression https://aws.amazon.com/about-aws/whats-new/2022/10/dark-mode-support-aws-management-console/ and https://aws.amazon.com/about-aws/whats-new/2022/10/amazon-sqs-increased-throughput-quota-fifo-high-throughput-ht-mode-6000-transactions-per-second-tps/ https://github.com/fffaraz/awesome-selfhosted-aws https://www.home-assistant.io/ https://www.iampulse.com/ https://github.com/iann0036/iamlive https://airiam.io/ https://hackingblogs.com/aws-security-tools-detail-guide/ https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-policy-generation.html
Why Terraform data sources are preferable over remote state, with use-cases using multiple filters based on tags to filter resources dynamically

Using the for expression in Terraform to filter, group, order and mutate information. With this knowledge in hand you will easily be able to construct complex objects based on existing information/configuration or from configuration passed in via input variables or ingested. Easily create multiple instances of resources or data sources using the for_each meta-argument.
