#aws (2022-01)

looks like it, you could also set a custom header and filter requests without it, it would still allow the backend to be discovered in some circumstances

Yeah, exactly. Thanks @Moritz (Dutch by any chance?)

(not quite, two countries over )

Largely depends on your cicd and what it is capable of. Github actions works as an oidc provider, which is pretty awesome. Setup the roles in AWS, and cicd automatically gets temporary credentials

From my reading, gitlab-ci can be made to do similar, but seems to require something like hashi vault as an intermediary

Yeah, I’m using GitHub actions… I have Hashicorp Vault so Actions authenticates to Vault using OIDC and can then uses vault-secrets action to pull secrets currently for HCP, TFC and DockerHub

I guess time to setup vault to also retrieve AWS creds, maybe?

But this is how I’m using GitHub actions with oidc and AWS, without vault… https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services

yeah that’s what I was thinking… thought I could use SSO to handle it but SSO login always needs the browser to complete. So looks like setting up a rather unprivileged IAM user in AWS mgmt acct with iam:AssumeRole action and setup the applicable role in the org accounts

that’s similar to how I have Vault connecting… https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-hashicorp-vault

I don’t have any iam user, just an oidc provider and an iam role that trusts the oidc provider and the claims

yeah I’m trying to avoid having to use IAM users as well as it never fails that credentials stagnate and fester

reading over that link it looks like it’s already assuming a role so I’d have to setup the OIDC in all member accounts to handle cross-account?

Exactly. I was surprised at how magical it was using oidc and an iam role

hmm… but I don’t think an iam role could assume another role… or and I missing something… I’ve been so far in the weeds to get this far my brain is fried and little sleep so that’s possible :)

You have two options, yeah. One, setup the oidc provider in every account and have the action assume role directly into each account to a role that does the things. Two, setup oidc provider in one account, assume role to that account to a role that can only assume-role to other accounts, then have the client do the assume role with the credential retrieved from the action

You can absolutely assume role from the credentials generated from an assumed role…

Lots of layers of abstraction, makes it all hard to manage. No easy button I know of

second option is more what I was envisioning… I’m using Terraform to deploy so I was wanting to give one set of creds to it and then have a map of the accounts it could then assume access to in order to do what it needs.

Yeah, it’s a little easier with terraform as the client… Use the assume_role config in the provider block to switch role to the target account

yeah exactly… ie- Route53 hosted zones are in one account, SES is already established in another, etc so I was going to have provider aliases and use assume_role to the various accounts

Yep that should work

accounts being split by roles so I’d be able to just have a map of accounts to role to assume and the role in each account would only have the iam actions allowed that they need

thanks for the sounding board and validation…

Sure thing!

was up to nearly 4am this morning working on several modules so operating on ~5 hours sleep right now… caffeine and adrenaline only thing keeping me going :)

won’t forget a bit of determination, or as the wife would call it stubbornness

@loren so you’ve used the GitHub OIDC for runners… I attempted to set it the OIDC IDP in my management account and run my GitHub action against it but I got the error Error: Not authorized to perform sts:AssumeRoleWithWebIdentity which I’m going with the assumption is based on the conditions in the trust policy for the IAM role. I’m attempting to have it only accept for a given GitHub org so I have the policy as this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GithubActionsAccess",
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<Account_ID>:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repository_owner:<GH_org>:*"
        }
      }
    }
  ]
}

This seems right as I’m reading the docs. Are you using something similar or do you see a flaw in this?

@Jeremy (UnderGrid Network Services) this is what i put in the action:

      - name: configure aws credentials
        uses: aws-actions/configure-aws-credentials@ea7b857d8a33dc2fb4ef5a724500044281b49a5e
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/${{ secrets.AWS_ROLE_NAME }}
          aws-region: us-east-1

      - name:  Validate credential
        run: aws sts get-caller-identity

and this is the trust policy on the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AssumeRoleOidc",
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:<GITHUB_ORG>/<GITHUB_REPO>:*",
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

if you want the entire github org instead a specific repo to have permissions to the role, i think it would look like:

"token.actions.githubusercontent.com:sub": "repo:<GITHUB_ORG>/*"

Okay I’ll try that… I had tried the repository_owner:<GITHUB_ORG>: as that was my understanding from reading the GitHub OIDC doc but I’ll try using a wildcard for the repo and use the repo: subject instead

post if it works!

Will do

I had also wrote my own code to setup the roles but I’ve since noticed there are some github actions oidc modules in the registry so I’ll probably go back and re-evaluate them

i’m just using the official action published by aws

if you’re rolling your own action, the aud is probably not [sts.amazonaws.com](http://sts.amazonaws.com)

no I’m not rolling my own. just talking about the a terraform module to set it up in AWS. Looks like I found 4 that each have some unique takes on the setup. I do see they all use repo:${org}/${repo} though for the sub

oh i see. yeah it wasn’t complex enough for a module in my usage. i just wrote a template for the trust policy and used the generic iam role module i already had

so changing the sub to be repo:${org}/*:* works fine… now my issue appears to be with GitHub itself and the GitHub App permissions to be able to comment back to pull-requests

Maybe this tool? https://github.com/snyk/driftctl

Thank you very much Johan

have a look at terraform plan -detailed-exitcode

mysql or mssql?

MS SQL

I have no clue, sorry

but I will imagine you can get a snapshot as you said and then need to import it to your new db server which will not be special if it is in aws

if you are going to use RDS or Aurora it is still the same way you will do on on-prem, the only difference will be the version that will be supported in RDS/Aurora

you can snapshot/dump your current db, then upload the file to an instance or s3 bucket and then use an instance to download the dump and then import it

there is no magical way to do this in AWS

there is something called Database migration service but that is more for when you need to massage some data before you put it in your db

DMS can definitely do a straight migration as well - it’s quite flexible. A “snapshot” is not going to handle database migration properly.

Agree, it all depends what type of migration you want to do I guess

I recommend you AWS DMS. It’s really flexible and can be used to move from GB to PB of data.

If you are confortable with Kafka there is also a tool called Debezium built on top of kafka connectors.

Thank you all for you kind responses. This is a starter.

looks like the data source for the acceptor doesn’t return anything

and why? what it’s trying to look?

https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account/blob/0cad62a594f9a116a3a2a395ea0c8e1a41e541d4/accepter.tf#L67

but what information is missing? i don’t see nothing clearly

it can’t find the acceptor route table

I’d work backwards and see how the module collects each argument to that data source

but there is route tables in the acceptor account

and verify manually that you have these values

it seems like it’s getting the subnets correctly

but it can’t find the route table with the same subnets that its collecting

mmmm

in the acceptor account.

i’ll take a look with this information, thank you!

still in the same status

Note: VPC peering requires a role from source account to assume a role in the destination account to accept the peering. It won’t work if it’s either unable to assume a role in the destination account or the assumed role in the destination account has insufficient permissions

is this real?

Yup. Tho we are still waiting for the detailed writeups

Also https://twitter.com/colmmacc/status/1481682859324760070

Turns out UnknownError means the aws javascript lib doesn’t know how to deal with the response it got from aws.

In this case one of our devs set the endpoint for all aws calls to the sqs endpoint instead of just setting it for the sqs object.

Same desire here!

I already contacted our AWS sales guy.

speak to your TAM and get added to their internal feature request

what do you mean by “full volume instead of a snapshot”? A snapshot is a complete image of a disk

unless the filesystem was striped across multiple EBS volumes, the snapshot will contain the full volume

ah yeah my mistake there, you’re right, edited

you can’t download EBS snapshots. If you want to download the data, you can create a volume from the snapshot, attach it to an already-running instance, and download the volume contents with whatever linux copy/backup tool you prefer

okay great, that seems to be the consensus in forums too, thanks a lot Alex.

would you consider the process of creating and privately sharing a full AMI too, or no?

no. What do you mean “privately sharing”? You can’t access AMI data outside AWS. All you can do with it is spin up an EC2 instance

this solution - https://aws.amazon.com/premiumsupport/knowledge-center/share-ami-account/

Hi @Michael I haven’t used AWS VPNs much.

https://docs.aws.amazon.com/vpc/latest/reachability/how-reachability-analyzer-works.html might help uncovers some hard to find cause…

That’s a good idea - thanks. I created a path from the EC2 instance to the VPN Gateway in the Reachability Analyzer. This results in the status “reachable” with a green tick but the VPN tunnels remain “DOWN”. I suspect that Reachability Analyzer does not go beyond the AWS infrastructure four outbound traffic

did you check if you have them in the other regions (don’t remember if they are regional or global)

It’s global, but yeah this is a brand new vpc setup on this account. So this is the only one

We reached out to aws licensing to attempt to get support that way. Worst case i suppose we will pay for their cheapest support tier and hope they can assist. Working fine on our UAT account

I’ve actually generated 2 PATs, one with repo, admin:repo_hook and admin:org_hook as described above. That is in var.GITHUB_TOKEN. (Actually TF_VAR_GITHUB_TOKEN in Terraform Cloud, then passed through to here.) The second just has repo permissions and is in AWS SSM with the key /Prod/GITHUB_OAUTH_TOKEN. But nothing changes when I put the one with more permissions in both places.

For anyone else who runs into this, my problem was that the user I had generated my PAT with had a Maintain role in the repo in question. The user required an Admin role. More detail in this GitHub thread.

i believe you need to install the aws alb controller helm chart

https://artifacthub.io/packages/helm/aws/aws-load-balancer-controller

you can do this using helm, helmfile, or terraform via helm-release module

Thanks for responding. I wiill try to incorporate that terraform helm-release module into my code, and then destroy/re-apply. Goal to is have an EKS cluster up and running with ingress, with one ‘terraform apply’.

…I got this successfully installed with Helm chart that was referenced above. At some point I need to come back and integrate that into my CP TF code.

Hey @Dan Garland ! Nice seeing you stop by

One of the major things off the bat is AWS partitions. All your modules that generate ARNs will need to be updated to support them

The other thing to double check is that the same services you depend on are available in the region

That makes sense

https://www.amazonaws.cn/en/about-aws/regional-product-services/

Keep in mind it will be a totally separate AWS account needed to operate in china

Looks like a massive headache already!

I am not sure how things like VPC peering work, transit gateway, IAM work between china and non china regions

I suppose one big thing as well is trying to keep data in sync across the great firewall… I suppose it’s not possible to have replicated services across the two so probably going to have to cobble something together

@Dan Garland you need to work closely with Legal on this — it’s not a matter of tech, but of law and regulations. Tech is like the 3rd or 4th priority here.

For example, you don’t only need a specific AWS-China account, but you’ll also need a company registered in China and a designated Chinese speaker to interact with the Government on your behalf, and more.

Most likely it will be a separate everything, not just a separate cell (if you’re doing cell-based architectures).

usually with ENIs, it’s not about permissions (the error message is just bad). It’s about some dependencies that the resource depend on. For example, that ENI is used somewhere, or you have a Security Group added to the ENI. Before you delete that SG (usually manually), you would not be able to delete the ENI and delete/update the VPC. Something like that we’ve seen before

Hmm, won’t let me change the SG either on the eni:

Failed to change security groups for network interface .
You do not have permission to access the specified resource.

check what permissions your user actually has. Just because it’s called Administrator doesn’t necessarily mean it is

for example, the standard “administrator” account at my work has limited permissions in production. There is a special break-glass “emergency” role which really has admin privileges

you ca null those out if you use ecs+ec2

Oh! So the only thing I need is the target_group_arn?

That could be much clearer in the docs, TBH.

it depends

for example

Oh, I have just learned about the Terraform state file (or States tab in Terraform Cloud).

But now I have a different question. I see that ssh_public_key_path is actually the only required input to key-pair. And yet when I use “/secrets” as suggested in the example, my Terraform Cloud apply fails with:

Error: mkdir /secrets: permission denied trying to create a local file with both public_key_openssh and private_key_pem

It looks like it works if I use a path of "."

@Nick Kocharhook the problem with using /secrets is that dir is not relative to where you are running terraform. TF would be looking for a dir named secrets at the root of the disk: /.

You would be better off using a fully qualified path. You can use the path.X feature to get a fully qualified path to the location you want: https://www.terraform.io/language/expressions/references#filesystem-and-workspace-info

Aah, thank you! I thought it was being tacked onto $PWD, but of course your explanation makes more sense.

In that case, I’m actually struggling to think of a time that the "/secrets" example would ever actually be what you wanted. Maybe in a Docker container? But maybe that example should be changed so it’s more likely to be useful.