#aws (2024-07)

An account per machine, seems like more overhead than required for a POC, but in general this seems like a good separation. Using jenkins seems like a bummer.

We may be moving the same setup to live, so I am thinking this way..

Also not sure about the costing with AWS, Is the aws accounts costs extra?

No. You pay per resource used.

Even organizations doesn’t cost extra, just the resources within the accounts attached to the organization.

thanks for the info @theherk, I will implement the same approach then…

For a POC… multiple accounts would be overkill.

However if your POC is to demonstrate account separation for a larger project, then yes, go for it. The org and the accounts are free.

I would think your breakout would be:

1. Production account for all production resources
2. UAT account for all non-production resources
3. Deployment account for automation. one thing that would be really great to acheive with this set up is only allowing deployments into Production or UAT via the services in the deployment account. That is, no manual changes unless absolutely necessary.

Using the UAT account resources as a deployment target, you would also realize all you would need to do to allow access to the production account resources — VPCs, Security Groups, Systems Manager connections, etc — from the deployment account.

However, if your POC is to only demonstrate deploying from Jenkins into two “environments” (not accounts) then the multi-account approach is overkill.

hi @managedkaos thanks for the response, We would move the same setup to live if everything is good. So i thought this approach in that longrun. By keeping that in mind, hope this approach is good? Or you are suggesting some other option?

Your approach is good, indeed! Not suggesting another option.

When you say it breaks, what do you mean. It might be that your code needs some changes to work with 3.11.

thanks for the reply. I get the below error. I used https://github.com/DataDog/datadog-serverless-functions/tree/master/aws/logs_monitoring#<i class="em em-~~~"https://github.com/DataDog/datadog-serverless-functions/tree/master/aws/logs_monitoring#:~~~ext=tabs%20%3E%7D%7D%20%[…]dFormation,-Log%20into%20your for installation.

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': cannot import name 'formatargspec' from 'inspect' (/var/lang/lib/python3.11/inspect.py)
Traceback (most recent call last):

See What’s new in Python 3.11. With respect to formatargspec:

The formatargspec() function, deprecated since Python 3.5; use the inspect.signature() function or the inspect.Signature object directly. And since that guide you shared says to use Python 3.10, perhaps you should. That would be before a feature it is using was removed. And once it imports it will pass, and maybe (probably) succeed at importing lambda_function which I presume is your entry point.

I just stumbled across that tab again and noticed it list varying runtime requirements based on the version you’re running. So while it says “Create a Python 3.10 Lambda function”, the version here is actually based upon version. So if you run version Upgrade an older version to +3.107.0 it would support Python 3.11, meaning it probably won’t try to import formatargspec.

Thanks, even I think upgrading the datadog application itself helps rather than i only upgrade the python runtime alone.

I will keep you posted

Hi, Post upgrade of the datadog application, I get this error

[ERROR]	2024-07-11T15:51:44.602Z	2d1a12b4-8337-4fec-af18-7aebda4d3a58	[dd.trace_id=597347809750705622 dd.span_id=5039695315866320206]	Failed to get log group tags from cache
Traceback (most recent call last):
  File "/opt/python/caching/cloudwatch_log_group_cache.py", line 125, in _get_log_group_tags_from_cache
    response = self.s3_client.get_object(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lang/lib/python3.11/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lang/lib/python3.11/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

Please suggest what is it? Thanks in advance

Looks like you’re going to need to troubleshoot why that key isn’t there or why your lambda can’t see it.

This is part of the baseline code of the Datadog Forwarder.

According to that method, it handle that exception. Im not sure how to add the key… before upgrading there was no issue

@Jeremy White (Cloud Posse)

I don’t immediately see why that wouldn’t work. I don’t recall ever having that same scenario, however.

Thanks Yeah it works

Interesting, looks like the Images are in the output of ctr -n [k8s.io](http://k8s.io) images list. Seems like I’ll need to get Image Builder to pull my image to that namespace with ctr

This is the solution I came up with. Haven’t tested it yet (as in launched a notebook) but I think it works (it’s pulling my image successfully to the same namespace that EKS uses)

phases:
  - name: build
    steps:
      - name: pull-machine-prospector
        action: ExecuteBash
        inputs:
          commands:
            - password=$(aws ecr get-login-password --region us-west-2)
              # Redirecting stdout because the process creates thousands of log lines.
            - sudo ctr --namespace k8s.io images pull --user AWS:$password acct.dkr.ecr.us-west-2.amazonaws.com/app:latest > /dev/null
            - sudo ctr --namespace k8s.io images list
  - name: test
    steps:
      - name: confirm-image-pulled
        action: ExecuteBash
        inputs:
          commands:
            - set -e
            - sudo ctr --namespace k8s.io images list | grep app

Didn’t seem to work, image still needed to pull.

@Ben Smith (Cloud Posse) @Igor Rodionov

Sorry for the delay, I meant to share this thread: https://archive.sweetops.com/aws/2024/04/#019d7c5e-1226-4a6b-ac03-5f4af8968f7d

TL;DR: we’re still looking for the silver bullet

:crossed_fingers: for AWS copilot CLI for ECS

We’ve had multiple iterations of our solution for ECS, and ecspresso is pretty nice. We chose it over other tools because it’s compiled as a single binary, easy to install, supports task definition templates out of the box, works with data sources to fetch data used in templates

Most of the family of ecs-deploy commands (there’s probably a half dozen or more) are scripts (shell, python, etc)

Escpresso also has a nice YAML-based configuration.

In the long run, want to see this win: https://aws.github.io/copilot-cli/

Oooh yeah the copilot CLI is interesting, was looking at it earlier and it manages everything so having custom stuff through Terraform might be hard.

because it’s compiled as a single binary This is a nice one

https://github.com/aws/amazon-ecs-cli

Somehow I missed this entirely

cc @Matt Gowie

Yeah, always interested in this discussion – Thanks for the ping @Erik Osterman (Cloud Posse).

Still feels like no silver bullet yet. My team did a recent re-implementation that we like. We should write it up into a blog post or video so we can share it out. I don’t think it’s perfect, but it at least doesn’t introduce drift and enables us to have a speedy CD for our downstream application team. I’ll follow up if we push forward with a writeup or video.

Ahaaa @Yangci Ou I was just about to cc you on this thread and then I come back to it and see you started it. I read all your comments and didn’t read the name

I’d like to hear what you came up with

I like the idea of just using Docker Compose files though. That way you get the same experience locally as you do when you deploy.

I know you said without using a third party tool, but I’d suggest considering Airbyte for the job rather than rolling your own. Otherwise the “simplest” solution is probably DynamoDB Streams -> Lambda. CDC/ETL jobs have a ton of different factors though (why Snowflake is named what it is), so it’s going to be hard for those without intimate knowledge thereof to be accurate. Aka, I could be wrong.

@Jeremy White (Cloud Posse)

I’d second that. Using a lambda is the easiest way. You could also try using a PITR with s3 export but there will be a lag time during which any deletes/updates to the data being restored would potentially get dropped. Being careful of that might work best, but again it has the risk of data integrity loss.

thank you so much for your insights. cheers!

I’m not familiar with this set up, but privatelink might be what you’re looking for. The service provider being the Thanos installation inside that separate VPC/AWS account and each of the Prometheus VPCs/Accounts being service consumers

Got it. I just looked at privatelink and you’re right, it does allow specific resource exposure. But wouldn’t it also share the same security concerns as VPC peering for this specific use case because prometheus will be running on the same eks cluster as my app. So if I share this cluster through privatelink, app still gets exposed to other environments?

I’m not as close to the how security works within EKS, but if you can associate a specific security group with individual workloads, then you can limit which traffic can reach the VPC endpoint ENIs on the Prometheus side

^^ Disregard - I believe I can expose prometheus pod as a service through NLB and use endpoint here.

Thank you so much @Joe Perez - privatelink’s the way to go.

No problem. I also wrote a couple of articles on what I’ve learned about PrivateLink

https://www.taccoform.com/posts/aws_pvt_link_1/

https://www.taccoform.com/posts/aws_pvt_link_2/

I’ll take a look! Appreciate it, Joe.