#geodesic (2024-02)

Current geodesic is constructed in such a way it cannot be used properly for dev containers (mainly the profile scripts relying on bash instead of sh)

We currently add some geodesic magic on top of our tooling container, using build stages this allows for a nice combo

I am not familiar with dev containers, but can confirm that Geodesic relies heavily on bash and its specific features and we have no plans to make it work under the POSIX Bourne shell.

Possibly it could be a different release of geodesic without the interactive elements. E.g. a dev-contiainer release.

It’s interesting though, I need to read up on it. Thanks for the suggestion @E-Love

@Hans D, can you provide a bit more context around your Bash vs sh comment? I haven’t run into that limitation yet and regularly use Bash as the primary shell in dev containers, but I fear I’m misunderstanding the limitation you’re referencing!

FWIW, I mostly use dev containers with Docker locally and Coder remotely, and not regularly with Codespaces.

I’ve been using it with jetbrains, so it could be implementation specific. Bash itself is not an issue, but the various commands - with jetbrains - arer executed using sh as main shell. The profile scripts will be executed, but in the case of Geodesic these are very bash specific (mainly arrays in variables), which makes it not working. (normally profile scripts are sh compatible).

Main difference I see is that the geodesic env is more operational focused, while the dev container is more … eh … dev focused. There is some overlap, but the whole wrapping that geodesic does for mounting etc. is handled differently with devcontainers.

We’ll have @Jeremy White (Cloud Posse) do a quick PoC, and see what we learn from it.

he can contact me as well if hee needs more info / my experience

same! I currently use dev containers for TF and Helm dev but my team is intrigued by the SweetOps approach and Geodesic feels like it has many of the “batteries included”.

I assume that, ideally, I would fork the devcontainers-contrib/templates repo

Yea, I like the idea of aligning with conventions like this.

The Terraform stuff in there uses “features”, which are quite different from the common (and Geodesic) way of building images. Features are a way to work around the single inheritance limitation of container images.

I would recommend using their base image and keeping the existing Dockerfile as similar as possible, instead of trying to replicate it using features.

Alternatively, there could be a “Geodesic feature” that could be applied to any image with proper prereqs, but that sounds like a bigger lift.

My experience so far: features are a nice start, but i prefer to roll my own layers. Way more control that way. (and that’s why the tooling container we are using is not build on top of Geodesic, but we import the bits we need). Allowed us to be running Bookworm way earlier (sorry Erik)

@E-Love Would you please educate me as to what makes dev containers attractive? What do you get with them that you don’t get with Docker or Nix? This would help me understand how we can best meet your needs.

I confess I still do not understand the use case, but it looks to me like you can get a good start on using Geodesic with dev containers with this configuration (Contents of devcontainer.json)

{
  "name": "Geodesic Dev Container",
  "image": "cloudposse/geodesic:2.9.0-debian",
  "mounts": [
    "source=${localEnv:HOME},target=/localhost,type=bind",
  ],
  "appPort": ["12345:12345"],
  "containerEnv": {
    "SHELL": "/bin/bash",
    "GEODESIC_PORT": "12345",
    "GEODESIC_HOST_CWD": "${localWorkspaceFolder}",
    "GEODESIC_WORKDIR": "${containerWorkspaceFolder}",
    "LOCAL_HOME": "${localEnv:HOME}",
  },
  "remoteEnv": {
    "LOGIN_SHELL": "/bin/bash"
  },
  "settings": {
    "terminal.integrated.defaultProfile.linux": "/bin/bash"
  },
}

Please let me know how it goes, or if you have other feedback.

NOTE: the mount of "source=${localEnv:HOME},target=/localhost is overkill and may cause performance issues, but it is what we currently do in Geodesic for the convenience of mounting all the dotfiles and dotdirectories in $HOME, so we can share configuration. You could pare it down to just the ones you want to use, but my personal list is kind of long:

.aws .bashrc.d .config .emacs.d .geodesic .kube .ssh .terraform.d

This also does not solve the problem for files like .bashrc or .inputrc

@Jeremy G (Cloud Posse), are you saying you don’t understand why the spec exists and why so many companies are buying into it? Do you want me to make the case for that?

@E-Love I am not saying it is not worthwhile, I’m just saying I don’t understand its value because it does not seem to be an improvement in my specific workflow. I don’t like using the terminal from within the IDE. For go development, a container with go is not that helpful because it doesn’t persist all the downloaded modules. None of this is to say that dev containers are not helpful to somebody for something, I am only saying that I don’t really get it yet.

Because I don’t get it yet, I cannot confidently evaluate whether or not Geodesic’s support of dev containers is good or not. So, I’m asking you to try it out and let me know if it works well for you, and if not, why not. I’m not really asking you to make the case for dev containers.

For example, the configuration I posted does not forward the SSH agent on the host. Do you care, or is that adequately handled by your IDE, or do you just not need it? For another example, the configuration assume that ${localEnv:HOME}/ contains configuration directories like .aws and .config. Does that work with remote workspaces? If not, what is the right way to import such configuration?

thanks for clarifying and I’m definitely down to try this out :slightly_smiling_face:

your last question is easiest to answer :slightly_smiling_face:

unfortunately, dotfiles support isn’t built into the spec. However, many of the remote environments like Codespaces and Coder have built-in support for them, as do editors like VSCode. For terminal-based dev tools and JetBrains IDEs, I typically use the CLI’s exec command to execute chezmoi (my dotfiles manager of choice), but I’ve been thinking there could be a way to build a postCreateCommand lifecycle script to do something similar.

for SSH, there are a number of approaches, but they’re unfortunately host-specific. For local dev, it’s relatively easy. For remote tools, every provider has their own mechanism for accessing exposed ports or, in Codespaces’s case, the gh CLI’s command.

Yes about SSH, that’s why I punted on that.

Looks like chezmoi has a Debian package, but not published in the Debian package repo. Maybe you can lobby them for that, then you can easily install it in Geodesic using apt-get.

I suspect installing from the .deb would work in the meantime

I’ll block off some time on Friday to try this out

I appreciate your effort. To keep things simpler here, please just focus on Geodesic in this thread. I think the above configuration will work for the use case where you are running the IDE and hosting the code on your local machine. I will look into making it easier to run remotely.

I’m happy to help with the same. My intent with this thread was to start a convo around feasibility, not create a feature request

With the above configuration, I was able to use Geodesic as a dev container with IntelliJ EAP. So it’s feasible. TBD is how useful it is and what would make it more useful.

I’ll happily provide feedback as I (and hopefully my team soon) work with it. Thanks a bunch!

The benefit if having devcontainers is that you can ship the minimal required bits together with the repo. Shipping geodesic as the base for that devcontainer could be done, but given its size might not the option, but multiple devcontainers can be used. Having the container minimal: fast pull, no interference because of special magic, keeping out personal preferences,… (could be added via Chezmoi Dotfile Manager / https://containers.dev/features).

@Hans D the special magic is a big part of Geodesic. Customized prompt, handling multiple AWS profiles, easy Kubernetes access, etc. If you don’t want that, then you can probably get away with just installing atmos and terraform (and maybe kubectl if you’re using Kubernetes) on your choice of Linux.

I know. And we use some of that special magic as well (we inject that from geodesic into our tooling container). But for some dev situation that might be a bit too much or too magical.

@Erik Osterman (Cloud Posse) public tutorial on Geodesic

Thanks @Venkat Sunil Minchala! We’ll review and bring it up in #office-hours

Maybe adding this gif would help: https://github.com/cloudposse/geodesic/blob/master/docs/demo.gif

(we keep that updated using VHS)

This looks great! One note I’d add is that you mention the “active community” but don’t link to it. It would be nice to have a link to this Slack channel included with that bullet point under “What people like about Geodesic” and how readers could get support from the community

And one small nitpick is that “Cloud Posse” is two words, not “CloudPosse”

@Erik Osterman (Cloud Posse) and @Dan Miller (Cloud Posse) Thanks for the quick input. I have addressed those suggestion and published. :)

@Venkat Sunil Minchala Thank you for the writeup on Geodesic. I’m so deeply involved with it I cannot possibly see it with fresh eyes, so your article was helpful to me and I’m sure it will be helpful to others, too.

I have a few minor comments: • It would be nice to have an explicit link to the Geodesic GitHub repo near the start of the article • Typo after Docker Primer: you have “Geodesic is based on advance features”, should be “advance_d_ features” • Under “Standalone Geodesic”, we really do not recommend running other commands under Geodesic in standalone mode, and it is not something we validate during our development and testing. So I suggest you remove the example of running terraform via

docker run -it --rm --volume $HOME:/localhost cloudposse/geodesic:latest-debian -c "terraform version"

• I think you want to add a heading before “Create your own Docker Image based on Geodesic”, like you do for the other 2 usage patterns. • Under “How to launch and use Geodesic?”, I think people will get it, but still, I think you should replace “Just type *Geodesic* to use it further” with “Just type geodesic to use it further”. Thank you again for helping to spread the word!

• @Jeremy G (Cloud Posse) Thank you for your feedback! I appreciate the suggestions and have implemented them in the published version. I’m glad my input was helpful.
https://medium.com/@venkatsunilm/infrastructure-automation-with-geodesic-your-devops-toolbox-made-easy-486ba9784596

Thanks, and I saw it. (already running bookworm since start of dec)

I would be willing to include GCP support (we already ship with the GCP SDK pre-installed), but I am cautious because I do not use GCP and would want to make sure anything we do does not interfere with our AWS support and that GCP users are not bogged down by our AWS support.

For example, we currently display the active AWS role in the prompt. It’s a rather convoluted process that accounts for the possibility that you may have changed roles in a number of ways. To do that, we check the setting of AWS_PROFILE on every prompt, and make further checks when it changes. The check (when it changes) takes about a second to complete, so we do not want to run any of this if you are using GCP.

We have other places where we just assume that ASSUME_ROLE is an AWS role, and I’m not sure what, if anything, needs to be adjusted there.

We have legacy support to help you switch roles via an assume-role command, and we would want that to work for GCP (if changing roles is a normal part of your workday under GCP) or be disabled (if not needed).

If you can help me out with use cases and testing, I can add this kind of support for GCP. Otherwise, I’m not sure exactly what to do or how to test it, even though your role_name prompt is a good start.

To start, • How fast is gcloud auth list? Is it just processing a local file, or does it have to make a remote API call? • Where does gcloud store configuration data? (AWS stores it in $HOME/.aws/

• How do you change roles? How often do you change roles?

@Michael In addition to :point_up: , on my computer, gcloud auth list takes 0.3s to run. That is too long to run on every prompt. So, like we do with AWS, we will want to have some quick way to cache the results and then check if the cache needs to be invalidated. (With AWS, we check to see if AWS_PROFILE has changed or if the modification timestamp on the AWS Credentials file has changed.) Can you help me out by figuring out what to use to invalidate the cache for gcloud auth list?

@Jeremy G (Cloud Posse) Those are great insights and I would love to help out. Thank you for the offer to assist with this feature too. From my understanding, GCP has a few different authentication flows, but when it comes to the credentials used by SDK tools like gcloud or gsutil, they are stored in a DB file located at ~/.config/gcloud/credentials.db . Whenever someone runs gcloud auth login, it appends an entry to this file. I also found credentials inside of access_tokens.db, and was able to extrapolate them using sqlite.

When I look in ~/.config/gcloud/configurations/config_default :

I see:

[core] account = [[email protected]](mailto:[email protected]) project = projectexample

Maybe this is another starting point, because most of the gcloud commands I ran hovered around 0.22s for completion as well

You should convert your modules to registry syntax.

module "stack-config" {
  source  = "cloudposse/stack-config/yaml"
  version = "0.22.3"
}

Note also that that version is from June, 2022, so you might want to think about upgrading.

@Matt Gowie Of course, you should also be fixing any underlying file ownership issues. All the files you touch in Geodesic should appear to be owned by root, and the ones that are actually on the host filesystem, when viewed from the host, should appear to be owned by you.

Huh. That is coming from the registry syntax:

module "account" {
  source  = "cloudposse/stack-config/yaml//modules/remote-state"
  version = "0.22.3"

  component = "account-map"
  stage     = "root"

  context = module.this.context
}

Can you expand more on the file ownership bit? Not sure what you’re getting at. I see the owner + group as 501 + dialout.

Inside Geodesic everything should appear to be owned by root:

⨠ ls -l .terraform*
-rw-r--r-- 1 root root 1043 Feb 20 08:58 .terraform.lock.hcl

.terraform:
total 4
drwxr-xr-x 4 root root 4096 Feb 20 21:01 e98s/

If you have done something to avoid running as the root user inside Geodesic, we do not support that.

Hm… we’re not doing anything special that I know of. I’m only referring to the /localhost/**** volume mount that is owned by “501” (never seen that) and the group “dialout”.

If you are using BindFS because of not running Docker in rootless mode, you may need to upgrade to Geodesic 2.8.1 or later. See PR 901

Just rebuilt our image using Geodesic 2.9.1 Debian. Still hitting the same issue. I’m running colima and not docker desktop though, so that may have something to do with it.

Yes, colima is not supported

Well, I’m not necessarily hitting the same issue, but if I check the owner of a file in /localhost/*** then I get the following:

What you are seeing is a failure to map UID and GID from host to container. I expect you are using a Mac because the default UID:GID on Mac is 501:20

Docker desktop manages translating UID:GID on host to the container user’s UID:GID inside the container. Apparently Colima does not. Refer to Colima docs about it and LMK what you find.

Yeah, host is mac. I’ll check it out.

If you are still using our wrapper to launch Geodesic, you might have success with

export GEODESIC_HOST_BINDFS_ENABLED=true

from the host before launching.

@Matt Gowie LMK if this works. (FYI, VirtioFS is the default for Docker Desktop for Mac, but there are still open issues relating to file ownership permissions when using VirtioFS under DD, so I still use gRPC FUSE.)

Yeah finding the same thing, it seems to be a problem with the default which is sshfs . Will try out your suggestion next time I sit down to my machine.

Here is just one of many issues with VirtioFS and Docker Desktop for Mac. You may run into the same kinds of things with Colima.

LMK how you are launching Geodesic. The more I think about it, the more I think that BindFS may be the best solution for Colima users.

Orbstack has good results as well on Mac

@Hans D Thanks for mentioning Orbstack. Potential users should beware that Orbstack will destroy Time Machine backups and effectively prevent other backups if the destination drive is not APFS formatted (such as an HFS+ drive or NAS), unless its data file is immediately excluded from backups. Several people have been affected. For example, pjv .

@Jeremy G (Cloud Posse) running coliam with VirtioFS did the trick for correct file permissions. All files are now owned by root.

I am launching Geodesic via the wrapper script.

@Matt Gowie Glad that worked. Would you please add a relevant comment to https://github.com/cloudposse/geodesic/issues/594 for other Colima users. I might update the issue title to be more comprehensive about file ownership issues.

Done Thanks for the pointers + help Jeremy

You’re welcome, and thank you for all your support for Cloud Posse open source.