#geodesic (2022-01)

@Erik Osterman (Cloud Posse) @Jeremy (UnderGrid Network Services) Originally, I was trying to add a docker argument as follows:

	if [ "${OS}" == 'Linux' ]; then
		DOCKER_ARGS+=(--user="$(id -u):$(id -g)")
	fi

This works ok, but you can’t sudo. I next have tested:

	if [ "${OS}" == 'Linux' ]; then
		DOCKER_ARGS+=(--user="$(id -u):$(id -g)")
		DOCKER_ARGS+=(--volume="/etc/group:/etc/group" --volume="/etc/passwd:/etc/passwd" --volume="/etc/shadow:/etc/shadow")
	fi

which also works, but it borks your groups in the docker container. You can sudo properly, but the system isn’t quite right.

@rb may be onto a better method, which would look like the following: • Update the docker image to run as a user with sudo privileges for *everybody*. I know this is a process change, but I think it’s actually a good idea: Dockerfile:

RUN \
  groupadd -g 1000 geouser && \
  useradd -d /conf -G sudo -g geouser -u 1000 geouser && \
  sed -i 's/sudo[[:space:]]ALL=(ALL:ALL) ALL/sudo ALL=(ALL) NOPASSWD: ALL/' /etc/sudoers

RUN chown -R 1000:1000 /conf /usr/local/bin /home /var/tmp

• Update for Linux (and Darwin, IMO – please jump in if you disagree) templates/wrapper:

 	if [ "${OS}" == 'Linux' ]; then
		DOCKER_ARGS+=(--user="$(id -u):$(id -g)")
	fi

• We would need to update the Documentation website to show the run line of:

docker run -it --rm --user="$(id -u):$(id -g)" --volume $HOME:/localhost cloudposse/geodesic:latest-debian --login

I think this will be pretty universal, though we may need to add a notation to remove that for WSL? I haven’t used WSL in ages, so I’m not sure if it supports id -u

I don’t know much about the original version of WSL as I had never really used it personally, with WSL2 it is for all intents a Hyper-V VM of whatever guest OS you’re running (Ubuntu, Debian, etc) with a custom kernel so all your typical commands are going to execute. id -u and id -g work fine under WSL2. This is, from what I’ve gathered, much different with WSL2 than originally as WSL2 has an virtual hard drive image and the Windows drives are mounted under /mnt/<drive letter> by default though this mountpoint could be changed in the /etc/wsl.conf if it is created inside the WSL guest itself. The initial user that is created for inside WSL does use UID:GID 1000:1000.

I do have multiple WSL guests (Ubuntu and Debian) that are using WSL2 so I can’t confirm if this works on WSLv1 or only WSL2 which could be a way to determine the different environments. But on both my WSL2 images there is the WSL_DISTRO_NAME env variable. The HOME env variable does point to the user home directory under WSL not Windows. The Windows mount points are all of type 9p not drvfs as the templates/wrapper currently looks for. While uname -r will include Microsoft if you use case-insensitive matching (ie- 5.10.60.1-microsoft-standard-WSL2) that elif block doesn’t seem to be reached when running under WSL2. Nor does the templates/wrapper account for situations where the Windows User home directories are on a drive other than C:\Users as in my case where it is actually H:\Users .

@rb @Mike Crowe @Jeremy (UnderGrid Network Services) I published my BindFS proposed solution as a comment on issue #594 and I am eager to have someone try it and report back. I don’t have access to appropriate platforms on which to test it myself.

@rb @Mike Crowe @Jeremy (UnderGrid Network Services) Please try Geodesic release candidate v0.152.0-rc1 and report back.

@rb @Mike Crowe @Jeremy (UnderGrid Network Services) OK, I have not been able to reproduce and fix the problem, so I need more information from you.

This is not an issue with Docker v20 on Ubuntu (tested on 20.04 LTS) when running in rootless mode. As with macOS, this configuration correctly translates file ownership between the root UID and GID inside the container to the user’s UID and GID on the host.

When running Docker as root, and launching Geodesic using sudo geodesic, the geodesic wrapper does not get the user’s UID and GID or home directory, so BindFS does not help and we do not get the right home directory mapped to /localhost anyway.

How are you launching Geodesic? Are you able to find the non-root user’s UID, GID, and $HOME?

Hey Stephen!

we also have this: https://docs.cloudposse.com/tutorials/atmos-getting-started/

we are working on a new README now

Thanks, will check back when you are done

thanks @stephen! yes, @Andriy Knysh (Cloud Posse) will have the updated readme soon. The public docs are pretty up to date though and we have a repo with examples as well https://github.com/cloudposse/tutorials

I will say WSL has been difficult for us to support since we don’t use it ourselves. Support was contributed by multiple different community members.

https://github.com/cloudposse/geodesic/blob/master/rootfs/templates/wrapper#L121-L149

If you look at this section, you’ll see some of the nasty hacks we had to do to figure out directory mounting on WSL

and the behavior changes with different versions of WSL

I was digging into the geodesic script and saw some of the WSL mount bits. I tried executing the commands myself manually and they didn’t seem to behave as I think the script expected them to. Maybe those are WSL compatible and WSL2 is not. I’ll dig into it a bit more and if I find something put a PR together

I found a solution, someone can tell me if it’s not the best but it seems to work. I just simply added export GEODESIC_DOCKER_EXTRA_ARGS="--volume /mnt:/mnt" to my WSL2 shell so it mounted the extra volume that points back to the Windows drives. Looking at lines commented above I determined it likely wouldn’t have worked on my workstation as my Windows home drive is on H: which is mounted as /mnt/h and the code never accounts for that. It does appear that changing L124 to use findmnt -S 'C:\' -t 9p -no target does find the /mnt path which the allows L126 to function but then L127 fails because it doesn’t have a path for cmd.exe. All that seemed for naught though as LOCAL_HOME was set so it never appeared to ender that elif block.

that seems like an elegant fix

let me run it by some WSLers

A bit investigation, it looks like the WSL support that is in the wrapper is for WSL-1 and does not work with WSL-2. • $mount_path/c/Users/${windows_user_name}/AppData/Local/lxss/ does not exist with WSL-2 on L132 • On L39 the /LocalState/rootfs${HOME} path doesn’t exist. In my case the only thing under /LocalState/ is ext4.vhdx

Also the way it’s written it would have only picked up Ubuntu WSL not any other like Debian

Yes, that sounds familiar

is WSL-1 EOL now?

I don’t know if it’s EOL yet but WSL-2 is definitely the recommended as it provides a better environment overall. There are some things that still work easier between Windows and the WSL guest when using v1 vs v2.

Biggest difference as I’m aware is that v1 was more of a Linux overlay running on Windows; while v2 is more of a full fledge virtual machine environment with it’s own boot kernel, etc

this is what we do in Dockerfile

ARG VERSION=0.147.11
ARG OS=debian
ARG ATMOS_VERSION=1.3.17

FROM cloudposse/geodesic:$VERSION-$OS

# Install atmos (<https://github.com/cloudposse/atmos>)
ARG ATMOS_VERSION
RUN apt-get update && apt-get install -y atmos="${ATMOS_VERSION}-*"

Okay so you do just build another image off the cloudposse/geodesic base image. Easy enough

yes we do

the Dockerfile is in each infrastructure repo

oh okay… the infrastructure repo

as in the archive *.cloudposse.co repos?

that’s very old. We usually create infrastructure repo with all the stacks and components in it, then in the Dockerfile we use the code above. Same structure as in https://github.com/cloudposse/atmos/tree/master/examples/complete + the Dockerfile

yeah I thought I’d heard Erik say the *.cloudposse.co was old and used envdir that was no longer your recommended way. I hadn’t looked at geodesic or atmos previously until it was discussed during the office hours this week

we use both in all client engagements (atmos is lacking good docs, we are working on it)

I liked the functionality with the first blush exposure during the zoom call. I’ve been looking at how to restructure my workflow going forward. I’d already been moving everything to Terraform and getting away from manual changes. Though I was still using Terraform Cloud for remote state

https://github.com/cloudposse/tutorials

Some live tutorials here

these are more current

@Erik Osterman (Cloud Posse) I was actually running through that repo and executing them and ran into issues with the AWS environment example and the s3-cdn module

can you share the error?

https://github.com/cloudposse/tutorials/issues/10

the fix there allowed the plan to succeed and apply but the end result was not visible using the cloudfront distribution. I suspect due to changes in S3/cloudfront and the policy assigned via the s3-cdn module without extra values being set

aha, I think this is actually changes in the AWS provider

https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/pull/207#issuecomment-1009958645

Yeah that’s possible

How did you start geodesic?

I used

docker run --rm cloudposse/geodesic:0.149.1-debian init | bash

I am using minikube as a docker server with hyperkit as the driver, if that is helpful

ok, so after doing that, it should install a wrapper script

then you ran the wrapper script?

yes, geodesic? it installed to /usr/local/bin/geodesic

there were no errors running the wrapper script, but nothing in my home directory gets mounted to /localhost

Start geodesic from your project folder, not your $HOME folder. I think because you fired it up from Users/matan, it only mapped the workdir. If you start from /Users/matan/terraform/tutorials (say where ~/terraform/tutorials is where you downloaded the CloudPosse tutorials repo, then you would see: • /Users/matan -> /localhost • (workdir) -> ~/terraform/tutorials/ -> /localhost/terraform/tutorials (I think…)

Good thought, trying

Hm that resulted in a new message, it is red but I’m not sure it’s an error

Hey just wanted to let you know there is something wrong with how my docker is mounting volumes in general https://docs.docker.com/engine/reference/commandline/run/#mount-volume--v---read-only Maybe it is because of running Docker on macOS with minkube

It appears to be a known issue https://github.com/kubernetes/minikube/issues/2481

This is the best workaround I can find for now - https://github.com/kubernetes/minikube/issues/2481#issuecomment-469736193

minikube mount /Users/xxxxx:/Users/xxxxx

or wherever you want to be available to run geodesic in. I need it for several workflow, so I’ll run it in the background

minikube mount /Users/xxxxx:/Users/xxxxx &

@Jeremy G (Cloud Posse) any ideas?

@Erik Osterman (Cloud Posse) This is a 4-year old bug in minikube, so while I’m sympathetic, with this being the first we have heard of it, and it in no way being specific to or triggered by Geodesic, I am not inclined to do anything about it, especially since there is a minikube-specific workaround.

I agree, I just wanted to throw up my solution here for the record

I imagine I’m not the only geodesic user experiencing the need to switch off Docker Desktop due to the license change

Thanks for the support! I look forward to discovering a lot more about Geodesic

Hm interesting aside about this: when I run geodesic now, with the minikube mount running in another terminal window, I get an error message about mounting /localhost , but I find interacting with the container, everything is mounted as expected. red message first:

########################################################################################
# No filesystem is mounted at /localhost which limits Geodesic functionality.
# EXIT THIS SHELL and on your host computer,
# run the following to install the script that runs
# Geodesic with all its features (the recommended way to use Geodesic):
#
#   docker run --rm cloudposse/geodesic:0.149.1-debian init | bash
#
# (On a Linux workstation, you might need to use "sudo bash" instead of just "bash")
#
# After that, you should be able to launch Geodesic just by typing
#
#   geodesic
#
########################################################################################


Disabling user customizations: GEODESIC_CUSTOMIZATION_DISABLED is '/localhost not a volume'

then a green message:

# Initial working directory set from host CWD to /localhost/dev/emr_app_auth

So it seems like either detecting the mount is off when doing this using minikube mount s the intermediary, or that the error message is displayed asynchronously to the mount finishing somehow

BUT I’m unblocked for now, just wanted to let you know it is still imperfect. I will also look into other alternatives to Docker Desktop that may help. Using VirtualBox instead of hyperkit as the virtualization layer seems to fix it as well, but that isn’t acceptable to us from a performance standpoint

@Matan Shavit Thank you for the additional information. I believe I know the cause of the warning message. If you are willing to give it a try, modifying and rebuilding Geodesic, I believe you can eliminate the

 No filesystem is mounted at /localhost...

warning by changing this line to

	if ! df -a | grep -q /localhost; then

We will make and publish that change eventually, but if you want to get a head start (and verify that it works for you), you can, and I would appreciate hearing how it goes.

I can give that a try, sounds fun

@Matan Shavit Please try https://github.com/cloudposse/geodesic/releases/tag/0.152.0-rc1 and report back

Tested this out briefly today.

At least in my setup, I had issues with atmos inside of geodesic and something to do with the file system. I could not get it to run plans that included modules pulled from git, as it always said there were changes.

I was using the docker backend, not the containerd backend.

I’ll probably poke at it more, but besides those hiccups, it was able to build my customized geodesic just fine and do basic stuff. Only ran into terraform/atmos quirks so far.

Could you open an issue with a captire of the errors.

I’ll see if someone can take a look at it

Try this to see my point in the latest shell:

1. cd /conf
2. cd /etc
3. Now quickly change back to your project folder. You have to cd /localhost/…/…/…/folder I think if workdir is /project this is much smoother

@Yonatan Koren I’m working thru your comments (thanks!), and wondering: All of your comments have been related to the Alpine version of geodesic, whereas I was using the Debian version. When we suggest changes (as I did), how should we test? I didn’t think I needed to check the Alpine version, but maybe I do

Some comments like the copying of kube-ps1.sh go for both flavors, because the functionality is identical (see here and here)

But you can easily build with debian instead by rebuilding with DOCKER_BASE_OS=debian

So, I wonder if the way I was adding scripts to my Dockerfile might have caused some of these issues:

Yes possibly

Testing now

Works fine — I invoked the wrapper script with GEODESIC_MAC_FORWARD_SOCKET=true after building the container. Added and removed ssh-agent identities on the host and they appear and disappear within Geodesic.

You tested this using the stock /etc/profile.d/ssh-agent.sh that ships with geodesic?

I checked out your branch and rebuilt the container and installed the wrapper script using make install, then invoked GEODESIC_MAC_FORWARD_SOCKET=true geodesic

My working tree when checking out your branch is entirely clean, I didn’t change anything or add anything

If you use the stock geodesic (w/o my changes), can you see the host identities added/removed? ie. is the MacOS default ssh-agent fine and it’s only Linux where the socket forwarding is the issue?

On the main branch I can perform the same test and it works fine — so I think if you’re experiencing issues with SSH Agent forwarding / socket mounting it’s only on Linux Hosts

Perfect – after we settle on the fix for #594, I’ll see if I can include this as well, or determine how to get it to work with Linux with minimal changes