#geodesic (2023-01)

I’m trying to manage infrastructure for a very large enterprise, and suddenly can’t do anything, any help would be very appreciated

here is the makefile with the enterprise clients name obfuscated out with ***

export HOME_DIR ?= /home/code
export STORAGE_DIR ?= /mnt/storage
export PROJECTS_DIR ?= $(STORAGE_DIR)/projects
export DOCKER_ORG ?= grizzlyforce
export DOCKER_IMAGE ?= $(DOCKER_ORG)/***-atmos
export DOCKER_TAG ?= latest
export DOCKER_IMAGE_NAME ?= $(DOCKER_IMAGE):$(DOCKER_TAG)
export APP_NAME = ***-atmos
GEODESIC_INSTALL_PATH ?= /home/code/.local/bin
export INSTALL_PATH ?= $(GEODESIC_INSTALL_PATH)
export SCRIPT = $(INSTALL_PATH)/$(APP_NAME)
export ADR_DOCS_DIR = docs/adr
export ADR_DOCS_README = $(ADR_DOCS_DIR)/README.md
BUILD_HARNESS_EXTENSIONS_PATH := $(CURDIR)/.build-harness-extensions

-include $(shell curl -sSL -o .build-harness "<https://cloudposse.tools/build-harness>"; echo .build-harness)

.DEFAULT_GOAL := default

## Initialize build-harness, install deps, build docker container, install wrapper script and run shell
all: init deps build install run
	@exit 0

## Install dependencies (if any)
deps:
	@exit 0

## Build docker image
build:
	@make --no-print-directory docker/build

## Push docker image to registry
push:
	@$(call fail,Refusing to push $(DOCKER_IMAGE_NAME) to docker hub)

## Install wrapper script from geodesic container
install:
	@docker run --rm $(DOCKER_IMAGE_NAME) | bash -s $(DOCKER_TAG) || (echo "Try: sudo make install"; exit 1)

## Start the geodesic shell by calling wrapper script
run:
	docker run -ti --rm -v $(HOME_DIR):/localhost -v $(PROJECTS_DIR):/localhost/projects  -v $(STORAGE_DIR)/cody/.aws:/localhost/.aws -v $(STORAGE_DIR)/cody/.ssh:/localhost/.ssh grizzlyforce/***-atmos

@Jeremy White (Cloud Posse) can you share how you fixed it?

(it’s related to the latest version of Make)

i had a feeling this was the case, thanks Erik. I’m on arch so i’ve rolled into the latest version i guess

Aloha, yeah, I had this happen. Can you share your make version with make --version ?

GNU Make 4.4

aha, I think this is the related issue: https://github.com/cloudposse/build-harness/issues/333

but I have a quick way to check

try running /usr/bin/make all

@Jeremy White (Cloud Posse) going to try this now, thank you for the info and patience, i got delayed on other things

I ran the make SHELL='sh -x' init and did experience the infinite loop

is the only option for now to rollback make @Jeremy White (Cloud Posse)?

i did find a half solution here https://stackoverflow.com/questions/19728933/continue-looping-over-submodules-with-the-git-submodule-foreach-command-after

haven’t tried to implement yet

in the build harness, it looks like git submodule –init –recursive could be replaced with

git submodule foreach --remote 'git fetch | 0'
git submodule foreach --remote 'git merge origin master | 0'

or something like this

to be clear, you tried with the old make binary? /usr/bin/make ?

i did not

but i was able to replicate the forever loop

from that github issue

/usr/bin/make for me is also 4.4

we’ve been able to recreate the loop issue as well, so this is definitely a confirmed regression with newer make

lol very annoying

aha, okay, that complicates things.

yeah, in arch there is a project for rolling back packages, not sure what other issues that will cause. I could also just build and install an older version myself i guess

and symlink to it or something

well, there is also the chance you can replace the formula in brew with the make 4.3 script

oh i dont use brew, im not on a mac

its arch linux

also, if you are use nix, I can give you a nix-shell command to try

well, endeavour

hmmm, i dont use nix, but i suppose i could do that as well

no time like the present to get hooked on another tool like nix lol

nix is honestly the easiest way to have an old binary for things like this while bugs are squashed. the command would be nix-shell -p gnumake42

which would hopefully give you a make 4.2 binary in your PATH to quickly unblock

but if you don’t have it set up, that can take time

i see, yeah, i’ve read plenty about it, but never attempted to set it up

haha i can’t remove make because dkms relies on it

so i think installing the downgrade beside it is my only option

of you don’t want to remember the command, you can commit a file shell.nix and put this inside it:

{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
  nativeBuildInputs = with pkgs; [
    gnumake42
  ];
}

such a huge project to get held up on, just my luck haha

well, we’re hoping to squash this bug soon

I just cannot say if it will be soon enough for you

makes sense, i provided a potential solution above. have you guys tried to approach this in any way yet?

https://sweetops.slack.com/archives/CB84E9V54/p1673560456014769?thread_ts=1673413948.047819&cid=CB84E9V54

also, have you tried the change you mentioned above? to alter the git submodule line?

i haven’t, but i could, if i can figure out where that gets called

i dont entirely understand where i would change something to call a different build-harness

i could fork the harness though if you can tell me where that gets pulled in

the include line is in the makefile, around line 25

I’m going to try to find the line you speak of… if I can test it, I’ll holler

i found the call for the build-harness, will attempt

Ok, I did the following: • commented out the line in the Makefile that downloads the build-harness makefile from cloudposse.tools/build-harness • altered the previously downloaded file to use my forked build-harness repo (grizzlyforce/build-harness) • updated the forked repo to use git fetch / git merge • ran make run successfully

i can’t run make all or make build though which tells me my solution is probably not a complete solution?

I think I need to host my own version of that cloudposse.tools/build-harness makefile

okay… can you link to any files you changed?

one sec, let me take this a step further, i just realised that file is actually in the repo, so i also have it hosted and can change to my version

gotcha

it still infinite loops, so it must be before my change

I searched geodesic and build-harness for ‘submodule’ and the only call i found was in build-harness/modules/git/Makefile

is there somewhere else that i’m missing?

I wasn’t able to find it anywhere else either

but

I am suspect of the bootstrap.Makefile in the git module. It has a trailing endif and I don’t like the way that looks…

the issue by rb claims that the bootstrapper is what causes the loop

and if you comment out the include for the bootstrap.Makefile includes, it works. I’ll test that now

confirmed that fixes it for me

can you too? just hop into the build-harness/Makefile and comment out line 42

it should be something about including bootstrap files

I commented out that line, pushed into my repo, ran make all, and still get the infinite loop

https://github.com/grizzlyforce/build-harness/blob/master/Makefile

I restored the modules/git/Makefile to original to test again, no difference

so, I noticed you said you ‘pushed’ the change. I didn’t have to push. I did the edit directly in the infra folder… since the build-harness directory should be present after running the include lines

it seems to try to download the .build-harness file everytime

oh, not the file

the folder

there should be a build-harness/Makefile

oh, you’re running make all in the build-harness project? sorry i was downstream in my own project

thats why i pushed

gotcha. I’m assuming you’re doing a geodesic project, so I have a similar geodesic project

ok, i was able to run make init in the build-harness project with that line commented out.

i was not able to run make init while that line is still active

yeah, we use atmos for managing our terraform projects

so, geodesic in the background

okay, so I think the smoking gun here is the bootstrap.Makefiles

and I definitely see the modules/git/bootstrap.Makefile` as being especially suspect…

fair enough. i am unblocked in my geodesic project by commenting out the line that downloads the build-harness everytime

at least i can run terraform now

an extremely large staffing company is white-labelling our software for their own use and i was in the middle of provisioning their infrastructure when everything stopped working

serious panic set in haha

i was on the verge of just building my own container and installing atmos

gotcha. you don’t have to roll any further with this. I just hoped to unblock you all the same

thank you very very much for your help, i cant thank you enough!

if i can help test in any way, i’m happy to do so, you know i face this problem now

absolutely, I think once I have a fix in place, I’ll mention the branch, and you should be able to set it in the harness or at least hop into the dir and git checkout

if there’s any other way I can help, just holler or add it to the github issue

have a good thursday!

thank you sir! you as well