#docker (2018-11)

docker

All things docker

Archive: https://archive.sweetops.com/docker/

2018-11-14

Nikola Velkovski avatar
Nikola Velkovski

Hi People, does anyone know a good and proper way to remove layers with secrets from a docker image ? e.g. the only way that this can be done currently is to use multi stage builds or to unset the ENV/ARGS and there’s this blog post here that mentions experimental secrets but not for strings https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066 TL;DR docker history <docker_images> exposes layers with secrets from build time.

Build secrets and SSH forwarding in Docker 18.09 – Tõnis Tiigi – Mediumattachment image

One of the complexities when using Dockerfiles has always been accessing private resources. If you need to access some private repository…

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Nikola Velkovski did you look at https://github.com/goldmann/docker-squash

goldmann/docker-squash

Docker image squashing tool. Contribute to goldmann/docker-squash development by creating an account on GitHub.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Handling secrets when building docker images is easy · Tomáš Tomeček

Technical blog. Read about Linux, Python, Containers, Red Hat, Fedora, tools.

Nikola Velkovski avatar
Nikola Velkovski

Nope, thanks will have a look.

Nikola Velkovski avatar
Nikola Velkovski
Build Enhancements for Docker

Docker Build is one of the most used features of the Docker Engine - users ranging from developers, build teams, and release teams all use Docker Build. Docker Build enhancements…

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I like this new output

Build Enhancements for Docker

Docker Build is one of the most used features of the Docker Engine - users ranging from developers, build teams, and release teams all use Docker Build. Docker Build enhancements…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i believe it’s also possible to squash an image without any thirdparty tools

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

docker build --squash

Nikola Velkovski avatar
Nikola Velkovski

Ah yes we discuissed it with @maarten but when using --squash there will be no caching.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ah, that’s true… let’s step back: why do you need the secrets in the image to begin with?

Nikola Velkovski avatar
Nikola Velkovski

in this particular case it’s for ruby gems, i need to bundle them from a private repo so I need the api key from it

Nikola Velkovski avatar
Nikola Velkovski

there are cases for npm etc…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Can you use build args? Those are not persisted in the layers unless the a build step writes it to disk.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Something like

FROM node:X
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > ~/.npmrc && \
  npm install && \
  rm ~/.npmrc
Nikola Velkovski avatar
Nikola Velkovski

@Erik Osterman (Cloud Posse) I am using args but it turns out those layers also persist

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, the layers persist, but that’s why you need to rm in the same layer

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

see the example above

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

need to chain with &&

Nikola Velkovski avatar
Nikola Velkovski

Argh that’s why… ok great good to know!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
docker image build
Description Build an image from a Dockerfile Usage docker image build [OPTIONS] PATHURL- Options Name, shorthand Default Description –add-host Add a custom host-to-IP mapping (host:ip) –build-arg…
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You can also have an internal proxy that you use in your build process

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that proxy then has the authentication token for github

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we wrote one for the docker registry to do this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we could pull images without authentication in a trusted environment

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

same concept would apply to github

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

here’s the simple service

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(you’d adapt something like this to proxy github requests instead)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

DUPE: ~@Nikola Velkovski~heck this out~<https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066~

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

~Posted just a few days ago… highly relevant~

Nikola Velkovski avatar
Nikola Velkovski

Ah yeah, I already read it and posted it yesterday

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hahah

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

sorry, missed that! you did post it

Nikola Velkovski avatar
Nikola Velkovski

Anyway thanks for the help, I was also under the impression that passing secrets as args is ok

Nikola Velkovski avatar
Nikola Velkovski

luckily I haven’t pushed any public images

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so regarding that experimental feature, that would be the cleanest, no?

Nikola Velkovski avatar
Nikola Velkovski

yes

2018-11-25

Nikola Velkovski avatar
Nikola Velkovski
wagoodman/dive

A tool for exploring each layer in a docker image. Contribute to wagoodman/dive development by creating an account on GitHub.

2018-11-28

tamsky avatar

Has anyone used docker as a test environment for starting/stopping services using their init manager? I can’t figure out why none of the “your-favorite-OS-in-docker” images… support their standard init process / service manager.

None of the vendor images (amazon 1 or 2, ubuntu) include their init system: (upstart, systemd, sysvinit) [ to their credit, CentOS does includes instructions on how to create your own “Dockerfile for systemd base image” on their dockerhub readme: https://hub.docker.com/_/centos/ ]

my concern is, that without such hoop jumping, native, installed packages that try to install/enable their /etc/init or systemd unit files or /etc/init.d files are going to behave differently than real systems running on the exact same OS.

tamsky avatar

not to mention the OSX support for running systemd is missing (it requires --cap-add=SYS_ADMIN, apparently only on OSX, to allow systemd to mount tmpfs) cf. https://github.com/moby/moby/issues/30723

Cannot run container with systemd in it on macOS · Issue #30723 · moby/moby

Hi, I did a bit of research around this and, event though it looks like there is a way to work around this problem, this is not applicable to Docker users running it on macOS. OS version: macOS Sie…

tamsky avatar

Actually it looks like I can (on OSX) at least get it to start without --cap-add and without --security-opt=seccomp:unconfined by using --tmpfs /tmp --tmpfs /run in the docker run command

Running systemd in a non-privileged container - RHD Blog

What is the scoop on running systemd in a container? A couple of years ago I wrote an article on Running systemd with a docker-formatted Container. Sadly, two years later if you google docker systemd this is still the article people see — it’s time for an update. This is a follow-up for my last article. Everything you …

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

is the requirement to use systemd inside of docker or to use systemd to start containers (e.g. like in CoreOS)?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

actually, i’m not quite clear on the problem statement.

tamsky avatar

goal I’m trying to work with is that packer treats a docker builder the same as an EC2 builder… assuming they both start with the same OS image…. that systems work as similarly as possible.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

aha, i see

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@catdevman does this sound familiar?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@catdevman was telling me yesterday that they use packer to build both their AMIs and docker images and using Amazon Linux as base for both.

tamsky avatar

thats where I’d like to be

tamsky avatar

Thanks Erik. That sounds close. @catdevman when you have a moment, I’m interested to hear your strategy for using packer to create both docker images and AMIs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

fwiw, when we’ve needed init in containers we’ve used s6 or dumb-init

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
s6 - skarnet's small supervision suite

s6 - skarnet’s small supervision suite

joshmyers avatar
joshmyers

The folks at Yelp are

2018-11-29

catdevman avatar
catdevman

yeah @tamsky I can share some of that information… the biggest part is that your starting point has to be the same so for us we went with amazon linux and ubuntu. Only problem with this strategy is docker and assuming everything is root and doesn’t have sudo which means become for ansible needs to be variable and needs to know what type of build you are doing.

catdevman avatar
catdevman

@tamsky I have made separate packer files for ami and docker but the ansible playbooks are the same just different variables passed in. if you want to know more we can do pm

tamsky avatar

Thanks @catdevman, that lines up 100% with my starting points (amazonlinux:2, ubuntu:18.04) and matches where I’m at with packer+ansible. I’d be interested to share my docker bits and ansible “hacks” that I’m using to remove the differences between container and AMI.

maybe we should have a #ansible channel? @Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@tamsky sure thing!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I will create in a few

2

2018-11-30

Nikola Velkovski avatar
Nikola Velkovski

Hello people! Any recommendations for a a docker centric CI that just builds and uploads images, and utilizes caching

Nikola Velkovski avatar
Nikola Velkovski

because I am losing my s*** with travis ci

Nikola Velkovski avatar
Nikola Velkovski

travis cannot into Docker

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Nikola Velkovski avatar
Nikola Velkovski

ahha I knew it !

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

all pipeline steps are containers

Nikola Velkovski avatar
Nikola Velkovski

soo I was checking out the pricing, and 3 concurent builds which is the pro subscription won’t do

Nikola Velkovski avatar
Nikola Velkovski

any idea how will it cost to have let’s say 20 concurent builds ?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you need to discuss that with @Erik Osterman (Cloud Posse), he can provide more info and connect you with the Codefresh guys

Nikola Velkovski avatar
Nikola Velkovski

(Y)

Nikola Velkovski avatar
Nikola Velkovski

Thanks

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

one example how we use our Docker images in Codefresh pipelines https://github.com/cloudposse/github-status-updater#integrating-with-codefresh-cicd-pipelines

cloudposse/github-status-updater

Command line utility for updating GitHub commit statuses and enabling required status checks for pull requests - cloudposse/github-status-updater

Nikola Velkovski avatar
Nikola Velkovski

thanks @Andriy Knysh (Cloud Posse)

tamsky avatar

I just found this today: https://github.com/just-containers/s6-overlay interesting approach vs trying to support multiple flavors of init(1) within docker: sysv, upstart, systemd

just-containers/s6-overlay

s6 overlay for containers (includes execline, s6-linux-utils & a custom init) - just-containers/s6-overlay

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That’s cool!

    keyboard_arrow_up