#docker (2018-11)
All things docker
Archive: https://archive.sweetops.com/docker/
2018-11-14
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
Hi People,
does anyone know a good and proper way to remove layers with secrets from a docker image ?
e.g. the only way that this can be done currently is to use multi stage builds or to unset the ENV/ARGS and there’s this blog post here that mentions experimental secrets but not for strings https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066
TL;DR docker history <docker_images>
exposes layers with secrets from build time.
![attachment image](https://cdn-images-1.medium.com/max/1200/1*vzJN8Wh-qVHKdINZsNXGCg.jpeg)
One of the complexities when using Dockerfiles has always been accessing private resources. If you need to access some private repository…
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
@Nikola Velkovski did you look at https://github.com/goldmann/docker-squash
Docker image squashing tool. Contribute to goldmann/docker-squash development by creating an account on GitHub.
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
here a a post how to do it https://blog.tomecek.net/post/docker-build-with-secrets/
Technical blog. Read about Linux, Python, Containers, Red Hat, Fedora, tools.
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
Nope, thanks will have a look.
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
I also found this https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-command-line-build-output which is more or less what the blog post is saying
Docker Build is one of the most used features of the Docker Engine - users ranging from developers, build teams, and release teams all use Docker Build. Docker Build enhancements…
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I like this new output
Docker Build is one of the most used features of the Docker Engine - users ranging from developers, build teams, and release teams all use Docker Build. Docker Build enhancements…
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
i believe it’s also possible to squash an image without any thirdparty tools
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
docker build --squash
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
Ah yes we discuissed it with @maarten but when using --squash
there will be no caching.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Ah, that’s true… let’s step back: why do you need the secrets in the image to begin with?
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
in this particular case it’s for ruby gems, i need to bundle them from a private repo so I need the api key from it
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
there are cases for npm etc…
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Can you use build args? Those are not persisted in the layers unless the a build step writes it to disk.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Something like
FROM node:X
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > ~/.npmrc && \
npm install && \
rm ~/.npmrc
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
@Erik Osterman (Cloud Posse) I am using args but it turns out those layers also persist
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
yes, the layers persist, but that’s why you need to rm
in the same layer
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
see the example above
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
need to chain with &&
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
Argh that’s why… ok great good to know!
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Description Build an image from a Dockerfile Usage docker image build [OPTIONS] PATH | URL | - Options Name, shorthand Default Description –add-host Add a custom host-to-IP mapping (host:ip) –build-arg… |
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
You can also have an internal proxy that you use in your build process
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
that proxy then has the authentication token for github
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
we wrote one for the docker registry to do this
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
so we could pull images without authentication in a trusted environment
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
same concept would apply to github
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
here’s the simple service
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
(you’d adapt something like this to proxy github requests instead)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
DUPE: ~@Nikola Velkovski~heck this out~<https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066~
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
~Posted just a few days ago… highly relevant~
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
Ah yeah, I already read it and posted it yesterday
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
hahah
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
sorry, missed that! you did post it
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
Anyway thanks for the help, I was also under the impression that passing secrets as args is ok
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
luckily I haven’t pushed any public images
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
so regarding that experimental feature, that would be the cleanest, no?
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
yes
2018-11-25
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
A tool for exploring each layer in a docker image. Contribute to wagoodman/dive development by creating an account on GitHub.
2018-11-28
![tamsky avatar](https://avatars.slack-edge.com/2019-10-31/817094217669_6e765cea39b456597957_72.jpg)
Has anyone used docker as a test environment for starting/stopping services using their init manager? I can’t figure out why none of the “your-favorite-OS-in-docker” images… support their standard init process / service manager.
None of the vendor images (amazon 1 or 2, ubuntu) include their init system: (upstart, systemd, sysvinit) [ to their credit, CentOS does includes instructions on how to create your own “Dockerfile for systemd base image” on their dockerhub readme: https://hub.docker.com/_/centos/ ]
my concern is, that without such hoop jumping, native, installed packages that try to install/enable their /etc/init
or systemd unit files or /etc/init.d
files are going to behave differently than real systems running on the exact same OS.
![tamsky avatar](https://avatars.slack-edge.com/2019-10-31/817094217669_6e765cea39b456597957_72.jpg)
not to mention the OSX support for running systemd is missing (it requires --cap-add=SYS_ADMIN
, apparently only on OSX, to allow systemd to mount tmpfs)
cf. https://github.com/moby/moby/issues/30723
Hi, I did a bit of research around this and, event though it looks like there is a way to work around this problem, this is not applicable to Docker users running it on macOS. OS version: macOS Sie…
![tamsky avatar](https://avatars.slack-edge.com/2019-10-31/817094217669_6e765cea39b456597957_72.jpg)
Actually it looks like I can (on OSX) at least get it to start without --cap-add
and without --security-opt=seccomp:unconfined
by using --tmpfs /tmp --tmpfs /run
in the docker run
command
What is the scoop on running systemd in a container? A couple of years ago I wrote an article on Running systemd with a docker-formatted Container. Sadly, two years later if you google docker systemd this is still the article people see — it’s time for an update. This is a follow-up for my last article. Everything you …
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
is the requirement to use systemd inside of docker or to use systemd to start containers (e.g. like in CoreOS)?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
actually, i’m not quite clear on the problem statement.
![tamsky avatar](https://avatars.slack-edge.com/2019-10-31/817094217669_6e765cea39b456597957_72.jpg)
goal I’m trying to work with is that packer treats a docker builder the same as an EC2 builder… assuming they both start with the same OS image…. that systems work as similarly as possible.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
aha, i see
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@catdevman does this sound familiar?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@catdevman was telling me yesterday that they use packer to build both their AMIs and docker images and using Amazon Linux as base for both.
![tamsky avatar](https://avatars.slack-edge.com/2019-10-31/817094217669_6e765cea39b456597957_72.jpg)
thats where I’d like to be
![tamsky avatar](https://avatars.slack-edge.com/2019-10-31/817094217669_6e765cea39b456597957_72.jpg)
Thanks Erik. That sounds close. @catdevman when you have a moment, I’m interested to hear your strategy for using packer to create both docker images and AMIs
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
fwiw, when we’ve needed init in containers we’ve used s6 or dumb-init
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
s6 - skarnet’s small supervision suite
![joshmyers avatar](https://avatars.slack-edge.com/2018-11-20/483958217281_8117d6f6c62807ce9912_72.jpg)
The folks at Yelp are
2018-11-29
![catdevman avatar](https://secure.gravatar.com/avatar/08dd2e177a984cc989a254eeb5479a4a.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0013-72.png)
yeah @tamsky I can share some of that information… the biggest part is that your starting point has to be the same so for us we went with amazon linux and ubuntu. Only problem with this strategy is docker and assuming everything is root and doesn’t have sudo which means become for ansible needs to be variable and needs to know what type of build you are doing.
![catdevman avatar](https://secure.gravatar.com/avatar/08dd2e177a984cc989a254eeb5479a4a.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0013-72.png)
@tamsky I have made separate packer files for ami and docker but the ansible playbooks are the same just different variables passed in. if you want to know more we can do pm
![tamsky avatar](https://avatars.slack-edge.com/2019-10-31/817094217669_6e765cea39b456597957_72.jpg)
Thanks @catdevman,
that lines up 100% with my starting points (amazonlinux:2
, ubuntu:18.04
)
and matches where I’m at with packer+ansible.
I’d be interested to share my docker bits and ansible “hacks” that I’m using to remove the differences between container and AMI.
maybe we should have a #ansible channel? @Erik Osterman (Cloud Posse)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@tamsky sure thing!
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
2018-11-30
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
Hello people! Any recommendations for a a docker centric CI that just builds and uploads images, and utilizes caching
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
because I am losing my s*** with travis ci
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
travis cannot into Docker
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
we use https://codefresh.io/
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
ahha I knew it !
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
all pipeline steps are containers
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
soo I was checking out the pricing, and 3 concurent builds which is the pro subscription won’t do
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
any idea how will it cost to have let’s say 20 concurent builds ?
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
you need to discuss that with @Erik Osterman (Cloud Posse), he can provide more info and connect you with the Codefresh guys
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
(Y)
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
Thanks
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
one example how we use our Docker images in Codefresh pipelines https://github.com/cloudposse/github-status-updater#integrating-with-codefresh-cicd-pipelines
Command line utility for updating GitHub commit statuses and enabling required status checks for pull requests - cloudposse/github-status-updater
![Nikola Velkovski avatar](https://avatars.slack-edge.com/2018-11-08/474538495603_cc9e62a39b3dbc9d8d65_72.png)
thanks @Andriy Knysh (Cloud Posse)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
set the channel topic:
![tamsky avatar](https://avatars.slack-edge.com/2019-10-31/817094217669_6e765cea39b456597957_72.jpg)
I just found this today: https://github.com/just-containers/s6-overlay interesting approach vs trying to support multiple flavors of init(1) within docker: sysv, upstart, systemd
s6 overlay for containers (includes execline, s6-linux-utils & a custom init) - just-containers/s6-overlay
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
That’s cool!