#docker

All things docker Archive: https://archive.sweetops.com/docker/

2019-10-17

2019-10-16

Mithra

For now I want to set up the Environment variables (Dev, Stage or Test and Production) in a single docker file instead of creating env variables for each environment. All I need is to get those environment variables in a single docker file and deploy through web application in azure portal. Please help me up on it.

Mithra

So that the variables has to pick up Dynamically.

2019-10-15

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Brought down our whole CI/CD in CircleCI. Looks like there is no out-of-the-box redundancy for when the hub is down.

Mithra

Hello can some one help me to Create a docker file with environment variables such as (production, QA ) in JSON.

aknysh

what do you mean by in JSON?

aknysh
Environment variables in Compose

There are multiple parts of Compose that deal with environment variables in one sense or another. This page should help you find the information you need. Substitute environment variables in…

aknysh

you can provide a file with ENV vars

aknysh
docker run -i --rm --env-file example.env
aknysh

but those are key-value pairs

aknysh

you can stringify a JSON object into a string, and then provide it as value to an ENV var

aknysh

and then in the app, decode the JSON string into a JSON object

Mithra

Thanks

you can also use tools like jq, gomplate to extract some data from JSON in any format you need

2019-10-01

@Nikola Velkovski Thanks for that. What puzzles me is that the container works on other hosts. At which point do permissions of the user within the container get constrained by the host permissions?

When I -it into it as root as su as node, everything works fine

node user has permissions and can run entrypoint and cmd

Nikola Velkovski

@ that is weird indeed, I am usually using debian and alpine, and I haven’t encountered similar issues.

Nikola Velkovski

The only thing I can think of is a MAC software

Nikola Velkovski

apparmor, SElinux or grsecurity

Nikola Velkovski

on the host that runs the dockers

disabled everything selinux, but maybe one of the others

It’s a hardened host

Nikola Velkovski

but wait there was one aditional step…

And I have no visibility atm on how it was setup

Nikola Velkovski

you need to disable it

Nikola Velkovski

and then either reboot or restart the service..

Nikola Velkovski

what was it..

Nikola Velkovski

<thinking>

Yup, done that, thanks for your input on this

Just annoying that there is nothing but a single exec user process caused "permission denied" error

I thought the purpose of docker was to avoid this kind of nonsense

Nikola Velkovski

haha you wish

Nikola Velkovski

it should be the other way around, SElinux is in place just to annoy the hell out of you

2019-09-30

re: above - turned out that switching user in the container is causing the issue (ie a node:10-alpine image is working fine, but adding USER node to it causes the error). Hoping someone may have an idea on what may be causing this

Nikola Velkovski

Hi Igor, well it’s kinda obvious that the issue is with the user not having enough permissions to start the process.

Nikola Velkovski

I would suggest you start the process yourself

Nikola Velkovski

e.g.

Nikola Velkovski

docker run -it node:10-alpine sh

Nikola Velkovski

that would start a shell with the user specificed in the USER directive

Nikola Velkovski

if you need root you can do the following

Nikola Velkovski

docker run -u 0 -it node:10-alpine sh

Nikola Velkovski

I guess from there you can deduct what kind of rights does the user node need.

2019-09-27

Has anybody run into a problem with exec user process caused "permission denied" on running a docker container? The image doesn’t work on a hardened RHEL host specifically. Also, an nginx host based on the same alpine image works fine, but not node/redis.

2019-09-17

I am testing the docker swarm configuration for the first time with and nginx+nodejs+redis combo

And a single t2.medium server without docker is showing significantly better results in performance testing to 2 t2.medium nodes running in swarm

Any idea on why docker isn’t performing up to par?

2019-09-15

Daniel Minella

Someone already face something like this:

'An assembly specified in the application dependencies manifest (x.Api.deps.json) was not found: ` package: ‘System.Private.ServiceModel’, version: ‘4.5.3’ path: ‘runtimes/unix/lib/netstandard2.0/System.Private.ServiceModel.dll’’`

My situation:

Dotnet 2.2 application, with this Dockerfile:

FROM <http2.2> AS base

Restoring

WORKDIR /app

Copy solution

COPY ./*.sln ./

Copy src projects

COPY src//.csproj ./ RUN for file in $(ls .csproj); do mkdir -p src/${file%.}/ && mv $file src/${file%.*}/; done

Copy tests projects

COPY tests//.csproj ./ RUN for file in $(ls .csproj); do mkdir -p tests/${file%.}/ && mv $file tests/${file%.*}/; done

Restore

RUN dotnet restore

Publishing

WORKDIR /app COPY src/. ./src/ RUN dotnet publish -c Release –no-restore -o /app/out

Testing

FROM base AS tester WORKDIR /app COPY tests/. ./tests/ RUN dotnet test –logger:trx –no-restore

Running

FROM <http2.2> AS runtime

EXPOSE 5001 EXPOSE 5002 ENV ASPNETCORE_ENVIRONMENT=Unset ENV ConnectionStrings__DefaultConnection=Unset ENV Sentry__Dsn=Unset ENV ELK__Elasticsearch__Dsn=Unset ENV TokenAuthSettings__Issuer=Unset ENV TokenAuthSettings__Key=Unset ENV TokenAuthSettings__Audience=Unset

WORKDIR /app COPY –from=base /app/out/* ./ ENTRYPOINT [“dotnet”, “x.Api.dll”]

aknysh
Could not load file or assembly 'System.Private.ServiceModel, Version=4.1.2.2 · Issue #2824 · dotnet/wcf

I am using Azure Function (C#) which is calling a .NET standard library to call an external WCF service and I am getting the below error. I am not sure what exactly happening inside the Function ca…

aknysh

apparently they fixed it in .NET Core 3.0

Daniel Minella

Hm, I upgraded but still with the same error

2019-09-06

We’re just starting with docker. Currently, we are planning to use CircleCi to build core images for our solution and distribute customized private images+data container combo (which is based on the core) to customers, or host in AWS ECS (leaning Fargate atm). I am currently planning to build the core and store in ECR. Then, have customer-specific build pull this image down and use as base, and produce either an image in ECR for ECS or an artifact in S3 to distribute to the customer. Looking for validation that this is a good approach. Is ECR is a good option, or is there a better alternative (I am concerned about having copies of images across all AWS accounts/regions)? Is saving the image and packaging it in S3 for customer delivery the right approach? ** Appreciate the feedback!

Steven

There’s nothing wrong with that. We use 100% ECR for private images. I would recommend not having a lot of duplicate ECR repos. I put them all in a AWS account for shared resources, then have all other accounts access them from there. As for across region ( I don’t have this need yet), it is mostly a performance question. It you need to pull 5 images an hour, you’re not going to worry much about latency. On the other hand, if you need to pull 100 in 5 minutes, you will want to replicate ECR across regions so it is always close to the running containers

Lee Skillen

@ If you’re looking for a service that is specialised for distribution, look at Cloudsmith (https://cloudsmith.io) (note: I work there, and happy to help out).

Lee Skillen

No matter what way you set it up, distributing it via S3 alone is probably not the right approach since that makes it awkward for your customers to use (i.e. external distribution); they’d have to docker load the image after pulling it down, rather than using docker pull directly. Very do-able, but not great since it pulls the entire image down rather than only the layers that it needs.

Thank you @Steven @Lee Skillen

2019-07-30

2019-07-26

Mike Nock

Hey all, question in case anyone might know: Running Gitlab with Docker, and since the docker:dind 19 image push, my pipeline broke like most others. I got the /certs/client volumes mounted with the runner and turned on tls-verify. Build and test phases go through just fine, but at the deploy container, after a successful login to the $CI_REGISTRY the docker container gets an error x509: certificate signed by unknown authority when trying to pull the image (which the test phase containers were able to do successfully with the same login). Thoughts? Anyone else run into this maybe?

vitaly.markov

@Mike Nock do you really need to enable TLS for Docker? did you try to disable TLS passing env variable DOCKER_TLS_CERTDIR with empty value?

Mike Nock

I don’t, but disabling it wasn’t working either. I ended up getting it working by mounting /certs as an environment variable, and a runner variable. Then migrating all services over to dind, and the images to docker:stable

Seemed to work well.

2019-06-28

I guess @btai meant, maintaining, building and updating docker images (and push the new tagged release to a registry). Do you know any CI/CD examples for that purpose?

meant moreso fo rsecutity

i use codefresh for wut ur talking about

2019-06-20

docker watchtower

2019-06-19

how do you guys periodically update your docker images? looking moreso for automated solutions

1
Erik Osterman

What specifically is the objective?

Erik Osterman

Pull in latest packages?

2019-06-09

has anyone tried running a desktop on docker. Target is to deploy a virtual desktop solution using containers. Tried LXC containers, but had issues with the networking when the core server was deployed on cloud. Seems docker will be a better alternative, then again not easier to configure as a workstation like lxc/lxd.

2019-06-06

2019-06-01

Maxim Tishchenko

hello everyone, I have a question about npm install -> postinstall inside docker I have super small config (exmple)

{
  "private": true,
  "scripts": {
    "postinstall": "echo test"
  }
}

when I execute npm i --unsafe-perm everything goes well but when I execute npm ci --unsafe-perm I’ve got error message npm ERR! Cannot read property 'length' of undefined

Maxim Tishchenko

does anybody know what to do?

Erik Osterman

Have you tried building the container without running this command in the Dockefile. Then start the container with a shell, and triage that way? Sometimes it’s easier to understand what is going on

Maxim Tishchenko

hmmm

Maxim Tishchenko

no

Maxim Tishchenko

I’ve tried. - the same error

Step 5/7 : RUN npm ci --unsafe-perm
 ---> Running in 8781a79481f0
npm ERR! Cannot read property 'length' of undefined

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-06-01T19_33_54_463Z-debug.log
Maxim Tishchenko

the problem is in postinstall

Maxim Tishchenko

but I have no any idea what is wrong with npm ci and postinstall

Maxim Tishchenko

seems like i found the reason!

Maxim Tishchenko

npm 6.5.0 works

Maxim Tishchenko

npm upper version doesn’t work

2019-05-28

do you guys create a non-root user in all your dockerfiles?

Nikola Velkovski

@btai yes

Nikola Velkovski

sometimes it just works with the nobody:nobody user/group present inside the docker image.

2019-05-21

tamsky

https://github.com/moby/moby/issues/2259#issuecomment-494662512

Superb recent comment on docker’s support for volume mount +(uid/gid/access bits) support, and how hacks around this missing basic feature are now appearing in helm charts.

Add ability to mount volume as user other than root · Issue #2259 · moby/moby

Use case: mount a volume from host to container for use by apache as www user. The problem is currently all mounts are mounted as root inside the container. For example, this command docker run -v …

2019-05-09

Exequiel Barrirero

For Alpine Linux container based implementations.

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5021

Versions of the Official Alpine Linux Docker images (since v3.3) contain a NULL password for the `root` user. This vulnerability appears to be the result of a regression introduced in December of 2015. Due to the nature of this issue, systems deployed using affected versions of the Alpine Linux container which utilize Linux PAM, or some other mechanism which uses the system shadow file as an authentication database, may accept a NULL password for the `root` user.
CVE - CVE-2019-5021

Common Vulnerabilities and Exposures (CVE®) is a list of entries — each containing an identification number, a description, and at least one public reference — for publicly known cybersecurity vulnerabilities. Assigned by CVE Numbering Authorities (CNAs) from around the world, use of CVE Entries ensures confidence among parties when used to discuss or share information about a unique software vulnerability, provides a baseline for tool evaluation, and enables data exchange for cybersecurity automation.

2019-05-05

johncblandii

Has anyone done docker image promotion on Artifactory? (specifically through jenkins or similar CI)

Blaise Pabon

Oh boy, it was about 2 yrs ago at my last job. We had a groovy method that promoted as part of the release process.

2019-04-26

Erik Osterman

Docker Hub Hacked. 190K accounts affected (~5%), GitHub tokens may be exposed.

2019-04-15

oscarsullivan_old

Damn, I’m still super stuck on volumes and changes in a container not being reflected on the host.

oscarsullivan_old

I have this volume mount taking place ./config/schema.json:/app/config/schema.json When /app/config/schema.json is updated in the container, it is not on my local… I want to ‘get’ the file from the container. I’ve tried with ./config/schema.json both existing and not on my local as well as mounting the directory above it and not the file, as schema.json is generated during the build of the container

oscarsullivan_old

oh have I just answered it… volume mounting is for when it is run not when it is being built.

2019-04-09

oscarsullivan_old

I’m volume mounting package-lock.json (a file) but it is mounting as a directory… Any ideas why its a dir not a file?

    volumes:
      - ./:/app
      - /app/node_modules
      - /app/package-lock.json
oscarsullivan_old

I can only think it is because volume is meant to be a dir…

oscarsullivan_old

I’m noticing that my yarn.lock file in /app (mounted above) doesn’t appear on my local though

oscarsullivan_old

It’s only files created on my local that appear in /app

Erik Osterman
Erik Osterman

Aha yes the file must first exist locally

oscarsullivan_old

In a dockerised node app (or any language, really) should the package manager lock file be source controlled when it is already inside the container? Really struggling to mount it and have that file update

Nikola Velkovski

When you need to install gems and/or packages or pip module ( or however they are called ) ideally one would use a multi stage docker builds in which in the initial stages you package the needed modules and then you copy them over to the last stage which is a docker image without the lock file, the build tools for them etc.. ( among many other things )

Nikola Velkovski
Use multi-stage builds

Multi-stage builds are a new feature requiring Docker 17.05 or higher on the daemon and client. Multistage builds are useful to anyone who has struggled to optimize Dockerfiles while keeping…

aknysh

this is how i did it:

aknysh
FROM node:11.2.0-alpine as builder
WORKDIR /usr/src/app
COPY package.json ./
COPY package-lock.json ./
RUN npm install --only=production
COPY server/ ./server/
COPY static/ ./static/
COPY views/ ./views/
COPY app.js ./

FROM node:11.2.0-alpine
WORKDIR /usr/src/app
COPY --from=builder /usr/src/app/ ./
EXPOSE 3000
CMD ["node", "app.js"]
oscarsullivan_old

thanks @aknysh I’ll give that dockerfile a go

oscarsullivan_old

here’s my local dev one:

oscarsullivan_old
# Stage 1 - build & run local environment
FROM node:8.15-alpine AS react-build

WORKDIR /app

ARG PORT
EXPOSE ${PORT}
ARG APP_ENV

COPY package.json package.json
RUN yarn

RUN ls -la

CMD ["yarn", "start"]
oscarsullivan_old
version: '3'
services:
  portal:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        - PORT=3070
        - APP_ENV=.env-local
    ports:
      - "3070:3070"
    volumes:
      - ./:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development
loren

i don’t use docker a whole lot yet, so not sure if this helps, but saw this article recently and it sounded to me like it might be dealing with something similar…. https://medium.com/build-acl/docker-deployments-using-terraform-d2bf36ec7bdf

Docker Deployments using Terraform

How to use Terraform to not only set up your orchestration platform, but also to deploy code updates as part of Continuous Delivery.

loren

ignore me if off base

loren
Using Docker Containers As Development Machines

How we did it and the lessons we learnt

Maciek Strömich
Another tradeoff is that now every command you run on the traditional non-docker environment will need to be run inside the container by SSH-ing into it. 

yeah… ssh-ing into containers

loren

that seemed like a terrible idea

Joe Presley

I’ve worked in a similar setup for a project. Ssh-ing into the containers isn’t bad. Whether it’s docker or vagrant, I’d rather use a standard environment for a project than each developer has a bespoke development environment.

Joe Presley

And you don’t really ssh into it. You just docker-compose <container> <command>

Joe Presley

Using bash for the <command> is if you want a dedicated terminal to run commands within the container.

oscarsullivan_old

#geodesic would like a word with you

1
Joe Presley

I could never wrap my mind around using it.

2019-04-03

Erik Osterman
Turn Your Code into Docker Images with Cloud Native Buildpacks

With Cloud Native Buildpacks, we’ve taken the same philosophies that made buildpacks successful and applied them towards creating Docker images.

2019-04-01

oscarsullivan_old

Has anyone noticed dockerised applications behaving different when using localhost vs IP

I have an app that works on localhost:80 but when using an IP it behaves differently. Really frustrating, because obviously when on a server the users will be using the ‘ip’ instead of ‘localhost’

oscarsullivan_old

Tried with both docker-compose and docker run

oscarsullivan_old

oscarsullivan_old

it was because of https’s less strict behaviour on localhost

Erik Osterman

aha! makes sense

Erik Osterman
Why your Docker multi-stage build is surprisingly slow

If you want your Docker images to be small and you still want fast builds, multi-stage images are the way to go. And yet, you might find that multi-stage builds are actually quite slow in practice, in particular when running in your build pipeline. If that’s happening, a bit of digging is likely to show that even though you copied your standard build script, somehow the first stage of the Dockerfile gets rebuilt every single time. Unless you’re very careful, Docker’s build cache often won’t work for multi-stage builds—and that means your build is slow. What’s going on? In this article you will learn: Why multi-stage builds don’t work with the standard build process you’re used to for single-stage images. How to solve the problem and get fast builds. A note: outside the specific topic under discussion, the Dockerfiles and build scripts in this article are not examples of best practices, since that would involve many extraneous details that would make the main point harder to understand.

1
mmuehlberger

Thanks for sharing, @Erik Osterman! I started using multi-stage images quite recently (we’re just getting started on using containers at work) and this is something I encountered yesterday with our backend.

Why your Docker multi-stage build is surprisingly slow

If you want your Docker images to be small and you still want fast builds, multi-stage images are the way to go. And yet, you might find that multi-stage builds are actually quite slow in practice, in particular when running in your build pipeline. If that’s happening, a bit of digging is likely to show that even though you copied your standard build script, somehow the first stage of the Dockerfile gets rebuilt every single time. Unless you’re very careful, Docker’s build cache often won’t work for multi-stage builds—and that means your build is slow. What’s going on? In this article you will learn: Why multi-stage builds don’t work with the standard build process you’re used to for single-stage images. How to solve the problem and get fast builds. A note: outside the specific topic under discussion, the Dockerfiles and build scripts in this article are not examples of best practices, since that would involve many extraneous details that would make the main point harder to understand.

1

2019-03-22

tamsky
GoogleContainerTools/kaniko

Build Container Images In Kubernetes. Contribute to GoogleContainerTools/kaniko development by creating an account on GitHub.

2019-03-14

@rohit can I help you?

what part are you struggling with?

2019-03-13

rohit

@aknysh thank you very much. I am still not able to connect all the dots

rohit

maybe because i am new to this

2019-03-12

2019-03-11

rohit

Hi. I am new to docker, we do have java web applications(java+tomcat) running nginx as reverseproxy. I am not sure how to run both tomcat and nginx in a same container, can anyone Please help me out ?

aknysh

@rohit are you using kubernetes?

rohit

i am just at the beginning stage but eventually would like to use k8s maybe EKS as we run our infrastructure in AWS

rohit

This is very helpful, will go through these articles

rohit

and how do you pass secrets to docker containers ?

rohit

Let’s say if i have a properties file and the value needs to be populated at run time, how would i do that in containers world ?

aknysh

we use https://github.com/segmentio/chamber to store secrets to SSM Parameter Store. Then, for example when deploying from a CI/CD pipeline, we use chamber to read the secrets from SSM and populate ENV vars with the secrets. Then we use helmfile to deploy a Kubernetes app that reads the ENV vars, e.g. https://github.com/cloudposse/helmfiles/blob/master/releases/datadog.yaml#L39

segmentio/chamber

CLI for managing secrets. Contribute to segmentio/chamber development by creating an account on GitHub.

cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

rohit

when you say use chamber to store secrets to SSM store, do you mean that you execute chamber commands to write to it ?

segmentio/chamber

CLI for managing secrets. Contribute to segmentio/chamber development by creating an account on GitHub.

cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

aknysh

yes

aknysh

write it first

aknysh

then read it when deploying

rohit

because chamber would already know about the variables/values that it stored in SSM ?

aknysh

you know the service (namespace) and the name, so you can read it

oscarsullivan_old

The access key to access SSM.. how secure is this? Can you store it in VCS or is it generated at runtime and added to SSM allowed keys via aws-vault auth?

endofcake

Where is your service running? Storing secrets in VCS is a bad idea.

rohit

@aknysh thanks for pointing me to the right tools. To begin with, if i just want to pass secrets to my docker container, how do i achieve that ?

endofcake

Have the container retrieve it from a secrets management service at init time.

aknysh
Runtime secrets with docker containers

We, at YP are using docker containers for quite some time now. Onboarding onto docker wasn’t always that easy. There are lots of things to account for before running a docker container in productio…

rohit

@aknysh Is it possible to use Chamber as a runtime secrets manager ?

aknysh

we use chamber from geodesic and from CI/CD pipelines (Codefresh)

aknysh

chamber is a CLI tool

aknysh

which works with AWS SSM Parameter Store

aknysh

so if you are asking if your app could use chamber, then probably not a good idea since you will have to call chamber from the app

aknysh

but AWS has SDKs for all languages, so you can just call SSM API from your app to get secrets if you need that

2019-03-06

Erik Osterman

I had naively tried to implement this 4 years ago using btsync and the docker v1 registry

Erik Osterman

lets just say, it didn’t work (at all!)

2019-03-05

Erik Osterman
Introducing Kraken, an Open Source Peer-to-Peer Docker Registry

Developed by Uber, Kraken is an open source peer-to-peer Docker registry capable of distributing terabytes of data in seconds.

2

2019-02-25

Erik Osterman
ivanilves/lstags

Manipulate Docker images across different registries - ivanilves/lstags

2019-02-21

Nikola Velkovski
Improve Build Performance and Save Time Using Local Caching in AWS CodeBuild | Amazon Web Services

AWS CodeBuild now supports local caching, which makes it possible for you to persist intermediate build artifacts locally on the build host so that they are available for reuse in subsequent build runs. Your build project can use one of two types of caching: Amazon S3 or local. In this blog post, we will discuss […]

Nikola Velkovski

anyone using codebuild ?

2019-02-15

Maciek Strömich

Is anyone using latest python image (doesn’t matter which tag) and latest celery? I’ve noticed that after yesterdays pull my celery workers die constantly without any meaningful notification in the debug logs

joshmyers

any tag including older releases?

Maciek Strömich

yeah

Maciek Strömich

I’ve checked so far python:3.4-jessie, python:3.6 and python:3.6-alpine

Maciek Strömich

but few minutes after writing this I found a difference in the kombu version library which was 4.2.2-post1

Maciek Strömich

and now it’s 4.3.0

Maciek Strömich

I’m rebuilding the image

Maciek Strömich

and it works

Maciek Strömich

so it’s not really related to docker image

2019-02-14

Erik Osterman
05:18:43 AM

@Erik Osterman set the channel purpose: All things docker Archive: https://archive.sweetops.com/docker/

2019-02-11

2019-01-30

Bogdan

how do you guys handle the ordered_placement_strategy in a ecs service module, from an input perspective (passing a list of maps or map) when passing several strategies? I couldn’t find an example in cloudposse ecs service modules

2019-01-28

tamsky

and here lies my error:

 => [4/4] RUN ssh-add -l:
#9 0.397 Could not open a connection to your authentication agent.
------
rpc error: code = Unknown desc = executor failed running [/bin/sh -c ssh-add -l]: exit code: 2

RUN command should read RUN --mount=type=ssh ssh-add -l \| tee /hello

tamsky

working build output from OSX:

# ( export DOCKER_BUILDKIT=1 && docker build --ssh default -f Dockerfile.ssh . )
[+] Building 1.6s (10/10) FINISHED                                                                                             
 => [internal] load .dockerignore                                                                                         0.4s
 => => transferring context: 2B                                                                                           0.0s
 => [internal] load build definition from Dockerfile.ssh                                                                  0.7s
 => => transferring dockerfile: 41B                                                                                       0.0s
 => resolve image config for <http<i class="em em-//docker.io/docker/dockerfile"></i>experimental>                                                     0.0s
 => CACHED <docker-image<i class="em em-//docker.io/docker/dockerfile"></i>experimental>                                                        0.0s
 => [internal] load metadata for <http<i class="em em-//docker.io/library/alpine"></i>latest>                                                          0.0s
 => [1/4] FROM <http://docker.io/library/alpine>                                                                                   0.0s
 => CACHED [2/4] RUN apk add --no-cache openssh-client git                                                                0.0s
 => CACHED [3/4] RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan <http://github.com> >> ~/.ssh/known_hosts                              0.0s
 => CACHED [4/4] RUN --mount=type=ssh ssh-add -l \| tee /hello                                                             0.0s
 => exporting to image                                                                                                    0.0s
 => => exporting layers                                                                                                   0.0s
 => => writing image sha256:7dcdf95d6e1745d9c12ca89b2209fd58fe7417c93acb1e3e5ce35a20ff544b14                              0.0s
tamsky

/hello does get populated, so we finally have a cross platform solution for the agent

1
tamsky

and for folks who want to know more about how “Docker for Mac” works… I found this great, and very detailed, post about it:

1

2019-01-24

i5okie

oh thanks

Erik Osterman

@i5okie let me know if you get that working. I haven’t tried it yet, but looks cool!

Erik Osterman

@tamsky have you tried this?

i5okie

reading right now

tamsky

I just upgraded my Docker Desktop on OSX to 2.0.0.2 to test this. The secrets file stuff works with “experimental features” enabled. The ssh_agent stuff, not so much.

tamsky
12:00:54 AM

experimental features flag

Erik Osterman

What happened when you tried using the SSH agent stuff?

tamsky
# make ssh
export DOCKER_BUILDKIT=1
docker build --ssh default -f Dockerfile.ssh .
[+] Building 2.4s (9/9) FINISHED                                                                                               
 => [internal] load build definition from Dockerfile.ssh                                                                  0.0s
 => => transferring dockerfile: 334B                                                                                      0.0s
 => [internal] load .dockerignore                                                                                         0.0s
 => => transferring context: 2B                                                                                           0.0s
 => resolve image config for <http<i class="em em-//docker.io/docker/dockerfile"></i>experimental>                                                     1.0s
 => CACHED <docker-image<i class="em em-//docker.io/docker/dockerfile"></i>[email protected]:2220efe9582e00cd8f6bbee8f4566e34d7f0388c0e10f2>  0.0s
 => [internal] load metadata for <http<i class="em em-//docker.io/library/alpine"></i>latest>                                                          0.0s
 => [1/4] FROM <http://docker.io/library/alpine>                                                                                   0.0s
 => CACHED [2/4] RUN apk add --no-cache openssh-client git                                                                0.0s
 => CACHED [3/4] RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan <http://github.com> >> ~/.ssh/known_hosts                              0.0s
 => ERROR [4/4] RUN ssh-add -l                                                                                            0.6s
------
 > [4/4] RUN ssh-add -l:
#9 0.397 Could not open a connection to your authentication agent.
------
rpc error: code = Unknown desc = executor failed running [/bin/sh -c ssh-add -l]: exit code: 2
tamsky

https://github.com/mariusgrigaitis/docker-mac-ssh-auth-sock might be a workable hack for interactive use, but buildkit docker build can’t use that hack.

mariusgrigaitis/docker-mac-ssh-auth-sock

SSH_AUTH_SOCK socket forwarding for Docker for Mac - mariusgrigaitis/docker-mac-ssh-auth-sock

tamsky
Support for sharing unix sockets · Issue #483 · docker/for-mac

Expected behavior When mounting a directory containing unix sockets the sockets should function the same as they do on a Linux host. Actual behavior The socket is &#39;there&#39;, but non-functiona…

$SSH_AUTH_SOCK is not being forwarded to docker · Issue #410 · docker/for-mac

Expected behavior OSX ssh-agent socket is available (for mount) in containers $ docker run -it -v ${SSH_AUTH_SOCK}:${SSH_AUTH_SOCK} -e SSH_AUTH_SOCK=&quot;${SSH_AUTH_SOCK}&quot; –rm alpine:3.4 /bi…

tamsky

iheartradio[1] has gotten around this entire mess using multiple stages and ephemerally tagged local images:

  • download source docker run … ; docker commit using a different author’s ssh-agent forwarding hack[2]
  • followed by a final docker build which can build the source and do what it wants with the artifacts.

[1] https://github.com/iheartradio/docker-node [2] https://github.com/avsm/docker-ssh-agent-forward

iheartradio/docker-node

iHeartRadio’s Nodejs Dockerfiles. Contribute to iheartradio/docker-node development by creating an account on GitHub.

avsm/docker-ssh-agent-forward

Forward SSH agent socket into a container. Contribute to avsm/docker-ssh-agent-forward development by creating an account on GitHub.

Erik Osterman

oh interesting using multistage like that

tamsky

also, for folks following along from home, here’s the “check if your setup is experimental”:

# docker info \| grep -i Experimental
Experimental: true
2

2019-01-23

i5okie

hey, question.. I have a git repo with submodules of our app repos. in this directory i’ve got docker-compose.yaml file.

Is it possible to use context to a git repo just for the Dockerfile, but when building make it use the local folder from submodule? I want to do docker-compose build app

  • get dockerfile / entrypoint file from repo
  • build image by copying things from submodule
Erik Osterman

ping me in a few hours if you don’t get a response

i5okie

ok

Erik Osterman

Ok, so it sounds like you’re having trouble pulling submodules from private git repo inside of Docker?

Erik Osterman

If so, see this:

Erik Osterman
Build secrets and SSH forwarding in Docker 18.09 – Tõnis Tiigi – Medium

One of the complexities when using Dockerfiles has always been accessing private resources. If you need to access some private repository…

2019-01-03

Erik Osterman

docker hub is down

2018-12-14

sarkis
Introducing the New Docker Hub - Docker Blog

Today, we’re excited to announce that Docker Store and Docker Cloud are now part of Docker Hub, providing a single experience for finding, storing and sharing container images. This means that: Docker Certified and Verified Publisher Images are now available for discovery and download on Docker Hub Docker Hub has a new user experience   Millions of individual users and more than a hundred thousand organizations use Docker Hub, Store and Cloud for their container content needs. We’ve designed this Docker Hub update to bring together the features that users of each product know and love the most, while addressing known Docker Hub requests around ease of use, repository and team management. Here’s what’s new: Repositories View recently pushed tags and automated builds on your repository page Pagination added to repository tags Improved repository filtering when logged in on the Continue reading…

2018-12-02

I’ll be at DockerCon BCN if anyone has anything I can facilitate or is going to be there just let me know

2018-11-30

Nikola Velkovski

Hello people! Any recommendations for a a docker centric CI that just builds and uploads images, and utilizes caching

Nikola Velkovski

because I am losing my s*** with travis ci

Nikola Velkovski

travis cannot into Docker

Nikola Velkovski

ahha I knew it !

aknysh

all pipeline steps are containers

Nikola Velkovski

soo I was checking out the pricing, and 3 concurent builds which is the pro subscription won’t do

Nikola Velkovski

any idea how will it cost to have let’s say 20 concurent builds ?

aknysh

you need to discuss that with @Erik Osterman, he can provide more info and connect you with the Codefresh guys

Nikola Velkovski

(Y)

Nikola Velkovski

Thanks

aknysh

one example how we use our Docker images in Codefresh pipelines https://github.com/cloudposse/github-status-updater#integrating-with-codefresh-cicd-pipelines

cloudposse/github-status-updater

Command line utility for updating GitHub commit statuses and enabling required status checks for pull requests - cloudposse/github-status-updater

Nikola Velkovski

thanks @aknysh

Erik Osterman
11:20:01 PM

@Erik Osterman set the channel topic:

tamsky

I just found this today: https://github.com/just-containers/s6-overlay interesting approach vs trying to support multiple flavors of init(1) within docker: sysv, upstart, systemd

just-containers/s6-overlay

s6 overlay for containers (includes execline, s6-linux-utils & a custom init) - just-containers/s6-overlay

Erik Osterman

That’s cool!

2018-11-29

catdevman

yeah @tamsky I can share some of that information… the biggest part is that your starting point has to be the same so for us we went with amazon linux and ubuntu. Only problem with this strategy is docker and assuming everything is root and doesn’t have sudo which means become for ansible needs to be variable and needs to know what type of build you are doing.

catdevman

@tamsky I have made separate packer files for ami and docker but the ansible playbooks are the same just different variables passed in. if you want to know more we can do pm

tamsky

Thanks @catdevman, that lines up 100% with my starting points (amazonlinux:2, ubuntu:18.04) and matches where I’m at with packer+ansible. I’d be interested to share my docker bits and ansible “hacks” that I’m using to remove the differences between container and AMI.

maybe we should have a #ansible channel? @Erik Osterman

Erik Osterman

@tamsky sure thing!

Erik Osterman

I will create in a few

2

2018-11-28

tamsky

Has anyone used docker as a test environment for starting/stopping services using their init manager? I can’t figure out why none of the “your-favorite-OS-in-docker” images… support their standard init process / service manager.

None of the vendor images (amazon 1 or 2, ubuntu) include their init system: (upstart, systemd, sysvinit) [ to their credit, CentOS does includes instructions on how to create your own “Dockerfile for systemd base image” on their dockerhub readme: https://hub.docker.com/_/centos/ ]

my concern is, that without such hoop jumping, native, installed packages that try to install/enable their /etc/init or systemd unit files or /etc/init.d files are going to behave differently than real systems running on the exact same OS.

tamsky

not to mention the OSX support for running systemd is missing (it requires --cap-add=SYS_ADMIN, apparently only on OSX, to allow systemd to mount tmpfs) cf. https://github.com/moby/moby/issues/30723

Cannot run container with systemd in it on macOS · Issue #30723 · moby/moby

Hi, I did a bit of research around this and, event though it looks like there is a way to work around this problem, this is not applicable to Docker users running it on macOS. OS version: macOS Sie…

tamsky

Actually it looks like I can (on OSX) at least get it to start without --cap-add and without --security-opt=seccomp:unconfined by using --tmpfs /tmp --tmpfs /run in the docker run command

Running systemd in a non-privileged container - RHD Blog

What is the scoop on running systemd in a container? A couple of years ago I wrote an article on Running systemd with a docker-formatted Container. Sadly, two years later if you google docker systemd this is still the article people see — it’s time for an update. This is a follow-up for my last article. Everything you …

Erik Osterman

is the requirement to use systemd inside of docker or to use systemd to start containers (e.g. like in CoreOS)?

Erik Osterman

actually, i’m not quite clear on the problem statement.

tamsky

goal I’m trying to work with is that packer treats a docker builder the same as an EC2 builder… assuming they both start with the same OS image…. that systems work as similarly as possible.

Erik Osterman

aha, i see

Erik Osterman

@catdevman does this sound familiar?

Erik Osterman

@catdevman was telling me yesterday that they use packer to build both their AMIs and docker images and using Amazon Linux as base for both.

tamsky

thats where I’d like to be

tamsky

Thanks Erik. That sounds close. @catdevman when you have a moment, I’m interested to hear your strategy for using packer to create both docker images and AMIs

Erik Osterman

fwiw, when we’ve needed init in containers we’ve used s6 or dumb-init

Erik Osterman
s6 - skarnet's small supervision suite

s6 - skarnet’s small supervision suite

joshmyers

The folks at Yelp are

2018-11-25

Nikola Velkovski
wagoodman/dive

A tool for exploring each layer in a docker image. Contribute to wagoodman/dive development by creating an account on GitHub.

2018-11-14

Nikola Velkovski

Hi People, does anyone know a good and proper way to remove layers with secrets from a docker image ? e.g. the only way that this can be done currently is to use multi stage builds or to unset the ENV/ARGS and there’s this blog post here that mentions experimental secrets but not for strings https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066 TL;DR docker history <docker_images> exposes layers with secrets from build time.

Build secrets and SSH forwarding in Docker 18.09 – Tõnis Tiigi – Medium

One of the complexities when using Dockerfiles has always been accessing private resources. If you need to access some private repository…

aknysh

@Nikola Velkovski did you look at https://github.com/goldmann/docker-squash

goldmann/docker-squash

Docker image squashing tool. Contribute to goldmann/docker-squash development by creating an account on GitHub.

aknysh
Handling secrets when building docker images is easy · Tomáš Tomeček

Technical blog. Read about Linux, Python, Containers, Red Hat, Fedora, tools.

Nikola Velkovski

Nope, thanks will have a look.

Nikola Velkovski
Build Enhancements for Docker

Docker Build is one of the most used features of the Docker Engine - users ranging from developers, build teams, and release teams all use Docker Build. Docker Build enhancements…

1
Erik Osterman

I like this new output

Build Enhancements for Docker

Docker Build is one of the most used features of the Docker Engine - users ranging from developers, build teams, and release teams all use Docker Build. Docker Build enhancements…

1
Erik Osterman

i believe it’s also possible to squash an image without any thirdparty tools

Erik Osterman

docker build --squash

Nikola Velkovski

Ah yes we discuissed it with @ but when using --squash there will be no caching.

Erik Osterman

Ah, that’s true… let’s step back: why do you need the secrets in the image to begin with?

Nikola Velkovski

in this particular case it’s for ruby gems, i need to bundle them from a private repo so I need the api key from it

Nikola Velkovski

there are cases for npm etc…

Erik Osterman

Can you use build args? Those are not persisted in the layers unless the a build step writes it to disk.

Erik Osterman

Something like

FROM node:X
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > ~/.npmrc && \
  npm install && \
  rm ~/.npmrc
Nikola Velkovski

@Erik Osterman I am using args but it turns out those layers also persist

Erik Osterman

yes, the layers persist, but that’s why you need to rm in the same layer

Erik Osterman

see the example above

Erik Osterman

need to chain with &&

Nikola Velkovski

Argh that’s why… ok great good to know!

Erik Osterman
docker image build

Description Build an image from a Dockerfile Usage docker image build [OPTIONS] PATH | URL | - Options Name, shorthand Default Description –add-host Add a custom host-to-IP mapping (host:ip) –build-arg…

Erik Osterman

You can also have an internal proxy that you use in your build process

Erik Osterman

that proxy then has the authentication token for github

Erik Osterman

we wrote one for the docker registry to do this

Erik Osterman

so we could pull images without authentication in a trusted environment

Erik Osterman

same concept would apply to github

Erik Osterman

here’s the simple service

Erik Osterman

(you’d adapt something like this to proxy github requests instead)

Erik Osterman

~Posted just a few days ago… highly relevant~

Nikola Velkovski

Ah yeah, I already read it and posted it yesterday

Erik Osterman

Erik Osterman

hahah

Erik Osterman

sorry, missed that! you did post it

Nikola Velkovski

Anyway thanks for the help, I was also under the impression that passing secrets as args is ok

Nikola Velkovski

luckily I haven’t pushed any public images

Erik Osterman

Erik Osterman

so regarding that experimental feature, that would be the cleanest, no?

Nikola Velkovski

yes

2018-10-03

Erik Osterman

an excellent exemplar of mutli-stage docker builds by @alebabai

Erik Osterman
cloudposse/bastion

Secure Bastion implemented as Docker Container running Alpine Linux with Google Authenticator & DUO MFA support - cloudposse/bastion

alebabai
09:11:44 PM

@alebabai has joined the channel

2018-09-18

08:01:11 PM

@ has joined the channel

Gabe
11:48:11 PM

@Gabe has joined the channel

2018-09-07

endofcake
09:29:16 PM

@endofcake has joined the channel

2018-09-05

loren
11:32:32 AM

@loren has joined the channel

loren

docker newb question… is there a way to see the prior digests of a given image/tag… centos:7 in particular is what i’m using

Erik Osterman

it’s a good question. Off the top of my head, not sure how. I think when they push a release for centos:7 it clobers the tag so theres no prior history

Erik Osterman

you would need to find another service that keep track of images/layers

loren

ok, thanks. was updating how we specify the image in our Dockerfiles to include the sha, now that dependabot supports that notation. was hoping to grab a prior sha to test immediately, but yeah, doesn’t seem possible to lookup prior digests. guess i’ll just have to wait until they publish a new image…

1

2018-09-04

siert
03:48:32 PM

@siert has joined the channel

2018-09-02

07:31:29 AM

@ has joined the channel

2018-08-29

Raghu
03:04:39 PM

@Raghu has joined the channel

2018-08-28

05:17:10 PM

@ has joined the channel

Arkadiy
06:30:25 PM

@Arkadiy has joined the channel

2018-08-27

11:14:36 AM

@ has joined the channel

loweryr
03:11:39 PM

@loweryr has joined the channel

2018-08-24

ruan.arcega
11:24:54 AM

@ruan.arcega has joined the channel

2018-08-22

04:17:15 AM

@ has joined the channel

2018-08-17

Erik Osterman

@Igor Rodionov would this be nice for our local dev harness stuff? https://github.com/bcicen/ctop

bcicen/ctop

ctop - Top-like interface for container metrics

Igor Rodionov
02:50:42 AM

@Igor Rodionov has joined the channel

2018-08-05

jylee
04:29:35 PM

@jylee has joined the channel

2018-08-02

Erik Osterman
09:37:46 PM

@Erik Osterman has joined the channel

Erik Osterman
09:37:46 PM

@Erik Osterman set the channel purpose: All things docker

aknysh
09:37:59 PM

@aknysh has joined the channel

09:38:06 PM

@ has joined the channel

krogebry
09:47:47 PM

@krogebry has joined the channel

So for everyone who is dealing with docker signalling issues, I had some pain with it today with sidekiq not stopping gracefully inside docker. Docker stop sends a SIGTERM to docker pid. In most cases, with /bin/sh -c being the entrypoint, the SIGTERM will only reach the pid of /bin/sh and not the child processes of it. Inside my bash script I was already using exec, but I had to use exec in the CMD line of the Dockerfile to make it to work.

Note the exec.

CMD exec /$APP_DIR/bin/dispatch.sh

And note the exec Inside dispatch.sh

SIDEKIQ_COUNT=3 SIDEKIQ_MAXMEM_MB=2000 SIDEKIQ_PRELOAD=sidekiq_swarm exec sidekiqswarm -t 25 -C config/sidekiq.yml

https://hochzehn.com/insights/docker-stopping-containers-softly.html

Erik Osterman

Aha… so basically, it’s running /bin/sh -c 'exec /$APP_DIR/bin/dispatch.sh', which then replaces PID1 with dispatch.sh I’ve also run into problems with signal handling and shell scripts with docker.

tamsky
02:55:12 AM

@tamsky has joined the channel

tamsky

it feels like the format of shell commands can convey a lot of meaning:

SIDEKIQ_COUNT=3 SIDEKIQ_MAXMEM_MB=2000 SIDEKIQ_PRELOAD=sidekiq_swarm && 
    exec sidekiqswarm \
        -t 25 \
        -C config/sidekiq.yml
tamsky


it’s running /bin/sh ...

does CMD (and hence RUN) always pick /bin/sh, or does it use the named shebang #! shell at the top of:/$APP_DIR/bin/dispatch.sh ?

    keyboard_arrow_up