Docker for Mac, Alpine & Glibc issues

Many Docker users are moving to use Alpine as the base for their images. And why not, it is a really slim base image but still provides decent package management and other needed features. Also when using Docker for Mac, the containers actually run on a very streamlined VM (called Moby) running on top of HyperKit. And what do you know, Moby is actually also Alpine based.

As most Alpine users know, it doesn't ship with the "standard" libc implementation. Instead it uses a library called musl.

A while ago, I wanted to run something on the host side while using Docker for Mac. While working, I started getting some weird errors which turned out to be quite difficult to debug and fix. These might be common issues for people migrating to Alpine based images. I hope this article is helpful for those of you having similar kind of issues.

The issue

When running in Docker for Mac host side I encountered following issue:

/ # ls -lah /opt/bin/cadvisor 
-rwsrwsrwx    1 root     root       22.0M Aug 19 10:10 /opt/bin/cadvisor
/ # whoami
root  
/ # /opt/bin/cadvisor 
sh: /opt/bin/cadvisor: not found  

What, the file is executable and all and clearly it exists. So why the not found error?

Maybe some of the libs are missing as the Moby linux is pretty streamlined setup.

# ldd /opt/bin/cadvisor 
    /lib64/ld-linux-x86-64.so.2 (0x561ba6da2000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x561ba6da2000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x561ba6da2000)

Everything seems to be in place. Next option, is the binary really compatible with the Moby linux?

/ # uname -a
Linux moby 4.4.15-moby #1 SMP Thu Jul 28 22:03:07 UTC 2016 x86_64 Linux  
/ # file /opt/bin/cadvisor 
/opt/bin/cadvisor: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=a3a50f1c2945950b396f4964ba1f14064b80a2a0, not stripped

Seems so. What on earth is going on, why can't we run an executable that seems to be valid in every sense.

When looking a bit closer on the file format output there's a clear hint:
interpreter /lib64/ld-linux-x86-64.so.2

# ls -lah /lib64/ld-linux-x86-64.so.2
ls: /lib64/ld-linux-x86-64.so.2: No such file or directory  

What? Wait a minute. Oh yes, the Moby is Alpine and musl based, so normal glibc stuff is not available thus the file execution is a no-go. Something more obvious as error message would have been nice, but what can you do.

Solution

Naturally, the solution is to get the glibc stuff installed into the Moby linux setup.

Step 1, get a terminal into the machine

There's no ssh available in the Moby linux so we need to do a little "trick" to get a shell into the VM. Luckily we have containers and Linux namespaces to do the trick:

docker run --rm -it --privileged --pid=host walkerlee/nsenter -t 1 -m -u -i -n sh  

The above command will launch a new container which only attaches the terminal into host side. Some explanation might be necessary for the flags.

 -t, --target <pid>     target process to get namespaces from
 -m, --mount[=<file>]   enter mount namespace
 -u, --uts[=<file>]     enter UTS namespace (hostname etc)
 -i, --ipc[=<file>]     enter System V IPC namespace
 -n, --net[=<file>]     enter network namespace

So basically we launch sh terminal process within the host pid 1 and attach it also to the other namespaces from that pid.

Now as we are on the Moby "console", we can put the glibc packages into place:

# wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk
# apk --allow-untrusted --force add glibc-2.23-r3.apk

Test if we can now execute the binary:

# /opt/bin/cadvisor -version
cAdvisor version 0.23.2 (7ddf6eb)  

Remarks

Not all errors seem to be what they tell they are. In this case we needed to go a bit deeper than expected to figure out the real error. :)

About Kontena

Kontena, Inc. is the creator of Kontena, an open source, developer-friendly container and microservices platform. Kontena is built to maximize developer happiness by simplifying running containerized applications on any infrastructure: on-premises, cloud or hybrid. It provides a complete solution for organizations of any size. Founded in March 2015, Kontena was recognized as one of the best new open source projects in the 8th annual Black Duck Open Source Rookies of the Year Awards. For more information, visit: www.kontena.io

Image credits: Inject by jaroh

Jussi Nummelin

Read more posts by this author.

Tampere, Finland
comments powered by Disqus