Docker, Mac/OSX, xhyve: Modify Container’s HostConfig

Hello Friends – Today I wanted to do some reorganization to my docker containers on my development box where I use Docker, Mac/OSX, xhyve for development. It was really very simple – I wanted to move my containers under a local directory (same as the namespace I like to use: sab).

A Simple Folder Rename…

Here’s the old structure:

CloudraticSolutionsLLCs-MacBook-Pro:docker l.abruce$ pwd
/Users/l.abruce/proj/docker

CloudraticSolutionsLLCs-MacBook-Pro:docker l.abruce$ ls -la
total 16
drwxr-xr-x  6 l.abruce  staff   204 Oct 15 15:33 .
drwxr-xr-x  9 l.abruce  staff   306 Jun  3 16:33 ..
-rw-r--r--@ 1 l.abruce  staff  6148 Jun  3 17:07 .DS_Store
drwxr-xr-x  3 l.abruce  staff   102 Jul 24 19:31 yum-repo

So a couple of commands later I have things rearranged:

CloudraticSolutionsLLCs-MacBook-Pro:sab l.abruce$ pwd
/Users/l.abruce/proj/docker/sab

CloudraticSolutionsLLCs-MacBook-Pro:sab l.abruce$ ls -la
total 0
drwxr-xr-x  6 l.abruce  staff  204 Oct 15 12:02 .
drwxr-xr-x  6 l.abruce  staff  204 Oct 15 15:33 ..
drwxr-xr-x  8 l.abruce  staff  272 Oct 15 12:00 yum-repo

No big change…but the Docker container now has a problem. I have moved a mounted folder around on it! Let’s first see the container (and it’s GUID):

CloudraticSolutionsLLCs-MacBook-Pro:~ l.abruce$ docker ps -a
CONTAINER ID        IMAGE                         COMMAND             CREATED             STATUS                            PORTS               NAMES
e56f5e722826        sab/yum-repo:1                "/bin/bash"         5 months ago        Exited (137) About a minute ago                       sab-yum-repo

Now let’s look at the mounted folder that will give me fits because I have moved it:

CloudraticSolutionsLLCs-MacBook-Pro:sab l.abruce$ docker inspect sab-yum-repo
[
    {
        [...output cut...]
        "Mounts": [
            {
                "Source": "/Users/l.abruce/proj/docker/yum-repo/yumrepo",
                "Destination": "/yumrepo",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        [...output cut...]
    }
]

…Leads to Problems!

You see the problem? It’s the reference to the old mount location. Now I could simply recreate the container (I have a Dockerfile and an automated init script that gets loaded from my local Git repo), but I realized that with the Integrated Docker for Mac/OSX which I’m using – I couldn’t easily get to the containers location where the JSON descriptor files are stored.

Docker, Container Metadata, and xhyve

Docker stores container metadata (depending on your backing filesystem) in a well-known location: /var/lib/docker/containers/[CONTAINER_ID]/ under the files config.v2.json and / or hostconfig.json. Since I moved my folders around on the Docker host, I need to update these files. This is simple on Linux – see this reference on modifying container options. And on “classic” Docker integrated with VirtualBox it’s still easy; you simply connect to the VirtualBox docker-machine VM (normally just called “default“) and we can get to the filesystem as we would on a native Linux installation of Docker.

However – there is one problem. I’m using the New Docker / xhyve Integration which eliminates the need for VirtualBox, docker-machine, or any of that cruft. But the question I had – how do I get to the lightweight Alpine Linux which Docker runs as an xhyve process?

ps and screen to the Rescue

The first thing I did was to learn how this xhyve integration works. I was half-expecting to find some CLI tools similar to virsh or esxcli but xhyve seems to work differently. Take a look at ps output:

CloudraticSolutionsLLCs-MacBook-Pro:~ l.abruce$ ps -efa | grep -i -e docker | grep -v grep
    0    99     1   0 Sun07AM ??         0:00.02 /Library/PrivilegedHelperTools/com.docker.vmnetd
  502 26682     1   0  4:08PM ??         0:00.26 /Applications/Docker.app/Contents/MacOS/Docker
  502 26694 26682   0  4:08PM ??         0:00.03 /Applications/Docker.app/Contents/MacOS/com.docker.osx.hyperkit.linux -watchdog fd:0
  502 26696 26694   0  4:08PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.osx.hyperkit.linux -watchdog fd:0
  502 26697 26694   0  4:08PM ??         0:00.25 com.docker.db --url fd:3 --git /Users/l.abruce/Library/Containers/com.docker.docker/Data/database
  502 26698 26694   0  4:08PM ??         0:00.11 com.docker.osxfs --address fd:3 --connect /Users/l.abruce/Library/Containers/com.docker.docker/Data/@connect --control fd:4 --volume-control fd:5 --database /Users/l.abruce/Library/Containers/com.docker.docker/Data/s40
  502 26699 26694   0  4:08PM ??         0:00.03 com.docker.slirp --db /Users/l.abruce/Library/Containers/com.docker.docker/Data/s40 --ethernet fd:3 --port fd:4 --vsock-path /Users/l.abruce/Library/Containers/com.docker.docker/Data/@connect --max-connections 900
  502 26700 26694   0  4:08PM ??         0:00.02 com.docker.osx.hyperkit.linux
  502 26701 26694   0  4:08PM ??         0:00.07 com.docker.driver.amd64-linux -db /Users/l.abruce/Library/Containers/com.docker.docker/Data/s40 -osxfs-volume /Users/l.abruce/Library/Containers/com.docker.docker/Data/s30 -slirp /Users/l.abruce/Library/Containers/com.docker.docker/Data/s50 -vmnet /var/tmp/com.docker.vmnetd.socket -port /Users/l.abruce/Library/Containers/com.docker.docker/Data/s51 -vsock /Users/l.abruce/Library/Containers/com.docker.docker/Data -docker /Users/l.abruce/Library/Containers/com.docker.docker/Data/s60 -addr fd:3 -debug
  502 26702 26700   0  4:08PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.osx.hyperkit.linux
  502 26703 26701   0  4:08PM ??         0:00.02 /Applications/Docker.app/Contents/MacOS/com.docker.driver.amd64-linux -db /Users/l.abruce/Library/Containers/com.docker.docker/Data/s40 -osxfs-volume /Users/l.abruce/Library/Containers/com.docker.docker/Data/s30 -slirp /Users/l.abruce/Library/Containers/com.docker.docker/Data/s50 -vmnet /var/tmp/com.docker.vmnetd.socket -port /Users/l.abruce/Library/Containers/com.docker.docker/Data/s51 -vsock /Users/l.abruce/Library/Containers/com.docker.docker/Data -docker /Users/l.abruce/Library/Containers/com.docker.docker/Data/s60 -addr fd:3 -debug
  502 26705 26701   0  4:08PM ??         0:05.47 /Applications/Docker.app/Contents/MacOS/com.docker.hyperkit -A -m 2G -c 4 -u -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-vpnkit,uuid=525496a2-c58d-4df2-93e5-2584885d63f5,path=/Users/l.abruce/Library/Containers/com.docker.docker/Data/s50,macfile=/Users/l.abruce/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/mac.0 -s 3,virtio-blk,file:///Users/l.abruce/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2,format=qcow -s 4,virtio-9p,path=/Users/l.abruce/Library/Containers/com.docker.docker/Data/s40,tag=db -s 5,virtio-rnd -s 6,virtio-9p,path=/Users/l.abruce/Library/Containers/com.docker.docker/Data/s51,tag=port -s 7,virtio-sock,guest_cid=3,path=/Users/l.abruce/Library/Containers/com.docker.docker/Data,guest_forwards=2376;1525 -l com1,autopty=/Users/l.abruce/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty,log=/Users/l.abruce/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/console-ring -f kexec,/Applications/Docker.app/Contents/Resources/moby/vmlinuz64,/Applications/Docker.app/Contents/Resources/moby/initrd.img,earlyprintk=serial console=ttyS0 com.docker.driver="com.docker.driver.amd64-linux", com.docker.database="com.docker.driver.amd64-linux" ntp=gateway mobyplatform=mac -F /Users/l.abruce/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/hypervisor.pid
  502 26711 26682   0  4:08PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.frontend {"action":"vmstateevent","args":{"vmstate":"running"}}
  502 26712 26711   0  4:08PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.frontend {"action":"vmstateevent","args":{"vmstate":"running"}}

There’s a great resource on Inside Docker / xhyve Integration which you can read up on, but to me the key points are the references to hyperkit (promising!) and the references to the folder /Users/l.abruce/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux – there are five of those references.

So now we take a look at that folder:

CloudraticSolutionsLLCs-MacBook-Pro:sab l.abruce$ ls -la /Users/l.abruce/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux
total 37451664
drwxr-xr-x@ 12 l.abruce  staff          408 Oct 15 16:08 .
drwxr-xr-x  20 l.abruce  staff          680 Oct 15 16:08 ..
-rw-r--r--   1 l.abruce  staff  19174981632 Oct 15 16:15 Docker.qcow2
-rw-r--r--   1 l.abruce  staff        65536 Oct 15 16:08 console-ring
-rw-r--r--   1 l.abruce  staff            5 Oct 15 16:08 hypervisor.pid
-rw-r--r--   1 l.abruce  staff            0 Jul 24 19:09 lock
drwxr-xr-x   5 l.abruce  staff          170 Aug 11 01:51 log
-rw-r--r--   1 l.abruce  staff           17 Oct 15 16:08 mac.0
-rw-r--r--   1 l.abruce  staff           36 Jul 24 19:09 nic1.uuid
-rw-r--r--   1 l.abruce  staff            5 Oct 15 16:08 pid
-rw-r--r--   1 l.abruce  staff       182996 Oct 15 16:08 syslog
lrwxr-xr-x   1 l.abruce  staff           12 Oct 15 16:08 tty -> /dev/ttys000

Folks, look at that reference to tty. Looks to me like it’s a console interface to *something* – and that something looks awfully close to being a VM managed by a hypervisor. So let’s go into screen:

CloudraticSolutionsLLCs-MacBook-Pro:sab l.abruce$ screen /Users/l.abruce/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

Welcome to Moby
Kernel 4.4.20-moby on an x86_64 (/dev/ttyS0)

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/

moby login: root
Welcome to Moby, based on Alpine Linux.
moby:~# 

Full Disclosure: To get the fact that there is no root password I had to search around a bit and found the answer from our StackOverflow friends at https://stackoverflow.com/questions/39455764/change-storage-driver-for-docker-on-os-x.

But now I am actually on the Alpine Linux VM that Docker uses to manage containers, and presumably I can find my container’s metadata in the usual location. Let’s go look!

moby:~# cd /var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161

moby:/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161# ls -la
total 44
drwx------    3 root     root          4096 Oct 15 20:08 .
drwx------    7 root     root          4096 Oct 15 16:13 ..
-rw-------    1 root     root          2674 Oct  4 08:34 .tmp-config.v2.json523583531
-rw-rw-rw-    1 root     root          2677 Oct 15 20:07 config.v2.json
-rw-r-----    1 root     root          3011 Oct 15 20:07 e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161-json.log
-rw-rw-rw-    1 root     root          1159 Oct 15 20:08 hostconfig.json
-rw-r--r--    1 root     root            13 Oct 15 19:38 hostname
-rw-r--r--    1 root     root           174 Oct 15 19:38 hosts
-rw-r--r--    1 root     root           230 Oct 15 19:38 resolv.conf
-rw-r--r--    1 root     root            71 Oct 15 19:38 resolv.conf.hash
drwx------    2 root     root          4096 Apr 25 02:10 shm

Profit! We have located the two config files. In my case, I know I have mounted the yumrepo folder to my original user directory, and the reference is indeed in the files:

moby:/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161# grep -l -e yumrepo *.json
config.v2.json
hostconfig.json

Let’s now stop and restart the Docker service:

moby:/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161# rc-status
Runlevel: default
 [...output cut...]
 docker
 [  started  ]

moby:/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161# rc-service docker stop
 * Stopping docker

Now we simply edit the two JSON files using good ol’ vi and update the directory mount point:

moby:/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161# grep -e yumrepo *.json
config.v2.json:{"State":{"Running":false,"Paused":false,"Restarting":false,"OOMKilled":false,"RemovalInProgress":false,"Dead":false,"Pid":0,"StartedAt":"2016-10-15T19:38:16.482788677Z","FinishedAt":"2016-10-15T20:07:04.961379487Z","Health":null},"ID":"e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161","Created":"2016-04-25T02:10:23.698922171Z","Managed":false,"Path":"/bin/bash","Args":[],"Config":{"Hostname":"e56f5e722826","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"ExposedPorts":{"80/tcp":{}},"Tty":true,"OpenStdin":true,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/bash"],"Image":"sab/yum-repo:1","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":{"build-date":"2016-03-31","license":"GPLv2","name":"CentOS Base Image","vendor":"CentOS"},"StopSignal":"SIGTERM"},"Image":"sha256:7ca8dd769605d08a61d88b11ab2e0f1b9355596b7519aaaf2ac8a97915dec6d4","NetworkSettings":{"Bridge":"","SandboxID":"06f060a6be3a37c5900b82570eeb3f0fd3db4235dbf91a0cb7dfa0dd0ecb3238","HairpinMode":false,"LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"NetworkID":"6c607797fe25f490c70b6f54bd2a0a2e306ad7340d40e4e45c226017409c6bff","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":""}},"Service":null,"Ports":null,"SandboxKey":"/var/run/docker/netns/06f060a6be3a","SecondaryIPAddresses":null,"SecondaryIPv6Addresses":null,"IsAnonymousEndpoint":false},"LogPath":"/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161-json.log","Name":"/sab-yum-repo","Driver":"aufs","MountLabel":"","ProcessLabel":"","RestartCount":0,"HasBeenStartedBefore":true,"HasBeenManuallyStopped":true,"MountPoints":{"/yumrepo":{"Source":"/Users/l.abruce/proj/docker/sab/yum-repo/yumrepo","Destination":"/yumrepo","RW":true,"Name":"","Driver":"","Relabel":"","Propagation":"rprivate","Named":false,"ID":""}},"AppArmorProfile":"","HostnamePath":"/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161/hostname","HostsPath":"/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161/hosts","ShmPath":"/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161/shm","ResolvConfPath":"/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161/resolv.conf","SeccompProfile":"","NoNewPrivileges":false}

hostconfig.json:{"Binds":["/Users/l.abruce/proj/docker/sab/yum-repo/yumrepo:/yumrepo"],"ContainerIDFile":"","LogConfig":{"Type":"json-file","Config":{}},"NetworkMode":"default","PortBindings":{"80/tcp":[{"HostIp":"","HostPort":"15080"}]},"RestartPolicy":{"Name":"no","MaximumRetryCount":0},"AutoRemove":false,"VolumeDriver":"","VolumesFrom":null,"CapAdd":null,"CapDrop":null,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IpcMode":"","Cgroup":"","Links":[],"OomScoreAdj":0,"PidMode":"","Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"SecurityOpt":null,"UTSMode":"","UsernsMode":"","ShmSize":67108864,"Runtime":"runc","ConsoleSize":[0,0],"Isolation":"","CpuShares":0,"Memory":0,"CgroupParent":"","BlkioWeight":0,"BlkioWeightDevice":null,"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"CpuPeriod":0,"CpuQuota":0,"CpusetCpus":"","CpusetMems":"","Devices":[],"DiskQuota":0,"KernelMemory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"OomKillDisable":false,"PidsLimit":0,"Ulimits":null,"CpuCount":0,"CpuPercent":0,"IOMaximumIOps":0,"IOMaximumBandwidth":0}

Maybe hard to read, but I hope you can see where I updated the mount / bind points.

Finally, restart the service:

moby:/var/lib/docker/containers/e56f5e7228263150ecd5e698d73a68481272b4551e6ea707b9ce30676e31c161# rc-service docker start
 * Starting Docker ...
 [ ok ]

The Final Frontier: Restart the Container

After all this work, we restart our container and check the binding:

docker start sab-yum-repo

CloudraticSolutionsLLCs-MacBook-Pro:~ l.abruce$ docker inspect sab-yum-repo
[
    {
        [...output cut...]
        "Mounts": [
            {
                "Source": "/Users/l.abruce/proj/docker/sab/yum-repo/yumrepo",
                "Destination": "/yumrepo",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        [...output cut...]
    }
]

Success. And we learned a lot about how Docker, xhyve, and Alpine Linux are playing together.

Happy Computing!