Docker Issues and Tips (aufs/overlay/btrfs..)
Picked up and categorized subjectively from https://github.com/docker/docker/issues. Comments and pull requests are welcome.
Storage Drivers
AUFS
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| Cannot access to a directory due to a permission error | Expected AUFS behavior. dirperm1 mount option fixes this issue. |
Update the kernel (AUFS >= 2008xxxx?) and Docker daemon (>= 1.7) | Confirm: `docker info | |||
| A process becomes a zombie and hangs up | Compatibility between the kernel and AUFS | Update the kernel (AUFS >= 20160111) | Java apps and MongoDB are known to be affected | |||
fcntl(F_SETFL, O_APPEND) is ignored and hence data can be corrupted |
AUFS bug | Update the kernel (AUFS >= 20160301) | Dovecot is known to be affected | |||
Weird permission even though dirperm1 is enabled |
AUFS bug | Update the kernel (AUFS >= 20160905) | ||||
Hang up related to O_DIRECT |
Unanalyzed | None | Percona is known to be affected | |||
| Unable to remove files previously committed | Unanalyzed | This article seems related, but perhaps slightly different(Japanese) | ||||
| AUFS + XFS hangs up | AUFS bug | Update AUFS |
Non-bug issues:
- AUFS is not available in the mainline kernelοΌOnly a few distros (Ubuntu, Boot2Docker, ..) support AUFS, but even for Ubuntu, Canonical says "AUFS will disappear".
- No support for extended attributes ("xattrs"), and might not ever get support (#1070, #8460).
rename(2)is not fully supported ( see also #aufs--overlay-common )
Overlay
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| RPMDB corruption | Expected overlay behavior | Use yum-{utils,plugins-ovl}-1.1.31-33.el7 (included in RHEL 7.2) or later. Kernel patch is also available. | Linux 4.6 or later prints human-friendly dmesg | |||
| Cannot use UNIX domain sockets | Overlay Bug | Use Linux 4.7-rc4 or later | ||||
| pip fails | Overlay Bug | Use Linux 4.5 or later | ||||
| Weird behavior after removing the current directory | Overlay Bug | Use Linux 4.5 or later | ||||
| Untar fails intermittently | Overlay Bug | Use Linux 4.13 with OVERLAY_FS_INDEX=y | Analysis is in progress in coreos/bugs#1095 | |||
| Container cannot be started | Unanalyzed | None | Possibly identical to #16902 | |||
| /dev/console: operation not permitted | Kernel Bug | Use recent Linux kernels | ||||
docker build fails intermittently (overlay1) |
DiffDriver bug | Use Docker 1.13 or later | Overlay2 doesn't have this issue by design | |||
| permissions broken after chown | Overlay Bug | Use Linux 4.6 or later | The overlay2 issue #28391 is due to the identical bug | |||
| opaque flag not reset after directory copy up | Overlay Bug | Resolved in Linux 4.8 and backported to 4.4.21 and 4.7.4 | npm is known to be affected | |||
| chmod fails with EPERM | Overlay Bug | Use Linux 4.5 or later | ||||
| file removal weird on overlay + XFS (ftype=0) | Expected behavior | Format xfs with ftype=1 | ||||
docker build produces weird images with CONFIG_OVERLAY_FS_REDIRECT_DIR=y |
DiffDriver issue | Apply #34342 (Docker 17.08?) |
Non-bug issues:
π± High inode usage (resolved in overlay2, which will be available in Docker 1.12)- Red Hat says "OverlayFS remains a Technology Preview in Red Hat Enterprise Linux 7.3 under most circumstances"
rename(2)is not fully supported ( see also #aufs--overlay-common )- MySQL doesn't work without
touch-ing files under/var/lib/mysql: docker/for-linux#72 (comment)
AUFS / Overlay common
Non-bug issue: rename(2) is not fully supported #25409
reports about the incompatible behavior of rename(2) from the real world
| Software | Report |
|---|---|
| Apache Kudu | https://issues.apache.org/jira/browse/KUDU-1419 |
| CernVM-FS | https://sft.its.cern.ch/jira/browse/CVM-651 |
| GPG | moby/moby#26317 |
| NPM | npm/npm#9863 |
| Samba | https://bugzilla.samba.org/show_bug.cgi?id=9966 |
BtrFS
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
sendfile(2) can be unkillable |
BtrFS bug | None | Not likely to happen in production, but needs consideration for public PaaS | |||
| cgroups kmem limit leads crash and data corruption | Btrfs bug | Avoid kmem limit configuration? |
Non-bug issues:
- Slow #10161
- No page sharing (e.g. same DLLs are loaded redundantly) http://comments.gmane.org/gmane.comp.sysutils.docker.devel/1384
- Docker says BtrFS is Experimental. Red Hat says BtrFS is Tech Preview.
ZFS
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
Some operations fail due to EBUSY |
Daemon bug | Update Docker daemon |
Non-bug issues:
- Docker says ZFS is not recommended for production.
DeviceMapper
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| Mount fails | udev sync disabled | Use a Docker daemon binary which supports udev sync | Confirm: `docker info | |||
| Infinite βmount/remountβ loop, which makes the system unresponsive | Unanalyzed (perhaps related to XFS) | None |
Non-bug issues:
- Slow #10161
- No page sharing (e.g. same DLLs are loaded redundantly) http://comments.gmane.org/gmane.comp.sysutils.docker.devel/1384
Storage driver test tool
- dmcgowan/dsdbench: Docker Storage Driver Benchmarks and Tests
So which storage driver should I use?
It totally depends on your workload, but Docker, Inc. says AUFS and Devicemapper (direct-lvm) are "production-ready".
Although not listed in the above table, VFS driver is also attractive for its robustness.
Links:
- https://jpetazzo.github.io/assets/2015-03-03-not-so-deep-dive-into-docker-storage-drivers.html#1
- http://www.projectatomic.io/docs/filesystems/
- https://blog.jessfraz.com/post/the-brutally-honest-guide-to-docker-graphdrivers/
Anyway...
You know, containers should be "immutable" and "disposable".
For persistent data and some special temporary data, you should better consider using an external volume (docker run -v).
Links:
Network
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
hang up with unregister_netdevice: waiting for lo to become free |
Kernel bug | Use Linux 4.8 or later | The patch will be backported to old kernels in major distros | |||
| TCP checksums are ignored | Kernel bug | Use Linux 4.4 or later | blog |
Logging
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| GELF driver saturates CPU | Compression | Disable compression | ||||
cat /dev/zero leads to out of memory |
logger's stdio handling issue | Use Docker 1.13 or later (or just disable the logging) | Related: #21181 | |||
| container cannot be stopped if many logs are being printed | logger's stdio handling issue | |||||
| logging blocks the container | logger's stdio handling issue | Use Docker 1.11 or later | affected versions: 1.10.0 |
Others
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| Docker daemon 1.9 serious performance issue | ? | Use Docker 1.10 | ||||
soft lockup related to show_mountinfo(), after frequent docker run |
Unanalyzed (Kernel bug related to the number of processors?) | None | ||||
/dev/pts unmounted on the HOST when you are using -v /dev:/dev (After that you can no longer open SSH nor xterm) |
daemon bug related to mount namespace | Use Docker 1.11.1. (Or Spawn the docker daemon from systemd. Or do not use -v /dev:/dev) |
||||
Daemon hangs up after frequent docker run |
Daemon bug | Use Docker 1.11.1 | ||||
| Strange permission issues with named containers on 1.12.3 | Daemon bug related to SELinux) | Use Docker 1.12.4 | ||||
fatal error: unaligned sysUnused on Azure |
? | Ubuntu linux-azure kernel bug | Use linux-azure 4.11.0-1013.13 or later |
Non-bug issues: