Managing Persistence for Docker Containers
A few years ago, when virtualization was introduced to IT administrators, there was an attempt to standardize the virtual machine (VM) as the unit of deployment. Each new build and version resulted in a new VM template. Eventually, developers and administrators started creating new images and provisioning VMs, which resulted in a phenomenon called “VM sprawl.” Each template and the provisioned VM occupied a few gigabytes of storage, which led to inefficient storage management. Given the large size of VM images, organizations realized that it wasn’t practical to create a new VM template for each version.
One of the goals for Docker was to avoid the pitfall illustrated by “VM sprawl.” The only way Docker could avoid the trap of fragmented images was to adopt a different storage mechanism for its images and containers.
Another goal that was critical to Docker was the separation of filesystems to create isolation between the host system and containers. This isolation was core to the security of containerized applications. To meet these requirements, Docker adopted a union filesystem architecture for the images and containers.
Union filesystems represent a logical filesystem by grouping different directories and filesystems together. Each filesystem is made available as a branch, which becomes a separate layer. Docker images are based on a union filesystem, where each branch represents a new layer. It allows images to be constructed and deconstructed as needed instead of creating a large, monolithic image.
Docker’s use of existing Linux union filesystems is ideal for running applications that can rapidly scale. Since they are self-contained, Docker containers can be launched on any host with no dependencies or affinity to a specific host. While this is certainly an advantage for web-scale workloads, it becomes a challenge to run stateful applications that deal with persistent data. Workloads, such as relational databases, NoSQL databases, content management systems, and big data stacks, demand persistence and durability of data. Some workloads, like content management systems, also require shared data access across multiple application instances.
When a Docker image is pulled from the registry, the engine downloads all the dependent layers to the host. When a container is launched from a downloaded image comprised of many layers, Docker uses the copy-on-write capabilities of the available union filesystem to add a writeable “working directory” — or temporary filesystem — on top of the existing read-only layers. When Docker first starts a container, this initial read-write layer is empty until changes are made to the file system by the running container process. When a Docker image is created from an existing container, only the changes made — which have all been “copied up” to this writeable working directory — are added into the new layer. This approach enables reuse of images without duplication or fragmentation.
When a process attempts to write to an existing file, the filesystem implementing the copy-on-write feature creates a copy of the file in the topmost working layer. All other processes using the original image’s layers will continue to access the read-only, original version of the layer. This technique optimizes both image disk space usage and the performance of container start times.
Read the entire article at The New Stack.
Janakiram MSV is an analyst, advisor, and architect. Follow him on Twitter, Facebook and LinkedIn.