Drooling Over Docker #1 - The Genesis of Containers

“I once heard that hypervisors are the living proof of operating system’s incompetence.” — Glauber Costa, LinuxCon Europe 2012

Effectively handling resource contention has been a primary challenge for Operating Systems.

Consider the following resource-allocation challenges:

• One process demands more memory from the Operating System which in turn pages out (read punishes) the memory used by some other process.
• It is possible that one application, running a group of related processes, gets better share of CPU cycles than an application running with lesser number of processes.
• A process with bugs (or fork bombs) causing a denial-of-service attackthat leaves the kernel resources exhausted and thus impacting other unrelated processes.

Hypervisors, and Containers attempt to overcome these (and even some more) challenges by isolating process execution space & controlling the system resources in their own ways that we’ll explore in this Docker Seriesfurther; however, our focus here will remain on Containers.

Genesis of Containers

A Container, as we know it today (Docker, Rkt etc), is a group of processes running on the underlying Operating System in its own isolated process space. It can be configured to have its own share of system resources; has its own file-system that, though mounted on some directory of the file-system of the host Operating System, is seen as a separate file-system tree with its own root directory, by the processes running inside it. This lightweight execution unit has its own copy of software, shared libraries, and any other necessary files required to run the software it is prepared to run.

Running Containerized software could only become possible because of the development of some underlying technologies in recent years.

Three such major technologies are -

#1 — CGroups — Controlling What You Can USE

• In 2006, a team of engineers at GOOGLE developed this Linux Kernel feature.
• Cgroups (Control Groups) feature was eventually introduced in Linux Kernel version 2.6.24 — released in January 2008.
• It allows to limit (apply quota), account for, and isolate the use of computing resources (CPU, memory, disk I/O, network, etc.).
• Containers management software would make use of CGroups for effective resource control by Containers.

The following diagrams shows limited system resources (CPU, Memory, N/W, Storage, etc.) being distributed amongst separate Control Groups of processes.

The above picture is taken from Mairin Duffey’s Blog

#2 — Namespaces — Controlling What You Can SEE

• Initial version released in 2002 with Linux Kernel version 2.4.19.
• Functionality usable by Containers was added with Linux Kernel version 3.8.
• Using Namespaces, it became possible to isolate and virtualize resources for processes e.g. separate process IDs, hostnames, user IDs, network access, IPCs, and filesystems.
• Namespaces are a fundamental aspect of containers on Linux.
• Namespaces would provide much needed process virtualization space to the Containers.

#3 — Union File Systems - Managing Software Image for Containers

• A file system that allows a collection of different file systems and directories (called branches) to be transparently overlaid (does a UNION of) into a single logical file system.
• Union File Systems show Containers a way to manage their software using a layered image and mounting them by doing a UNION of underlying file systems branches.

Note — This concept deserves a separate detailed description and I would cover it in a chapter of its own.

CGroups and Namespaces, when used together, provide an isolated environment within a Linux System where the running processes can only see the boot directory, from related processes, own user id, and related network interfaces (common NAMESPACE) & a control over CPU, memory, network and IO usage (common CGroup). These functionalities were combined with easy to manage command line interface under the name of LXC (Linux Containers).

Arrival of Docker

In 2013, Docker Inc. introduced Docker — a technology to create and run Docker Containers.

Docker makes use of CGroups, Namespaces, & Union File Systems to package an application and its dependencies to make a Docker image which can be run as a Container on a Linux Operating System — a much more portable and efficient method of running an application than running it on a Virtual Machine.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

1. Resource Isolation: The Failure of Operating Systems & How We Can Fix It — Glauber Costa, Parallels
2. Evolution of Linux Containers and Future — Imesh Gunaratne
3. The failure of operating systems and how we can fix it — Michael Kerrisk
4. How Linux Kernel Cgroups And Namespaces Made Modern Containers Possible — Duncan Macrae
5. Cgroups @ Wikipedia
6. Introduction to Control Groups (CGROUPS) @ RedHat
7. How I Used CGroups to Manage System Resources In Oracle Linux 6
8. Linux Kernel Documentation on CGroups
9. Namespaces in operation
10. Resource management: Linux kernel Namespaces and cgroups by Rami Rosen
11. Union File System
12. Docker — Wikipedia
13. What is Docker?