Opened questions about container live-migration


One Line Summary

How to integrate P.Haul with Docker and how to fix criu pre-dump.


= P.Haul =
Many users asks us how to migrate Docker containers. Unfortunately this feature isn’t implemented yet. You probably know about the P.Haul project, which is an extension for CRIU to implement a live-migration logic. Its first version is used in OpenVZ, but we found that it may be better to have another version in Go to implement live-migration of Docker containers. Pavel introduced a second version a few month ago and now we are looking for a volunteer, who are ready to integrate it with Docker (Containerd?). In this part, I’m going to briefly describe how P.Haul works and how to integrate it with Docker. Any ideas about live-migration are welcome.

= Pre-dump vs Pipes =
It’s one more topic about live-migration. Current pre-dump mode suffers from several issues
It keeps all the memory in pipes, and their number can be huge due to limited one pipe size
It keeps all the memory in pipes and this memory is unreclaimable for that period
It infects tasks to drain memory from
We can try to use sys_read_process_vm() syscall to mitigate all of the above
No pipes, just copy data into temp buffer and send
Memory is always reclaimable
No infection is needed, just freeze, reset the tracker and proceed
The second way is less effective, because data are copied into a temporary user-space buffer, so a new syscall  sys_process_vmsplice() was introduced, it allows to splice memory of any process into a pipe. There is a few other ideas how to optimize live-migration.

= Nested-namespaces =
Last year we did a lot of work to support nested-namespaces. I want to describe the current status of this work. The main question here is how many people are interested in this feature. The code becomes much more complex and it is a big question whether to merge it to the master branch or not. By the way, this work may be useful to integrate CRIU with Kubernetes.


kernel, CRIU, docker, container, syscall

Presentation Materials



  • 20151022_100252-zzz


    Developer in the CRIU and OpenVZ projects.

    Andrew Vagin is interested in Container Virtualization (LXC, OpenVZ). He started to write autotest for OpenVZ in 2006, when he was a student at the Moscow Institute of Physics and Technology (MIPT). Now Andrew works in OpenVZ kernel team. In addition he is an active developer in the CRIU (Checkpoint/Restore in Userspace) project.