Nicolas LIMARE / pro / notes / 2013 / Experiences with the KVM Virtual Machines Boot Time

Brief: Booting a minimal KVM virtual Linux system takes 2s, plus 5 seconds for the KVM/qemu startup.

I am investigating different solutions to implement some sort of system isolation for the back-ends of the IPOL demos. The reason for that is the possibility to exploit, via maliciously crafted image files, security issues is the demo codes. These exploits could target the libraries (libpng, libtiff for example) or, more likely, the algorithm codes themselves. Given that the authors don't know much about the possible consequences of a buffer overflow, and that the code is publicly available and will probably never be updated, the possibility of a security hole is real.

I am not paranoid about it, and I think many other web services are more exposed than us and would provide more benefits for attackers, but the risk exists and I have been thinking for a long time about reducing it by some form of sandboxing.

The first version of our demo server was running in a Xen virtual machine without any outgoing network capacity, but this network restriction made it a pain to maintain and update. My new idea would be to run the demo back-ends in a toy environment with controlled access to the computer resources (limited disk and memory space and CPU time, no network) and throw it away away after the computation. In addition to the resource control, it adds the guarantee that the computing environment is always clean and identical when a demo starts.

But this sandboxing on demand needs to have a very low overhead. Some algorithms are very fast and we should not add 5 seconds of fixed cost for those running in .5s (see for example LSD on typical images).

I did my first tests on KVM virtual machines:

  • basic Debian Wheezy amd64 system, kernel 2.6.32-5-amd64
  • single filesystem on a virtio device image stored on ramdisk
  • boot directly without bootloader from kernel and initrd stored on ramdisk too
  • minimal system services (atd, cron and rsyslogd), disabled networking and console-setup init
  • only disk, serial and network devices, every other device (CDROM, video/VNC, ...) unnecessary and removed
  • auto-login on the serial link via a wrapper script to /bin/login -f user with a timestamp

The host machine runs on an Intel i5 M520 CPU at 2.4GHz, no hyperthreading, frequency scaling set at maximum performance, no other demanding task running. The kernel is the stock Debian 3.2.0-4-amd64, with qemu 1.1.2.

Then I started the virtual machine via virsh with a console access, and did a few reboots. On average, a reboot cycle takes around 8 seconds. In the sample log reproduced at the end of this message, the reboot starts at 07:21:29 and the new console is available at 07:21:37. I recorded the console session with script, you can replay it with the attached log files and scriptreplay kvm-boottest.time kvm-boottest.script.

Looking at the reboot procedure in detail (ie counting in my head while watching the screen), I observe that the proper system boot time is around 2 seconds, with 5 other seconds waiting for the kvm and qemu init.

So my conclusion for the moment is that KVM virtual machines can not be started on-demand for short computations taking about one second. In this situation, I definitely have to use process-level isolation. The next candidate will be LXC Linux Containers.

kvm-boottest.time and kvm-boottest.script

root@kvm-boottest:~# reboot

Broadcast message from root@kvm-boottest (ttyS0) (Thu Jan 24 07:21:29 2013):The system is going down for reboot NOW!
INIT: Switching to runlevel: 6
INIT: Sending processes the TERM signal
root@kvm-boottest:~# Using makefile-style concurrent boot in runlevel 6.
Stopping deferred execution scheduler: atd.
Asking all remaining processes to terminate...done.
All processes ended within 1 seconds....done.
Stopping enhanced syslogd: rsyslogd.
Cleaning up ifupdown....
Saving the system clock.
Deactivating swap...done.
Will now restart.
[    9.364537] Restarting system.
Loading, please wait...
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Starting the hotplug events dispatcher: udevd.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...[    0.872152] Error: Driver 'pcspkr' is already registered, aborting...
Setting preliminary keymap...done.
Activating swap...done.
Checking root file system...fsck from util-linux-ng 2.17.2
/dev/vda1: clean, 27786/59392 files, 170325/237312 blocks
Loading kernel modules...done.
Cleaning up ifupdown....
Setting up networking....
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting kernel variables ...done.
Cleaning up temporary files....
Setting console screen modes.
cannot (un)set powersave mode
Skipping font and keymap setup (handled by console-setup).
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting ACPI services....
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.

Thu Jan 24 07:21:37 EST 2013
Last login: Thu Jan 24 07:21:25 EST 2013 on ttyS0