|Download Linux On Linux||Virtual Machine Monitor Documentation|
Table of Contents
- Licence and Copyright
- Building and Installation
- Running the Virtual Machine
- Change Log
- Future Work
- Further Information
Linux-on-Linux is Copyright 2005-2007 to National ICT Australia, the University of NSW, Matthew Chapman and Peter Chubb. It is licensed under the GNU Public Licence, version 2.0.
- Get the virtual machine monitor (VMM) and patches as a release snapshot from this directory.
Install the prerequisites:
gcc(version >= 4.1)
Unpack the virtual machine monitor source:
$ tar xjvf LinuxOnLinux-1.0.tar.bz2
Create two copies of Linux kernel source. The patches from the CVS repository are usually against a recent kernel.org kernel. One tree will be for the host, and one for the guest. Call them, e.g.,
I usually do:
$ cg-clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6-import
$ cg-clone -l linux-2.6-import linux-2.6-guest
$ cg-clone -l linux-2.6-import linux-2.6-host
.../LinuxOnLinux/scripts/prepare.shon each source tree to patch it and to create the symbolic links.
$ sh LinuxOnLinux/scripts/prepare.sh linux-2.6-host host
$ sh LinuxOnLinux/scripts/prepare.sh linux-2.6-guest guest
prepare.shcarries out the following steps:
patches to each tree using quilt.
Patches are in
Creates symbolic links from
.../LinuxOnLinux/userspace/vmm-offsets.hinto the guest Linux
include/asm-ia64/vmm-offsets.h, and from
- Applies appropriate patches to each tree using quilt. Patches are in
Configure the host OS for your machine, build it, install it,
and reboot. Include
CONFIG_VMMHOSTin the configuration, and exclude
There are sample configurations in the LinuxOnLinux source for HP and SGI systems, but you will have to adjust these for the filesystems and hardware you actually have
The sample configuration also enables ethernet bridging. If you want to use it, install
iptablesto set up NAT if you want your virtual machines to talk the the rest of the internet.
Build the virtual machine monitor. The build process needs to be able to get at the output directory of the patched host kernel.
$ cd LinuxOnLinux/userspace
$ KERNEL=/lib/modules/`uname -r`/build sh autogen.sh
(to rebuild, just type
Configure the guest OS as if for the SKI simulator, but also select
CONFIG_AFTERBURNER, and exclude
CONFIG_HUGETLBFS(again, there are sample configurations in the LinuxOnLinux source tree).
I prefer not to configure modules, because it's painful to get the modules into an appropriate place for the virtual machine to use.
Build with a command line that looks like this:
$ make EXTRA=-Bpath_to_LinuxOnLinux
Get or build yourself a root image. You can
build one using debootstrap: see SkiDiskImage on the Itanium WiKi for details).
Call the result path_to_disk
If you prefer, there is a script
runvmmin the source tree that you can start with to create and configure virtual machines. To use it, set the environment variable
VMMPATHto the base of the LinuxOnLinux source tree. Or read the manual page and create a
.vmmrcfile in your home directory.
The VMM uses
/dev/tap? as an ethernet
interface; you can bridge these with
assign an IP address and use IP forwarding.
- Faster. The main bottlenecks now are in the host's signal path (guest pagefaults are signalled by a SIGSEGV for instance), and scheduling for spinlocks. When a spinlock is entered on one virtual processor, ideally, the running virtual CPU would donate the rest of its timeslice to the non-running virtual cpu that is currently holding the spinlock, if any.
Current features being worked on include:
- Direct access to PCI devices, using the User Level Device Driver framework.
- NUMA support, by providing better firmware emulation (including an SRAT table), and by better memory policies in the virtual machine monitor.
- Better packaging to make LinuxOnLinux easier to buld and deploy.
- Better tools and techniques for debugging, tracing and profiling virtual machines.
- New guests: FreeBSD is the first target.
- Improved overall speed by 30% by adding multiple address spaces per virtual processor.
vmm and runvmm now cleans up after themselves.
-koption gives old behaviour.
- Got rid of ptrace thread on suitably capable kernels; the
result is that one can run
straceon the hypervisor.
runvmmscript; it now works better in environments other than mine, and is more configurable.
- Altix supported `out of the box'
Release 0.9First release.
Troubleshooting is harder than it should be at present; this is one of the things we're working on.
Getting more information
The first thing to do when troubleshooting is to run with
-k. If you're using
it'll print stuff to standard error; if you're using
runvmm the trace will be in
/tmp/xNNNNN where NNNNN is the
process number of
You should get an error message of some sort in that trace.
If it's not enough, edit
change the definition of the
DEBUG symbol to
(~0UL), and rebuild.
You'll now get a very verbose trace from the virtual machine monitor: every page fault, every privileged instruction, etc., will be logged.
The commonest problems are because the guest kernel hasn't been
built correctly. Check that all the patches applied cleanly, that
CONFIG_USRVMM is defined and that the afterburner was used
when building. I've had problems using distcc where the
afterburner wasn't installed in the same place on every machine;
also you should disable
ccache as it cannot see the
afterburner. gcc-4.0 sometimes miscompiles, as well. I use
gcc-4.1 without problems.
If you see a message like:
Cannot map from 0x8000000100000000,0x8000000103ffffff to 0x4000000--- mmap: Invalid argument PANIC in VMM: tlbop23 failed
it's because you've configured
in the host kernel. Note that the address is in region 4, which on a
standard kernel is used for hugetlb pages, but for the virtual machine
is used for the guest kernel's virtual memory area.