Find out how ICT can support biomedical and clinical researchFind out more. Managing complexity by developing new tools and processes. Managing Complexity
Download Linux On Linux Virtual Machine Monitor Documentation

Itanium Linux-on-Linux

Table of Contents

Licence and Copyright

Linux-on-Linux is Copyright 2005-2007 to National ICT Australia, the University of NSW, Matthew Chapman and Peter Chubb. It is licensed under the GNU Public Licence, version 2.0.

Building and Installation

  1. Get the virtual machine monitor (VMM) and patches from the Gelato CVS repository or get a release snapshot from this directory.
  2. Install the prerequisites:
    • libatomic-ops-dev
    • gcc (version >= 4.1)
    • make
    • quilt
    • automake, and
    • autoconf
  3. Unpack the virtual machine monitor source:

    $ tar xjvf LinuxOnLinux-1.0.tar.bz2

  4. Create two copies of Linux kernel source. The patches from the CVS repository are usually against a recent kernel.org kernel. One tree will be for the host, and one for the guest. Call them, e.g., linux-2.6-host and linux-2.6-guest.

    I usually do:
    $ cg-clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6-import
    $ cg-clone -l linux-2.6-import linux-2.6-guest
    $ cg-clone -l linux-2.6-import linux-2.6-host

  5. Run .../LinuxOnLinux/scripts/prepare.sh on each source tree to patch it and to create the symbolic links.

    $ sh LinuxOnLinux/scripts/prepare.sh linux-2.6-host host
    $ sh LinuxOnLinux/scripts/prepare.sh linux-2.6-guest guest

    prepare.sh carries out the following steps:

    1. Applies appropriate patches to each tree using quilt. Patches are in .../vmm/patches/patches-host and .../vmm/patches/patches-guest
    2. Creates symbolic links from .../LinuxOnLinux/userspace/vmm-offsets.h into the guest Linux include/asm-ia64/vmm-offsets.h, and from .../LinuxOnLinux/afterburner/burn.S to arch/ia64/kernel/burn.S.
  6. Configure the host OS for your machine, build it, install it, and reboot. Include CONFIG_TUN and CONFIG_VMMHOST in the configuration, and exclude CONFIG_HUGETLBFS.
    There are sample configurations in the LinuxOnLinux source for HP and SGI systems, but you will have to adjust these for the filesystems and hardware you actually have
    The sample configuration also enables ethernet bridging. If you want to use it, install bridge-utils otherwise install iptables to set up NAT if you want your virtual machines to talk the the rest of the internet.

  7. Build the virtual machine monitor. The build process needs to be able to get at the output directory of the patched host kernel.

    $ cd LinuxOnLinux/userspace
    $ KERNEL=/lib/modules/`uname -r`/build sh autogen.sh

    (to rebuild, just type make).

  8. Configure the guest OS as if for the SKI simulator, but also select CONFIG_USRVMM and CONFIG_AFTERBURNER, and exclude CONFIG_HUGETLBFS (again, there are sample configurations in the LinuxOnLinux source tree).
    I prefer not to configure modules, because it's painful to get the modules into an appropriate place for the virtual machine to use.
    Build with a command line that looks like this:

    $ make EXTRA=-Bpath_to_LinuxOnLinux/afterburner EXTRA_AFLAGS=-Bpath_to_LinuxOnLinux/afterburner boot

    (note that you cannot use the tilde for home directory shortcut in path_to_LinuxOnLinux)
  9. Get or build yourself a root image. You can use a prebuilt RedHat or Debian image for the simulator, or you can use debootstrap on a Debian system (see SkiDiskImage on the Itanium WiKi for details). Call the result path_to_disksda.

  10. Run the resulting system with
    $ path_to_LinuxOnLinux/userspace/vmm vmlinux maxcpus=2 simscsi=path_to_disk/sd simeth=eth0

    If you prefer, there is a script runvmm in the source tree that you can start with to create and configure virtual machines. To use it, set the environment variable VMMPATH to the base of the LinuxOnLinux source tree. Or read the manual page and create a .vmmrc file in your home directory.

The VMM uses /dev/tap? as an ethernet interface; you can bridge these with brctl, or assign an IP address and use IP forwarding.

Current and Future Work

We're currently enhancing LinuxOnLinux to make it:
  • Faster. The main bottlenecks now are in the host's signal path (guest pagefaults are signalled by a SIGSEGV for instance), and scheduling for spinlocks. When a spinlock is entered on one virtual processor, ideally, the running virtual CPU would donate the rest of its timeslice to the non-running virtual cpu that is currently holding the spinlock, if any.
  • More featureful. Current features being worked on include:
    • Direct access to PCI devices, using the User Level Device Driver framework.
    • NUMA support, by providing better firmware emulation (including an SRAT table), and by better memory policies in the virtual machine monitor.
    • Better packaging to make LinuxOnLinux easier to buld and deploy.
    • Better tools and techniques for debugging, tracing and profiling virtual machines.
    • New guests: FreeBSD is the first target.

More information

More information is available on the Gelato@UNSW wiki and in the Virtualisation research pages.

Change Log

Release 1.0

  • Improved overall speed by 30% by adding multiple address spaces per virtual processor.
  • vmm and runvmm now cleans up after themselves. New -k option gives old behaviour.
  • Got rid of ptrace thread on suitably capable kernels; the result is that one can run gdb or strace on the hypervisor.
  • More functional runvmm script; it now works better in environments other than mine, and is more configurable.
  • Altix supported `out of the box'

Release 0.9

First release.

Troubleshooting

Troubleshooting is harder than it should be at present; this is one of the things we're working on.

Getting more information

The first thing to do when troubleshooting is to run with -k. If you're using vmm directly, it'll print stuff to standard error; if you're using runvmm the trace will be in /tmp/xNNNNN where NNNNN is the process number of runvmm.

You should get an error message of some sort in that trace. If it's not enough, edit userspace/debug.h, change the definition of the DEBUG symbol to (~0UL), and rebuild.

You'll now get a very verbose trace from the virtual machine monitor: every page fault, every privileged instruction, etc., will be logged.

Common Problems

The commonest problems are because the guest kernel hasn't been built correctly. Check that all the patches applied cleanly, that CONFIG_USRVMM is defined and that the afterburner was used when building. I've had problems using distcc where the afterburner wasn't installed in the same place on every machine; also you should disable ccache as it cannot see the afterburner. gcc-4.0 sometimes miscompiles, as well. I use gcc-4.1 without problems.

If you see a message like:
Cannot map from 0x8000000100000000,0x8000000103ffffff to 0x4000000--- mmap: Invalid argument PANIC in VMM: tlbop23 failed
it's because you've configured CONFIG_HUGETLBFS on in the host kernel. Note that the address is in region 4, which on a standard kernel is used for hugetlb pages, but for the virtual machine is used for the guest kernel's virtual memory area.