Moving from XEN to KVM

RedHat Enterprise Linux 6 doesn’t include Xen support out of the box.  Fedora and Ubuntu also feature KVM.  For a non-business user like myself, it was time to follow suit.  It was 8 hours worth of work and 26 virt-install attempts later, that I finally found the magic combo.

When reading this note that the machine that was used as a host is running CentOS 5.  There seem to be some changes between it and the packages that would come with a system based on RedHat Enterprise Linux 6.

Note: After writing this, it seems as though the greatest benefit would come from reading this completely, reading other resources, and then starting your migration.

Step 0: Copy

This step was easy.  My current virtual machines sit on a LVM partition.  After shutting down the virtual machines.  Running the dd command created a backup easily.

dd if=/dev/vms/webRoot of=/backups/webRoot.img

Step 1: Migration

Things went bad shortly thereafter. The initial plan was to run virt-v2v and simply migrate the a CentOS 4 virtual machine that was running as a web server. But, alas, I had committed a great configuration “error” that I wasn’t even aware of. Instead of having the LVM block devices represent an entire virtual disk, I had them configured to be just a partition. virt-v2v was unable to read the partition. At least, that was the last error that was given. I found virt-v2v to be unforgiving. The error messages that it produced were cryptic. I had to look at the perl scripts to actually see what it was failing on. After taking into acount that CentOS 4 was only going to be supported for another few months and the fact that I really wanted to install Scientific Linux for something, I did something that is usually not in my vocabulary….I gave up. The migration step was a failure. I’ve seen sites where people made it work (they were not using block devices as partitions as I was), it just didn’t work for me.

Step 2: Creating a new VM from an install disk

This isn’t hard to do, the part where this fell apart was that I was trying to do this in headless mode. The entire install was to be done from a console. Again, I’ve seen posts where people have this working. The man page for virt-install gives us a hint as to how to do this

No graphical console will be allocated for the guest. Fully
virtualized guests (Xen FV or QEmu/KVM) will need to have a text
console configured on the first serial port in the guest (this can be
done via the –extra-args option). Xen PV will set this up
automatically. The command virsh console NAME can be used to
connect to the serial device.

It says we need to create a console to connect to. No problem. In fact, here is a sample command that _should_ work.

virt-install -v --connect qemu:///system -n vm24 -r 512 --vcpus=2 --disk="path=/var/lib/libvirt/images/vm24.qcow2,size=25" --location /var/lib/libvirt/images/SL-61-x86_64-2011-07-27-Install-DVD --os-type linux --os-variant rhel6 --accelerate --network=bridge:br0 --prompt --extra-args="text console=tty0 console=ttyS0,115200" --nographics

It almost does work, but something has happened to my drives!

This ended my first evening of work on the project.  It was an absolute failure.  The only work that was done was to discover out what didn’t work. I again gave up trying to do the install from the console.

Day two was much better.  Most example you will see on the internet for virt-install use vnc to provide a virtual monitor’s view into the installation.  This is the approach I now recommend.

If you were trying to do this completely headless from a console, good luck to you :). For everyone else, break down and run the following:

yum install virt-viewer xauth

Do not forget xauth! Without it ssh -X to your sever will not work! You will look for a solution and will find that people recommend running ssh -Y instead. The real problem is that you are missing xauth. (Yes, this did cost me an hour.)

Now run virt-install and leave off the –nographics and console part

virt-install -v --connect qemu:///system -n vm26 -r 1024 --vcpus=2 --disk="path=/var/lib/libvirt/images/vm26.qcow2,size=25" --cdrom /var/lib/libvirt/images/SL-61-x86_64-2011-07-27-Install-DVD.iso --os-type linux --accelerate --network=bridge:br0 --prompt --vnc

All of the drives are accessible. Hmmmm. I’m guessing that it works because of… well, I don’t know. Perhaps is has something to do with the cdrom drive being defined differently. It doesn’t make sense to further peruse this when the vnc method works just fine.

Step 3: OS install

This is done as normal. There were no installation modifications.

Step 4: Enable a console (not required)

VNC and virt-viewer work just find. But, I was used to running virsh console to get a console on machine that is able to see the machine’s startup output. The method for this varies by OS. For Scientific Linux 6, I this to the end of the kernel parameter in /etc/grub.conf

text console=tty0 console=ttyS0,115200

In debian squeeze, this line was modified in /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet text console=tty0,115200"

After running update-grub, the console worked.  Day (or should I type “night”) 2 was over and I had a working web server.

A note about transient domains

Are you getting this error:

“internal error cannot set autostart for transient domain”

When a domain is created just by running virsh create ${domain}.xml the domain is considered transient. The biggest issue with this is that it cannot be set to autostart with virsh autostart ${domainname}. Ensure that if you would like a domain to autostart, and you already have the xml file, run these commands
virsh define ${domain}.xml
virsh autostart ${domainname}

Then, the domain will be autostarted on boot. Run virsh start ${domainname} to start it up manually. This was somewhat talked about in the official documentation, but the actual commands that make it happen are not listed.

A Note About Paravirtualization

The virtual network and disk can be paravirtualized.    This will result in increased performance.  I’m actually writing this so I know how this can be done in the future.  This is covered extensively in other places.

Here are the guides to help make this happen ( guide1, guide2).

Additional Resources

This is not a complete resource. I haven’t even mentioned firewalling or bridging. These topics are covered in other places on the web pretty throughly.

A great guide to get you started can be found here.
You will most likely have to play with the VM’s XML.  The details are here.
Another conversion document that I found helpful can be found here.
Debian images that can be used with KVM can be found here.

Related posts:

Tags: ,

  • I am actually thinking about building a small test server and running KVM so I have room to play and learn. I have been reading up on how I can get KVM to run Mac OSX on non-apple hardware.. and since I bought the server software that would be perfect. 🙂 Thanks for the write up Tom!

  • Dan D.


    Didn’t know you were back in MI. I bumped into your dad on Sat. with Mike’s boys at Home Depot. I talked with your dad for about 5 minutes. Wondered how life was treating you. Kinda hoped I could swap some stories over the email type communication, or get together with you and Sarah sometime. I look forward to hearing from you.


  • jk

    Completely headless KVM install script:

    virt-install \
    –name=[enter_vm_name] \
    –ram=2048 \
    –vcpus=1 \
    –os-type=linux \
    –os-variant=rhel6 \
    –disk path=/dev/[enter_vg_name]/[enter_lv_name],device=disk,bus=virtio,sparse=true,format=raw \
    -w bridge=br0,model=virtio \
    –nographics \
    -x console=ttyS0 \

  • Dan Purcell

    Thank you very much for this information, it has answered a lot of my questions and has solved a lot of the trouble I had.