RHEV/ovirt – can’t switch SPM role – async_tasks are stucked

On the host with the SPM role

$ vdsClient -s 0 getAllTasksStatuses
{'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {'feb3aaa5-ec1c-42a6-8f17-f7c94891b43f': {'message': '1 jobs completed successfully', 'code': 0, 'taskID': '631fd441-0955-49da-9376-1cba24764aa7', 'taskResult': 'success', 'taskState': 'finished'}, 'b4fe0c6d-d458-4ed2-a9e2-2c0d41914b8f': {'message': '1 jobs completed successfully', 'code': 0, 'taskID': '67e1a2e8-3747-43fa-b0dd-fc469a6f6a02', 'taskResult': 'success',
'taskState': 'finished'}}}

On the RHEV/ovirt manager

$ for i in b4fe0c6d-d458-4ed2-a9e2-2c0d41914b8f feb3aaa5-ec1c-42a6-8f17-f7c94891b43f; do psql --dbname=engine --command="DELETE FROM async_tasks WHERE vdsm_task_id='${i}'"; done
$ for j in b4fe0c6d-d458-4ed2-a9e2-2c0d41914b8f feb3aaa5-ec1c-42a6-8f17-f7c94891b43f; do vdsClient -s 0 clearTask ${j}; done

Choosing the right scheduler on a virtual maschine (kvm)

The default i/o scheduler is the Completely Fair Queuing (cfq) in the 2.6 kernel. This is not the first choice for a virtual machine/hypervisor. The combination of the noop and the deadline scheduler is much better for a virtualization host.

virtual machine: noop
hypervisor: deadline

Set the scheduler temporarily (vm)

$ echo noop > /sys/block/sda/queue/scheduler

Set the scheduler permanently (vm)

kernel /vmlinuz-3.8.11 root=/dev/vgsystem/lvroot elevator=noop

(For the hypervisor replace noop with deadline!)

And don’t forget to use virtio & raw devices for guest and noatime & nodiratime in fstab wherever it’s possible.

Hint: VMware also recommends the noop scheduler for the guests.

Building qemu & libvirt from scratch…


git clone git://git.qemu.org/qemu.git
git clone git://github.com/sahlberg/libiscsi.git
http://libvirt.org/sources/libvirt-1.0.0.tar.gz (4/12/12)
aptitude install libvdeplug2 libvdeplug2-dev libaio1 libaio-dev libcap-dev libattr1-dev libsdl-dev libxml2-dev



make && make install


./configure --enable-kvm --enable-vde --enable-virtfs --enable-linux-aio --enable-libiscsi --enable-sdl --target-list=i386-softmmu,x86_64-softmmu,i386-linux-user,x86_64-linux-user --audio-drv-list=alsa
make && make install


(maybe you should build libvirt with python < v3.2)

configure: Drivers
configure:       Xen: no
configure:      QEMU: yes
configure:       UML: yes
configure:    OpenVZ: yes
configure:    VMware: yes
configure:      VBox: yes
configure:    XenAPI: no
configure:  xenlight: no
configure:       LXC: yes
configure:      PHYP: yes
configure:       ESX: yes
configure:   Hyper-V: no
configure: Parallels: yes
configure:      Test: yes
configure:    Remote: yes
configure:   Network: yes
configure:  Libvirtd: yes
configure: Interface: yes
configure:   macvtap: yes
configure:  virtport: yes
configure: Storage Drivers
configure:      Dir: yes
configure:       FS: yes
configure:    NetFS: yes
configure:      LVM: yes
configure:    iSCSI: yes
configure:     SCSI: yes
configure:    mpath: yes
configure:     Disk: no
configure:      RBD: no
configure: Sheepdog: no
configure: Libraries
configure:   libxml: -I/usr/include/libxml2   -lxml2  
configure:   dlopen: -ldl
configure:  libcurl: -DCURL_DISABLE_TYPECHECK   -lcurl  
configure: openwsman: no
configure:  libssh2:   -lssh2  
configure:   gnutls: -I/usr/include/p11-kit-1   -lgnutls   -lgcrypt
configure:     sasl:   -lsasl2
configure:     yajl:   -lyajl
configure:  sanlock: no
configure: firewalld: yes
configure:    avahi: no
configure:   polkit: /usr/bin/pkcheck (version 1)
configure:    audit: no
configure:  selinux: no
configure: apparmor: no
configure:  numactl: no
configure:    capng:  -lcap-ng
configure:      xen: no
configure:   xenapi: no
configure: xenlight: no
configure:      hal: no
configure:     udev:   -ludev -lrt     -lpciaccess  
configure:    netcf: no
configure:     pcap: -I/usr/include -L/usr/lib64 -Wl,-rpath,/usr/lib64 -lpcap
configure:       nl: -I/usr/include/libnl3   -I/usr/include/libnl3   -lnl-3   -lnl-route-3 -lnl-3  
configure:    mscom: no
configure:      xdr: 
configure:      rbd: no
make && make install

If you getting errors like this one

make[3]: Entering directory `/home/rmichel/src/libvirt-1.0.0/python'
  GEN    generator.py.stamp
  File "./generator.py", line 21
    print "test"
SyntaxError: invalid syntax
make[3]: *** [generator.py.stamp] Error 1

you are using python 3.2.x (or newer) and should switch back to python 2.7.x!

The problem is the ‘print’ function. In python 2.7.x the syntax is print "foobar" but in 3.x that’s not valid anymore! Instead you should be use print("foobar") (which is also valid in 2.7.x) !!



qemu [...] -drive file=iscsi://,if=scsi,bus=0,unit=3


vde_switch -s /tmp/vdesock1 &
qemu [...] -net nic -net vde,vlan=0,sock=/tmp/vdesock1

“QEMU has two asynchronous I/O mechanisms: POSIX AIO emulation using a pool of worker threads and native Linux AIO.”

qemu [...] -drive file=foo.img,aio=[threads|native]

libvirt – QEMU Monitor Protocol (QMP)

Because libvirt self is using the qemu monitor to mange the guests, it is not available for the user. Since version 0.8.6 [1] its possbile to send a command throught libvirt to the monitor.

virsh qemu-monitor-command <domain> <command>

The problem is that you must use the QMP format to send a command like

$ virsh qemu-monitor-command <domain> \
    '{"execute":"human-monitor-command","arguments":{"command-line":"info kvm"}}'

The solution: Human Monitor Protocol / --hmp

With this nifty switch [2] its possible to use qemu monitor commands without the QMP format like

$ virsh qemu-monitor-command --hmp <domain> 'info kvm'
kvm support: enabled