RHEV/ovirt – can’t switch SPM role – async_tasks are stucked

On the host with the SPM role

$ vdsClient -s 0 getAllTasksStatuses
{'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {'feb3aaa5-ec1c-42a6-8f17-f7c94891b43f': {'message': '1 jobs completed successfully', 'code': 0, 'taskID': '631fd441-0955-49da-9376-1cba24764aa7', 'taskResult': 'success', 'taskState': 'finished'}, 'b4fe0c6d-d458-4ed2-a9e2-2c0d41914b8f': {'message': '1 jobs completed successfully', 'code': 0, 'taskID': '67e1a2e8-3747-43fa-b0dd-fc469a6f6a02', 'taskResult': 'success',
'taskState': 'finished'}}}

On the RHEV/ovirt manager

$ for i in b4fe0c6d-d458-4ed2-a9e2-2c0d41914b8f feb3aaa5-ec1c-42a6-8f17-f7c94891b43f; do psql --dbname=engine --command="DELETE FROM async_tasks WHERE vdsm_task_id='${i}'"; done
$ for j in b4fe0c6d-d458-4ed2-a9e2-2c0d41914b8f feb3aaa5-ec1c-42a6-8f17-f7c94891b43f; do vdsClient -s 0 clearTask ${j}; done

RHEV/ovirt – find stucked / zombie tasks

Random notes

$ vdsClient -s 0 getAllTasksStatuses
$ vdsClient stopTask <taskid>
$ vdsClient clearTask <taskid>
$ su - postgres
$ psql -d engine -U postgres
> select * from job order by start_time desc;
> select DeleteJob('702e9f6a-e2a3-4113-bd7d-3757ba6bc4ef');

or

/usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "select * from job;"

entropy inside a virtual machine

Sometimes my ceph-(test!)deployments inside a VM failed.

The Problem is that the kernel/cpu can not provide enough entropy (random numbers) for the ceph-create-keys command – so it stuck/hang. It is not a ceph problem! This can also happen with ssl commands.

But first things first – we need to check the available entropy on a system:

cat /proc/sys/kernel/random/entropy_avail

The read-only file entropy_avail gives the available entropy.
Normally, this will be 4096 (bits), a full entropy pool (see man 4 random)

Values less than 100-200, means you have a problem!

For a virtual machine we can create a new device – virtio-rng. Here is a xml example for libvirt.

<rng model='virtio'>
  <backend model='random'>/dev/random</backend>
</rng>

That is ok for ONE virtual machine on the hypervisor. Usually we find more than one virtual machine. Therefore we need to install the rng-tools package on the virtual machines.

$pkgmgr install rng-tools
systemctl enable rngd
systemctl start rngd

That’s it! That solved a lot of my problems 😉

Openstack Horizon – leapyear bug

Switching the language in the dashboard ends with a error.

day is out of range for month

eg. https://bugs.launchpad.net/horizon/+bug/1551099

[Mon Feb 29 09:20:05 2016] [error] Internal Server Error: /settings/
[Mon Feb 29 09:20:05 2016] [error] Traceback (most recent call last):
[Mon Feb 29 09:20:05 2016] [error]   File "/usr/lib64/python2.6/site-packages/django/core/handlers/base.py", line 112, in get_response
[Mon Feb 29 09:20:05 2016] [error]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Mon Feb 29 09:20:05 2016] [error]   File "/usr/lib64/python2.6/site-packages/horizon/decorators.py", line 36, in dec
[Mon Feb 29 09:20:05 2016] [error]     return view_func(request, *args, **kwargs)
[Mon Feb 29 09:20:05 2016] [error]   File "/usr/lib64/python2.6/site-packages/horizon/decorators.py", line 52, in dec
[Mon Feb 29 09:20:05 2016] [error]     return view_func(request, *args, **kwargs)
[Mon Feb 29 09:20:05 2016] [error]   File "/usr/lib64/python2.6/site-packages/horizon/decorators.py", line 36, in dec
[Mon Feb 29 09:20:05 2016] [error]     return view_func(request, *args, **kwargs)
[Mon Feb 29 09:20:05 2016] [error]   File "/usr/lib64/python2.6/site-packages/django/views/generic/base.py", line 69, in view
[Mon Feb 29 09:20:05 2016] [error]     return self.dispatch(request, *args, **kwargs)
[Mon Feb 29 09:20:05 2016] [error]   File "/usr/lib64/python2.6/site-packages/django/views/generic/base.py", line 87, in dispatch
[Mon Feb 29 09:20:05 2016] [error]     return handler(request, *args, **kwargs)
[Mon Feb 29 09:20:05 2016] [error]   File "/usr/lib64/python2.6/site-packages/django/views/generic/edit.py", line 171, in post
[Mon Feb 29 09:20:05 2016] [error]     return self.form_valid(form)
[Mon Feb 29 09:20:05 2016] [error]   File "/srv/www/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/settings/user/views.py", line 38, in form_valid
[Mon Feb 29 09:20:05 2016] [error]     return form.handle(self.request, form.cleaned_data)
[Mon Feb 29 09:20:05 2016] [error]   File "/srv/www/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/settings/user/forms.py", line 89, in handle
[Mon Feb 29 09:20:05 2016] [error]     expires=_one_year())
[Mon Feb 29 09:20:05 2016] [error]   File "/srv/www/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/settings/user/forms.py", line 32, in _one_year
[Mon Feb 29 09:20:05 2016] [error]     now.minute, now.second, now.microsecond, now.tzinfo)
[Mon Feb 29 09:20:05 2016] [error] ValueError: day is out of range for month

SUSE Openstack Cloud – sleshammer – pre/post scripts – pxe trigger

Enable root login for the sleshammer image

(it is used by the suse cloud as a hardware discovery image)

The sleshammer image will mount “/updates” over nfs from the admin node and execute the control.sh. This script will check if there are some pre/post-hooks and will possibly execute them.

root@admin:/updates # cat /updates/discovered-pre/set-root-passwd.hook
#!/bin/bash
echo "root" | passwd --stdin root

echo
echo
echo "ROOT LOGIN IS NOW ENABLED!"
echo
echo
sleep 10

Make sure that the hook set as executable!

SUSE Openstack Cloud supports only pre and post scripts. discovered is the state – discovery or hardware-installed should also work.

BTW: You can also create custom control.sh-script (and also hooks) for a node!

mkdir /updates/d52-54-00-9e-a6-90.cloud.default.net/
cp /updates/control.sh /updates/d52-54-00-9e-a6-90.cloud.default.net/

Some random notes – discovery/install

default pxelinux configuration
(see http://admin-node:8091/discovery/pxelinux.cfg/)

DEFAULT discovery
PROMPT 0
TIMEOUT 10
LABEL discovery
  KERNEL vmlinuz0
  append initrd=initrd0.img crowbar.install.key=machine-install:34e4b23a970dbb05df9c91e0c1cf4b512ecaa7b839c942b95d86db1962178ead69774a9dc8630b13da171bcca0ea204c07575997822b3ec1de984da97fca5b84 crowbar.hostname=d52-54-00-8b-c2-17.cloud.default.net crowbar.state=discovery
  IPAPPEND 2

allocated node

The sleshammer-image will wait for this entry (.*_install) on the admin-node once you allocate a node.

DEFAULT suse-11.3_install
PROMPT 0
TIMEOUT 10
LABEL suse-11.3_install
  KERNEL ../suse-11.3/install/boot/x86_64/loader/linux
  append initrd=../suse-11.3/install/boot/x86_64/loader/initrd   crowbar.install.key=machine-install:34e4b23a970dbb05df9c91e0c1cf4b512ecaa7b839c942b95d86db1962178ead69774a9dc8630b13da171bcca0ea204c07575997822b3ec1de984da97fca5b84 install=http://192.168.124.10:8091/suse-11.3/install autoyast=http://192.168.124.10:8091/nodes/d52-54-00-8b-c2-17.cloud.default.net/autoyast.xml ifcfg=dhcp4 netwait=60
  IPAPPEND 2

Gentoo – initramfs with busybox, lvm and some more…

Preparations

mkdir -p /usr/src/initramfs/{bin,lib/modules,dev,etc,mnt/root,proc,root,sbin,sys}
cp -a /dev/{null,console,tty,sda*} /usr/src/initramfs/dev/

busybox

USE="static make-symlinks -pam -savedconfig" emerge --root=/usr/src/initramfs/ -av busybox

LVM
LVM provides already a static binary 🙂

cp /sbin/lvm.static /usr/src/initramfs/lvm

ldap initial configuration

A more or less initial configuration for openldap (>2.4)

##
# to import run:
# ldapmodify -Y EXTERNAL -H ldapi:/// -f $filename
#
# to verfiy run:
# ldapsearch -Y EXTERNAL -H ldapi:/// -b "olcDatabase={1}hdb,cn=config"
#
# to create a password:
# slappasswd -h {SSHA} -s admin
##

dn: olcDatabase={1}hdb,cn=config
changetype: modify
replace: olcSuffix
olcSuffix: dc=example,dc=de
-
replace: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange by dn="cn=admin,dc=example,dc=de" write by anonymous auth by self write by * none
olcAccess: {1}to dn.base="" by * read
olcAccess: {2}to * by self write by dn="cn=admin,dc=example,dc=de" write by * read
-
replace: olcRootDN
olcRootDN: cn=admin,dc=example,dc=de
-
replace: olcRootPW
olcRootPW: {SSHA}4RHgrU6ghLqA21CNI8biQblHtEodToyd

TLS config

dn: cn=config
changetype: modify
add: olcTLSCipherSuite
olcTLSCipherSuite: AES128+EECDH:AES128+EDH
-
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/ssl/ca.crt
-
add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/ssl/cert.crt
-
add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/ssl/keyfile.key
-
add: olcTLSVerifyClient
# never - allow - try - demand
olcTLSVerifyClient: demand

Refs
openldap – tls config
openldap – access

Monitoring a fritzbox via upnp – FritzOS 6.20 // FritzBox 7490

curl "http://fritz.box:49000/igdupnp/control/WANCommonIFC1" -H "Content-Type: text/xml; charset="utf-8"" -H "SoapAction:urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1#GetAddonInfos" sion='1.0' encoding='utf-8'?> <s:Envelope s:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/' xmlns:s='http://schemas.xmlsoap.org/soap/envelope/'> <s:Body> <u:GetAddonInfos xmlns:u='urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1' /> </s:Body> </s:Envelope>"
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:GetAddonInfosResponse xmlns:u="urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1">
<NewByteSendRate>0</NewByteSendRate>
<NewByteReceiveRate>0</NewByteReceiveRate>
<NewPacketSendRate>0</NewPacketSendRate>
<NewPacketReceiveRate>0</NewPacketReceiveRate>
<NewTotalBytesSent>270290911</NewTotalBytesSent>
<NewTotalBytesReceived>1577019202</NewTotalBytesReceived>
<NewAutoDisconnectTime>0</NewAutoDisconnectTime>
<NewIdleDisconnectTime>1</NewIdleDisconnectTime>
<NewDNSServer1>81.xxx.xxx.xxx</NewDNSServer1>
<NewDNSServer2>81.xxx.xxx.xxx</NewDNSServer2>
<NewVoipDNSServer1>81.xxx.xxx.xxx</NewVoipDNSServer1>
<NewVoipDNSServer2>81.xxx.xxx.xxx</NewVoipDNSServer2>
<NewUpnpControlEnabled>0</NewUpnpControlEnabled>
<NewRoutedBridgedModeBoth>0</NewRoutedBridgedModeBoth>
</u:GetAddonInfosResponse>
</s:Body>
</s:Envelope>
curl "http://fritz.box:49000/igdupnp/control/WANIPConn1" -H "Content-Type: text/xml; charset="utf-8"" -H "SoapAction:urn:schemas-upnp-org:service:WANIPConnection:1#GetExternalIPAddress" -d "<?xml version='1.0' encoding='utf-8'?> <s:Envelope s:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/' xmlns:s='http://schemas.xmlsoap.org/soap/envelope/'> <s:Body> <u:GetExternalIPAddress xmlns:u='urn:schemas-upnp-org:service:WANIPConnection:1' /> </s:Body> </s:Envelope>"
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:GetExternalIPAddressResponse xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1">
<NewExternalIPAddress>xxx.xxx.xxx.xxx</NewExternalIPAddress>
</u:GetExternalIPAddressResponse>
</s:Body>
</s:Envelope>
curl "http://fritz.box:49000/igdupnp/control/WANCommonIFC1" -H "Content-Type: text/xml; charset="utf-8"" -H "SoapAction:urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1#GetCommonLinkProperties" -d "<?xml version='1.0' encoding='utf-8'?> <s:Envelope s:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/' xmlns:s='http://schemas.xmlsoap.org/soap/envelope/'> <s:Body> <u:GetCommonLinkProperties xmlns:u='urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1' /> </s:Body> </s:Envelope>"
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:GetCommonLinkPropertiesResponse xmlns:u="urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1">
<NewWANAccessType>DSL</NewWANAccessType>
<NewLayer1UpstreamMaxBitRate>11964000</NewLayer1UpstreamMaxBitRate>
<NewLayer1DownstreamMaxBitRate>100014000</NewLayer1DownstreamMaxBitRate>
<NewPhysicalLinkStatus>Up</NewPhysicalLinkStatus>
</u:GetCommonLinkPropertiesResponse>
</s:Body>
</s:Envelope>

apache and tomcat – packetsize per request

During a consulting project i had some trouble with the following environment

Infrastructure

Loadbalancer >> Apache >> Tomcat / JBoss

Each request ended in a 400 Bad Request but the configuration looks good and works fine with other projects.

In the end we made some configuration modifications to the apache and tomcat.

Configuration

Apache

LimitRequestFieldSize 16384
ProxyIOBufferSize 16384

Tomcat

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector address="10.a.b.c" port="8009" protocol="AJP/1.3" redirectPort="8443" packetSize="16384" />

Refs

Tomcat AJP Docu
Apache 2.2 mod_proxy

systemd – abstract

Rescue Mode

cmdline=…systemd.unit=rescue.target

Analyzing the boot process

* systemd-analyze
* systemd-analyze blame
* systemd-analyze plot > /tmp/plot.svg

Start/Stop/Disable services

* systemctl start/stop/restart/mask [service]
* systemctl daemon-reload
* systemctl list-units –type=[timer,service,target,mounts,…]

Journal

* journalctl -u ssh
* _PID=1
* -b

Custom unit

http://www.freedesktop.org/software/systemd/man/systemd.unit.html