SUSE Cloud – missing cinder key on computes – part2

I’ve found the root cause for the missing cinder key on the computes.

chef-client output – without any files:

[2017-11-30T09:37:33+01:00] INFO: Processing package[ceph-common] action install (nova::ceph line 50)
[2017-11-30T09:37:33+01:00] INFO: Ceph configuration file is missing; skipping the ceph setup for backend ceph-hdd
[2017-11-30T09:37:33+01:00] INFO: Ceph configuration file is missing; skipping the ceph setup for backend ceph-ssd

chef-client output – only with ceph.conf:

[2017-11-30T09:40:00+01:00] INFO: Processing package[ceph-common] action install (nova::ceph line 50)
[2017-11-30T09:40:00+01:00] INFO: Ceph user keyring wasn't provided for backend ceph-hdd
[2017-11-30T09:40:00+01:00] INFO: Ceph user keyring wasn't provided for backend ceph-ssd

Still not the right secret. The correct name should be “ceph crowbar-#uuid# name”

root@d98-f2-b3-9e-d6-30:~ # virsh secret-list
 UUID                                  Usage
--------------------------------------------------------------------------------
 5b7c1b36-9093-4a13-b14d-da8b8cbdd8a6  ceph client.cinder secret

chef-client output – now with the key and the ceph.conf:

[2017-11-30T09:51:16+01:00] INFO: Processing package[ceph-common] action install (nova::ceph line 50)
[2017-11-30T09:51:16+01:00] WARN: Cloning resource attributes for ruby_block[save nova key as libvirt secret] from prior resource (CHEF-3694)
[2017-11-30T09:51:16+01:00] WARN: Previous ruby_block[save nova key as libvirt secret]: /var/chef/cache/cookbooks/nova/recipes/ceph.rb:94:in `block in from_file'
[2017-11-30T09:51:16+01:00] WARN: Current  ruby_block[save nova key as libvirt secret]: /var/chef/cache/cookbooks/nova/recipes/ceph.rb:94:in `block in from_file'

Yeah. Finally!

root@d98-f2-b3-9e-d6-30:~ # virsh secret-list
 UUID                                  Usage
--------------------------------------------------------------------------------
 5b7c1b36-9093-4a13-b14d-da8b8cbdd8a6  ceph crowbar-5b7c1b36-9093-4a13-b14d-da8b8cbdd8a6 secret
 7003682d-80fe-4258-b2bb-e6c1b628aa5e  ceph crowbar-7003682d-80fe-4258-b2bb-e6c1b628aa5e secret

SUSE Cloud – missing cinder key on computes

2017-11-01 14:30:53.970 27835 ERROR nova.virt.libvirt.driver [instance: c5618826-98cb-4fd6-9d6f-b8899bd320b7] libvirtError: Secret not found: no secret with matching uuid '5b7c1b36-
9093-4a13-b14d-da8b8cbdd8a6'
2017-11-01 14:30:53.970 27835 ERROR nova.virt.libvirt.driver [instance: c5618826-98cb-4fd6-9d6f-b8899bd320b7] 
2017-11-01 14:30:53.971 27835 ERROR nova.virt.block_device [req-9f046c95-fecf-46e5-874d-43b42da1e63f 62169e96ed4b485aa2dfb2ca3235305c 05f20019f1c94952937a7f34087f5471 - - -] [instan
ce: c5618826-98cb-4fd6-9d6f-b8899bd320b7] Driver failed to attach volume 9f33b42f-79ba-472f-8e10-9525f186cde1 at /dev/vdb

Unless you find a key on the compute like (something with crowbar-$ID)

# virsh secret-list 
 UUID                                  Usage
--------------------------------------------------------------------------------
 5b7c1b36-9093-4a13-b14d-da8b8cbdd8a6  ceph crowbar-5b7c1b36-9093-4a13-b14d-da8b8cbdd8a6 secret

you can/have to fix it on your own:

#!/bin/bash

ID="5b7c1b36-9093-4a13-b14d-da8b8cbdd8a6"
# get cinder key from ceph cluster - ceph auth get-key client.cinder
CINDERKEY="AQA4cw1aa2tAAhAAxYl2l/lCaer3squRBdXBYg=="
FILE="<secret ephemeral='no' private='no'><uuid>$ID</uuid><usage type='ceph'><name>client.cinder secret</name></usage></secret>"
FILENAME="/tmp/secret.xml"

for host in 01 02 03 04 05; do
	dest="compute${host}"
	echo "Verifiy host $dest:"
	if ! ssh $dest virsh secret-get-value $ID; then
		echo "Create secret for cinder user."
		ssh $dest "echo \"$FILE\" > $FILENAME"
		ssh $dest virsh secret-define --file $FILENAME
		ssh $dest virsh secret-set-value --secret $ID --base64 $CINDERKEY
	fi
	echo "ok!"	
done

[notepad] ceph journal size/ssd speed

ceph journal size (doc)

osd journal size = {2 * (expected throughput * filestore max sync interval)}

The default for filestore max sync interval is 5 therefore for a 10Gbit network the “perfect” size would be

osd journal size = { 2 * ( 1280 * 5 ) } = 12.5 GB

ceph ssd speed (journal)

The optimum would be sum of all disk seq write speeds – 11 disks with ~110mb/s = ~1210mb/s – an Intel P3520 might would fit.

How many journals per ssd?

Oh thats easy.

Journals = (ssd seq write speed) / (hdd seq write speed)

Journals = 1350 / 115 = ~11

(For the Intel P3520 with 11 hdds)

ROSE Xeon CW (2015) & power2max Rotor 3D+

Rahmen: ROSE Xeon CW 2015

Innenlager: Rotor Pressfit 4630 (PF46-68-30)

Kurbel: Rotor 3D+ mit p2m Spider

Spacer laut Specs: 1x A + 1x E auf der Ds, 1x A auf der NDs

Spacer verbaut: 2x A auf der Ds

Mit den von Rotor vorgesehen Spacern schleift der Spider am Rahmen. Laut ROSE und einem Sportlabor ist es kein Problem den 2.5mm Spacer von der NDs auf die Ds zustecken.

openvswitch and OpenFlow

openflow

Layer 1

ovs-ofctl del-flow BRIDGE
ovs-ofctl add-flow BRIDGE priority=500,in_port=1,actions=output:2
ovs-ofctl add-flow BRIDGE priority=500,in_port=2,actions=output:1
ovs-ofctl dump-flows BRIDGE

Layer 2

ovs-ofctl del-flow BRIDGE
ovs-ofctl add-flow BRIDGE dl_src=00:00:00:00:00:01,dl_dst=00:00:00:00:00:02,actions=output:2
ovs-ofctl add-flow BRIDGE dl_src=00:00:00:00:00:02,dl_dst=00:00:00:00:00:01,actions=output:1
ovs-ofctl add-flow BRIDGE dl_type=0x806,nw_proto=1,actions=flood
ovs-ofctl dump-flows BRIDGE 

Layer 3

ovs-ofctl del-flow BRIDGE
ovs-ofctl add-flow BRIDGE priority=500,dl_type=0x800,nw_src=10.0.0.0/24,nw_dst=10.0.0.0/24,actions=normal
ovs-ofctl add-flow BRIDGE priority=800,ip,nw_src=10.0.0.3,actions=mod_nw_tos=184,normal
ovs-ofctl add-flow BRIDGE arp,nw_dst=10.0.0.1,actions=output:1
ovs-ofctl add-flow BRIDGE arp,nw_dst=10.0.0.2,actions=output:2
ovs-ofctl add-flow BRIDGE arp,nw_dst=10.0.0.3,actions=output:3
ovs-ofctl dump-flows BRIDGE 

Layer 4

ovs-ofctl del-flow BRIDGE 
ovs-ofctl add-flow BRIDGE arp,actions=normal
ovs-ofctl add-flow BRIDGE priority=500,dl_type=0x800,nw_proto=6,tp_dst=80,actions=output:3
ovs-ofctl add-flow BRIDGE priority=800,ip,nw_src=10.0.0.3,actions=normal
ovs-ofctl dump-flows BRIDGE 

 


 

Priority rules

When no priority is set is the default – 32768! Allowed values are from 0 to 65536. A higher priority will match at first.

 


dl_type and nw_proto

dl_type and nw_proto are filters to match a specific network packet. Generally dl_type is for L2 (matches ethertype) and nw_proto (matches IP protocol type) for L3 actions. For example:

dl_type=0x800 – for ipv4 packets

dl_type=0x86dd – for ipv6 packets

dl_type=0x806 and nw_proto=1 – match only arp requests (ARP opcode, see layer 2)

dl_type=0x800 or ip (as keyword, see layer 3) has the same meaning

ip and nw_proto=17 – udp packets

ip and nw_proto=6 – tcp packets


Parameters for actions can be (excerpt)

  • normal – Default mode, OVS acts like a normal L2 switch
  • drop – drops all packets
  • output – define the output port for a packet/rule
  • resubmit – useful for multiple tables, resend a packet to a port or table
  • flood – forword all packets on all port except the port on which it was received
  • strip_vlan – remove a vlan tag from a packet
  • set_tunnel – set a tunnel id (gre & vxlan)
  • mod_vlan_vid – add a vlan tag for a packet
  • learn – complex foo 😉

ovs-ofctl man page


Example from a openstack node (w/ GRE, see table 22) – ovs flows from the br-tun device

[root@node1 ~]# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=1221.218s, table=0, n_packets=0, n_bytes=0, idle_age=1221, priority=0 actions=drop
cookie=0x0, duration=1221.323s, table=0, n_packets=747, n_bytes=54800, idle_age=0, priority=1,in_port=1 actions=resubmit(,2)
cookie=0x0, duration=1220.226s, table=0, n_packets=0, n_bytes=0, idle_age=1220, priority=1,in_port=2 actions=resubmit(,3)
cookie=0x0, duration=1221.126s, table=2, n_packets=0, n_bytes=0, idle_age=1221, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
cookie=0x0, duration=1221.051s, table=2, n_packets=747, n_bytes=54800, idle_age=0, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
cookie=0x0, duration=1220.974s, table=3, n_packets=0, n_bytes=0, idle_age=1220, priority=0 actions=drop
cookie=0x0, duration=1218.706s, table=3, n_packets=0, n_bytes=0, idle_age=1218, priority=1,tun_id=0x3f7 actions=mod_vlan_vid:1,resubmit(,10)
cookie=0x0, duration=1217.462s, table=3, n_packets=0, n_bytes=0, idle_age=1217, priority=1,tun_id=0x442 actions=mod_vlan_vid:2,resubmit(,10)
cookie=0x0, duration=1220.898s, table=4, n_packets=0, n_bytes=0, idle_age=1220, priority=0 actions=drop
cookie=0x0, duration=1220.821s, table=10, n_packets=0, n_bytes=0, idle_age=1220, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0-&gt;NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]-&gt;NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
cookie=0x0, duration=1220.742s, table=20, n_packets=0, n_bytes=0, idle_age=1220, priority=0 actions=resubmit(,22)
cookie=0x0, duration=1220.666s, table=22, n_packets=137, n_bytes=21860, idle_age=13, priority=0 actions=drop
cookie=0x0, duration=1220.093s, table=22, n_packets=610, n_bytes=32940, idle_age=0, hard_age=1217, dl_vlan=2 actions=strip_vlan,set_tunnel:0x442,output:2
cookie=0x0, duration=1219.970s, table=22, n_packets=0, n_bytes=0, idle_age=1219, hard_age=1218, dl_vlan=1 actions=strip_vlan,set_tunnel:0x3f7,output:2

mutt: daily use (still in progress)

Tag messages matching
shift-t -> “search string”

Limit messages matching (pattern)
l > ~T (tagged)
l > ~A (all)
l > ~N (new)
l > ~U (unread)
l > ~F (flagged)
l > “search string”

Random commands
;d > Delete tagged messages
s > Move message
;s > Move tagged messages
b > Bounce messages
w/W > Set/Clear Flag
:source /path/to/muttrc > Reload mutt configuration

Firefox is loading painfully slowly…

aka Firefox, DNS lookups and IPv6 (under Arch)

Since my laptop is running with Arch, sometimes my Firefox makes me crazy when i’m trying to open a new page because it show a message like Looking up heise.de… for minutes !

After googling a few seconds i’ve found someting that is working for me…

  1. Go to about:config
  2. Search for network.dns.disableIPv6
  3. Set it to true

That is only a workaround because i have no time to debug the root cause.