ceph – wrong osd id with lvm+filestore

Not sure why…but i’ve found a strange ceph-volume behavior with lvm and filestore.

ceph-volume lvm list shows the wrong osd id while the affected osd is online with a another id.

$ mount | grep ceph-2
/dev/mapper/vg00-datalv1 on /var/lib/ceph/osd/ceph-2 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
$ cat /var/lib/ceph/osd/ceph-2/whoami 
$ sudo ceph osd metadata osd.2 | egrep "id|objectstore"
    "id": 2,
    "osd_objectstore": "filestore",
$ sudo ceph-volume lvm list
====== osd.8 =======

  [data]    /dev/vg00/datalv1

      type                      data
      journal uuid              XqM6CP-embw-gIfs-UN2Q-gRDR-TVWP-y1q5Te
      osd id                    8
      cluster fsid              ed62dbfb-f0f7-4b13-ace0-4ccea0c4a6bf
      cluster name              ceph
      osd fsid                  38e7bfb3-ad57-4979-b8a9-3f875e6cb6f5
      encrypted                 0
      data uuid                 W3h12f-xg3y-ij1Z-F70h-yx2n-SyD9-ioNEC7
      cephx lockbox secret
      crush device class        None
      data device               /dev/vg00/datalv1
      vdo                       0
      journal device            /dev/vg00/journallv1

  [journal]    /dev/vg00/journallv1

      type                      journal
      journal uuid              XqM6CP-embw-gIfs-UN2Q-gRDR-TVWP-y1q5Te
      osd id                    8
      cluster fsid              ed62dbfb-f0f7-4b13-ace0-4ccea0c4a6bf
      cluster name              ceph
      osd fsid                  38e7bfb3-ad57-4979-b8a9-3f875e6cb6f5
      encrypted                 0
      data uuid                 W3h12f-xg3y-ij1Z-F70h-yx2n-SyD9-ioNEC7
      cephx lockbox secret
      crush device class        None
      data device               /dev/vg00/datalv1
      vdo                       0
      journal device            /dev/vg00/journallv1

And if you try to start the osd via ceph-volume lvm trigger with the “wrong” ID 8 it will…

$ sudo ceph-volume lvm trigger 8-38e7bfb3-ad57-4979-b8a9-3f875e6cb6f5
Running command: mount -t xfs -o rw,noatime,inode64 /dev/vg00/datalv1 /var/lib/ceph/osd/ceph-8
Running command: ln -snf /dev/vg00/journallv1 /var/lib/ceph/osd/ceph-8/journal
Running command: chown -R ceph:ceph /dev/dm-2
Running command: systemctl enable ceph-volume@lvm-8-38e7bfb3-ad57-4979-b8a9-3f875e6cb6f5
Running command: systemctl start ceph-osd@8
--> ceph-volume lvm activate successful for osd ID: 8

$ sudo cat /var/log/ceph/ceph-osd.8.log
2018-07-04 19:28:34.754576 7f346e67fd80  0 set uid:gid to 167:167 (ceph:ceph)
2018-07-04 19:28:34.754598 7f346e67fd80  0 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable), process (unknown), pid 3755
2018-07-04 19:28:34.754872 7f346e67fd80 -1 OSD id 2 != my id 8

FAIL! Same with the correct ID 2…

[vagrant@ceph-osd2 ~]$ sudo ceph-volume lvm trigger 2-38e7bfb3-ad57-4979-b8a9-3f875e6cb6f5
-->  RuntimeError: could not find osd.2 with fsid 38e7bfb3-ad57-4979-b8a9-3f875e6cb6f5

To fix that problem we need to adjust the datatag: ceph.osd_id on the LVM device.

$ sudo lvs -o lv_tags vg00/datalv1
  LV Tags                                                                                                                                                                                                                                                                                                                                                                                                                                     
$ sudo lvs -o lv_tags vg00/journallv1
  LV Tags                                                                                                                                                                                                                                                                                                                                                                                                                                        
  1. Remove the old datatag

    lvchange --deltag ceph.osd_id=8 vg00/datalv1
    lvchange --deltag ceph.osd_id=8 vg00/journallv1
  2. Add the correct datatag

    lvchange --addtag ceph.osd_id=2 vg00/datalv1
    lvchange --addtag ceph.osd_id=2 vg00/journallv1

And et voilà

$ sudo ceph-volume lvm trigger 2-38e7bfb3-ad57-4979-b8a9-3f875e6cb6f5
Running command: mount -t xfs -o rw,noatime,inode64 /dev/vg00/datalv1 /var/lib/ceph/osd/ceph-2
Running command: ln -snf /dev/vg00/journallv1 /var/lib/ceph/osd/ceph-2/journal
Running command: chown -R ceph:ceph /dev/dm-2
Running command: systemctl enable ceph-volume@lvm-2-38e7bfb3-ad57-4979-b8a9-3f875e6cb6f5
 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-2-38e7bfb3-ad57-4979-b8a9-3f875e6cb6f5.service to /usr/lib/systemd/system/ceph-volume@.service.
Running command: systemctl start ceph-osd@2
--> ceph-volume lvm activate successful for osd ID: 2
$ sudo cat /var/log/ceph/ceph-osd.2.log
2018-07-04 19:40:04.075588 7fa9cbf6bd80  0 set uid:gid to 167:167 (ceph:ceph)                                                                                                                                                                                                                                                
2018-07-04 19:40:04.075608 7fa9cbf6bd80  0 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable), process (unknown), pid 4165                                                                                                                                                                     
2018-07-04 19:40:04.080821 7fa9cbf6bd80  0 pidfile_write: ignore empty --pid-file                       
2018-07-04 19:40:04.109636 7fa9cbf6bd80  0 load: jerasure load: lrc load: isa                                                                                                                                                                                                                                                
2018-07-04 19:40:04.110273 7fa9cbf6bd80  0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)                                                                                                                                                                                       
2018-07-04 19:40:04.121305 7fa9cbf6bd80  0 filestore(/var/lib/ceph/osd/ceph-2) start omap initiation

ceph-ansible: minimal containerized deployment (docker)

tested with v3.0.26


monitor_interface: eth1
radosgw_interface: eth1
        osd scrub during recovery: false
ceph_docker_image: "ceph/daemon"
ceph_docker_image_tag: latest
containerized_deployment: true


crush_location: true
osd_crush_location: "\"root={{ ceph_crush_root }} rack={{ ceph_crush_rack }} host={{ ansible_hostname }}\""
osd_objectstore: bluestore
osd_scenario: non-collocated
- /dev/sdb
- /dev/sdc
- /dev/sdd
- /dev/sde
- /dev/sdf
- /dev/sdg
- /dev/sdh
- /dev/nvme0n1
- /dev/nvme0n1
- /dev/nvme0n1
- /dev/nvme0n1
- /dev/nvme0n1
- /dev/nvme0n1
- /dev/nvme0n1

group_vars/mons.yml (optional)

openstack_config: true
  name: images
  pg_num: "{{ osd_pool_default_pg_num }}"
  rule_name: ""
  - "{{ openstack_glance_pool }}"

ovirt/rhev/rhv – custom branding (web ui)

Step 1 – create a custom folder in /etc/ovirt-engine/branding

[root@rev1 branding]# ll
total 0
lrwxrwxrwx. 1 [..] 00-ovirt.brand -> /usr/share/ovirt-engine/branding/ovirt.brand
lrwxrwxrwx. 1 [..] 50-rhev-2.brand -> /usr/share/ovirt-engine/branding/rhev-2.brand
drwxr-xr-x. 3 [..] 99-custom.brand

Step 2 – copy your layout and create a ‘branding.properties’

[root@rev1 branding]# ll 99-custom.brand/
total 8
-rw-r--r--. 1 [..] branding.properties
-rw-r--r--. 1 [..] common.css
drwxr-xr-x. 2 [..] images

Step 3 – branding.properties – don’t remove the version parameter

[root@rev1 branding]# cat 99-custom.brand/branding.properties 
#style sheets.


Step 4 – my example for a common.css – we only want to replace the logo!

[root@rev1 branding]# cat 99-custom.brand/common.css 
/* LoginSectionView.ui.xml:
   app logo, positioned in the top right of login screen */
.obrand_loginPageLogoImage {
    background-image: url(images/logo.png);
    width: 137px;
    height: 44px;
    border: 0px;
    display: block;

Step 5 – restart ovirt-engine and check the sourcecode

# systemctl restart ovirt-engine
<link rel="stylesheet" type="text/css" href="/ovirt-engine/theme/00-ovirt.brand/welcome_style.css">
<link rel="stylesheet" type="text/css" href="/ovirt-engine/theme/99-custom.brand/common.css">

SUSE Openstack Cloud – debugging sleshammer

To get a login shell during the discovery and before the nfs is mounted:


root@admin:~ # crowbarctl proposal edit provisioner default
  "id": "provisioner-default",
  "description": "Created on Thu, 09 Nov 2017 15:43:20 +0100",
  "attributes": {
    "provisioner": {
      "discovery": {
        "append": "DISCOVERY_ROOT_PASSWORD=replace-with-your-password"
root@admin:~ # crowbarctl proposal commit provisioner default

ceph metasearch – elasticsearch backend – part 2


  • ceph cluster (kraken release)
  • elasticsearch

The rgw syncer is only used/triggered in multisite configurations – so we need to setup a second zone for the metasearch.

environment / settings

export rgwhost=""
export elastichost=""
export realm="demo"
export zonegrp="zone-1"
export 1zone="zone1-a"
export 2zone="zone1-b" # used for metasearch
export sync_akey="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1 )"
export sync_skey="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1 )"
export user_akey="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1 )"
export user_skey="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1 )"

setup (see also part1)

create first zone
# radosgw-admin realm create --rgw-realm=${realm} --default
# radosgw-admin zonegroup create --rgw-realm=${realm} --rgw-zonegroup=${zonegrp} --endpoints=http://${rgwhost}:80 --master --default
# radosgw-admin zone create --rgw-realm=${realm} --rgw-zonegroup=${zonegrp} --rgw-zone=${1zone} --endpoints=http://${rgwhost}:80 --access-key=${sync_akey} --secret=${sync_skey} --master --default
# radosgw-admin user create --uid=sync --display-name="zone sync" --access-key=${sync_akey} --secret=${sync_skey} --system
# radosgw-admin period update --commit
# systemctl restart ceph-radosgw@rgw.${rgwhost}
create second zone
# radosgw-admin zone create --rgw-realm=${realm} --rgw-zonegroup=${zonegrp} --rgw-zone=${2zone} --access-key=${sync_akey} --secret=${sync_skey} --endpoints=http://${rgwhost}:81
# radosgw-admin zone modify --rgw-realm=${realm} --rgw-zonegroup=${zonegrp} --rgw-zone=${2zone} --tier-type=elasticsearch --tier-config=endpoint=http://${elastichost}:9200,num_replicas=1,num_shards=10
# radosgw-admin period update --commit

Restart the first radosgw and the start the second radosgw. For example:

# screen -dmS rgw2zone radosgw --keyring /etc/ceph/ceph.client.admin.keyring -f --rgw-zone=${2zone} --rgw-frontends="civetweb port=81"

Check elasticsearch for the new index:

# curl http://${elastichost}:9200/_cat/indices | grep rgw-${realm}
yellow open rgw-demo    z0UiKOOFQl682yILobYbMw 5 1 1 0 11.7kb 11.7kb

modify header/metadata

create a user

radosgw-admin user create --uid=rmichel --display-name="rmichel" --access-key=${user_akey} --secret=${user_skey}

upload some test data….

s3cmd is configured with the ${user_akey} + ${user_skey} and the ${rgwhost}:80 as the endpoint.

# s3cmd modify --add-header x-amz-meta-color:green s3://bucket1/admin.key
modify: 's3://bucket1/admin.key'
# s3cmd info s3://bucket1/admin.key
s3://bucket1/admin.key (object):
   File size: 63
   Last mod:  Thu, 27 Apr 2017 21:14:55 GMT
   MIME type: text/plain
   Storage:   STANDARD
   MD5 sum:   ee40e385a45c4855bd360cfbdbd48711
   SSE:       none
   policy:    <?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>bucket1</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>admin.key</Key><LastModified>2017-04-27T21:14:55.494Z</LastModified><ETag>&quot;ee40e385a45c4855bd360cfbdbd48711&quot;</ETag><Size>63</Size><StorageClass>STANDARD</StorageClass><Owner><ID>rmichel</ID><DisplayName>rmichel</DisplayName></Owner></Contents></ListBucketResult>
   cors:      none
   ACL:       rmichel: FULL_CONTROL
   x-amz-meta-color: green
   x-amz-meta-s3cmd-attrs: uid:0/gname:root/uname:root/gid:0/mode:33152/mtime:1493326171/atime:1493326171/md5:ee40e385a45c4855bd360cfbdbd48711/ctime:1493326171

query elasticsearch

The radosgw creates a index with the name rgw-${realm} (ref ceph.git)

In my case the url is http://${elastichost}:9200/rgw-${realm}/

# curl | python -m json.tool
    "_shards": {
        "failed": 0,
        "successful": 5,
        "total": 5
    "hits": {
        "hits": [
                "_id": "d9b0c7a5-f9e5-4c6e-a0c2-48642840c98b.14125.1:admin.key:",
                "_index": "rgw-demo",
                "_score": 0.23691465,
                "_source": {
                    "bucket": "bucket1",
                    "instance": "",
                    "meta": {
                        "content_type": "text/plain",
                        "custom": {
                            "color": "green",
                            "s3cmd-attrs": "uid:0/gname:root/uname:root/gid:0/mode:33152/mtime:1493326171/atime:1493326171/md5:ee40e385a45c4855bd360cfbdbd48711/ctime:1493326171"
                        "etag": "ee40e385a45c4855bd360cfbdbd48711",
                        "mtime": "2017-04-27T21:14:55.483Z",
                        "size": 63,
                        "x-amz-copy-source": "/bucket1/admin.key",
                        "x-amz-date": "Thu, 27 Apr 2017 21:14:55 +0000",
                        "x-amz-metadata-directive": "REPLACE"
                    "name": "admin.key",
                    "owner": {
                        "display_name": "rmichel",
                        "id": "rmichel"
                    "permissions": [
                "_type": "object"
        "max_score": 0.23691465,
        "total": 1
    "timed_out": false,
    "took": 102

ceph metasearch – elasticsearch backend

Fetch zonegroup configuration (json struct)
# radosgw-admin zonegroup get > /tmp/zonegroup.json

change the tier_type to elasticsearch

Import the configuration
# radosgw-admin zonegroup set --infile /tmp/zonegroup.json
Fetch zone configuration (json struct)
# radosgw-admin zone get > /tmp/zone.json

Add the following parameter endpoint & {url} for the section tier_config

    "tier_config": [
            "key": "endpoint",
            "val": "http:\/\/"
Import the configuration
# radosgw-admin zone set --infile /tmp/zone.json


# radosgw-admin zone modify --rgw-zonegroup={zonegroup-name} --rgw-zone={zone-name} --tier-config=endpoint={url}
Update & Commit
# radosgw-admin period update --commit

to be continued… part2