ceph - down pgs after lost osd

Posted on Wed 19 August 2020 in Ceph • 1 min read

Initial situation / Issue

  • pool size = 1 (Don't ask why)
  • lost/crashed osd
  • pgs down/incomplete


$ ceph pg <id> query
    "snap_trimq": "[]",
    "snap_trimq_len": 0,
    "state": "active+clean",
    "epoch": 348,
    "acting": [
         "blocked": "peering is blocked due to down osds",
         "down_osds_we_would_probe": [
         "peering_blocked_by": [
               { "osd": 60,
                 "current_lost_at": 0,
                 "comment": "starting or marking this osd lost may let us proceed"


Mark the missing osd as lost:

$ ceph osd lost osd.<id> --yes-i-really-mean-it

You can try to create the pg:

$ ceph pg force-create-pg <id>

But because the PG is already created, this will usually fail or have no effect.

In the meantime ceph already tries to start the PG on a new OSD - without success (PG is incomplete). Therefore the faulty PG must be created/marked as an empty PG (data loss!)

systemctl stop ceph-osd@87
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-87 \
  --pgid <id> --op info
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-87 \
  --pgid <id> --op mark-complete
systemctl start ceph-osd@87

Root Cause

The pool size is == 1.