A few days ago our Ceph Cluster died - nobody knows why, but it seems that the monitors just got deleted. When trying to re-add them to the ceph.conf we get timeouts when starting ceph and these errors:7f1b04382700  0 -- :/3908692206 >> 172.16.50.2:6789/0 pipe(0x7f1b0005a060 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f1b0005e350).fault(172.16.50.2 is a node where a ceph monitor should be running...)Since we were not able to restart the cluster (Proxmox) we wanted to extract the images from all our VMs from the Ceph OSDs. To do so I wrote a script collecting information about the data from the OSDs and reassemble the collected files.The files have names likecurrent/1.382_head/DIR_2/DIR_8/DIR_3/rbd\udata.147082238e1f29.0000000000009238__head_CB25F382__1Where 147082238e1f29 seems to be the ID of the VM-Image and 0000000000009238 seems to be the offset. So I listed all files with the same ID and sorted them respecting their offset.The problem is: There are gaps... Not all offsets are present on the OSDs.So when putting all these files together with dd, the image seems to be corrupted - because there are gaps.Can anyone help us with the timeouts when starting ceph or at least help us with extracting the VM images?
Recover VM Images from dead ceph cluster
data recovery;proxmox;ceph
After some more prayers I actually got it:I specified a wrong block size and I changed to copying with offsets instead of inserting 0-blocks.To extract images from OSDs only (nothing else working):https://gitlab.lbader.de/kryptur/ceph-recovery/tree/masterNow I am able to extract the VMs from the OSDs and copy the images to a fresh cluster.