title: Glusterfs Replace Brick in Dispersed
body: <p>I have a Dispersed Glusterfs volume comprised of 3x bricks on 3x servers. Recently one of the servers experienced a hard drive failure and dropped out of the cluster. I am trying to replace this brick in the cluster but i cant get it to work.</p>
  
  <p>First up here is the version info:</p>
  
  <pre><code>$ glusterfsd --version
  glusterfs 3.13.2
  Repository revision: git://git.gluster.org/glusterfs.git
  Copyright (c) 2006-2016 Red Hat, Inc. &lt;https://www.gluster.org/&gt;
  GlusterFS comes with ABSOLUTELY NO WARRANTY.
  It is licensed to you under your choice of the GNU Lesser
  General Public License, version 3 or any later version (LGPLv3
  or later), or the GNU General Public License, version 2 (GPLv2),
  in all cases as published by the Free Software Foundation.
  </code></pre>
  
  <p>It is running on Ubuntu 18.04. </p>
  
  <p>Here is the existing info:</p>
  
  <pre><code>Volume Name: vol01
  Type: Disperse
  Volume ID: 061cac4d-1165-4afe-87e0-27b213ea19dc
  Status: Started
  Snapshot Count: 0
  Number of Bricks: 1 x (2 + 1) = 3
  Transport-type: tcp
  Bricks:
  Brick1: srv02:/srv/glusterfs/vol01/brick &lt;-- This is the brick that died
  Brick2: srv03:/srv/glusterfs/vol01/brick
  Brick3: srv04:/srv/glusterfs/vol01/brick
  Options Reconfigured:
  nfs.disable: on
  transport.address-family: inet
  </code></pre>
  
  <p>I wish to replace the srv02 brick with a brick from srv05 using the following:</p>
  
  <pre><code>gluster volume replace-brick vol01 srv02:/srv/glusterfs/vol01/brick srv05:/srv/glusterfs/vol01/brick commit force
  </code></pre>
  
  <p>However when I run this command (as root) I get this error:</p>
  
  <pre><code>volume replace-brick: failed: Pre Validation failed on srv05. brick: srv02:/srv/glusterfs/vol01/brick does not exist in volume: vol01
  </code></pre>
  
  <p>As far as I know it should work, srv05 is connected: </p>
  
  <pre><code># gluster peer status
  Number of Peers: 3
  
  Hostname: srv04
  Uuid: 5bbd6c69-e0a7-491c-b605-d70cb83ebc72
  State: Peer in Cluster (Connected)
  
  Hostname: srv02
  Uuid: e4e856ba-61df-45eb-83bb-e2d2e799fc8d
  State: Peer Rejected (Disconnected)
  
  Hostname: srv05
  Uuid: e7d098c1-7bbd-44e1-931f-034da645c6c6
  State: Peer in Cluster (Connected)
  </code></pre>
  
  <p>As you can see srv05 is connected and in the cluster, srv02 is not and disconnected...</p>
  
  <p>All the bricks are the same size on a XFS partitions. The brick on srv05 is empty. </p>
  
  <p>What am I doing wrong? I would prefer not to have to dump the whole FS and rebuild it if possible...</p>
  
  <p><strong>EDIT 2019-01-01:</strong>
  After following this tutorial here: <a href="https://support.rackspace.com/how-to/recover-from-a-failed-server-in-a-glusterfs-array/" rel="nofollow noreferrer">https://support.rackspace.com/how-to/recover-from-a-failed-server-in-a-glusterfs-array/</a> to replace the dead server brick (srv02) with the new one. </p>
  
  <p>The server and brick are recognized by the cluster:</p>
  
  <pre><code># gluster volume status
  Status of volume: vol01
  Gluster process                             TCP Port  RDMA Port  Online  Pid
  ------------------------------------------------------------------------------
  Brick srv02:/srv/glusterfs/vol01/brick      N/A       N/A        N       N/A
  Brick srv03:/srv/glusterfs/vol01/brick      49152     0          Y       21984
  Brick srv04:/srv/glusterfs/vol01/brick      49152     0          Y       16681
  Self-heal Daemon on localhost               N/A       N/A        Y       2582
  Self-heal Daemon on srv04                   N/A       N/A        Y       16703
  Self-heal Daemon on srv03                   N/A       N/A        Y       22006
  </code></pre>
  
  <p>The brick however on the replacement SRV02 is not coming online!</p>
  
  <p>After much searching I found this in the brick log on the new srv02: </p>
  
  <pre><code>[2019-01-01 05:50:05.727791] E [MSGID: 138001] [index.c:2349:init] 0-vol01-index: Failed to find parent dir (/srv/glusterfs/vol01/brick/.glusterfs) of index basepath /srv/glusterfs/vol01/brick/.glusterfs/indices. [No such file or directory]
  </code></pre>
  
  <p>Not at all sure how to fix this one as its a blank brick that I am looking to bring online and heal!</p>