Determining the secret using the Blakley's scheme?
I don't see any problem with your matrix, it is invertible. 

Once you've setup the matrix, you can work out the solution of your system. Just don't forget you are working modulo 87, so:

$$
\begin{aligned}&amp;
\begin{pmatrix}
    7       &amp; 84 &amp; -1  \\
    17       &amp; 5 &amp; -1  \\
    37   &amp; 46 &amp; -1 
\end{pmatrix} 
\begin{pmatrix} x_{0} \\ y_{0} \\ z_{0} \end{pmatrix}
\equiv
\begin{pmatrix} -41 \\ -3 \\ -66 \end{pmatrix}
\pmod{87} \\
\equiv&amp;
\begin{pmatrix}
    7       &amp; 84 &amp; 86  \\
    17       &amp; 5 &amp; 86  \\
    37   &amp; 46 &amp; 86 
\end{pmatrix} 
\begin{pmatrix} x_{0} \\ y_{0} \\ z_{0} \end{pmatrix}
\equiv
\begin{pmatrix} 46 \\ 84 \\ 21\end{pmatrix}
\pmod{87}
\end{aligned}
$$ 

And for example the inverse of $5 \bmod{87}$ is $35$, so I would multiply $\bmod{87}$ the second row with $35$ to get a $1$ in the second column:
$$\begin{pmatrix}
    7       &amp; 84 &amp; 86  \\
    \color{red}{73}       &amp; \color{red}{1} &amp; \color{red}{52}  \\
    37   &amp; 46 &amp; 86 
\end{pmatrix} 
\begin{pmatrix} x_{0} \\ y_{0} \\ z_{0} \end{pmatrix}
\equiv
\begin{pmatrix} 46 \\ \color{red}{69} \\ 21\end{pmatrix}
\pmod{87}$$

As mentioned, since the matrix has an invertible determinant $\bmod{87}$, the matrix is invertible, so if there is a solution (don't worry, there is one), you can find it using Gaussian Elimination, I think you should try it too, until you obtain in the end a system of the form:
$$
\begin{pmatrix}
    1   &amp; 0 &amp; 0  \\
    0   &amp; 1 &amp; 0  \\
    0   &amp; 0 &amp; 1 
\end{pmatrix} 
\begin{pmatrix} x_{0} \\ y_{0} \\ z_{0} \end{pmatrix}
\equiv
\begin{pmatrix} ?? \\ ?? \\ ??\end{pmatrix}
\pmod{87}
$$ 

PS: Note that the rank's theorem for a matrix does not exist in a commutative ring.