I'm currently working on quantum computing. 
In a literature source that I have, it is said that the following matrix $U$ is unitary. 
My question is, how can we show that in this matrix? I know that a matrix is unitary if:
$U^{\*}U=I$

The matrix is an NxN matrix:

$U=\begin{pmatrix} -1+\frac{2}{N} & \frac{2}{N} & ...& \frac{2}{N} \\ 
\frac{2}{N} & -1+\frac{2}{N} & ...& \frac{2}{N} \\ \vdots & \vdots & \ddots & \vdots \\
\frac{2}{N} & \frac{2}{N} & ...& -1+\frac{2}{N} \end{pmatrix} $

my first thought would be to pull the 1 out:

$U=\begin{pmatrix} \frac{2}{N} & \frac{2}{N} & ... & \frac{2}{N} \\ 
\frac{2}{N} & \frac{2}{N} & ... & \frac{2}{N} \\ \vdots & \vdots & \ddots & \vdots \\
\frac{2}{N} & \frac{2}{N} & ... & \frac{2}{N} \end{pmatrix} - I $

Then you would have to show now that the matrix is unitary, but I can not get any further here.

I would be very happy if someone could help me to show that the matrix is unitary. I hope that my question is clear and understandable.