AUC-ROC of a random classifier
Premise:
The confusion matrix that you mention above is only correct if

there is as much Positive and Negative cases in the data
we use a random classifier that assigns Positive and Negative class both with probability 0.5

So you derive the confusion matrix for a very particular random classifier for a very particular dataset. This very particular random classifier is indeed, as you point out, only a point in the ROC plot. To obtain the whole ROC curve, we have to vary the probability with which we assign the positive class, from 0 to 1.
So in effects, the ROC curve is a graphical evaluation of the performance of infinitely many classifiers!
Each one of these random classifiers with a different probability will have a different expected confusion matrix.

Derivation: Here I derive the AUC of a random classifier on a dataset with an arbitrary class imbalance.
Assuming that there is a fraction $x$ of positive cases and a fraction $1-x$ of negative cases, and that our classifier consists of randomly assigning the positive class with probability $\rho$ and the negative class with probability $1-\rho$. The confusion matrix of a random classifier will have the following expected proportions
$$TP = \rho x \\ FP = \rho (1-x) \\ FN = (1-\rho)x \\ TN = (1-\rho)(1-x)$$
Then we calculate the True Positive Rate (sensitivity) and False Positive Rate (1-specificity) of our random classifier
$$TPR=\frac{TP}{TP+FN}=\frac{\rho x}{\rho x + (1-\rho)x}=\rho\\ FPR=\frac{FP}{TN+FP}=\frac{\rho(1-x)}{\rho(1-x)+(1-\rho)(1-x)}=\rho$$
So as you can see TPR and TFR do not depend on the class proportion $x$, which means that the ROC-AUC will also be independent of $x$. Moreover TPR=FPR, meaning that the ROC-curve will look like a straight line from (0,0) to (1,1). Now we calculate the AUC as the integral between 0 and 1 of the area under TPR as a function of FPR as we vary the threshold $\rho$.
$$AUC=\int_0^1 \rho d\rho' = \frac{\rho'^2}{2}|_0^1 = 1/2$$
So the area under the ROC curve for a random classifier is 0.5 regardless of the class proportion.