How to solve ADMM for TV Minimization Problem For Different Sizes $A$ and $x$ in $Ax=b$
The Error in the Model

The problem is in the dimensions of the Linear Operator $ A $ in your model compared to the Data Matrix $ X $. The number of columns of the matrix $ A $ must match the number of pixels in each column of $ X $ (Each image). While in your case it matches the number of images.

Matrix Form of 2D Linear Operator

Let's try to build the model correctly. Assume our data (Images) is given by the set of 2D matrices $ {\left\{ {X}_{i} \right\}}_{i = 1}^{m} $ where $ {X}_{i} \in \mathbb{R}^{p \times q} $. Given a Linear Operator $ \mathcal{A}: \mathbb{R}^{p \times q} \to \mathbb{R}^{k \times l} $ and we have $ {B}_{i} = \mathcal{A} \left( {X}_{i} \right) $.

In order to create the matrix form we should represent the Linear Operator by the matrix $ A \in \mathbb{R}^{\left( k l \right) \times \left( p q \right)} $ which is the matrix which applies the operator on the Column Stacked (See Vectorization Operator) vectors. So we have $ \boldsymbol{x}_{i} = \operatorname{Vec} \left( {X}_{i} \right) $ and $ \boldsymbol{b}_{i} = \operatorname{Vec} \left( {B}_{i} \right) $.

By defining $ X = \left[ \boldsymbol{x}_{1}, \boldsymbol{x}_{2}, \ldots \boldsymbol{x}_{m} \right] $ and $ B = \left[ \boldsymbol{b}_{1}, \boldsymbol{b}_{2}, \ldots \boldsymbol{b}_{m} \right] $ we have $ A X = B $.

With this definition everything will work as expected. Same logic will work for $ F $.

Remark

If you want $ F $ to work along the rows of $ X $ then set $ Z = F {X}^{T} $.

ADMM for the Vector Case

First, pay attention that the ADMM works on vectors (You can work with Matrices but then you need to update the Prox operations accordingly.

So $ \boldsymbol{x} \in \mathbb{R}^{n}, A \in \mathbb{R}^{m \times n}, \boldsymbol{b} \in \mathbb{R}^{m} $ and $ F \in \mathbb{R}^{o \times n} $ where $ o $ is set to match the operation of TV on the image columns and rows which $ \boldsymbol{x} $ is a column stack of.

So now the terms do align propely: $ {\left( {A}^{T} A + p {F}^{T} F \right)}^{-1} \in \mathbb{R}^{n \times n} $ and $ \left( {A}^{T} \boldsymbol{b} + p {F}^{T} \left( \boldsymbol{x} - \boldsymbol{u} \right) \right) \in \mathbb{R}^{n} $. So the calculation of $ \boldsymbol{x}^{k} $ is well defined.

Given that $ \boldsymbol{z} \in \mathbb{R}^{o} $ and $ \boldsymbol{u} \in \mathbb{R}^{o} $ their calculation is also well defined.

I guess something in your code doesn't match. But if the code have the terms $ A \boldsymbol{x} - \boldsymbol{b} $ and $ F \boldsymbol{x} $ well defined then everything else will work.