According the the paper, VQ-VAE goes through two stage training. First to train the encoder and the vector quantization and then train an auto-regressive model for discrete estimation.

```
commitment_loss = self.beta * tf.reduce_mean(
            (tf.stop_gradient(quantized) - x) ** 2
        )
        codebook_loss = tf.reduce_mean((quantized - tf.stop_gradient(x)) ** 2)
        self.add_loss(commitment_loss + codebook_loss)

```

Through out the training there is a reconstruction loss term with a VQ loss term

My question is why do we not replace the VQ posterior with the autoregressive model and give it the VQ-Loss and it should optimize for that same estimation.

I ask this because VQ process is a modified version of K-means clustering for a specific orientation.
clustering could be estimated with generative models. Since an auto-regressive model is a generative model and it is used to estimate discrete values given encoded X and discrete values as labels for them. (I am referring the the second stage training).

A clustering generative model with the VQ Loss as a training signal should hypothetically reach the same goal.

I might be in over my head as I am no Deep learning or Statistics expert.

If my question is not clear enough please let me know.