where 𝑓 𝑖𝑗 𝛼𝛽 and 𝑓 𝑖 𝑎, 𝑓 𝑏 𝛽 are the second and first-order statistics for columns i, j of the alignment, respectively. Each covariance term measures the difference between the joint frequency for pairs of amino acids and the product of the frequencies of the residues at each site, i.e. the expected counts under the statistical hypothesis of independence. Therefore, zero 𝐶 𝑖𝑗 𝛼𝛽 for all 𝛼𝛽 would imply frequences one would observe if positions i, j were independent. Reproducing pairwise covariance in protein alignments is an important aspect of generative models as it measures how well the model captures interactions between distant amino acids, an essential indicator for the likely stability and function of the generated proteins .