The regular inner product of two vectors is invariant to orthogonal transformations (rota-tion and reflection). In contrast, the RF kernel is not invariant to orthogonal transformation because decision trees generate axis-aligned splits. Therefore, the decision boundary is not equivariant with respect to feature rotation. In other words, it is to be expected that an RF used for calculating the RF kernel will not yield good classification performance when trained on embeddings generated from training an MLP. Therefore, the RF kernel loses its ability to adequately consider label information for partitioning and, in consequence, the ability to consider the label for CKA.