Dimensions of Transformer - dmodel and depth
d_model is the dimensionality of the representations used as input to the multi-head attention, which is the same as the dimensionality of the output. In the case of normal transformers, d_model is the same size as the embedding size (i.e. 512). This naming convention comes from the original Transformer paper.

depth is d_model divided by the number of attention heads (i.e. 512 / 8 = 64). This is the dimensionality used for the individual attention heads. In the tutorial you linked, you can find this as self.depth = d_model // self.num_heads. Each attention head projects the original representation into a smaller representation of size depth, then computes the attention, and then all the attention head results are concatenated together, so that the final dimensionality is again d_model. You can find more details on the individual computations in this other answer.


Note that the implementation of the multi-head attention in the tutorial is not a straightforward implementation from the original paper but it is equivalent: in the original paper, there are different matrices $W_i^Q, W_i^K, W_i^V$ for each attention head $i$, while in the implementation of the tutorial there are combined matrices $W^Q, W^K, W^V$ that compute the projection for all attention heads, which is then split into the separate heads by means of the function split_heads.