Do the rows of the design matrix refer to the observations or predictors?
Based on my understanding, in common machine learning, we formulate our matrices, from vectors, as rows to be observations, columns to be predictors.

The rows (or, in general, the first dimension of your tensor) are typically the observations. For example, in TensorFlow, the first dimension of the input tensor typically refers to the batch size, i.e. the number of observations. If you are using Pandas (a Python library to manipulate data), the rows are typically the observations and the columns are the predictors.
However, in general, it does not really matter which convention you use, provided that you use one of the conventions consistently in your implementation (i.e. you choose one of the conventions and you stick with it throughout all your code, to avoid complexity), and make it clear in the documentation. So, you can have a matrix where either the rows or columns are observations and, consequently, the columns or, respectively, rows are the features (aka predictors or independent variables).
Anyway, it is probably a good idea to be consistent with existing literature and implementations/libraries, so you should probably use the rows for the observations.