While the formula language is great for interactive model-fitting and exploratory data analysis, there are times when we want a different or more systematic interface for creating design matrices. If you ever find yourself writing code that pastes together bits of strings to create a formula, then stop! And read this chapter.
Our first option, of course, is that we can go ahead and write some code to construct our design matrices directly, just like we did in the old days. Since this is supported directly by dmatrix() and dmatrices(), it also works with any third-party library functions that use Patsy internally. Just pass in an array_like or a tuple (y_array_like, X_array_like) in place of the formula.
However, there is also a middle ground between pasting together strings and going back to putting together design matrices out of string and baling wire. Patsy has a straightforward Python interface for representing the result of parsing formulas, and you can use it directly. This lets you keep Patsy’s normal advantages – handling of categorical data and interactions, predictions, term tracking, etc. – while using a nice high-level Python API. An example of somewhere this might be useful is if, say, you had a GUI with a tick box next to each variable in your data set, and wanted to construct a formula containing all the variables that had been checked, and letting Patsy deal with categorical data handling. Or this would be the approach you’d take for doing stepwise regression, where you need to programatically add and remove terms.
And then wrap the whole thing up in a ModelDesc.
# something there if we were calling dmatrices.
Notice that no intercept term is included. Implicit intercepts are a feature of the formula parser, not the underlying represention. If you want an intercept, include the constant INTERCEPT in your list of terms (which is just sugar for Term()).
Another option is to just pass your term lists directly to design_matrix_builders(), and skip the ModelDesc entirely – all of the highlevel API functions like dmatrix() accept DesignMatrixBuilder objects as well as ModelDesc objects.
# if one of the column names in extra_columns is named like "weight.in.kg"
# or "sys.exit()" or "LittleBobbyTables()".
If LookupFactor and EvalFactor aren’t enough for you, then you can define your own factor class.
This must return a short string describing this factor. It will be used to create column names, among other things.
A patsy.Origin if this factor has one; otherwise, just set it to None.
If your factor will ever contain categorical data or participate in interactions, then it’s important to make sure you’ve defined __eq__() and __ne__() and that your type is hashable. These methods will determine which factors Patsy considers equal for purposes of redundancy elimination.
Return the number of passes through the data that this factor will need in order to set up any Stateful transforms.
If you don’t want to support stateful transforms, just return 0. In this case, memorize_chunk() and memorize_finish() will never be called.
state is an (initially) empty dict which is maintained by the builder machinery, and that we can do whatever we like with. It will be passed back in to all memorization and evaluation methods.
eval_env is an EvalEnvironment object, describing the Python environment where the factor is being evaluated.
Called repeatedly with each ‘chunk’ of data produced by the data_iter_maker passed to design_matrix_builders().
state is the state dictionary. which_pass will be zero on the first pass through the data, and eventually reach the value you returned from memorize_passes_needed(), minus one.
Called once after each pass through the data.
Evaluate this factor on the given data . Return value should ideally be a 1-d or 2-d array or Categorical() object, but this will be checked and converted as needed.
Do not store evaluation-related state in attributes of your factor object! The same factor object may appear in two totally different formulas, or if you have two factor objects which compare equally, then only one may be executed, and which one this is may vary randomly depending on how build_design_matrices() is called! Use only the state dictionary for storing state.
eval() is called zero or more times.
Define the magic __patsy_get_model_desc__ method on your formula object. dmatrix() and friends check for the presence of this method on any object that is passed in, and if found, it is called (passing in the EvalEnvironment), and expected to return a ModelDesc. And your ModelDesc can, of course, include your special factor object(s).
The only downside to this approach is that you can’t control the names of individual columns. (A workaround would be to create multiple terms each with its own factor that returns a different pieces of your overall matrix.) If this is a problem for you, though, then let’s talk – we can probably work something out.