# Residualize a binary variable to remedy multicollinearity?

Imagine a regression model where there is a continuous-valued response variable and three continuous-valued explanatory variables. For concreteness, imagine that we are interested in the effects of "Predictability," "Length," and "Frequenty," on the reading times "RT" of words.

Suppose further that we know that there is collinearity between "Frequency" and the other two explanatory variables. A way of dealing with this in the model is to residualize "Frequency," along the lines of the R code given (where the second line is the form of the model inputted in to a suitable regression algorithm):

r. <- function (formula, ...) rstandard(lm(formula, ...))
RT~Predictability+Length+r.(Frequency~Length+Predictability)


Now suppose that one of the the variables of interest was binary-valued. Let's say that we wanted to run a model with the form:

RT~Education+ReadsDaily


but we found that neither variable had a significant effect when both were included, but when a saturated systematic component, Education*ReadsDaily was used, the model found highly significant results for both coefficients as well as the interaction term. When the two models are compared, the inclusion of the interaction term decreases the deviance by 16.7. When a VIF analysis is run on the model with an interaction terms, very high values are reported:

ReadDailyY            Education          ReadDailyY:Education
8.693957             4.266084            15.607665


The model without an interaction term, which fits poorly, has low values in the VIF analysis:

ReadDailyY  Education
1.227842    1.227842


I think this means that there is multicollinearity between the two predictors (cf. Agresti 2007:138):

...models with several predictors often suffer from multicollinearity – correlations among predictors making it seem that no one variable is important when all the others are in the model. A variable may seem to have little effect because it overlaps considerably with other predictors in the model, itself being predicted well by the other predictors. Deleting such a redundant predictor can be helpful, for instance to reduce standard errors of other estimated effects.

We would, however, like to try to separate their effects on "RT" rather than combining the variables or discarding one, if at all possible, since the distinction between the two variables happens to be of theoretical importance. It would be possible to do something like the following in R with no error messages:

r. <- function(formula) rstandard(glm(family=binomial(),formula))


The residualized "ReadsDaily" parameter in the model would then be continuous-valued. I suspect that this strategy for dealing with collinearity is methodologically suspect, but I do not know enough to give a reason for why or why not the strategy is to be dispreferred.

Agresti, A. (2007) An Introduction to Categorical Data Analysis. Wiley.

• You can run linear regression with binary outcomes if you like. (Though I'd recommend ensuring that Gender is indeed binary, and not stored as a factor.) But why do you want to do this? In your example the Gender coefficient in glm(RT~Predictability+Length+Gender, family=binomial) tells you about the (log) ratio of odds of RT, comparing Genders, in observations among which Predictability and Length have the same values. There's no need to adjust or rezidualize the Gender variable before doing logistic regression – guest Mar 11 '12 at 0:47
• @guest Thanks for this. I realized that the example I gave was poorly chosen, since there is no obvious real-world situation where Gender would correlate with some experimentally controlled variable. I've changed the example to give a better idea of why it appear to be necessary to residualize a binary variable. (If I'm not understanding properly, thanks for being patient ;-) – user9437 Mar 11 '12 at 1:36
• I do not see why multicollinearity is implicated here. If I read this correctly, you note an improvement in the model when you include a new variable: namely, the Education#ReadsDaily interaction. Have you assessed multicollinearity in a quantitative way, such as VIF or condition number, to verify your assumption that multicollinearity is indeed a problem? – whuber Mar 12 '12 at 21:42
• @whuber thanks for your attention. I've edited the post to give figures from running a vif diagnostic on the model. – user9437 Mar 12 '12 at 22:11
{"ft_lang_label":"__label__en","ft_lang_prob":0.92843723,"math_prob":0.8815592,"size":3631,"snap":"2021-04-2021-17","text_gpt3_token_len":823,"char_repetition_ratio":0.11717673,"word_repetition_ratio":0.0,"special_character_ratio":0.21646929,"punctuation_ratio":0.123853214,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9899466,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2021-01-28T02:39:53Z\",\"WARC-Record-ID\":\"<urn:uuid:e6d9f87d-40a2-4dbb-b7ef-9627f936153f>\",\"Content-Length\":\"155541\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:dbc5fa7c-bfec-4d95-8cfa-49f03dbda6ea>\",\"WARC-Concurrent-To\":\"<urn:uuid:8d21ad37-e887-4db6-8d2c-e57a58984c73>\",\"WARC-IP-Address\":\"151.101.129.69\",\"WARC-Target-URI\":\"https://stats.stackexchange.com/questions/24331/residualize-a-binary-variable-to-remedy-multicollinearity\",\"WARC-Payload-Digest\":\"sha1:K3HEY2BME33CETKHVGSPZO23N3NIGUOM\",\"WARC-Block-Digest\":\"sha1:PCUM3T5U5TEC4GK52LT75NZYQY24GXGS\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2021/CC-MAIN-2021-04/CC-MAIN-2021-04_segments_1610704835583.91_warc_CC-MAIN-20210128005448-20210128035448-00230.warc.gz\"}"}