Dropping one category for regularized linear models
When you do linear regression you have to leave out one column as it's a singular matrix and hence columns are linearly dependent and we cannot calculate the inverse.
But when you do regularization it take cares of singularity. The matrix is almost surely nonsingular. Hence we don't need to drop a column and if you drop different columns from each feature it could lead to different prediction as it would lead to bias.
Refer this.