Simple Imputer cannot impute by column
I don't think it's trying to impute across rows; rather, it's trying to impute in the dates column.  You may want to use ColumnTransformer to select which columns to impute.