Adding time as a feature with xgboost/random forests
You have already constructed features that indicate relevant events like weekday, holiday and so on, this is good. Create as many such features you can come up with.

As for the time of day, I'd try using the hour only and combination of hour+minute. Probably less important is the minute alone, unless there is something that causes spikes at the same times of every hour.

Since you use tree-based models, you don't need to normalize. Mark your time features as categorical (can you do that in xgboost? I forgot, but I think you can in lightgbm) or as boolean where applicablen. And then check feature importances, which you can with gradient boosted tree type of models.

Finally, make sure your validation splits are good, which can be a challenge whenever you have some kind of time involved. Make sure your don't leak information and that your validation set is representative.