Removing USB Plugs Daily
USB is hot-pluggable, so you don't actually have to power down to remove/insert those (I don't know about HDMI though, so I'd keep powering down). 

From an ELECTRICAL/ELECTRONIC/SOFTWARE standpoint there is NO problem with doing this. None of the equipment or the operating systems care.

From a MECHANICAL standpoint, you are adding wear and tear to the connectors. I did a quick search and I think your average USB connector is rated at somewhere in the neighborhood of 10,000 insertions. (That's two a day for 13 years). Some will wear out quicker than rated, some longer, but I'd say your computer USB connectors won't (from wear) be the first things to go. The hard drives will probably die first.

My bigger worry at that point would actually be that you're taking an increased risk of accidentally screwing something up by twisting a plug or accidentally leaning on a connector , or whatever. As long as you are careful, you should be able to do this safely for as long as you can stand swapping the cables back and forth.