Samsung Galaxy Note 8.0 (GT-N5100) USB connection & charging problem
I think you've hit the nail on the head. 

It sounds like you have a faulty port/connection. If you fiddle around with the cable in the device does it connect/disconnect? Have you tried using other cables etc?

The part that concerns me is that you say it starts to work after 3-4 hours. If this is the case every time; it could be even stranger than purely a faulty connection.