Remove/Hide the block if the View is empty?
I had the same behavior with "empty" fields. Unless you don't hide the fields themselves, the design/ markup still renders.
Solution for me was to filter for fields being "NOT NULL".
(Maybe it depends on the "fences"-module i have installed, too. Not sure, because not tested ...)