Is this an abandoned wasp's nest?
Wasps only use these nests for a year and then abandon them.  If you haven't seen any activity it should be safe to just knock it down.  By the looks of your picture and your description it does seem like an old, unused nest.  Have a fly swatter with you to be safe, but a nest that small isn't going to have wasps pouring out of it.  I'm sure you could just run around the corner to get away even if there was a stray wasp or two.