Remove a single tab on a new Finder Window (Yosemite)
Cmd ⌘   Shift ⇧   T  will show/hide the tab bar (if there is not more than one tab already).

It appears that if you close a Finder window with the Tab bar open, then subsequent window will start with it open, even for a single tab.  

So I guess the trick is to always close tabbed windows one tab at a time, or remember the above key-combo for when you forget.