I cannot see Time Machine in Finder to eject it
If you want to be safe, power down your Mac and then disconnect the drive. If something is stuck, the os will alert you to the specific program so you can decide to wait. Most drives can be repaired in disk utility over night, but time machine drives can take days to "repair" so I would recommend not pulling the drive if you are uncertain that it ejected cleanly. 

The restart also fixes 95% of the cases where thing should like this happen and your Finder sidebar should be accurate again.