Some Applications stay in Dock after closing
I don't think it' malware. I have the same problem after installing 10.11.5. 
I found a (not so nice) workaround: 

When the app is still running: Save all changes and force quit the Application

When already quit the app: Open the Activity Monitor and force quit the Widowserver Process. You'll be logged out. Now restart the mac (don't just log in again) and you can use it without having the icons in the dock. 

I think it's an 10.11.5 issue.