Removing Reminders and Notes from the dock?
The only unremovable Dock icon is the Finder, removing anything else should result in it staying removed.  There is nothing special about Reminders and Notes, and it's always annoyed me that they put in icons for things like launchpad etc when they have system wide keystrokes or gestures for the same task.

One point to bear in mind tho, is that to prevent people accidentally removing items from the dock (accidental drags are apparently more common on Magic Mouse/Trackpad users) the distance you have to drag the icon away from the dock is significantly increased.  Try it, drag an icon away, and see how far it has to travel before you get the "puff of smoke" icon that indicates it will remove it.  Letting go before seeing this cursor change will just boomerang your icon back into the dock.