Removes the connection between Actor and the given entity. Does all the required book keeping (as opposed to straight up RemoveHandleForActor call). If the Handle doesn't match Actor no action is taken.