Do I need an Availability Group Listener if I have "Failover Partner" in my connection string?
Failover partner is for when you are using database mirroring, you should not use it for Availability Groups. While it, strictly speaking, will work, it will only do so when the primary server is offline, and is not designed for this kind of scenario.

Using a listener within an Availability Group is the way to go. This is a virtual network resource that will point you to the correct primary (and is required in the event that you want to use read-routing at any point in the future). Using a listener you should not have to perform an app restart to have client redirection, it should just work.