Customized Search Results Page
It sounds like you are almost there but without writing custom templates for each content type I would look at display suite. That way, when you click manage display for each different content type you can choose "custom display settings" and then "search result". This will let you build a different layout for each different content type as a search result (and if you like eventually build your own template files easily based on display suites templates).

Then you go back to your view and choose "Show: Display Suite -> Search Result" and each result will be shown as you have set up in the manage display screen for that type. You can still use all the filters and sorts as you like.