How to improve search performance on front-end?
Have you looked into eager loading the Matrix field if you are on 2.6? This should vastly cut down on the number of queries needed.

The other thought could be to use the Preparse field plugin to push the matrix fields into simple text, and then direct the search query to only look at the title field and preparse fields.