Performance issues from matrix vs rich text?
Matrix does have additional overhead, but it's unlikely that it is the sole reason for any slowdown.

If you enable devMode, you can see profiling information for SQL queries as well as code execution in your browser's console to try and track down where any slowness is occurring.

If it is ultimately coming from you template, you can look into the {% cache %} tag to help with initial load time.