Does using Views & Panels on every page will effect site performance (speed)?
Panels is actually sometimes faster than core is on it's own.

http://www.codeenigma.com/en/blog/performance-panels

And that's without utilizing caching, something Panels can do much better than core, because it allows you to do it from on a page by page basis, down to on a block by block basis.