How Do I improve the performance of my Drupal7 Site?
These are notes from my experiences and might vary from what others experience. I predominantly use LAMP stack and have considered the same in my suggestions.

Thumb rules for caching that I generally follow.


Process Once Use Multiple Times.
Live with stale data when possible
Clear Caches infrequently and keep it very specific.
When possible do the changes at the lowest level in the stack.
LAMP - DCCc : Linux,    Apache, Mysql, PHP, Drupal Core, Contrib and custom module.


Improve Performance of a Drupal Site (In the increasing order of complexity)


Keep the core updated, contrib module and themes updated. Yes it
matters.
Install APC on your server. (Moved to top based on suggestion from
Letharion)
Page Caching : admin/config/development/performance Difference between Minimum cache lifetime and Expiration of cached pages  
Block Caching https://drupal.org/project/blockcache_alter Caching options for all the blocks.
Aggregate javascript and css files - Front End Improvements 
https://www.drupal.org/project/advagg
Disable Unnecessary modules. Every module adds to the amount of code
that needs to be available for a page load. And it also increases
the number of lookups. Wherver possible use a generic module in
place of multiple module that does specific functionalities.
Cache Views content - Content aware caching for Views
https://www.drupal.org/project/views_content_cache
Disable DB logging - Use https://drupal.org/project/syslog_ng
Reduce 404 Errors -  http://www.brokenlinkcheck.com/
Fast 404 Responses  - https://drupal.org/project/fast_404 - Try
handling at server level.
Client Side Validations -
 https://www.drupal.org/project/clientside_validation
Compress Image - https://www.drupal.org/project/imageapi_optimize
Lazy Loading of Images - Don’t load unnecessary images - 
https://www.drupal.org/project/lazyloader
Use Sprite Sheets - https://www.drupal.org/project/spritesheets
Set Minimum Cache Life Time Value to a higher number and use cache clearing modules to clear the caches for specific pages - Whenever I edit/update a node all the page caches for anonymous user are lost
Use Devel Module to watch queries.
Rewrite Views Queries /  avoid Views if its a overkill.
XHProf - https://www.drupal.org/project/XHProf
FPM, HHVM.
DB Profiling and Tuning - https://www.drupal.org/project/dbtuner
Use Boost, don't Bootstrap DB if not required.
 https://drupal.org/project/boost For most of the small to medium sites Boost is good enough and you may not need Reverse Proxies or so.
Use CDNs - https://www.drupal.org/project/cdn Its easy to set up.
If your cache tables are huge use Memcached - If you can install
 memcached and set up RAM for it, it is not as complex as it sounds.
Etags - Configure Etags properly.
https://developer.yahoo.com/blogs/ydnfiveblog/high-performance-sites-rule-13-configure-etags-7211.html
Use Reverse Proxy Server - Varnish(at-least for assets). Helps a lot if most of your users are anonymous.
Compressed transfer - Enable gzip compression
Keep Alive - Use Persistent Connections where possible.
Progressive JPEGS -
CACHING IN CODE - Eaton’s blog is awesome.
 http://www.lullabot.com/blog/article/beginners-guide-caching-data-drupal-7
Implement Cache Warming - https://www.drupal.org/project/cache_warmer - Cache Warm the pages before the end user hits them.
Master Slave DB Config - https://www.drupal.org/project/autoslave makes it easier for you to set up one.
Database Clusters - https://stackoverflow.com/questions/1163216/database-cluster-and-load-balancing
Load Balancers -http://en.wikipedia.org/wiki/Load_balancing_(computing)
Use heuristic Cache Warming - https://www.drupal.org/project/cache_graceful
Authenticated User Caching - https://www.drupal.org/project/authcache