New SQL instance and migration What are the best practices of creating new SQL Server instance on SQL Server Clustering 2008 R2 and migrating a database with high transactions?
You can start from this list from Jonathan Kehayias.

http://sqlblog.com/blogs/jonathan_kehayias/archive/2010/03/22/sql-server-installation-checklist.aspx

Can you elaborate on "migrating a database with high transactions"? This is very open ended question and not sure how to address this. There are lot of good practices to follow here.


Making sure data, logs, tempdb are on separate disk and following best practices for Files i.e Instant file initialization, pre-allocating sizes so that autogrow won't happen, partition alignment, having good RAID http://blogs.msdn.com/b/jimmymay/archive/tags/disk+partition+alignment/
Good index creation &amp; maintenance procedures 
Good application + sql code
Getting a head start on http://msdn.microsoft.com/en-us/library/dd672789(v=sql.100).aspx
Having decent memory size so that disk access is limited
Looking for disk latency using virtual file stats DMV
Watching for WAIT STATISTICS


There are so many other things here but it takes a while to list them all and it depends.