SQL Server 2008 R2 Reporting Services: "Exception of type System.OutOfMemoryException was thrown"
Upgrading the server to a 64 bit version of Windows, and a 64 bit version of SQL Server might help.

However, what are you doing with a report with 600,000 rows?  If you are using this to export data, you should use SSIS or some other tool designed for the job.  If you are summarizing the data inside the report, re-write the source query to summary the data at the server prior to it arriving at the SSRS instance.