SQL Server - failed to update records when RAM is fully occupied
As suggested set the “Maximum Server Memory”. 

https://sqlserverperformance.wordpress.com/2009/10/29/suggested-max-memory-settings-for-sql-server-20052008/

Additionally, please note that your error is more likely related to locking or timeouts rather than “running out memory”. An extensive exclusive lock on the target table combined with missing indexes for the update statement can result in what you are experiencing.

1)  Tune your report query and ensure that its not having redundant excessive reads. Verify that hash and sort operators in your execution plan are not spilling to disk.

2)  Make sure that your update statement has logical supporting indexes.

Tuning both process might enhance concurrency of your application.

Good luck :)