Optimizing this Excel automation
There is definetely a way to improve that even though the excel sheet has a lot of formulas.
In VBA there is an:

Application.Calculation = xlManualCalculation


That can be used to help dramatically improve the time it takes, when the problem is many formulas in a sheet. And then in the end you should turn it back to automatic. 

As you are using VB with a connection to the Excel Workbook through a COM, the way to do it is very similar. It is:

ExcelApp.Calculation = XlCalculation.xlCalculationManual


Where ExcelApp is the excel application object.