Can I remove unused core modules from Drupal? I am about to start a new website that is supposed to handle thousands of users at once. It is having some mobile application, and I need to use web service.
As a matter of performance, and the link I found here, I am planning to remove core modules such as the Blog module, the Forum module, and maybe the Search module too. 

Is it a best practice? What can I do to increase the performance, as I am not planning to purchase a VPS or something like that for hosting?
Modules that have never been enabled are not a performance issue.  

The issue report you linked is referring to what Drupal does when it doesn't find an enabled module; in such case, Drupal rescan the directories where the modules are placed, causing performance issues. To avoid it, modules should be uninstalled from the module page, and then deleted from the server directories; if you do this, you will never have any performance issue.

If you are thinking of removing the database rows found in the "system" table referring to those core modules that you don't use, then it is useless, as Drupal adds those rows back every time you visit the module page.
If Drupal were not doing that, you would not be able to install new modules. When you visit the modules page (admin/modules in Drupal 7), Drupal rescans the directories where modules are placed, to list the new modules you could have copied on the server. If it finds a module that   doesn't have a row in the "system" table, it adds it; it cannot know if you removed that row from the table, or it is a module that you copied on the server, and you want to install.

If you are thinking of removing the module files for those core modules you don't use, you should keep doing it every time you update Drupal; as there isn't any gain, it's useless doing it.