Do I have to delete the default folder?
Unless you're running a multi-site or another non-standard configuration (and the way the question is phrased, I do not think you do), you can't delete the sites/default folder or the files within it.

The reason you can't delete it is because: This directory (unless you've set up a multi-site or similar) contains (among other things) the settings.php for your site. If you delete it, all you settings including the database credentials will be lost, and your site will stop working.

Please also note that the sites/default folder has nothing whatsoever to to with the theming of your site.

If you have installed a new theme, and have the problem that the new theme is not used, this problem is not related to the presence of the sites/default folder on your site.