Issues installing SQL Server 2012
The error message is the generic "I can't reach the SQL instance you've specified."  There are no components to "uninstall/reinstall", mixed mode/Windows mode are simply configuration settings.  I generally recommend you run in mixed mode.

Two things to check:


If this is a SQL Express instance, you should make sure your connecting to [Computer Name]\SQLEXPRESS.  SQL Express installs as a named instance, so as such needs to be connected to as a named instance.
By default, SQL Server installs without TCP protocols enabled, which would prevent most connections.  Open your SQL Server Configuration Manager, browse to SQL Server Network Configuration, and then Protocols for SQLEXPRESS.  Select that and look for TCP/IP and make sure it's enabled.


As for your file being larger than 10 GB, you'll need to use Standard Edition or higher, nothing you can do to get around that.  You can have a separate instance running alongside the Express instance, but if you want you can simply uninstall your Express instance as you would any other Windows program.