Performance of a linked server between two instances of SQL Server on the same virtual machine
should I expect a significant performance drop during queries between
  them (large joins)? Is there a way to overcome this?


This answer addresses directly your question about using a linked server to run queries through.  And the answer is YES, you can expect a performance drop, especially with large joins.  

Usually when SQL processes a query, it can use statistics, indexes and other things to define an optimal execution plan, using later joins and where statements to eliminate required rows before they are even read (or at least making a good estimate for how much memory is needed).  But the linked server is a black box for those things.  With queries of even minimal complexity, you will quickly see a drop in performance.  This is due to SQL bringing the entire table contents over the wire into a worktable and then performing the join operation(s).  

So, a few thoughts for you:

You can do log shipping of Database A on Server A to Database B on Server A (the same server, but the database is a different name).  You will have to roll your own, but it's just RESTORE LOG WITH STANDBY.  You won't be able to separate cpu/memory for each instance, but you would keep reporting transactions from interfering with your OLTP transactions.  If these are going to live on the same host (just as different instances) then this is no great loss.

What joins are you doing through the linked server?  If all the data you need is available in the reporting database, then you could use OPENQUERY() to make sure that the linked server handles all of the query.  Otherwise, you could use OPENQUERY to pull minimal data from the remote server to the local (temp table) for later joining/work.