First CiviCRM API call is slower, are unnecessary resources loaded?
They are several distinct issues:


Calling civicrm_api3, this isn't REST, and (unless requested) the code runs as admin, so probably not linked with authentication


Here, a common issue is the database access, the first time you access the underlying tables (civicrm_contact and co), mysql might not have kept them in the cache, but the second time they are and it will be several order of magnitude faster.

My guess would be that most of the difference in time is due to the database layer (that might has well be kept some cached in the DAO layer) but mostly the difference between an empty mysql cache. 

When you compare with running directly the executeQuery, which one do you run? They are likely quite a few sql queries inside the api call. Do you run them after having called the api? 

if you want to dig into the detail of where the time is spent, I'd recommend you to use a profiler, because it's hard enough th it, and black magic and speculation without


The REST interface. It's always going to be slower, both because of the overhead of the network and the extra authentication step (based on the keys you provide)
It will be faster if you provide a return param to the api call, to specify which fields you want. by default, it will returns tons of fields that you probably won't use, some that are from related joined tables (eg. email, phone, address, name for options or pseudo constants...).


adding a return="first_name,last_name,email" would avoid fetching stuff you don't need.