How will speed be affected by entity_metadata_wrapper()?
Entity API does a lot of things, but most of them are cached.

For example, entity_metadata_wrapper() calls entity_get_info(), which uses the cache to retrieve the data. If the cache is empty, other functions are called:

drupal_schema_fields_sql()
cache_set()
...


Other modules may be involved through hooks:

module_invoke_all('entity_info');
drupal_alter('entity_info', $entity_info);


But note that no data is fetched from the database, this is because when you call entity_metadata_wrapper() you already have the entity loaded -it's a parameter to the function, indeed. 

So if you are running critical or long processes such as importing data or running thousands of iterations inside a loop, you should better use entities directly. But even then, the cache should help a lot.

Look how Drupal looks without Entity API:

$field = '';
if (isset($entity-&gt;field_name[LANGUAGE_NONE]) 
    and isset($entity-&gt;field_name[LANGUAGE_NONE][0])) {
  $field = $entity-&gt;field_name[LANGUAGE_NONE][0]['value'];
}


It's ugly. This is why I use Entity API almost everywhere, and I haven't had any performance issues so far. For the benefits, I think it's worth using it.