Minimizing the time taken for entity_load while using for apachesolr indexing I am working on a site with a specific node type that has close to 50 fields, most of which are taxonomy term references, a couple of node references and a few conventional text and integer values. Almost all of them accept multiple values. 
While indexing the nodes with apachesolr I observe that most of the time in the indexing process is consumed by calls made to entity_load function, almost 3.5 - 4 minutes for loading 200 nodes when checked with timer based logging while the total indexing takes about 50 seconds more (4.5 minutes overall).

Adding a custom 'status back' with hook_apachesolr_entity_info_alter to query and return the node status directly instead of loading it, I was able to negate the entity_load in one instance, however another function apachesolr_index_entity_to_documents() where the drupal node to solr document conversion takes place had entity_load calls consumed more time since they could no longer use the cached result of the status callback.

My aim is to reduce the time taken for indexing to at least half of what it currently which now is completely dependant on the time taken by entity_load. Since entity_load is one of the critical aspects of Drupal 7 around which functions like node_load/node_load_multiple are derived from I'm unable to find a different option to proceed or determine ways to improvise entity_load usage.

Note: The site functions such that these nodes get created almost all the time and none of the fields can be pruned out, negating the option of pre-loading the entities. Also the site uses entity cache module with memcache 

I'm hoping to get suggestions on improving the node load time programatically with entity_load or using alternative approach to load the respective entities to bring down my indexing time.
You could very well form an array of the nodes to be indexed after the query execute and do a single entity_load instead of doing a entity_load for each and every node separately and I hope that will reduce the time and below is the patch that could help you



Index: apachesolr/apachesolr.index.inc

--- apachesolr/apachesolr.index.inc     (revision 9348)
+++ apachesolr/apachesolr.index.inc     (working copy)
@@ -636,9 +636,21 @@
     $query->condition('aie.entity_type', $entity_type);
   }
   $query->range(0, $limit);
-  $records = $query->execute();
-  $status_callbacks = apachesolr_entity_get_callback($entity_type, 'status callback');
+  $records = $query->execute()->fetchAll();
+  foreach ($records as $node_record) {
+    $entities[] = $node_record->entity_id;
+  }
+  $arr_entity = entity_load($entity_type, $entities);
+  foreach ($records as $record) {
+    $record->status = $record->status &amp;&amp; $arr_entity[$record->entity_id]->status;
+    $rows[] = $record;
+  }
+  // Commenting as we have already used the entity_load
+  /*$status_callbacks = apachesolr_entity_get_callback($entity_type, 'status callback');
   foreach ($records as $record) {
     // Check status and status callbacks before sending to the index
     if (is_array($status_callbacks)) {
@@ -651,7 +663,7 @@
       }
     }
     $rows[] = $record;
-  }
+  }*/
   return $rows;
 }


As you have said most of your fields are taxonomy term it is idle that you check the term_load method and also whether the term_data table is properly indexed.