What is the preferred way to delete la arge amount of data from a Drupal site? We have a client site that currently has a very, very large data set in Drupal. We are working on generating a 'stripped' copy of the database to be used for testing new code by eliminating the bulk of the site content.

To this end, we need to periodically delete 40 million+ nodes and field collection entities from Drupal to strip down a copy of the production database so it can be used more readily on our staging and local development instances.

This is somewhat beyond what VBO or Bulk Delete (Nodes) or other similar contrib solutions can handle in a reasonable time, not to mention that many of the field collections are not on node objects. I suspect that using node_delete_multiple() and entity_delete_multiple() for millions of items may be a non-starter time-wise, even if batched through either the queue API or the batch API.

Is there any way to 'cheat' by TRUNCATING the field_data_* and field_revision_* tables in MySQL to speed the record deletions?
I'm not seeing any simple way to truncate (or truncate from X) a table like this.   The tables that would contain field data would be all mixed up and not nicely sorted.

In this case I would;


Start with a view that generated a list of node IDs based on your
select criteria (content type, status, older than date published,
etc)
Write a custom module with a drush command 
Load the node IDs
Either node_delete($nid) OR...
Examine what node_delete() does and delete * from {table} where nid= $nid


The tables node_delete() may touch will largely be dependant on your content types.  node_delete() will do the job, doing it my hand will take more time to prepare but may be faster to run. With the volume of records you're talking about a few milliseconds per delete may save a decent chunk of time.