Asset indexing failing with large asset source
I've done this before with a combination of a Craft console command that replicates the indexing process, and a external way that processes each row.

&lt;?php
namespace Craft;

class BusinessLogicCommand extends BaseCommand
{
    /**
    * ./craft/app/etc/console/yiic businesslogic --offset=x --sourceId=y --sessionId=z
    */
    public function actionIndex ($offset = '', $sourceId = '', $sessionId = '')
    {
        if ( $offset ) {
            $index = craft()-&gt;assetIndexing-&gt;processIndexForSource($sessionId, $offset, $sourceId);

            if ( $index ) {
                echo sprintf("Processed offset: %s for source %s", $offset, $sourceId);

                return 1;
            }
        }

        return 0;
    }

    /**
    * ./craft/app/etc/console/yiic businesslogic getIndexOverview
    */
    public function actionGetIndexOverview ($sourceIds = '')
    {

        // Get batches
        $batches   = array();
        $sessionId = craft()-&gt;assetIndexing-&gt;getIndexingSessionId();

        // Selection of sources or all sources?
        if ( $sourceIds ) {
            $sourceIds = explode(',', $sourceIds);
        }
        else {
            $sourceIds = craft()-&gt;assetSources-&gt;getViewableSourceIds();
        }

        $missingFolders = array();
        $grandTotal     = 0;

        foreach ($sourceIds as $sourceId) {
            // Get the indexing list
            $indexList = craft()-&gt;assetIndexing-&gt;getIndexListForSource($sessionId, $sourceId);

            if ( !empty($indexList['error']) ) {
                echo json_encode($indexList);

                return 0;
            }

            if ( isset($indexList['missingFolders']) ) {
                $missingFolders += $indexList['missingFolders'];
            }

            $batch = array();

            for ($i = 0; $i &lt; $indexList['total']; $i++) {
                $batch[] = [
                    'sessionId' =&gt; $sessionId,
                    'sourceId'  =&gt; $sourceId,
                    'total'     =&gt; $indexList['total'],
                    'offset'    =&gt; $i,
                    'process'   =&gt; 1,
                ];
            }

            $batches[] = $batch;
        }


        // Overview

        $assetsSourcesBeingIndexed = $sourceIds;
        $assetsMissingFolders      = $missingFolders;

        $job = [
            'batches' =&gt; $batches,
            'total'   =&gt; $grandTotal,
        ];

        echo json_encode($job);

    }


    public function actionCleanUpIndex ()
    {
        // Get overview
        $sourceIds      = $assetsSourcesBeingIndexed;
        $missingFiles   = craft()-&gt;assetIndexing-&gt;getMissingFiles($sourceIds, $sessionId);
        $missingFolders = $assetsMissingFolders;

        $responseArray = array();

        if ( !empty($missingFiles) || !empty($missingFolders) ) {
            $responseArray['confirm'] = craft()-&gt;templates-&gt;render('assets/_missing_items', array( 'missingFiles' =&gt; $missingFiles, 'missingFolders' =&gt; $missingFolders ));
            $responseArray['params']  = array( 'finish' =&gt; 1 );
        }
        // Clean up stale indexing data (all sessions that have all recordIds set)
        $sessionsInProgress = craft()-&gt;db-&gt;createCommand()
                                        -&gt;select('sessionId')
                                        -&gt;from('assetindexdata')
                                        -&gt;where('recordId IS NULL')
                                        -&gt;group('sessionId')
                                        -&gt;queryScalar();

        if ( empty($sessionsInProgress) ) {
            craft()-&gt;db-&gt;createCommand()-&gt;delete('assetindexdata');
        }
        else {
            craft()-&gt;db-&gt;createCommand()-&gt;delete('assetindexdata', array( 'not in', 'sessionId', $sessionsInProgress ));
        }

    }

}


This is done in 3 steps:


Trigger getIndexOverview, which returns an JSON object string that lists all your index rows.
Use the JSON object in some way, and create a job for each row, which should trigger step #3.
Trigger the index command with the parameters  --offset=x --sourceId=y --sessionId=z


I have used Laravel + Queue backed by Redis for this. If you have huge files, or a lot of data to process, think about spinning up a big VPS somewhere.

That way you can run 10-12 jobs in parallel, export your database/file structure, and tear the VPS down after. With hourly pricing, you'd end up paying a few $.