title: Core Data memory usage while importing large dataset
body: <p>I'm now stuck for about two weeks with a nasty Core Data problem. I read lots of blogpost, articles and SO questions/answers but I'm still not able to solve my problem.</p>
  
  <p>I ran lots of tests and was able to reduce the larger problem to a smaller one.
  It's going to be a large explanation so keep with me!</p>
  
  <h2>Problem - datamodel</h2>
  
  <p>I have to got following <strong>datamodel</strong>:</p>
  
  <p>Object A has one-to-many relation with object B which has another one-to-many relation with object C. Because of Core Data recommendations I have to create inverse relations so each instance of B points to its parent A and the same for C which points to its parent B.</p>
  
  <pre><code>A &lt;-&gt;&gt; B &lt;-&gt;&gt; C
  </code></pre>
  
  <h2>Problem - MOC setup</h2>
  
  <p>To keep responsiveness smooth as butter I created a three-level managedObjectContext structure.</p>
  
  <ol>
  <li>Parent MOC - Runs on its own private thread using <code>NSPrivateQueueConcurrencyType</code>, is tight to the <code>persistentStoreCoordinator</code></li>
  <li>MainQueue MOC - Runs on the mainThread using <code>NSMainQueueConcurrencyType</code> and has parent MOC 1</li>
  <li>For each parsing operation I create a third MOC which also has its private queue and has parent mainQueue MOC</li>
  </ol>
  
  <p>My main datacontroller is added as an observer to the <code>NSManagedObjectContextDidSave</code> notification of MOC 2 so every time MOC 2 saves a <code>performBlock:</code> on MOC1 is triggered which performs a save operation (asynchronously because of <code>performBlock:</code>).</p>
  
  <h2>Problem - Parsing</h2>
  
  <p>To perform parsing a large JSON file into my Core Data structure I wrote a recurrent parser. This parser starts by creating a new MOC (3). It then takes the data for object A and parses its properties. Then the parser reads out the JSON relations for B and create the corresponding objects which are filled with data. These new objects are added to A by calling <code>addBObject:</code> on A. 
  Because the parser is recurrent, parsing B means parsing C and here also new objects are created and attached to B.
  This all happens in the <code>performBlock:</code> on MOC 3.</p>
  
  <ul>
  <li>Parse (creates 'A'-objects and starts parsing B)
  <ul>
  <li>Parsing A (creates 'B'-objects, attaches them to A and starts parsing C)
  <ul>
  <li>Parsing B (creates 'C'-objects, attaches them to B)
  <ul>
  <li>Parsing C (just stores data in a C-object)</li>
  </ul></li>
  </ul></li>
  </ul></li>
  </ul>
  
  <p>After each parsing operation I save MOC 3 and dispatches on the mainThread a save operation of the main MOC (2). Because of the <code>NSManagedObjectContextDidSave</code> notification MOC 1 will autosave asynchronously.</p>
  
  <pre><code>        if (parsed){
              NSError *error = nil;
              if (![managedObjectContext save:&amp;error])
                  NSLog(@"Error while saving parsed data: %@", error);
          }else{
              // something went wrong, discard changes
              [managedObjectContext reset];
          }
  
          dispatch_async(dispatch_get_main_queue(), ^{                
              // save mainQueueManagedObjectContext
              [[HWOverallDataController sharedOverallDataController] saveMainThreadManagedObjectContext];
          });
  </code></pre>
  
  <p>To release my memory footprint and because I do not need to parsed data for now I am performing:</p>
  
  <pre><code>[a.managedObjectContext refreshObject:a mergeChanges:NO];
  </code></pre>
  
  <p>for each A I just parsed.</p>
  
  <p>Because I need to parse about 10 A's which all have about 10 B's which have all about 10 C's a lot of managedObject's are generated. </p>
  
  <h2>Problem - Instruments</h2>
  
  <p>Everything works fine. The only thing is: when I turn on the Allocations tool I see unreleased A's, B's and C's. I don't get any useful information from their retainCounts or whatsoever. 
  And because my actual problem regards a more complex dataModel the living objects become a serious memory problem.
  Can someone figure out what I'm doing wrong? Calling refreshObjects on the other managedObjectContexts with the correct managedObject does not work either. Only a hard <code>reset</code> seems to work but then I loose my pointers to living objects used by the UI.</p>
  
  <h2>Other solutions I tried</h2>
  
  <ul>
  <li><p>I tried creating unidirectional relations instead of bidirectional ones. This create a lot other problems which cause Core Data inconsistencies and weird behavior (such as dangling objects and Core Data generating 1-n relations instead of n-n relations (because the inverse relation is not known).</p></li>
  <li><p>I tried refreshing each changed or inserted object when I retrieve a <code>NSManagedObjectContextDidSave</code> notification on any object</p></li>
  </ul>
  
  <p>These both 'solutions' (which don't work by the way) seems also a bit hacky. This should not be the way to go. There should be a way of getting this to work without raising the memory footprint and by keeping the UI smooth, though?</p>
  
  <h2>- CodeDemo</h2>
  
  <p><a href="http://cl.ly/133p073h2I0j">http://cl.ly/133p073h2I0j</a></p>
  
  <h2>- Further Investigation</h2>
  
  <p>After refreshing every object ever used (which is tedious work) in the mainContext (after a mainSave) the object their sizes are reduced to 48 bytes. This indicates that the objects are all faulted, but that there is still a pointer left in memory. When we have about 40.000 objects which are all faulted there is still 1.920 MB in memory which is never released until the persistentManagedObjectContext is reset. And this is something we don't want to do because we loose every reference to any managedObject.</p>