title: Core data bulk insert suddenly slows to 1/10th the speed
body: <p>I am bulk inserting into core data. I have a person object, and this person object has a relationship called "otherPeople" that is an NSSet of people. When bulk inserting data from a download, things were great until about 10,000 people are read in at which point the bulk insert speed slows down to a crawl. I am saving and resetting my NSManagedObjectContext every 500 inserts.</p>
  
  <p>If I comment out the part that inserts the "otherPerson" relationships, the bulk insert is speedy through the entire download. parseJSON is called in batches of 500 JSONKit dictionaries.</p>
  
  <p>Any ideas what might be causing this? Possible solutions?</p>
  
  <p>Code:</p>
  
  <pre><code>- (NSArray*) getPeople:(NSArray*)ids
  {
      NSFetchRequest* request = [[[NSFetchRequest alloc] init] autorelease];
      NSEntityDescription* entityDescription = [NSEntityDescription entityForName:@"Person" inManagedObjectContext:context];
      [request setEntity:entityDescription];
      [request setFetchBatchSize:ids.count];
  
      //Filter by array of ids
      NSPredicate* predicate = [NSPredicate predicateWithFormat:@"externalId IN %@", ids];
      [request setPredicate:predicate];
  
      NSError* _error;
      NSArray* people = [context executeFetchRequest:request error:&amp;_error];
  
      return people;
  }
  
  - (void) parseJSON:(NSArray*)people
  {
      NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
      NSMutableArray* idsToFetch = [NSMutableSet setWithCapacity:CHUNK_SIZE * 3];
      NSMutableDictionary* existingPeople = [NSMutableDictionary dictionaryWithCapacity:CHUNK_SIZE * 3];
  
      // populate the existing people dictionary first, that way we know who is already in the context without having to do a fetch for each person in the array (externalId IS indexed)
      for (NSDictionary* personDictionary in people)
      {
          // uses JSON kit to parse out all the external ids...
          [PersonJSON addExternalIdsToArray:idsToFetch fromDictionary:personDictionary];
      }
  
      // see above code for getPeople implementation...
      NSArray* existingPeopleArray = [self getPeople:idsToFetch];
      for (Person* p in existingPeopleArray)
      {
          [existingPeople setObject:p forKey:p.externalId];
      }
  
      for (NSDictionary* personDictionary in people)
      {
          NSString* externalId = [personDictionary objectForKey:@"PersonId"];
          Person* person = [existingPeople objectForKey:externalId];
  
          if (person == nil)
          {
              // the person was not in the context, make a new person in the context
              person = [[self newPerson] autorelease];
              person.ancestryId = externalId;
              [existingPeople setObject:person forKey:person.externalId];
          }
  
          // use JSON kit to populate the core data object...
          [PersonJSON populatePerson:person withDictionary:personDictionary inContext:[self context]];
  
          // these are just objects that contain an externalId, showing that the link hasn't been setup yet
          for (UnresolvedOtherPerson* other in person.unresolvedOtherPeople)
          {
              Person* relatedPerson = [existingPeople objectForKey:other.externalId];
  
              if (relatedPerson == nil)
              {
                  relatedPerson = [[self newPerson] autorelease];
                  relatedPerson.externalId = other.externalId;
                  [existingPeople setObject:relatedPerson forKey:relatedPerson.externalId];
              }
  
              // add link - if I comment out this line, everything runs very fast
              // if I don't comment out, things slow down gradually and then exponentially
              [person addOtherPersonsObject:relatedPerson];
          }
  
          self.downloaded++;
      }
  
      [pool drain];
  }
  </code></pre>