title: Handling massive contact book on Android
body: <p>Background:
  Seems like a solved problem:  A chat application wishes to upload your local phonebook to their own servers to crosscheck the numbers so that in case one of your friends has registered with the service, they show up in a list that says "Hey chat with these people, they also have App X".</p>
  
  <p>Naive solution:
  <a href="https://stackoverflow.com/questions/6587674/android-contacts-display-name-and-phone-numbers-in-single-database-query">Android contacts Display Name and Phone Number(s) in single database query?</a></p>
  
  <p>Approaches like the above query the Contacts ContentProvider for all the contact data on the phone.  This data is then combined into contact entities, converted to JSON, and sent to a server where the phone and email credentials are checked against the database there.  </p>
  
  <p>Problem:
  This solution doesn't scale for a number of reasons.  Consider something like a phone with 25,000 contacts (this happens with corporate users with something like LDAP) </p>
  
  <ol>
  <li>The query itself will be saved to a sqlite database in the app.  This is needed so that domain related data model stuff can be saved with contacts that are registered and those that are not registered can also be displayed with invite buttons (think WhatsApp etc). Querying this table as a whole becomes an issue since a Cursor can only handle a result set of &lt;= 1MB.  </li>
  <li>Even if you can get all the Contacts into memory and convert them to JSON, you are now looking at like a 2MB String to upload to the server for sync.  On poor networks, this could take a REALLY long time.  Mostly this can be done in the background but it will become very obvious on the first time load and any subsequent changes to the local phone book (if you add a new contact who is a user of App X for instance).  </li>
  </ol>
  
  <p>Let's say that there is a way of uploading deltas after the initial launch as opposed to the entire contact book again each time the phonebook changes.  Even then, you have the initial upload that either you block the user on to wait for or you end up with the user in an app with no contacts being shown.</p>
  
  <p>Long story short, my questions are:</p>
  
  <ol>
  <li>How is this best done?  Are large contact lists just not being considered for the majority of messaging apps?</li>
  <li>How can I make waiting for the initial upload a non-terrible UX?  </li>
  <li>How can I test this case?  Is there a tool I can use to programatically add thousands of contacts into the address book or do I have to roll my own? </li>
  </ol>