title: How to crawl, process and index social media profiles?
body: <p>For a new tech-startup venture we want to set up a so called "Social Crawler". With this piece of software we want to give clients the ability to search for potential employees. The results  (E.g. when searched for "iOS developer") should be aggregated social profiles from crawled sources like: twitter, linkedin, etc. based on bio's and other relevant public information.</p>
  
  <p>Here is how i think the process should work:</p>
  
  <ol>
  <li>Crawl websites like Twitter, Linkedin, etc.</li>
  <li>Process and match document data</li>
  <li>Index aggregated profiles</li>
  <li>Access indexed information via a search server</li>
  </ol>
  
  <p>The setup should be an ongoing aggregator that allows the database to grow automatically. Developing these kind of setups are beyond my current knowledge. I would like to know where to start or who to talk to, or even who can develop this for us.</p>
  
  <p>I did stumble upon <a href="http://www.crawl-anywhere.com/" rel="nofollow">Crawl Anywhere</a>. Is this something that might work for us?  I am curious what your thoughts are!</p>
  
  <p>Cheers!</p>