title: Scrape Facebook in Python
body: <p>I'm interested in getting the number of friends each of my friends on Facebook has. Apparently the official Facebook API does not allow getting the friends of friends, so I need to get around this (somehwhat sensible) limitation somehow. I tried the following:</p>
  
  <pre><code>import sys
  import urllib, urllib2, cookielib
  
  username = 'me@example.com'
  password = 'mypassword'
  
  cj = cookielib.CookieJar()
  opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
  login_data = urllib.urlencode({'email' : username, 'pass' : password})
  request = urllib2.Request('https://login.facebook.com/login.php')
  request.add_header('User-Agent','Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Fedora/3.6.12-1.fc14 Firefox/3.6.12')
  opener.open(request, login_data)
  resp = opener.open('http://facebook.com')
  print resp.read()
  </code></pre>
  
  <p>but I only end up with a captcha page. Any idea how FB is detecting that the request is not from a "normal" browser? I could add an extra step and solve the captcha but that would add unnecessary complexity to the program so I would rather avoid it. When I use a web browser with the same User-Agent string I don't get a captcha.</p>
  
  <p>Alternatively, does anyone have any saner ideas on how to accomplish my goal, i.e. get a list of friends of friends?</p>