Ask HN: Scraping Facebook – Will this work?
I had an idea for a nice app, which would in part rely on listing Facebook posts from your friends.
I thought this would be easy enough, so I designed it before I prototyped, which is something I never do.<p>Sure enough, when I had finished the design and finally got to prototyping, I realised that Facebook simply no longer allow access to the read_stream API endpoint, unless you get authorisation from them (seems like no one does).
Info here: https:&#x2F;&#x2F;developers.facebook.com&#x2F;docs&#x2F;facebook-login&#x2F;permissions&#x2F;v2.5#reference-read_stream<p>Fuelled by ingenuity, and because I had the design ready, I thought I&#x27;d try and simply load the user&#x27;s news feed on a UIWebView, and read the data I need from elements in the DOM. I&#x27;m pretty sure this is against FB&#x27;s ToS and wouldn&#x27;t fly for long, but I kind of want to give it a go anyway.<p>I&#x27;ve gotten to a place where I&#x27;ve proven it works, but not always reliably and it&#x27;s certainly hacky.<p>If you want to give it a go, load up https:&#x2F;&#x2F;m.facebook.com on your favourite browser, and then copy&#x2F;paste + run the JS code in this jsFiddle (https:&#x2F;&#x2F;jsfiddle.net&#x2F;Letwernb&#x2F;), to your console.<p>It&#x27;ll list whatever posts it finds on your feed, and give you some info on them.
I believe at the moment I&#x27;m skipping ads and not so relevant posts, such as &quot;friend shared a link&quot;.<p>I&#x27;ve also got a bit of code that lets me load more posts, until I&#x27;ve reached the 20 I need to display in the app.<p>I&#x27;ve got some challenges though. Like I said, it&#x27;s hacky and relies on FB not changing certain class names, and because the date for each post comes as a string (&quot;2 hours ago&quot;), I need to find a way to convert that back to a timestamp so I can re-order the posts.<p>I thought I&#x27;d post here to get some thoughts on this approach, and maybe get some ideas on how to do it better.<p>Is this something you&#x27;ve tried before, or wanted to try? How would you have done it?<p>Looking forward to hearing your thoughts.