Facebook not consistently scraping information as per OG meta tags
Question: I automatically post adverts for people's jobs on member's facebook pages
using Python's facebook module. I have applied for and got Facebook's
authority to do this. Now when I post to most people's pages, the image and
description are not being scraped, however when I post with the FB site
owner's access token of the business site associated with the application,
they are. The code I use to post the pages:

    
    
      import facebook
      graph       = facebook.GraphAPI(access_token)
      # record the id of the wall post so we can like it
      fbWallPost  = graph.put_wall_post(message=fbmsg,  attachment=fbpost, profile_id=pageID)
    

Running this in the python shell posts a URL that gets scraped, but running it
from within the application it very frequently does not. The page has the
following meta tags:

    
    
    <!DOCTYPE html>
    <html lang="en">
    <head>
      <meta charset="utf-8">
    
      <title>Fair Work</title>
    
    
      <meta name="viewport" content="width=device-width, initial-scale=1.0">
      <meta name="author" content="Fair Work">
        <!-- Facebook crawler uses these JRT -->
      <meta property="og:site_name" content="Fair Work"/>
      <meta property="fb:app_id" content="322643981237986" />
    
    
      <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" />
      <meta http-equiv="Pragma" content="no-cache" />
      <meta http-equiv="Expires" content="0" />
    
    
      <meta property="og:url" content="http://fairwork.com.au/job/final-job-on-thursday-april-30th/" />
      <meta property="og:description" content="My last test today"/>
      <meta property="og:type" content="article"/>
    
      <meta property="og:image" content="http://fairwork.com.au/media/cache/cd/b0/cdb05f8dd8885351925bf43076870937.jpg"/>
      <meta property="og:image:width" content="400" />
      <meta property="og:image:height" content="400" />
    

The application ID is correct. I go to Facebook's debugger, and I enter the
URL ![My URL given to FB debugger](http://i.stack.imgur.com/pKFZT.png)  
I click on "Debug", and I get an error that the "og:type" meta tag is missing,
which it is not as you can see: ![enter image description
here](http://i.stack.imgur.com/Vawl2.png) I then click on "Fetch new scrape
information" ... ![enter image description
here](http://i.stack.imgur.com/CLqq6.png) ...and the image and description are
loaded up fine. From this point if I post the link on my site, the image and
description display, however all previously posted links do not have the
information scraped. The two successive posts of the same link in the next
image were done before (bottom) and after (top) running through the debug
process mentioned previously: ![enter image description
here](http://i.stack.imgur.com/QmNUZ.png)

The page linked to is publicaly available, and served by Django. I want to be
able to automatically post the ad's (when this is requested by the FB page
owner) and have the image and description scraped by FB on posting. Any ideas?


Answer: It seems that NGINX is causing the http calls to the Facebook graph API not to
work properly. When

    
    
    fbWallPost  = graph.put_wall_post(message=fbmsg,  attachment=fbpost, profile_id=pageID)
    

is run through Celery (which runs separately to django, and not through the
web server), facebook scrapes the posted link and shows the image and
description linked to by the OG meta tags. When django runs put_wall_post, the
post occurs, but facebook does not scrape the link. Other calls to the API
also don't work when run from a Django view. I can only think that NGINX is
interfering with the HTTP request in some way.

