title: How to perform unauthenticated Instagram web scraping in response to recent private API changes?
body: <p>Months ago, Instagram began rendering their public API inoperable by removing most features and refusing to accept new applications for most permissions scopes. <a href="https://later.com/blog/instagram-api/" rel="noreferrer">Further changes were made this week</a> which further constricts developer options.</p>
  
  <p>Many of us have turned to Instagram's private web API to implement the functionality we previously had. One standout <a href="https://github.com/ping/instagram_private_api" rel="noreferrer">ping/instagram_private_api</a> manages to rebuild most of the prior functionality, however, with the publicly announced changes this week, Instagram also made underlying changes to their private API, requiring in magic variables, user-agents, and MD5 hashing to make web scraping requests possible. This can be seen by <a href="https://github.com/ping/instagram_private_api/releases" rel="noreferrer">following the recent releases on the previously linked git repository</a>, and the exact changes needed to continue fetching data can <a href="https://github.com/ping/instagram_private_api/compare/1.5.0...master" rel="noreferrer">be seen here</a>.</p>
  
  <p>These changes include:</p>
  
  <ul>
  <li>Persisting the User Agent &amp; CSRF token between requests.</li>
  <li>Making an initial request to <code>https://instagram.com/</code> to grab an <code>rhx_gis</code> magic key from the response body.</li>
  <li>Setting the <code>X-Instagram-GIS</code> header, which is formed by magically concatenating the <code>rhx_gis</code> key and query variables before passing them through an MD5 hash.</li>
  </ul>
  
  <p>Anything less than this will result in a 403 error. These changes have been implemented successfully <a href="https://github.com/ping/instagram_private_api/compare/1.5.0...master" rel="noreferrer">in the above repository</a>, however, my attempt in JS continues to fail. In the below code, I am attempting to fetch the first 9 posts from a user timeline. The query parameters which determine this are:</p>
  
  <ul>
  <li><code>query_hash</code> of <code>42323d64886122307be10013ad2dcc44</code> (fetch media from the user's timeline).</li>
  <li><code>variables.id</code> of any user ID as a string (the user to fetch media from).</li>
  <li><code>variables.first</code>, the number of posts to fetch, as an integer.</li>
  </ul>
  
  <p>Previously, this request could be made without any of the above changes by simply GETting from <code>https://www.instagram.com/graphql/query/?query_hash=42323d64886122307be10013ad2dcc44&amp;variables=%7B%22id%22%3A%225380311726%22%2C%22first%22%3A1%7D</code>, as the URL was unprotected. </p>
  
  <p>However, my attempt at implementing the functionality to successfully written in the above repository is not working, and I only receive 403 responses from Instagram. I'm using superagent as my requests library, in a node environment. </p>
  
  <pre class="lang-js prettyprint-override"><code>/*
  ** Retrieve an arbitrary cookie value by a given key.
  */
  const getCookieValueFromKey = function(key, cookies) {
          const cookie = cookies.find(c =&gt; c.indexOf(key) !== -1);
          if (!cookie) {
              throw new Error('No key found.');
          }
          return (RegExp(key + '=(.*?);', 'g').exec(cookie))[1];
      };
  
  /*
  ** Calculate the value of the X-Instagram-GIS header by md5 hashing together the rhx_gis variable and the query variables for the request.
  */
  const generateRequestSignature = function(rhxGis, queryVariables) {
      return crypto.createHash('md5').update(`${rhxGis}:${queryVariables}`, 'utf8').digest("hex");
  };
  
  /*
  ** Begin
  */
  const userAgent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5';
  
  // Make an initial request to get the rhx_gis string
  const initResponse = await superagent.get('https://www.instagram.com/');
  const rhxGis = (RegExp('"rhx_gis":"([a-f0-9]{32})"', 'g')).exec(initResponse.text)[1];
  
  const csrfTokenCookie = getCookieValueFromKey('csrftoken', initResponse.header['set-cookie']);
  
  const queryVariables = JSON.stringify({
      id: "123456789",
      first: 9
  });
  
  const signature = generateRequestSignature(rhxGis, queryVariables);
  
  const res = await superagent.get('https://www.instagram.com/graphql/query/')
      .query({
          query_hash: '42323d64886122307be10013ad2dcc44',
          variables: queryVariables
      })
      .set({
          'User-Agent': userAgent,
          'X-Instagram-GIS': signature,
          'Cookie': `rur=FRC;csrftoken=${csrfTokenCookie};ig_pr=1`
      }));
  </code></pre>
  
  <p>What else should I try? What makes my code fail, and the provided code in the repository above work just fine?</p>
  
  <h1>Update (2018-04-17)</h1>
  
  <p>For at least the 3rd time in a week, Instagram has again updated their API. The change no longer requires the CSRF Token to form part of the hashed signature.</p>
  
  <p>The question above has been updated to reflect this.</p>
  
  <h1>Update (2018-04-14)</h1>
  
  <p>Instagram has again updated their private graphql API. As far as anyone can figure out:</p>
  
  <ul>
  <li>User Agent is no longer needed to be included in the <code>X-Instagram-Gis</code> md5 calculation.</li>
  </ul>
  
  <p>The question above has been updated to reflect this.</p>