title: Is this robots.txt syntax with an empty "Disallow:" correct?
body: <p>Today whilst improving my web crawler to support the robots.txt standard, I came across the following code at <a href="http://www.w3schools.com/robots.txt" rel="noreferrer">http://www.w3schools.com/robots.txt</a></p>
  
  <pre><code>User-agent: Mediapartners-Google 
  Disallow: 
  </code></pre>
  
  <p>Is this syntax correct? Shouldn't it be <code>Disallow: /</code> or <code>Allow: /</code> depending on the intended purpose?</p>