title: Where does CloudFlare detect web and terminal requests?! On equal terms
body: <p>My problem is opening sites on <code>CloudFlare</code> in <code>CLI</code>. <br />
  I do not mean when there is a challenge and I do not want to solve the challenge.</p>
  <p>Consider this site as an example: <a href="https://pegaxy.io" rel="noreferrer">https://pegaxy.io</a> <br />
  When opened for the first time on a newly installed any web browser. It opens without any problems. Code <code>200</code> is received.</p>
  <p><a href="https://i.stack.imgur.com/qS7zK.png" rel="noreferrer"><img src="https://i.stack.imgur.com/qS7zK.png" alt="open with web browser" /></a></p>
  <p>But when I click <code>Copy as cURL</code> and get a <code>403</code> error in the terminal.</p>
  <p><a href="https://i.stack.imgur.com/yMhfD.png" rel="noreferrer"><img src="https://i.stack.imgur.com/yMhfD.png" alt="open in terminal" /></a></p>
  <p>CURL code:</p>
  <pre><code>curl 'https://pegaxy.io/' \
    -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0' \
    -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' \
    -H 'Accept-Language: en-US,en;q=0.5' \
    -H 'Connection: keep-alive' \
    -H 'Upgrade-Insecure-Requests: 1' \
    -H 'Sec-Fetch-Dest: document' \
    -H 'Sec-Fetch-Mode: navigate' \
    -H 'Sec-Fetch-Site: none' \
    -H 'Sec-Fetch-User: ?1' \
    -H 'Pragma: no-cache' \
    -H 'Cache-Control: no-cache' \
    --compressed --verbose
  </code></pre>
  <p>Log:</p>
  <pre><code>$ curl 'https://pegaxy.io/' \
  &gt;   -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0' \
  &gt;   -H 'Upgrade-Insecure-Requests: 1' \
  &gt;   -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' \
  &gt;   -H 'Accept-Language: en-US,en;q=0.5' \
  &gt;   -H 'Connection: keep-alive' \
  &gt;   -H 'Upgrade-Insecure-Requests: 1' \
  &gt;   -H 'Sec-Fetch-Dest: document' \
  &gt;   -H 'Sec-Fetch-Mode: navigate' \
  &gt;   -H 'Sec-Fetch-Site: none' \
  &gt;   -H 'Sec-Fetch-User: ?1' \
  &gt;   -H 'Pragma: no-cache' \
  &gt;   -H 'Cache-Control: no-cache' \
  &gt;   --compressed --verbose
  *   Trying 172.67.10.157:443...
  * Connected to pegaxy.io (172.67.10.157) port 443 (#0)
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
    CApath: none
  * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  * TLSv1.3 (IN), TLS handshake, Server hello (2):
  * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  * TLSv1.3 (IN), TLS handshake, Certificate (11):
  * TLSv1.3 (IN), TLS handshake, CERT verify (15):
  * TLSv1.3 (IN), TLS handshake, Finished (20):
  * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  * TLSv1.3 (OUT), TLS handshake, Finished (20):
  * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  * ALPN, server accepted to use h2
  * Server certificate:
  *  subject: CN=*.pegaxy.io
  *  start date: Mar  3 05:22:24 2022 GMT
  *  expire date: Jun  1 05:22:23 2022 GMT
  *  subjectAltName: host &quot;pegaxy.io&quot; matched cert's &quot;pegaxy.io&quot;
  *  issuer: C=US; O=Let's Encrypt; CN=E1
  *  SSL certificate verify ok.
  * Using HTTP2, server supports multi-use
  * Connection state changed (HTTP/2 confirmed)
  * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  * Using Stream ID: 1 (easy handle 0x671500)
  &gt; GET / HTTP/2
  &gt; Host: pegaxy.io
  &gt; accept-encoding: deflate, gzip
  &gt; user-agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0
  &gt; accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
  &gt; accept-language: en-US,en;q=0.5
  &gt; connection: keep-alive
  &gt; upgrade-insecure-requests: 1
  &gt; sec-fetch-dest: document
  &gt; sec-fetch-mode: navigate
  &gt; sec-fetch-site: none
  &gt; sec-fetch-user: ?1
  &gt; pragma: no-cache
  &gt; cache-control: no-cache
  &gt;
  * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  * old SSL session ID is stale, removing
  * Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
  &lt; HTTP/2 403
  &lt; date: Fri, 18 Mar 2022 13:03:04 GMT
  &lt; content-type: text/html; charset=UTF-8
  &lt; cache-control: max-age=15
  &lt; expires: Fri, 18 Mar 2022 13:03:19 GMT
  &lt; x-frame-options: SAMEORIGIN
  &lt; expect-ct: max-age=604800, report-uri=&quot;https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct&quot;
  &lt; set-cookie: __cf_bm=rlU7vb3eTQzw02vcpzOo6gweMMadJXkNxsft3MqPLSY-1647608584-0-AXk3yx3EOmlDZ+tIGWB3S+1ud6hWmykBwT7IwKtO+e+eCdY36JjTgyM3SkdIyBeWvtZphzvnBZLCVE4R6YogbxI=; path=/; expires=Fri, 18-Mar-22 13:33:04 GMT; domain=.pegaxy.io; HttpOnly; Secure; SameSite=None
  &lt; vary: Accept-Encoding
  &lt; server: cloudflare
  &lt; cf-ray: 6ede2a920d7392c5-FRA
  &lt; content-encoding: gzip
  &lt;
  &lt;!DOCTYPE html&gt;
  &lt;!--[if lt IE 7]&gt; &lt;html class=&quot;no-js ie6 oldie&quot; lang=&quot;en-US&quot;&gt; &lt;![endif]--&gt;
  &lt;!--[if IE 7]&gt;    &lt;html class=&quot;no-js ie7 oldie&quot; lang=&quot;en-US&quot;&gt; &lt;![endif]--&gt;
  &lt;!--[if IE 8]&gt;    &lt;html class=&quot;no-js ie8 oldie&quot; lang=&quot;en-US&quot;&gt; &lt;![endif]--&gt;
  &lt;!--[if gt IE 8]&gt;&lt;!--&gt; &lt;html class=&quot;no-js&quot; lang=&quot;en-US&quot;&gt; &lt;!--&lt;![endif]--&gt;
  &lt;head&gt;
  &lt;title&gt;Attention Required! | Cloudflare&lt;/title&gt;
  &lt;meta charset=&quot;UTF-8&quot; /&gt;
  &lt;meta http-equiv=&quot;Content-Type&quot; content=&quot;text/html; charset=UTF-8&quot; /&gt;
  &lt;meta http-equiv=&quot;X-UA-Compatible&quot; content=&quot;IE=Edge,chrome=1&quot; /&gt;
  &lt;meta name=&quot;robots&quot; content=&quot;noindex, nofollow&quot; /&gt;
  &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width,initial-scale=1&quot; /&gt;
  &lt;link rel=&quot;stylesheet&quot; id=&quot;cf_styles-css&quot; href=&quot;/cdn-cgi/styles/cf.errors.css&quot; type=&quot;text/css&quot; media=&quot;screen,projection&quot; /&gt;
  &lt;!--[if lt IE 9]&gt;&lt;link rel=&quot;stylesheet&quot; id='cf_styles-ie-css' href=&quot;/cdn-cgi/styles/cf.errors.ie.css&quot; type=&quot;text/css&quot; media=&quot;screen,projection&quot; /&gt;&lt;![endif]--&gt;
  &lt;style type=&quot;text/css&quot;&gt;body{margin:0;padding:0}&lt;/style&gt;
  
  
  &lt;!--[if gte IE 10]&gt;&lt;!--&gt;
  &lt;script&gt;
    if (!navigator.cookieEnabled) {
      window.addEventListener('DOMContentLoaded', function () {
        var cookieEl = document.getElementById('cookie-alert');
        cookieEl.style.display = 'block';
      })
    }
  &lt;/script&gt;
  &lt;!--&lt;![endif]--&gt;
  
  
  &lt;/head&gt;
  &lt;body&gt;
    &lt;div id=&quot;cf-wrapper&quot;&gt;
      &lt;div class=&quot;cf-alert cf-alert-error cf-cookie-error&quot; id=&quot;cookie-alert&quot; data-translate=&quot;enable_cookies&quot;&gt;Please enable cookies.&lt;/div&gt;
      &lt;div id=&quot;cf-error-details&quot; class=&quot;cf-error-details-wrapper&quot;&gt;
        &lt;div class=&quot;cf-wrapper cf-header cf-error-overview&quot;&gt;
          &lt;h1 data-translate=&quot;block_headline&quot;&gt;Sorry, you have been blocked&lt;/h1&gt;
          &lt;h2 class=&quot;cf-subheadline&quot;&gt;&lt;span data-translate=&quot;unable_to_access&quot;&gt;You are unable to access&lt;/span&gt; pegaxy.io&lt;/h2&gt;
        &lt;/div&gt;&lt;!-- /.header --&gt;
  
        &lt;div class=&quot;cf-section cf-highlight&quot;&gt;
          &lt;div class=&quot;cf-wrapper&quot;&gt;
            &lt;div class=&quot;cf-screenshot-container cf-screenshot-full&quot;&gt;
  
                &lt;span class=&quot;cf-no-screenshot error&quot;&gt;&lt;/span&gt;
  
            &lt;/div&gt;
          &lt;/div&gt;
        &lt;/div&gt;&lt;!-- /.captcha-container --&gt;
  
        &lt;div class=&quot;cf-section cf-wrapper&quot;&gt;
          &lt;div class=&quot;cf-columns two&quot;&gt;
            &lt;div class=&quot;cf-column&quot;&gt;
              &lt;h2 data-translate=&quot;blocked_why_headline&quot;&gt;Why have I been blocked?&lt;/h2&gt;
  
              &lt;p data-translate=&quot;blocked_why_detail&quot;&gt;This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting
  a certain word or phrase, a SQL command or malformed data.&lt;/p&gt;
            &lt;/div&gt;
  
            &lt;div class=&quot;cf-column&quot;&gt;
              &lt;h2 data-translate=&quot;blocked_resolve_headline&quot;&gt;What can I do to resolve this?&lt;/h2&gt;
  
              &lt;p data-translate=&quot;blocked_resolve_detail&quot;&gt;You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.&lt;/p&gt;
            &lt;/div&gt;
          &lt;/div&gt;
        &lt;/div&gt;&lt;!-- /.section --&gt;
  
        &lt;div class=&quot;cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300&quot;&gt;
    &lt;p class=&quot;text-13&quot;&gt;
      &lt;span class=&quot;cf-footer-item sm:block sm:mb-1&quot;&gt;Cloudflare Ray ID: &lt;strong class=&quot;font-semibold&quot;&gt;6ede2a920d7392c5&lt;/strong&gt;&lt;/span&gt;
      &lt;span class=&quot;cf-footer-separator sm:hidden&quot;&gt;&amp;bull;&lt;/span&gt;
      &lt;span class=&quot;cf-footer-item sm:block sm:mb-1&quot;&gt;&lt;span&gt;Your IP&lt;/span&gt;: 46.62.217.20&lt;/span&gt;
      &lt;span class=&quot;cf-footer-separator sm:hidden&quot;&gt;&amp;bull;&lt;/span&gt;
      &lt;span class=&quot;cf-footer-item sm:block sm:mb-1&quot;&gt;&lt;span&gt;Performance &amp;amp; security by&lt;/span&gt; &lt;a rel=&quot;noopener noreferrer&quot; href=&quot;https://www.cloudflare.com/5xx-error-landing&quot; id=&quot;brand_link&quot; target=&quot;_blank&quot;&gt;Cloudflare&lt;/a&gt;&lt;/span&gt;
  
    &lt;/p&gt;
  &lt;/div&gt;&lt;!-- /.error-footer --&gt;
  
  
      &lt;/div&gt;&lt;!-- /#cf-error-details --&gt;
    &lt;/div&gt;&lt;!-- /#cf-wrapper --&gt;
  
    &lt;script type=&quot;text/javascript&quot;&gt;
    window._cf_translation = {};
  
  
  &lt;/script&gt;
  
  &lt;/body&gt;
  &lt;/html&gt;
  * Connection #0 to host pegaxy.io left intact
  </code></pre>
  <p>Tested on <code>windows</code> and <code>linux</code>.</p>
  <pre><code>$ curl -V
  curl 7.70.0 (x86_64-w64-mingw32) libcurl/7.70.0 OpenSSL/1.1.1g (Schannel) zlib/1.2.11 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.40.0
  Release-Date: 2020-04-29
  Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
  Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP
  </code></pre>
  <p>Note that talking about is exactly the first request. So this can not be related to cookies or a piece of JavaScript code that checks the status in the browser.</p>
  <p>Also I'm even aware of CloudFlare's sensitivities about the IP and rate of requests, but these are for when the requests are consecutive and confusing, and eventually the challenge is displayed. But in my tests, the same IP on the web is not a problem and only the first request occurs.</p>
  <p>No problem on <code>Chrome</code>, <code>Firefox</code>, <code>Edge</code>, <code>Brave</code>, <code>Tor</code> web browsers. <br />
  But there is a problem with <code>CURL</code>, <code>wget</code>, <code>nghttp</code>, <code>lynx</code> on the command line.
  I also tested several <code>nodejs</code> packages that had problems.</p>
  <p><strong>Question: When all the conditions are the same, how does CloudFlare find out that the request is not from the browser, and how can it be simulated or bypassed on the command line without the use of a web browser?</strong></p>
  <p>Keep in mind that I know about <code>Selenium</code>, <code>Flaresolverr</code>, <code>pupflare</code>, etc. and I do not intend to use the browser because they render the page and slow down the operation.</p>
  <p>Things I did and did not get an answer:</p>
  <ul>
  <li>I thought maybe the problem is the lack of <code>push server http2</code> support in the CURL command line. So I wrote it in PHP and implement push server in it, but the problem was not solved.</li>
  <li>I thought the problem was with the certificate, so I downloaded the browser certificate and converted it to <code>pem</code> file and used it in CURL, but the problem was not solved.</li>
  </ul>
  <p><strong>I just want to get the same 200 response code on the terminal in the first case if I getting the code 200 in the first request on the web browser!</strong></p>