title: Google CDN random not serving gzip content
body: <p>I need some advanced people to give me an advice is this is an Google CDN bug or i miss something. I discover this bug like 4 months ago, tried to contact their support, but they were so rude, that i don`t want to even speak here about that. They accepted, at least they told me that they will gonna send the problem to back-end team but after that they deleted the issue tracker and they dont response to my emails anymore. That is the main reason why i ask here.</p>
  
  <p><strong>Problem</strong></p>
  
  <p>Google CDN randomly not serving gzip content to end user. So instead of ~70KB they download 500KB files. I can not produce this problem directly to my origin, but i can produce this problem very easy on Google CDN.</p>
  
  <p>Here is example request to CDN:</p>
  
  <p>Request:</p>
  
  <pre><code>Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
  Accept-Encoding:gzip, deflate, sdch, br
  Accept-Language:en-US,en;q=0.8,bg;q=0.6,hr;q=0.4,mk;q=0.2,sr;q=0.2
  Cache-Control:no-cache
  Connection:keep-alive
  Cookie: example
  Host: example.com
  Pragma:no-cache
  Upgrade-Insecure-Requests:1
  User-Agent:Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36
  </code></pre>
  
  <p>Response:</p>
  
  <pre><code>Accept-Ranges:bytes
  Age:58422
  Alt-Svc:clear
  Cache-Control:public, max-age=604800
  Content-Length:550158
  Content-Type:text/css
  Date:Tue, 04 Apr 2017 03:45:53 GMT
  Expires:Tue, 11 Apr 2017 03:45:53 GMT
  Last-Modified:Sun, 19 Mar 2017 01:50:22 GMT
  Server:LiteSpeed
  Via:1.1 google
  </code></pre>
  
  <p>As you can see, my request have accept-encoding:gzip header but i receive not gzip content. Instead of 70KB i receive 500KB. Also note Age header, that item is cached/exist on CDN for 58422 seconds!</p>
  
  <p>Here is the same request from another machine (US)</p>
  
  <p>Request:</p>
  
  <pre><code>:authority: xxx
  :method:GET
  :path:/wp-content/themes/365/style.css
  :scheme:https
  accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
  accept-encoding:gzip, deflate, sdch, br
  accept-language:en-US,en;q=0.8
  cache-control:no-cache
  cookie: xxx
  pragma:no-cache
  upgrade-insecure-requests:1
  user-agent:Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
  </code></pre>
  
  <p>Response:</p>
  
  <pre><code>accept-ranges:bytes
  age:58106
  alt-svc:clear
  cache-control:public, max-age=604800
  content-encoding:gzip
  content-length:72146
  content-type:text/css
  date:Tue, 04 Apr 2017 03:49:28 GMT
  expires:Tue, 11 Apr 2017 03:49:28 GMT
  last-modified:Sun, 19 Mar 2017 01:50:22 GMT
  server:LiteSpeed
  status:200
  vary:Accept-Encoding
  via:1.1 google
  </code></pre>
  
  <p>As you can see, i got an gzip content from my other server.</p>
  
  <p>I have a tons of HAR files and also videos that i prove this bug, but lets keep it simple. Google CDN logs are available in GCP dashboard, check how they looks like.</p>
  
  <p><a href="https://i.stack.imgur.com/vmD5N.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vmD5N.jpg" alt="enter image description here"></a></p>
  
  <p>If all my visitors does not support gzip, what about GoogleBot?</p>
  
  <p><a href="https://i.stack.imgur.com/syDBC.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/syDBC.jpg" alt="enter image description here"></a></p>
  
  <p>I also analyzed my server logs and i found stats like 99% response size for that file is as gzip, only few requests are not gzip. Very logical, as some visitors or i prefer to say robots requested that file without gzip header.</p>
  
  <p><strong>Temporary solve the problem</strong> </p>
  
  <p>If i purge the CDN cache, this problem does not exist in next minutes/hours. After some time, it still happens. Also this problem does not always happens, but randomly. I got system that parse CDN logs and show me graphs, that is actually how i discover this bug.</p>
  
  <p><a href="https://i.stack.imgur.com/SQnog.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SQnog.jpg" alt="enter image description here"></a>  </p>
  
  <p>Whenever i see chart bandwidth increase (double as normal), when i log into google dashboard and check logs, i find those 500KB logs like 50% of that file requests, and it`s easy to produce the bug in browser, i just login to my servers, request the file and get random results.</p>
  
  <p>Ill be so happy if the problem is in my origin since ill solve in 1 minute, but i think it is Google CDN bug. I`ll be happy if any person more into CDN technology to assist me or some person from Google Cloud.</p>
  
  <p>EDIT:</p>
  
  <p>As i said, this bug happens in random time frame, here is an video i recorded now that show us an 'NO BUG TIME FRAME'. As you can see, every response is compressed.</p>
  
  <p><a href="https://www.youtube.com/watch?v=UC29yg-1WQ0&amp;feature=youtu.be" rel="nofollow noreferrer">NO BUG TIME FRAME CDN VIDEO</a> </p>
  
  <p>EDIT2:</p>
  
  <p>Here is an graph that shows number of gzip and not gzip responses for an single .css url test.</p>
  
  <p><a href="https://i.stack.imgur.com/YFOsx.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YFOsx.jpg" alt="stacking lines"></a></p>
  
  <p>EDIT3:</p>
  
  <p>On first graph image, lines are stack-able, here is same graph without stacking. As you can see, some hours have near 100% not gzip responses.</p>
  
  <p><a href="https://i.stack.imgur.com/JUkAU.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JUkAU.jpg" alt="not stacking lines"></a></p>
  
  <p>EDIT4:</p>
  
  <p>Here are my origin parsed logs for the same css file.</p>
  
  <p>1060 requests were served with response size below 100KB. 200,304,206 response codes.
  32 requests were served with response size above 100KB. 200 and 206 response codes.</p>
  
  <p><a href="https://i.stack.imgur.com/jtTY0.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jtTY0.jpg" alt="origin server"></a></p>
  
  <p>EDIT5:</p>
  
  <p>Analyzing 1-7 April logs here are some additional stats for single .css url:</p>
  
  <p>19803 CDN requests were served with > 100KB (not gzip)</p>
  
  <p>41004 CDN requests were served with &lt; 100KB (gzip)</p>
  
  <p>29 Cache Fill from origin with > 100KB (not gzip)</p>
  
  <p>924 Cache Fill from origin with &lt; 100KB (gzip)</p>
  
  <p>423 Cache-To-Cache fill with > 100KB (not gzip)</p>
  
  <p>2295 Cache-To-Cache fill with &lt; 100KB (gzip)</p>
  
  <p>I'm surprised how Cache-To-Cache fill is very effective, amazing. </p>
  
  <p><strong>SOLUTION</strong></p>
  
  <p>There is no bug in origin not even in Google CDN. Problem is when Google CDN receive an cache-able entity without 'Vary:Accept-Encoding' when request did not send 'Accept-Encoding:gzip', so Google CDN will store that uncompressed response and will <strong>overwrite all stored compressed cache entities</strong>. So next time when user try to get some file for example .css, Google CDN will answer like:</p>
  
  <ol>
  <li>I received this file from origin and its not vary by anything.</li>
  <li>Send uncompressed response.</li>
  </ol>
  
  <p>Be aware that web servers are not configured to send 'Vary:Accept-Encoding' headers on requests that does not have 'Accept-Encoding:gzip' headers. I tested this on Litespeed, Apache, Nginx and Cloudflare Nginx.</p>
  
  <p><strong>I highly recommend Google team to update the documentation about this</strong>. There is an statement about 'Vary headers' but no one will get the point regarding this issue since not me, not Google first level support (i also had 20 days communication on Google issue tracker with two Google support persons), stack-overflow or other person answer the problem.</p>
  
  <p>Additional the documentation says:</p>
  
  <pre><code>In addition to the request URI, Cloud CDN respects any Vary headers that instances include in responses.
  </code></pre>
  
  <p>But nothing when request does not have 'Vary' header.</p>
  
  <p>This is how i fix it:</p>
  
  <pre><code>&lt;FilesMatch '.(js|css|xml|gz|html|txt|xml|xsd|xsl|svg|svgz)$'&gt;
      Header merge Vary Accept-Encoding
    &lt;/FilesMatch&gt;
  </code></pre>