title: Sitemap Generation Solution for a dynamic website
body: <p><strong>CONTEXT</strong><br>
  Assume my website contains a listing of shops organized across country, state, city and locality. Each shop has a dynamically generated web page. The total number of shops would eventually reach ~1.5 million. I use NDB to store shop data. I plan to use XML sitemap and submit them manually to the search engines. I use GAE Python. </p>
  
  <p><strong>PROBLEM</strong><br>
  I want to maintain (generate and keep updated) url links in the sitemap for all shop pages. 
  Each unique url link for a shop page contains the following: 
  Country, State, City, Locality, Shop Name, Unique Index </p>
  
  <p>eg, wwww.example.com/country--state--city--locality--shop_name--unique_index </p>
  
  <p>Shops could be added, deleted or their data(eg, their name or city, etc) could be changed. 
  I need to design a solution which helps me have the latest sitemap with updated links for all the shops. I intend to submit a new sitemap as soon as possible after any shop has been added/deleted/updated. </p>
  
  <p><strong>My Approaches</strong><br>
  <strong>Approach 1</strong><br>
  Generate the sitemap on the fly by querying the information from NDB models.<br>
  <strong>Cons of Approach 1</strong>  </p>
  
  <ol>
  <li>NDB fetch limit of 10,000.  </li>
  <li>Read operation free quota of 50,000. </li>
  <li>High consumtion of Frontend instance hours. </li>
  <li>Operation completion time of 60 seconds.</li>
  </ol>
  
  <p><strong>Approach 2</strong><br>
  Generate and store the sitemap on my laptop using a program(say X, written by me in Perl/Python). Whenever a shop gets added / deleted / updated on my website, I would update a GCS(Google Cloud Storage) stored file with mnemonics like:  </p>
  
  <pre><code>ADD &lt; shop data like name, etc &gt;  
  DELETE &lt; shop data like name, etc &gt;  
  UPDATE &lt; shop data like name, etc &gt;
  </code></pre>
  
  <p>I would download and feed this file to my local program X to generate the sitemap by updating the older stored sitemap file.<br>
  <strong>Cons of Approach 2</strong>  </p>
  
  <ul>
  <li>GCS does not allow append of data to a file. Entire file needs to be written every time. So, with increase in the number of shops from 0 to 1.5 million, RAM usage and frontend instance hours consumtipn would peak.  </li>
  <li>Operation completion time of 60 seconds. </li>
  </ul>
  
  <p><strong>Approach 3</strong><br>
  The sitemap.xml file would contain:</p>
  
  <ul>
  <li><p>Entries with URLs for other Sitemap Index files on Country basis.
  These country sitemap files would contain entries for URLs of State
  sitemap files. Similarly, state sitemap files would contain entries
  for URLs of City sitemap files. Similarly, city sitemap file would
  contain entries for URLs of locality sitemap files. Locality sitemap
  files would contain entries for URLs of shop pages.</p></li>
  <li><p>Entries with URLs for all the static pages(like, FAQ, About Us, etc).</p></li>
  </ul>
  
  <p><strong>Pros of Approach 3</strong>  </p>
  
  <ul>
  <li>When a shop page gets added / deleted / updated, I need to update
  that particular sitemap file.</li>
  </ul>
  
  <p><strong>Doubts with Approach 3</strong>  </p>
  
  <ul>
  <li><p>Can I store all the sitemap.xml files in GCS? Do you foresee any
  problem with that?</p></li>
  <li><p>Is it allowed to have multiple levels of sitemap index files pointing
  to other sitemap index files?</p></li>
  </ul>
  
  <p>I am not able to find a good solution. I have seen similar questions on SO and Nick's blog but in vain. 
  I wish to remain within the free quota if possible. 
  Please provide your suggestions. </p>