title: Neo4j : Performance issue
body: <p>We are using Neo4j 3.4 Enterprise Edition as our application’s data store. It is running on EC2 instance with an 8 core CPU and 16GB RAM.</p>
  
  <p>We have uploaded these datasets :</p>
  
  <ol>
  <li>Sales: Consists of facts like Dollars, Discount, Volume and Price per Volume etc., visualized against dimensions Brand, Week and Business Unit etc.</li>
  <li>Weather:  Consists of facts Temperature, Precipitation and Snowfall, visualized against dimensions Day, Month and Weather Station</li>
  <li>Media: Consists of facts Spend, TRPs and Impression, visualized against dimensions Week, Business Unit, Brand, Channel, Audience and Campaign</li>
  </ol>
  
  <p>It is a generic model that we have designed that could accommodate different types of datasets.</p>
  
  <p>The model is based on the following Arrow diagram -</p>
  
  <p><a href="https://i.stack.imgur.com/Oac2g.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Oac2g.png" alt="Domain Diagram"></a></p>
  
  <p><a href="https://i.stack.imgur.com/Oac2g.png" rel="nofollow noreferrer">Arrow Diagram</a></p>
  
  <p>The type of queries we are running on it are not dependent on the type of application that dataset belongs to. It is a general use case wherein we are trying to fetch all details related to events that happened at same ‘time’ or ‘location’.</p>
  
  <p>Design philosophy :</p>
  
  <ol>
  <li>Each node represents a row of data in the fact table. The values of those facts are captured as properties. If a dataset has 2 million rows in fact table, it will have 2 million corresponding nodes following the pattern above. Check PosSales node.</li>
  <li>On what basis did we create relationships – all measures from one dataset are on the same node as properties, while dimensions are connected to the nodes. Each dimension is a separate node. </li>
  <li>Conformed dimensions for time and location are used to establish paths/relationships between disparate datasets like Sales, Weather and Media Delivery - time, location, or dataset specific dimensions</li>
  </ol>
  
  <p>We are looking to perform JOINs between these above-mentioned datasets. The result set should return an INNER JOIN between these three mentioned datasets to present say, one fact each from the three datasets, like these – Dollar Sales, Daily Weather, and Impression. These would be presented grouped by a combination of dimensions from all three datasets.
  The database has 1M Nodes and 2M relationships.</p>
  
  <p>Neo4j is configured to use maximum up to 80% of total system memory 16 GB.
  Apparent problem – </p>
  
  <p>Our queries are running into combinatorial explosion as highlighted in below diagram. We looked at query execution plan that reflects count of db above 100M hits in few stages. </p>
  
  <p><a href="https://i.stack.imgur.com/Rnpbn.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Rnpbn.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/Rnpbn.png" rel="nofollow noreferrer">Arrow Diagram</a></p>
  
  <p>Here are few statistics of each labels</p>
  
  <ol>
  <li>PosSales – 178k nodes</li>
  <li>SKU_IRI – 237 nodes </li>
  <li>SKU_Dist – 90 nodes</li>
  <li>DistributorShipments – 680k nodes</li>
  </ol>
  
  <p>We want to get facts from two datasets i.e. Sales and Shipments. When we connect PosSales and DistributorShipments nodes through SKU_IRI and SKU_Dist, combinatorial explosion is happening. </p>
  
  <p>How should we address the combinatorial explosion? </p>
  
  <p>Is there any better ways of modeling our graph to avoid these types of issues?</p>
  
  <p>Is there some configuration or setting that I can tweak on Neo4j server that can allow it to use more RAM and CPU which are clearly underutilized, so that queries run faster?</p>
  
  <p><a href="https://docs.google.com/document/d/159-MroRGH7wSVTgovVvlA7i5ILxE_Qsap2j9qoQhzkw/edit" rel="nofollow noreferrer">Attaching document which has few sample queries</a></p>