title: How to circumvent SPICE limitations (500 M rows) to create a QuickSight dashboard for a big data set?
body: <p>My goal is to quickly &amp; dynamically visualize a big data set (&gt; 500 M rows) using QuickSight. To achieve quick query times, it's necessary to load all of the data into SPICE. However, AWS currently has a hard limit for the maximum number of rows that can be imported into SPICE for a single data set, which is 500 M rows. I currently don't see any option that could be used to visualize all of the data. Here are things that I already considered:</p>
  <ul>
  <li><strong>Splitting the full data set into individual QS datasets:</strong> the problem with this approach is that QuickSight requires that each visual has a single dataset as an input, so values from multiple datasets cannot be shown in the same visual. I'm aware that multiple datasets can be used within one dashboard but that would not suit the use-case of having a single plot visualizing the data.</li>
  <li><strong>Pivoting the table:</strong> the input table has a lot of rows, so changing the format from long to wide table would circumvent the SPICE row limitations. However, QuickSight doesn't seem to support using an array of columns a y-values to be plotted.</li>
  <li><strong>Creating a dataset per visualization:</strong> Certain visualizations can theoretically be defined using fewer values than in the original data set. For example, to create a box plot over a set of groups, we mainly need the quartile values for each of the groups to be plotted, rather than the full data set, which would allow us to be below the SPICE limitation. However, QuickSight doesn't allow creating <em>custom</em> plots such as creation of a box plot where quartiles are already pre-processed.</li>
  </ul>
  <p>Currently, the only viable approach I see is to create a dashboard per user, since most users would only be interested in a subset of rows from the full data set.</p>