title: Application Insights with big data
body: <p>I am looking to use a jvm-profiler (<a href="https://github.com/uber-common/jvm-profiler" rel="nofollow noreferrer">https://github.com/uber-common/jvm-profiler</a>) on my spark clusters and correlate this with my application logs. On a large cluster (about 1000 executors), this will produce a lot of data. I tried dumping it into an influxDb which worked well. However, then I have to manage the influxDb. My stack is all in Azure, and we use Application Insights.</p>
  <p>Could application insights handle this kind of load, or is there a better Azure alternative. My plan would then be to use Power BI to create dashboards around this data.</p>
  <p>A current alternative is to use influxDb with Chronograf to visualize the data.</p>