Website remote render d3.js server side
Question: Looking for a solution to an arguably strange problem. Ok, so we are using
d3.js to plot charts and graphs. However our data sets can be very small, to
intensely massive. Right now most of what we are doing is internal and just
prototyping. However, we do show clients these charts and draw them in real
time for them, quite often and rapidly change their inputs.

Doing this in D3 looks great, but can be slow as expected. I'm more interested
in what the possibilities are for this process. Go to our website, loging, and
show an instance of our dashboard being rendered remotely on the server. Our
server cluster is a super demon beast so I'm not worried about it doing any
heavy lifting. It can do these processes about 100x faster than our best pc so
it seems if we could setup our website to create instances on the fly of our
dashboard, BUT only have access to that user accounts data.

This is getting a bit convoluted so let me explain. We have a database, full
of millions of data points. We have about 10 user accounts. Each have access
to different pieces of this data. One has access to all of it, the other some
of it. All of this is not the issue we are looking for a solution on. We are
more interested in the ability of our server to create multiple instances of
our site, through a window essentially, that the user is remotely controlling.
Like a Remote Desktop in a way. We could even start with the user login form
being part of the remote render. Where our system is fully hosted and operates
on the server itself, and the we page is essentially a KVM on the server in a
way. However it needs to handle multiple users at the same time.

We are using Centos 6.4 lots of python for the back end stuff, php HTML and a
mixture of Postgres and SQLite, but I doubt any of this is important. Just
want to cover my bases.


Answer: It seems unlikely to me that you'd be able meaningfully display millions of
datapoints on a single screen without grouping and summarizing them in some
way. Do the processing and summarize the data on the server and ship the
resulting smaller datasets to the client, which will then plot your graphs and
charts from that. It's likely you'll have more than one set of data now, but
it should result in much better client performance. e.g.

  * {millions of points} -> transform on server -> data for bar chart to client
  * {millions of points} -> transform on server -> data for XY-scatter chart
  * etc.

What you've proposed is not really a programming issue, and isn't going to
scale very well.

