Reconstruction of a signal from non-uniform samples
The problem you have to solve is having a model for how your samples uniquely represent actually continuous input.  For equidistant samples, the usual model is to stipulate (and then guarantee by analog filtering, oversampling, digital filtering, and decimation) that there are no signal components exceeding the Nyquist frequency.

What is your model going to be for the kind of non-uniform sampling you are working with?  If the information you work with is not sufficient for that model, you are going to have non-uniquely interpretable results, namely aliasing.