How to scale up algorithm development? In working on exploratory data analysis, and developing algorithms, I find that most of my time is spent in a cycle of visualize, write some code, run on small dataset, repeat.   The data I have tends to be computer vision/sensor fusion type stuff, and algorithms are vision-heavy (for example object detection and tracking, etc), and the off the shelf algorithms don't work in this context.  I find that this takes a lot of iterations (for example, to dial in the type of algorithm or tune the parameters in the algorithm, or to get a visualization right) and also the run times even on a small dataset are quite long, so all together it takes a while. 

How can the algorithm development itself be sped up and made more scalable?

Some specific challenges: 

How can the number of iterations be reduced?  (Esp. when what kind of algorithm, let alone the specifics of it, does not seem to be easily foreseeable without trying different versions and examining their behavior)

How to run on bigger datasets during development?  (Often going from small to large dataset is when a bunch of new behavior and new issues is seen)

How can algorithm parameters be tuned faster?

How to apply machine learning type tools to algorithm development itself?  (For example, instead of writing the algorithm by hand, write some simple building blocks and combine them in a way learned from the problem, etc)
First off, if your data has as many variations (in function of time, context, and others) as to make it hard to apply a single strategy to cope with it, you may be interested in doing a prior temporal/contextual/... characterization of the dataset. Characterizing data, i.e., extracting information about how the volume or specifics of the content varies according to some criteria, usually provides with a better understanding (more consise and precise) than simply inferring algorithms on a brute-force fashion.

So, answering each question:


characterization is definitely a means of reducing the number of iterations while trying to select proper algorithms for specific data;
if you have a discrete set of criterias on which your data varies, it becomes much easier to scale up solutions, as will know what information you'd gain/lose if simpler/specific solutions were applied;
after a characterization, you should be also easier to select parameters, since you'd know what kind of specific data you'd be dealing with;
finally, you may use data mining/machine learning algorithms to support this characterization. This includes using:

clustering algorithms, to reduce the dimensionality of data;
classification algorithms, to help deciding on specific properties the data in function of time/context/... may present;
association rules, to predict particular knowledge from the dataset, while also improving/fine-graining the data used for later analysis;
and other possible strategies and analyses.



And here is a list of some criterias on which to analyse data, which you may find helpful.