How to compare experiments run over different infrastructures I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to reduce the time spent with I/O; and with an efficient load balance policy, we can distribute tasks without much data replication overhead.

There are many studies on the literature that deal with the same problem, and each of them runs different experiments to evaluate their proposal. Some experiments are specific of the strategy presented, and some others, like weak scaling (scalability) and strong scaling (speedup), are common to all of the works.

The problem is the experiments are usually executed over entirely different infrastructures (disks, processors, # machines, network), and depending on what is being evaluated, it may raise false/unfair comparisons. For example, I may get 100% of speedup in my application running on 10 machines with Infiniband connection, whereas I could get the same or even worse results if my connection was Ethernet.

So, how can one honestly compare different experiments to point out efficiency gains?
This is a very good question and a common situation.

In my opinion there are three different factors that must be controlled:


Data: There exist already different benchmarks in order to evaluate algorithms and architectures. These data must be publicly available so that everybody can contrast their approaches.
Architecture: My suggestion is to test everything on the cloud, so that everyone can contrast their results and also there is no doubt the same machines and software is used.
Algorithms: If you have developed a distributed algorithm, it would be great to compare your algorithm on a specific data. In this case, algorithms must not be public.


So, answering your question, if you want to compare different experiments and state to what extent your distributed algorithm outperforms others, you should try to replicate as accurate as possible the same environment (data and architecture) where the experiments were carried out.

If this is not possible, my suggestion is that you test your algorithm with public data and cloud architecture so that you become a referent as you are facilitating the comparison of future algorithms.