CSMR for large-scale text-prcessing I'm working on a project for large-scale text-processing, which is a first implementation of the basic idea of CSMR. CSMR is an algorithm that measures the similarity between documents by calculating their cosine in the vector space in parallel manner, using MapReduce model.

Here is my implementation on GitHub.

Some review on the code, the algorithm, corrections and maybe proposals for changes would be gratefully appreciated. Below are the 3 most important classes:

Mapper

public class CSMRMapper extends Mapper&lt;Text,VectorWritable,IntWritable,
        DocumentWritable &gt;{

    @Override
    public void map(Text key, VectorWritable value, Context context) 
            throws IOException, InterruptedException{   
            DocumentWritable p = new DocumentWritable(new Text(key.toString())
                    ,new VectorWritable(value.get()));
            context.write(new IntWritable(1),p);

    }
}


Reducer (Pair Generator)

public class CSMRReducer extends Reducer&lt;IntWritable, DocumentWritable ,Text, 
        VectorArrayWritable&gt; {

    private ArrayList&lt;DocumentWritable&gt; al;
    private VectorWritable[] val;
    private VectorArrayWritable vaw; 

    @Override
    public void reduce(IntWritable key, Iterable&lt;DocumentWritable&gt; values, 
            Context context) throws IOException, InterruptedException{
        al = new ArrayList();
        vaw = new VectorArrayWritable();

        /* Storing each key-value pair (document) in a java.util.ArrayList */
        for (DocumentWritable v : values){
            al.add(new DocumentWritable(v.getKey(),v.getValue()));
        }

        /* Generating all the possible combinations of documents */
        if (al.size()&gt;0){
            for (int i=0;i&lt;al.size();++i){
                for (int j=i+1;j&lt;al.size();++j){
                    val = new VectorWritable[2];

                    /* Generating the key for the current document pair with
                        the format "doci_name@docj_name" */
                    String k = al.get(i).getKey().toString()+
                            "@"+al.get(j).getKey().toString();

                    //First Document (doci)
                    val[0] = new VectorWritable(al.get(i).getValue().get());
                    //Second Document (docj)
                    val[1] = new VectorWritable(al.get(j).getValue().get());
                    vaw.set(val);

                    context.write(new Text(k), vaw);
                }
            }  
        }
    }

}


The Final Reducer (Cosine Calculator)

public class CosineSimilarityReducer 
    extends Reducer&lt;Text, VectorArrayWritable, Text, DoubleWritable&gt;{

    @Override
    public void reduce(Text key, Iterable&lt;VectorArrayWritable&gt; value, Context context)
    throws IOException, InterruptedException{

        CosineDistanceMeasure cdm = new CosineDistanceMeasure();
        VectorWritable docX,docY;
        double cosine;
        for (VectorArrayWritable v : value){
            docX = (VectorWritable)v.get()[0];
            docY = (VectorWritable)v.get()[1];
            cosine = cdm.distance(docX.get(), docY.get());
            context.write(key, new DoubleWritable(cosine));
        }

    }        
}
public class CSMRReducer extends Reducer&lt;IntWritable, DocumentWritable ,Text, 
        VectorArrayWritable&gt; {

    private ArrayList&lt;DocumentWritable&gt; al;
    private VectorWritable[] val;
    private VectorArrayWritable vaw; 

    @Override
    public void reduce(IntWritable key, Iterable&lt;DocumentWritable&gt; values, 
            Context context) throws IOException, InterruptedException{
        al = new ArrayList();
        vaw = new VectorArrayWritable();

        /* Storing each key-value pair (document) in a java.util.ArrayList */
        for (DocumentWritable v : values){
            al.add(new DocumentWritable(v.getKey(),v.getValue()));
        }

        /* Generating all the possible combinations of documents */
        if (al.size()&gt;0){
            for (int i=0;i&lt;al.size();++i){
                for (int j=i+1;j&lt;al.size();++j){
                    val = new VectorWritable[2];

                    /* Generating the key for the current document pair with
                        the format "doci_name@docj_name" */
                    String k = al.get(i).getKey().toString()+
                            "@"+al.get(j).getKey().toString();

                    //First Document (doci)
                    val[0] = new VectorWritable(al.get(i).getValue().get());
                    //Second Document (docj)
                    val[1] = new VectorWritable(al.get(j).getValue().get());
                    vaw.set(val);

                    context.write(new Text(k), vaw);
                }
            }  
        }
    }

}


You can remove the fields and just declare the variables in the function. There's no need for class variables here.

You can also remove the al.size()&gt;0 check, because that's automatically checked by the for loop.

On the performance end, you could prepare al.get(i) and al.get(i).getKey().toString()+"@" before going into the j loop.