How to apply upperquartile normalization on RSEM expected counts?
You can use the quantile function in base R to get the value of a particular quantile (e.g. 0.75 for the upper quartile). This can then be used as a factor for normalisation: divide the observed expression by this number.

## set up simulated data
data.mat &lt;- floor(10^data.frame(sapply(1:5,function(x){rnorm(100)})));
colnames(data.mat) &lt;- paste0("S",1:5);
data.mat[data.mat &gt; 0] &lt;- floor(data.mat[data.mat &gt; 0]);


It's generally a good idea to first remove genes that have no counts in any samples. Rows can be removed using a logical selection based on the total expression in any given row.

## Only retain rows that have a positive total expression per gene
data.mat &lt;- data.mat[rowSums(data.mat) &gt; 0,];


Two methods of upper quartile determination are presented here, one which is based on the total counts, and one based on the expressed counts (i.e. excluding values with no expression). These both use the quantile function of R. The apply function calculates statistics that are based on a single specified dimension (or multiple dimensions) of the data. The first dimension of a 2D matrix in R corresponds to the row, while the second corresponds to the column. By using 2 in the apply function, data are collected along the second dimension (i.e. the column), and statistics generated from that:

data.quantileAll &lt;- apply(data.mat, 2, function(x){quantile(x, 0.75)});
data.quantileExpressed &lt;-
  apply(data.mat, 2, function(x){quantile(x[x&gt;0], 0.75)});


Normalisation is applied by dividing the values by the column-based upper quartile statistic. R will repeat values along the last dimension first (which in this case is the column), so the matrix needs to be transposed first (with columns as genes) to get the arithmetic working properly, with a final transposition to return the orientation back to the original direction (with columns as samples).

data.norm &lt;- t(t(data.mat) / data.quantileExpressed);