CodeChef- Chef and Strange Formula
As youself noted F(x) can be reduced to:

x*x*(x+1)/2 + (x+1)! - 1


As @Daniel Sokolov pointed, you don't need to calculate full factorial for each p[i]. If p is a sorted array, you can use previously calculated factorial in the current (x+1)! calculation.

Furthermore it would be better to get modulo after each arithmetic operation. This ensure that the current result after each operation will always &lt; m*m &lt; 10^14.

So the code might look like this:

/**
 *  @param m modulo
 *  @param n number of integers
 *  @param p sorted array of integers
 *  @return sum of F(pn) series modulo m
*/
function sum(unsigned long m, unsigned n, unsigned long long int *p) {
    unsigned long long int
        x, // current element of p array
        x0 = 1, // previous element of p array
        a, // sum of arithmetic progression
        fac = 1, // factorial of (x + 1)
        ret = 0; // result
    for (unsigned i = 0; i &lt; n; i++) {
        x = p[i];
        if (x &lt; x0) return 0;
        a = x % m * x % m * ((x + 1) / 2 % m) % m;
        for (var k = x0 + 1; k &lt;= x + 1; k++)
            fac = fac * (k % m) % m;
        ret = ((ret + a) % m + fac - 1) % m;
        x0 = x;
    }
    return ret;
}


Note that it assumes p is a sorted array.