Casting the left and right numbers as integers or doubles
One option would be to do your math operations on a double.  You can use a generic method that accepts any derivative of the Number class and use the doubleValue() method for your calculation.  The caller of the method can cast to Integer as needed:

enum Operators {
    ADD,
    SUB,
    MUL,
    DIV
}

public static &lt;T1 extends Number&gt; Double doMath(T1 left, T1 right, Operators operator) {
    switch (operator) {
        case ADD:
            return left.doubleValue() + right.doubleValue();
        case SUB:
            return left.doubleValue() - right.doubleValue();
        case MUL:
            return left.doubleValue() * right.doubleValue();
        case DIV:
            return left.doubleValue() / right.doubleValue();
        default:
            return 0.0;
    }
}