Counter mode secure hash algorithm
First of all, I think I want to correct you at one point; in step 2, you aren't actually that interested in whether the operation is commutative, what you're actually interested in is that the operation is associative, that is, if $(a \oplus b) \oplus c = a \oplus (b \oplus c)$.  In essence, your operator $\oplus$ in step 2 turns out to be a group operation.

Hence, your proposed hash function can be expressed as:

$H(P_1, ..., P_n) = G( \sum_i F( P_i, i ) )$

where:


F is the your so-called 'cipher mimicking counter-mode encryption' (bad terminology, btw) in step 1, which outputs group members
$\sum$ is the group operation sum over all the group members
G is finalization operation in step 3, which takes as input a group member, and outputs the actual hash (and may also take the file length as input; that detail is irrelevant)


Now, assuming that F and G are well-chosen (for example, that it's infeasible to find collisions or preimages in F), the obvious way to attack this construct is to find a collision at the summation step;


for second-preimage on a hash of $P_i$, find a distinct set of blocks $Q_i$ such that $\sum_i F(P_i, i) = \sum_i F(Q_i, i)$
for a collision attack, find distinct sets of blocks $P_i$, $Q_i$ such that $\sum_i F(P_i, i) = \sum_i F(Q_i, i)$


Now, I have good news, and I have bad news.

The good news is that there exists groups where the above problems are NP-hard (!), and even better, one such group is integer addition modulo a large number.  There are also groups where the above problems are easy (such as the group of bitstrings where the group operation is bit-wise exclusive or's), so we'll need to be careful when choosing a group.

Now, the bad news; just because the problem is NP-hard doesn't mean that there aren't shortcuts.  One such short-cut to the second-preimage problem is a square-root attack which works by dividing the blocks $Q_1, ..., Q_n$ into two sets $Q_1, ..., Q_{n/2}$ and $Q_{n/2+1}, ..., Q_n$, collecting sums from both subsets, and scanning the sums for a pair that sum to the target value.  This attack allows us to find a set that sums to the target N-bit value in $\sqrt N$ time; hence the group members must be — at least — twice as big in order to keep the second-preimage problem as difficult as expected.

The update of this observation is that for an $N$ bit hash, the internal sum must be over $2N$ bit values; that means that your $F$ function must generate strong $2N$ bit values, and so ends up being considerably harder to compute than the leaf compression function in a hash-tree construction.