Merge Replication from SQL 2008 R2 to SQL 2005 Express: SQL Server version (10.50) is not supported
Try to change setting,

Step 1: Launch the DFS Management Console (on the cluster node ‘PrimaryNode’).

Step 2: Click on the ‘New Replication Group…’ action.

Step 3: Select the type of replication group.

Step 4: Select the name and domain for the replication group.

Step 5: Specify the branch office file server (replication member)

Step 6: Select the folders to replicate from the branch office server.

Step 7: Specify the hub server (other replication member)

Step 8: Specify the path to the replicated folder on the hub server.

Step 9: Configure the replication schedule and bandwidth utilization.

then you apply and hope your problem will be solve.

http://www.techyv.com/questions/2008-r-2-replication-errors