Is there an aerodynamic force that would keep this experimental WW2 era prop from flying as fast as an early jet?
First of all, there were several propeller aircraft which were much faster than the XB-42. But let's stick with it and scale it to see where that brings us.

For that we need two things: Equations for drag $D$:
$$D = \frac{\rho\cdot v^2}{2}\cdot S\cdot\left(c_{D0}+\frac{c_L^2}{\pi\cdot AR\cdot\epsilon}\right) = \frac{\rho\cdot v^2}{2}\cdot S\cdot c_{D0}+\frac{(m\cdot g)^2}{\pi\cdot AR\cdot\epsilon\cdot\frac{\rho\cdot v^2}{2}\cdot S}$$
and for thrust $T$:
$$T = \frac{P}{v}\cdot\eta_p$$

The symbols are (values are for the XB-42 at 7,140 m):
$\kern{5mm} \rho\:\:\:\:\:$ air density = 0.581 kg/m³
$\kern{5mm} v\:\:\:\:\:$ velocity = 410 mph = 183.3 m/s
$\kern{5mm} S\:\:\:\:\:$ wing surface area = 51.6 m²
$\kern{5mm} c_{D0} \:$  zero-lift drag coefficient
$\kern{5mm} c_L \:\:\:$  lift coefficient
$\kern{5mm} \pi \:\:\:\:\:$  3.14159$\dots$
$\kern{5mm} AR \:\:$  aspect ratio of the wing = 8.95
$\kern{5mm} \epsilon \:\:\:\:\:\:$  the wing's Oswald factor = 0.9
$\kern{5mm} m \:\:\:\;$  aircraft mass = 15,000 kg
$\kern{5mm} g \:\:\:\:\;$  gravitational acceleration
$\kern{5mm} P \:\:\:\:\:$  engine shaft power = 3800 hp = 2833.66 kW
$\kern{5mm} \eta_P \:\:\;$  propeller efficiency = 0.85

In order to determine the unknown zero-lift drag, drag is set equal to thrust, the values above are inserted and the equation is solved for $c_{D0}$:
$$c_{D0} = \frac{\frac{P}{v}\cdot\eta_p - \frac{(m\cdot g)^2}{\pi\cdot AR\cdot\epsilon\cdot\frac{\rho\cdot v^2}{2}\cdot S}}{\frac{\rho\cdot v^2}{2}\cdot S} = 0.02272$$
which looks about right.

Then we need some clever formula how mass will change with a scaled airframe. Since both the engine and the pilot will stay constant, to assume that mass will scale with the square of the length is quite generous. So we have a second aircraft of 70.7% size which will have half the frontal area, half the mass but the same power. Now we need to solve for the unknown speed with the now known zero-lift drag:
$$v^4 = \frac{2\cdot P\cdot v\cdot\eta_p}{\rho\cdot ½\cdot S\cdot c_{D0}} - \frac{(2\cdot ½\cdot m\cdot g)^2}{\pi\cdot AR\cdot\epsilon\cdot\left(\rho\cdot ½\cdot S\right)^2\cdot c_{D0}}$$
$$v^4 = 14.144.659.448\cdot v - 167,497,545.8$$
$$v = 237.76\:\text{m/s} = 531.85\:\text{mph}$$

Now all what is left to be solved is how to squeeze the existing engines and their cooling system into half the fuselage cross section.


  I guess at the most basic level is this concept correct?


To answer your question directly: I used a slightly different approach and got an even higher speed. If I leave mass constant, the higher induced drag will limit the speed to 223.37 m/s, which is just shy of 500 mph. I wonder how you got a speed of 517 mph. The car example on the linked site does not include induced drag, so I guess your concept needs a little improvement.