What is the theoretical maximum RPM of a typical aircraft piston engine?
The size and weight of the pistons is going to be a major hurdle. With a 5.25 in bore and a 4.25 inch stroke, the pistons are massive and don't change directions very easily—there's only so many Gs you can put a shape this large through at the top and bottom of each stroke before the metal starts deforming. Bounce an open umbrella up and down really fast, and you'll start to get the picture.

Cooling would also be a major factor for non-aviation use. Aircraft engines (more or less rightly) assume that there's going to be a large fan out front and/or 100+ mph of airflow. Liquid cooling would allow the heads to maintain a more constant temperature, but would also add weight and complexity.

If you want a faster-revving flat six that will put out more than 100hp and is suitable for automotive use, I might suggest pulling an engine out of a Honda Goldwing or Valkyrie. Given the edited description, though, I'm guessing you're thinking of a drag racer.