Swapping 24v dc motor with 240 ac motor
It would help to know who manufactured the original motor and the model number, the listed power rating doesn't mean anything other than that 8000W/10HP is the maximum power of the motor under load, in other words, how much voltage at some current you can put through it safely. You would only encounter that sort of situation if the motor had stalled/was jammed by debris, which is very rare. You would practically have to run aground and have the prop get stuck from debris, in which cases you have more to worry about than a burnt out motor. You can see this with a cheap 5V DC motor hooked up to a power supply, while you watch the current change as you squeeze at varying pressures with your fingers on the motor shaft. One that I have, for example, draws only .3A without any load, but spikes up to 2 or 3A with a load before stalling.

You shouldn't expect your motor to be pulling a continuous 333 amps while running. It will probably draw more on the order of ten or so amps unless accelerating or spinning up for the first time, it's hard to tell without knowing more about the load/boat itself. It will pull a fair amount when starting up, but then again lead acid deep cycle marine batteries are quite resilient and should be able to handle even 333A at 24V, as they were designed to do. If you are really worried about that, you can simply just wire up a few more 24V batteries/battery arrays in parallel to distribute the load, you can use heavier gauge wiring if you so choose, you simply have to make sure that either your cables and their dielectrics can handle 24V at 333A, or that you parallel a few cables together of lesser current capability. You can test your motor with a multimeter set to the resistance setting to find out if it is safe to use, a DC motor should not have a lot of resistance on it's coils, usually in the milliOhm range. If it has a high resistance, for the sake of argument, 2kOhms, then the motor is bad. You should not turn the shaft while measuring resistance. You can also put some power on it, say 5V, and then measure the current in which case Ohm's law will give you the approximate resistance (R=V/I).

As far as replacing the 24VDC motor with a 240VAC motor, you would be taking on a project  that would probably cost twice as much as it would to simply retain the 24VDC motor, and would probably cost more and weigh more than any commercial product that runs on 12V-48VDC batteries/battery arrays, as well as the fact that it will probably kill you. You would at minimum need a VFD if you want to be able to vary the motor speed, which is neither cheap nor simple to design yourself, and it converts the input AC voltage to DC anyway. On the flip side, a DC motor is easily controllable with a cheap full bridge driver hooked up to what I imagine would be an array of paralleled high-power MOSFETS to ensure that the motor can run all the way to max, possibly with a water cooled heatsink, with a current sense shutoff to prevent damage, all controlled with a cheap microcontroller hooked up to whatever throttle lever is used for it. 

Believe it or not, a 24V DC motor is probably a better, more efficient option than a single phase 240VAC motor. Im assuming one of the reasons they are using a 24VDC system to begin with is that, #1, it's low voltage - the risk of killing yourself with 24VDC is less (but still possible especially with water involved) than with 240VAC. 240V, either AC or DC, especially on a piece of equipment that is already exposed to water, is an instant death sentence, there is no chance of surviving being electrocuted with 240AC in water, even 120VAC would be instantly fatal - which is probably the reason why the manufacturer refused to help. In addition, you will be placing everyone on board at risk, as well as rescuers who attempt to rescue you only to be electrocuted themselves. In addition, the likelihood of you being revived successfully is low, as your body's cells, your blood vessels, and your organs themselves will be damaged, electrically cooked, burst, or otherwise. #2, the batteries themselves act like a reserve of power, similar to a capacitor or a buffer. If you hooked an AC motor directly to the generator, more than likely your generator's output power will fluctuate as a load is put onto the AC motor. As I understand it, 16kVa is somewhere on the order of 13kW output, or 54A at 240VAC. If you replace that 24VDC motor with a 240AC motor of equivalent ratings, you would be drawing at most 7A/HP, or 7A*10HP = 70A = 16.8kW, so unless you are planning on replacing the entire power system..... DC is your best bet. You should follow whatever the manufacturer or local boat mechanic recommends, it honestly sounds as if your boat needs a complete overhaul.

To answer your final question on how to know what RPM the DC motor runs at.... the RPM it runs at will be dependent on input voltage if the current is not limited, whereas a 240VAC motor is dependent on the frequency of the AC sine wave, hence why you would need a fairly expensive VFD to control it. At 60Hz, your AC motor is limited to 3600RPM. Frankly, anything over 3600RPM with a 120-240V single phase AC motor is not really worth it, as torque begins to drop off sharply after that. You can "estimate" the RPM of a DC motor, but the only way to know for sure is if it is stamped on the data plate or if you have an encoder or some other means to measure it's RPM. You can probably expect the RPM of your 24V motor to be exponentially higher than that of a 240VAC motor.