Is there a formula to calculate how big a ventral fin should be?
There is no single formula - the number of factors to consider are too numerous and the load cases are too diverse. Here is an incomplete list:


Directional stability: Depending on fuselage size and shape, the tail must be sized to pull the aircraft into the wind direction. The fuselage is destabilising in yaw and the tail is needed to counteract that. Depending on factors like wing sweep, some of that stabilisation can be contributed by other parts of the airframe.
In multi-engined aircraft: Yaw from asymmetric thrust. If one engine fails, the fin must be large enough to create an equal but opposite yawing moment to that of the still running engine on the other side. Multi-engined aircraft have a minimum speed below which directional control with one engine inoperative is lost.
In high aspect ratio designs: Counteract adverse yaw. When ailerons are deflected and the aircraft rolls, the direction of lift on the side with the trailing-edge-up aileron tilts forward and vice versa. This horizontal lift component creates an undesired yawing moment that the tail needs to neutralize. This effect varies with speed, so if you want good low-speed maneuverability, this load case determines vertical tail size (mostly of gliders).
Dutch roll damping: Sometimes static stability in yaw is not sufficient and dynamic stability sizes the tail.
Maximum flight Mach number: At high Mach, the upper side of the plane flies in less dense air and a traditional vertical tail loses much of its efficiency. Here a ventral tailfin can help to reduce tail size considerably.


Add to this the fact that the destabilizing effect of the fuselage increases with sideslip angle and the fact that the vertical tail as the higher aspect ratio surface of the two will reach its maximum lift first and suffer a noticeable decrease in effectiveness at larger sideslip angles. Now details like a highly swept fillet are needed to maintain tail effectiveness at high sideslip angles.

Finally, it depends which of the load cases is sizing your tail, and that can be determined by factors like fuselage length. Tail effectiveness for directional control grows linearly with fuselage length and for yaw damping with the square of fuselage length. If you want to fly the same tail on fuselages of different length (like most airliner models do), your tail might be sized by a different load case on a different aircraft than the one it is mounted on.

If you want to fly low and fast, it might be that the loads on the vertical deform the structure so much that the efficiency of the vertical tail is greatly reduced. In that case it does not help to make the tail bigger, instead ventral fins must be added. Now you need not only to look at the best tail size but at the best overall tail configuration as well.