Is there a formula describing the relationship between bit error rate, frame size, and normalised effective throughput?
Yes, you will be able to arrive at a formula which will at best be as useful as the accuracy of the assumptions that you make, and which will have a statistical component due to effects of stat variation of noise and resync and ... on the results. 

You'd have to define things rather more tightly before you could get a useful formula.
 eg the data getting through is D_sent x (1-BER)
 (BER fractional rate eg 1 in 10^6 = 0.000001). 

BUT ANY error either needs a resend or has forward correction overhead.
 If any error occur at all then you need a resend and
 throughput drops to D_sent - (packets with error(s) in them per second = pweitps) x (packet data content length )
 And you also have the back channel overhead which may be part of your total bandwidth budget or may not.

To convert BER to pweitps you either assume every single error destroys a packet (as above), or you need to know statistical grouping of errors. 

If each error bit loses one packet and if net error free data rate is D and if packet length is P then you lose BER x P bits/second due to errors so rate simplistically reduces to ~= (D - BER x P)/ D of previously. 

If you lose synchronisation  due to errors and must resynchronise that is a rate reduction and you need to know how resync varies wih error rate and types of error hit. 

And much more. 

SO the answer is yes, a formula can be derived which is as useful as the assumptions that you make and which will hav a ststaistical component due to effects of stat variation of noise and resync and ... on results.