How does bitcoin conversion (to/from $) handle round off error? If the transaction rate were $300 for each Bitcoin, and I try to convert $100 to Bitcoins then :

$100 x 1 BTC/ 300=  1/3 of a bitcoin (.33333 repeating)

How does conversion between Fiat and Bitcoin work when truncating recurring decimals?
The Bitcoin protocol doesn't recognize the concept of a dollar. Conversions of the kind you mention do not exist in the Bitcoin protocol.

Conversions might exist with services that recognize dollars (and bitcoins), but how they handle it is entirely up to them, there is no general answer.

But I expect that most will simply round down to an amount of satoshis. So in your example it would be rounded to 0.33333333 (8 places).