What are the problems with Diffie-Hellman groups 3 and 4 (RFC 2409 + RFC 2412)? In RFC 2409 and RFC 2412 Diffie-Hellman groups 3 and 4 were defined.
These are groups over elliptic curves based on Galois Fields with   and  elements respectively.

I know that these sizes are considered as too small for modern cryptography.

But anyway, there is very little info about these groups on the internet. Are there other reasons besides the fields' sizes for not using these groups?
There is a relatively obscure attack, named the GHS attack after its authors Gaudry-Hess-Smart, that applies to binary curves where the exponent is not prime. In this case $155 = 5\cdot 31$ and $185 = 5\cdot 37$.

The main idea of the GHS attack is to map the discrete logarithm from an elliptic curve over, say,  $\mathbb{F}_{2^{155}}$ to a hyperelliptic curve of genus $16$ over $\mathbb{F}_{31}$.

Unlike elliptic curves, hyperelliptic curves of high enough genus have better-than-generic algorithms for computing discrete logarithms, and therefore the security of many curves over $\mathbb{F}_{2^{155}}$ and $\mathbb{F}_{2^{185}}$ is going to be smaller than you would expect from a proper strong curve. There are a few papers analyzing the $\mathbb{F}_{2^{155}}$ and $\mathbb{F}_{2^{185}}$ cases, and they largely conclude that such fields—that is, binary fields of composite degree—should not be used for elliptic curve cryptography.