RFID system for insect tracking - range of reading
There is no easy way this can be achieved with RFID. RTLS (Real Time Location System) is probably the most difficult subject which can be achieved by RFID system (where size of tags is not a factor); you are complicating it even further by inserting size of tags as a constraint.
The only thing which seems feasible is cell based location given an expected read range of one antenna, and basically make a chess grid from your enclosure. If the insects can fly, then your problem complicates even further as you need an antenna that covers vertical distance, not only ground surface.

It now comes to mind another aspect: types of tags that can be used. You definitely can't use active or BAP tags. Given the distances the greatest that can be achieved is with UHF tags. The problem is that UHF needs large antenna foot print. Smallest UHF tag with antenna is something like 7 * 7 mm diameter label (TagsysRFID MuTRAK), but will have less ID distance than ordinary UHF. They say up to 10m but you should expect less than that.

It really depends what is considered "sufficient achievement" and how precise you need the results to be.

Another problem which you need to think about is that the insects will permanently be in a dense field. Sure, RFID is not harmful to humans or other animals, but another recommendation you will always hear is "Spend as little time as possible between antennas as stay as far as possible from emitting antennas." But this might be a great idea for a research: "Is the behavior of insects modified by permanent exposure to RFID?"