How should I deal with a vocabulary with many terms?
Since i can't answer it in comment. i'll try to answer here. I've solve similar problem, my solution is using redis caching and entity cache module

Since then, my site going well with 256MB RAM configuration in php.ini.