Which gives better performance in render/viewport: mask modifier or assinging a transparent material?
A transparent material still costs vertex transformation, BVH calculation, and fill rate, so a mask modified mesh will have better performance.  The difference between the two will probably not be large enough to notice.
Viewport vs. render doesn't matter-- a viewport is a render-- although you'll have slightly different costs depending on rendering engine (and a solid view is workbench, a mat view, Eevee, and a rendered preview, whatever you have set.)  For example, only Cycles does BVH calculation, and fill rate is really a rasterizer (workbench or eevee) characteristic (as far as I understand it, I'm not a raytracing guru.)