What prevents backflow of blood from right atrium to vena cava during atrial systole
Pressure differences. Atrial contractions are much more gentle than ventricular ones, and in normal circulation the venous pressure in the vena cava often stays higher than the right atrium, even during atrial systole. If the pressure in the atrium does rise above the vena cava pressure at the peak of systole, there can be some backflow, although it is slight and brief in normal physiology.

The major veins are typically somewhat distended so they are constantly pushing blood into the right atrium. Blood continues to flow from the veins into the right atrium and from the right atrium into the right ventricle throughout ventricular diastole, including during most of atrial systole.

References:

Anderson, R. M. (1993). The gross physiology of the cardiovascular system. Robert M. Anderson.

Wexler, L., Bergel, D. H., Gabe, I. T., Makin, G. S., &amp; Mills, C. J. (1968). Velocity of blood flow in normal human venae cavae. Circulation Research, 23(3), 349-359.