How are the system health extended events files rolling over?
How is the mechanism working in SQL Server of rolling over the files

The files will roll over when any of the following occurs:

the file reaches its max size (in this case, 5 MB)
the XE session is stopped and restarted for any reason

This could be someone actually running the command:
ALTER EVENT SESSION [session_name]  
ON SERVER  
STATE = STOP;

Or it could be that the server or SQL Server instance was restarted (this seems unlikely given you mentioned this is a critical production server).


and is there a way I can check when and how this is happening?

If all of your files are at or around 5 MB, your server is just generating a lot of events.  I have a production server that's not very busy, and it still takes less than 24 hours to fill up one of the 4 XE files with 5 MB of events.  Here's a screenshot, check out the modified dates on those files:

To see what events are included in the event session in SSMS, you can go to Management → Extended Events → Sessions → right-click &quot;system_health&quot; and choose &quot;Script Session as&quot; → &quot;CREATE to&quot; → &quot;New Query Editor Window&quot;.
However, note this from the Microsoft docs article Use the system_health Session:

We recommend that you do not stop, alter, or delete the system health session.

So your best bet would be to create a new XE session with just the subset of events you want, and the specific Event File Target settings that work for your system.
If the files are less than 5 MB, then the server or event session is being restarted.  You can check the SQL Server error log for when restarts are happening, and you'll have to check with those who have CONTROL SERVER permissions to see who is stopping the XE session if that's the case.


Also, should we be increasing file sizes say 200 MB 4 files or just keep small size files like 10 MB 80 files? Which one is better?

It depends.  Querying (reading) from a bunch of different files can be inconvenient.  So I'd lean towards a lower total number of files, and just increasing the size of each one.
That being said, I've never tried parsing through a really large XE file, so there might be issues with that.  You'll have to test and see which one is better for your system.