Shrinking the SYSTEM tablespace in Oracle
Query the DBA_EXTENTS view to see what objects are present towards the end of the data files. It is possible that they are not system objects, or are system objects which can be safely modified so that they are towards the beginning of the files.

However I suspect that you have a big problem here -- you may be looking at creating a new database and moving all your user objects to that.