Is it possible to encrypt system partition with custom rom?
Android does not encrypt system partition as there is no confidential data on the /system. There are only standard ROM files which are already available in public domain. 

The encryption should only apply to the userdata partition /data or SD Card (if mounted as internal storage). Moreover /system is mounted as read-only. So, nothing can be tampered there.