exclusive lock mysql
I'm making assumptions here based on the available data.  I do not know your table name so I'm going to assume it's called institute_enroll_table for now.  I also called your insert variables by fake names myid and my_instutute_id since I don't know what your current insert looks like.

The easiest way to fix this is going to be to do the MAX(enroll_no) in your INSERT query rather than do it as a separate query.  The second INSERT will wait on the first and therefore will get a different MAX.  The COALESCE makes the value 1 when there is no record found.

INSERT INTO institute_enroll_table(id, institute_id, enroll_no)
SELECT myid, my_institute_id, 
    COALESCE(
        (SELECT MAX(enroll_no)+1 FROM institute_enroll_table ie WHERE 
ie.institute_id=my_institute_id)
     ,1)AS enroll_no