Is it reasonable for an institution to ask for families total income in application form?
I don't have any direct experience with educational systems in India, so this is a very general answer.

In response to a question, you said that you view the practice as being potentially unreasonable 


  Because income is a private matter and not related to admission. 


But if they are using this information to figure out whether you deserve a scholarship it does seem to be related to admissions.  According to their website, all students are considered for fellowships, so this could be it.  Admittedly there are less reasonable, um, reasons they might want to have this information: e.g. they may want to give preferential treatment towards or against students with high family incomes.


  [I]n that case the question could have been asked in a different way. 


I think you're probably right about this.  And the online representative does not seem to have been at all helpful (we can all sympathize with that!).  It seems you have the following options:


Make a more concerted effort to speak to someone in the admissions department -- either by email, phone or in person -- to find out the reasons behind this request.  If appropriate, you could also make clear that you are not seeking any sort of financial aid and see if that changes the situation.
You can feel that the practice is a bit unreasonable and invasive but decide that it is in your best interest to comply with it anyway.  
You can decide not to apply because of this, in which case I think it would be a good idea to contact someone (higher up than an online representative) in the admissions department.



  What if I fill in a random amount or enter zero?


That is a bad idea.  Many institutions (academic and otherwise) view lying during the application process as grounds for termination of your position into the indefinite future.  Saying that you have zero family income in particular is liable to get you in trouble: they may wonder how you are going to pay for the program, or they may come back with a financial aid / fellowship offer that it would be very unethical for you to accept.