if an academic writer critiques your study using an approach you are not familiar with, should you bother responding?
You are asking about cases where an idea you have put forward is criticised on the basis of a theory that you are not familiar with, which might require some investment of time to understand.  The example you give of two philosophy professors is a good one.

There is a time-cost involved in any attempt to learn a new discipline area, even to the level that would allow you to understand an argument directed against your work.  While the time cost might be too steep in some rare cases, as a general principle, if you are going to set out arguments and ideas on an academic subject, you should be willing to invest the required time to understand the arguments that bear on your subject matter.  This is a normal part of academic work; it requires you to read outside your own speciality on occasion to understand some outside idea.  This is also a useful way of expanding your knowledge and acquiring some interdisciplinary knowledge to allow some communication between different fields.

Although there is a time-cost to this kind of study, it is highly beneficial in giving a more rounded understanding of a subject area.  Having said this, there is generally a large difference between the amount of time required to learn a subject well enough to understand a criticism (a short time), and the amount of time required to learn a subject well enough to create new work in that area (a long time).  It is reasonable in cases like this to at least learn enough to understand and evaluate the argument made against your work, and be able to respond cogently to this argument.  If this piques further interest then it might lead to deeper study, but that is not a necessity.

Your example with the philosophy professors: In this example there is a time-cost to the first professor to learn enough about formal models of normative ethical operators (presumably some kind of modal logic) to work in that field.  However, he shouldn't need much time to learn enough to understand the particular argument made in that formalism that is deployed against his argument.  This is effectively just a translation of an argument in another language (mathematical formalism) into terms that he can understand.  It will require him to go through bit-by-bit and learn the specific aspects of the formalism that are deployed in that case, which is a lot smaller task than becoming an expert in modal logic.  Once he has done this, he should be able to understand and respond to the argument (in his own words - not necessarily in the formalism!).  If that argument is important or interesting it may pique his interest and lead to further study in modal logic ---e.g., he might decide he wishes to learn enough to couch his own ideas in the formalism, and thereby provide a contrary model in the formalism as a counter-argument.