Ethical grounds of letting supervisor hear recorded interviews with research subjects
Ordinarily, when research involving human subjects is conducted, it needs to go through an ethics clearance with the university, and that should involve stipulation of which staff members will have access to the data, and what protections are imposed on the data.  Usually the scope of this access would be broad enough to allow access for anyone on the research team (researcher, supervisor, relevant administrators, etc.), so long as they comply with the relevant data protections and only access the data for legitimate purposes.  These matters really should have been stipulated clearly in the ethics application for the research, and it is an oversight if they were not.  (You should think about this for next time.)

You mention your concern that giving access to your supervisor goes beyond the consent stipulated to the participants under the data protection section of your project, but you have not stated the content of that section, or the parties to that document.  (If you can elaborate on the specifics, they may help.)  Usually a data-agreement for research will be made between the research participants and the university, rather than being an agreement between the participants and the specific researcher.  Members of the public who consent to participate in research through a university usually expect that they are giving consent to the institution to record and access their data, rather than giving consent to specific researchers.  They will therefore expect that various professionals in the university will have access to their data, for the purposes of conducting or supervising the research, and the university will ensure that access is only done for a proper purpose.

Now, obviously, if your data agreement has a section that stipulates an access restriction, then you should obey that restriction.  However, in the absence of any clear stipulation that access is restricted to specific researchers, the more natural interpretation would be that the university has consent to record and access the data, and this would usually be expected to extend to a supervisor with a duty to supervise the research of a student researcher.

As to whether your supervisor can "demand" access to the data, if the research agreement with the participants is between them and the university, then your supervisor can probably legitimately demand access, and the university will almost certainly grant him access.  If you are a PhD student at the university, and this person is your graduate supervisor, then your supervisor has a duty to supervise your research.  In any case, if you are not sure of the access rules here, just send an inquiry back to the ethics board that approved your application, and ask for their guidance.