REST Export Serializer adds quotation marks to my field
I had to change my overall approach to the problem, but managed to solve the serialising problem by using two additional modules:

https://www.drupal.org/project/views_field_view

https://www.drupal.org/project/rest_export_nested

The description of the REST Export Nested explains everything.
The only thing that might provide problematic is the performance as this involves potentially calling a lot of sub-views. So maybe writing a custom normaliser as indicated in one of the comments is a better approach.