How to export data in Oracle database into CSV file with UTF-8 format?
UTL_FILE.PUT_LINE_NCHAR Procedure


  This procedure writes the text string stored in the buffer parameter
  to the open file identified by the file handle. With this function,
  you can write a text file in Unicode instead of in the database
  character set.


So you could try something like this:

-- Write the row to the file
UTL_FILE.PUT_LINE_NCHAR(file, TO_NCHAR(row_value));