why from_utf8 does not exist in postgresql?
As documented in the the manual the function you are looking for is convert_from()

Converts a binary string representing text in encoding src_encoding to text in the database encoding

So the query should probably be:
SELECT convert_from(name, 'UTF8') AS name, 
       COUNT(DISTINCT target) AS cnt
FROM directory_entry_file 
GROUP BY convert_from(name, 'UTF-8')
ORDER BY cnt DESC
LIMIT 1;