How to model "name similarity"?
There are existing dictionaries that list pairs of characters that are visually similar.  See, e.g.,


https://security.stackexchange.com/q/36257/971
https://security.stackexchange.com/q/128286/971
https://stackoverflow.com/q/4846365/781723


Therefore, I suggest you try modified edit distance: compute the edit distance, but with a cost function that treats the distance between a pair of visually similar characters as zero or much smaller than the distance between a pair of different characters.

I recommend using the Damerau–Levenshtein edit distance. It includes transpositions among the changes it considers.  There's some reason to believe this corresponds better to mistakes made by humans.  So, maybe it'll correspond better to what humans consider similar.



Another plausible approach would be to render the two strings to images (using whatever font you think will be used to display them), then compare the images.

To compare how visually similar the images are, a basic starting point would be to use L1 or L2 distance (both have been shown to correspond well to human perception in some situations).  This probably won't be enough because it will fail badly if the images aren't perfectly aligned.  To address that, I suggest you apply a sort of edit distance that allows inserting or deleting entire columns of pixels (a full-height column), in addition to changing individual pixels.



It's not clear whether machine learning will be useful here.  The obvious approach to try first would be to compute the edit distance and compare it to some threshold.  That might be sufficient.