How do I change the icon of an alias to a custom ICO file?
As you've already found out you cannot customize the icon of an alias.

What you'll need to do, is create an application bundle using either Applescript, Automator, included in OS X, or use third-party app like Platypus.

You can then use custom .incs files or copy and paste supported images into the Get Info Sheet Icon and then add the application bundle to the Dock.

I have 10 WinApps I've done this for using Automator, creating an Application and adding a Run Shell Script Action to it.

One App in particular is GIF Movie Gear and the command added to the Run Shell Script Action is:

open "$HOME/Wine Files/drive_c/Program Files/GIF Movie Gear/movgear.exe"


I can't remember where I got the GIF Movie Gear Icon File however it is an .icns file that I renamed to AutomatorApplet.icns in the application bundle replacing the original.



Notes: I do not have any ICO files to test with, so I can't say whether or not that file type is supported directly and if not, then it will need to be converted into a supported type.

Also if you do replace the AutomatorApplet.icns, do this last before adding it to the Dock and note that if you edit the Automator Application afterwards, it will replace the .icns file with a default one.

Doing the copy and paste method into the Get Info Sheet Icon I believe survives edits.