Custom keyboard layout problem High Sierra
Some things to try based on others who have had this kind of problem with other keyboards


Make sure you do not have Siri turned on and that its keyboard shortcut is set to none.
If your layout is in Home/Library, try Library instead.  And vice versa, and try it in both at same time.
Disable any antivirus apps
Safe Boot    
Set up new user account and see if it works when logged into that