Why is my toffee grainy in cracker-toffee "Crack"?
Brown sugar can be used to good effect in candy making.

With the texture issue, something is crystallizing the sugar. Try using your ingredients with the more classic method.