Glue for sides of cross stitch work
You could try fray check. These glues tend to be thinner, which would give you a less noticeable and stiff edge. They are usually clear, but may yellow over time.

If that isn't strong enough, you could try fabric glue, which as the name suggests, is specifically made for bonding fabrics.

You can also finish the back of your piece without glue! After mounting on a board or in a hoop, fold the raw edges under and stitch across from one side to another, keeping the fabric taut and in place. You can then cover up the back by sewing on another piece of fabric. This is often done with felt which does not fray, but you could use any fabric as long as you tuck the raw edges under.