I am working on a project, and I've been trying to come up with a custom loss/score function. Would really appreciate some help
Instead of computing $A$, make your algorithm compute $10\times A.$
Then use your favourite loss function for continuous variables. The most common one is the squared $L^2$-norm $l(10\times A, B) = (10\times A-B)^2.$

Here is a plot of $l$ that I made with Wolfram Alpha:.