Algorithm to find the size of a quotient of a free group
I think you're asking: given a group presentation $\langle X | R \rangle$, determine the size of the group.

Unfortunately, it is undecidable to determine whether the group is finite or infinite.  See the Adian-Rabin theorem.

For some additional references, see also https://math.stackexchange.com/a/606/14578, https://math.stackexchange.com/q/3106606/14578, https://math.stackexchange.com/q/2756802/14578, https://math.stackexchange.com/q/1516849/14578, or the following lecture notes: Presentation of Groups.  Derek Holt.