Finding the values of x and y using base 8
I think the issue with your method is that your subtracts are neglecting to account for the carry values that happen during the addition. If we instead go right to left and account for carries, we get:

(abc+010 def+101 110+abc 111+101)=(111 001 def abc)
                         100 
                 (110 + 100 + 1) // carrying the 1 here from the addition in the previous place
                 011
         (011 + 101 + 1) // carrying a 1 from the previous place again
         001
(100 + 010 + 1) // again, carrying a 1
111


Thus, we have that:

111 001 011 100 = (111 001 def abc)


Meaning that 100_2 = abc_2 and 011_2 = def_2, implying x=4 and y=3.