Use the given information to write the formula of this compound
For a molecule AB2, 3.5g of A represents one molar fraction, and 8.00g B represents 2 molar fractions (or 4.0+4.0). Therefore, a direct ratio can be given as 3.5:4.0, or 1:1.14.  This means a molecule AnBm will give a mass ratio for A:B of n:1.14xm

For a molecule AB, for every 1g of A, you will have 1.14g of B.

For a molecule AB2, for every 1g of A, you will have 2.28g of B.

For a molecule A2B3, for every 1g of A, you will have (1.14x3/2) 1.71g of B.

etc etc

So for 5.0g of A, in order to have 17.1g of B, you will have 1.14 x (m/n) = 17.1/5

=> m/n = 17.1 / 5 / 1.14 = 3

This means m/n=3, or AB3 (alternatively it could also be A2B6)