How do I figure out the composition of a material given specific densities?
Your mistake is that you have calculated the mass of a sample with the same volume as ours consisting entirely of sand, and a sample with the same volume as ours consisting entirely of sand.
volume = mass_total / density_total = 0.2 kg / 2300 kg/m^3 = 0.00008696 m^3
gives the initial volume of the sample before the salt is removed, however by multiplying this by the densities of sand and plastic, we calculate the mass of a 0.0000869 m^3 sample of sand and a 0.0000869 m^3 sample of plastic.
First we need to find the new volume and density of the sample after the salt has been removed.  This can be done by calculating the volume of salt removed:
volume_salt=0.03kg_salt/2100 kg/m^3=0.00001429m^3
By subtracting this volume from the initial volume you already calculated:
0.00008696m^3-0.00001429m^3=0.00007267m^3
dividing the new mass by this gives:
0.170kg/0.00007267m^3=2,339kg/m^3
Now we need to use this to calculate the proportion.  We can do this by creating and solving a system of equations.  We know that some mass of sand plus some mass of plastic gives us a 170 gram mixture, and that some volume of sand plus some volume of plastic gives us a mixture with a volume of 0.00007267m^3.
xkg_sand+ykg_plastic=0.170kg
am^3_sand+bm^3_plastic=0.00007267m^3
Now, we know that the volume and mass of each material are related by their densities, and because of this it is possible to replace the terms in one equation with the terms in the other by relating them through density, so if we decide to solve for mass, our volume equation becomes:
(xkg_sand/(2700 kg/m^3))+(ykg_plastic/(1400 kg/m^3))=0.00007267m^3
Because the x and y in these equations are the same, we can solve a system, which yields:
x=0.1418kg
y=0.0282kg
Dividing these by our starting mass gives:
proportion_sand=0.709
proportion_plastic=0.141
These can be rounded to match your answer if needed, however these proportions combined with your calculated salt proportion adds up to 1.