Percent error and significant figures?
Actually the % error is calculated as:
$\frac{\text{experimental} - \text{actual}}{\text{actual}}\cdot100\%$
That way

if experimental &lt; actual then the error is negative
if experimental &gt; actual then the error is positive

So my question is if the experimental is close enough to the actual that the subtraction would give zero significant figures, would then the percent error then be zero?

Not exactly zero...

EX: measured density of 0.997g/ml and actual density of 0.997171g/ml then, 0.997171-0.997=0.000171 which would be rounded to 0.000 due to sig. figs

Correct for subtraction you have used. Addition and subtraction round to the the last common significant decimal place of all the measurements. That would be the 8 in 10.98 in the following example.
So $10.9\bar8 + 0.6754 + 0.4795 = 12.1349 \approx 12.13 $
In actually you should have used the reverse 0.997 - 0.99717 = -0.000171

there are no significant figures so this would mean the calculated 0.0171% error would be zero correct.

Now this gets sticky. You could write that the error was 0% but that would indicate only one part in 100 precision. If you write 0.00% it isn't entirely clear if that is two or three significant figures. But you have claimed that the hundredth place in the % is significant, which it is not.
Remember that the minimum nonzero error possible was +/- 0.001 which yields an error of 0.1%.
So the error would be best written as 0.0% indicating that the error is 0 to 1 part in a thousand.

Reference: Significant Figures at Wikipedia
I'll leave the propagation of errors for a different exercise.