Anonymity problem after voting with blind signatures
The problem is, that you are actually checking a tautology and this is no danger to deanonymization. Let me explain it to you with a simple example of 2 signatures:

Lets say we want to blindly sign distinct messages $m_1$ and $m_2$ with distinct randomness $r_1$ and $r_2$.

Now during the blind signing you produce the transcripts (I omit $\mod N$):

$m_1'=r_1^e m_1$, $sm_1=r_1 m_1^d$ and 
$m_2'=r_2^e m_2$, $sm_2=r_2 m_2^d$ 

and the final message signature 
pairs are $(m_1,sm_1=m_1^d)$ and $(m_2,sm_2=m_2^d)$.

Now, lets apply your strategy:

lets look at one case where you start from one of the final signatures, say $(m_1,sm_1)$ and check it against both transcripts of the blind signing process  (and the second case is analogous due to symmetry):

$sm_1' sm_1^{-1}= (m_1^d r_1) (m_1^d)^{-1} = r_1$ and 

$sm_2' sm_1^{-1}= (m_2^d r_2) (m_1^d)^{-1} = r_1'$

now you have two candidates and lets check it with every $m_i'$:

$r_1^e m_1 = r_1^e m_1 = m_1'$ and
$(r_1')^e m_1 = ((m_2^d r_2) (m_1^d)^{-1})^e m_1 = (m_2^d)^e r_2^e (m_1^{-1})^{ed} m_1 = r_2^e m_2 = m_2'$

So, now you have checked all transcripts of the blind signing process against the signature of the first message and with your candidates you have found out 
that the first message corresponds to both transcripts with exactly the same probability. So you have learned nothing. As said above, the 
same argument (due to symmetry) applies to the other case and you can easily generalise this to more messages.