Why won't the Milky Way colliding with Andromeda affect the Solar System?
Outside my field, but collecting information from the Wiki article, I can make an attempt nevertheless:

You're right that we will probably pass through Oort-like clouds of other stars. But the largest of the planetesimals of the Oort Cloud is measured in kilometres, whereas the distance between these objects is measured in AU, so the total cross section of object compared to that of the cloud is still miniscule. Smaller objects are larger in number, but still it's a very low probability.

Gravitationally, the clouds also won't affect us. The total mass of the Oort Clouds is probably of the order of 10 Earth masses, i.e. much smaller than the Sun.

The outer parts of the Oort Cloud is only loosely bound to the Sun. I suppose the best evidence that we won't have any major collisions is that then we would already have similar objects bombarding us from our neighboring stars' clouds.