Clausius–Clapeyron relation for states of water and vapor
The answer may be somewhat counterintuitive. Yes, an inert gas does very much exert pressure on a solid or liquid. The problem is that the chemical potential of the substance in the gas phase is not affected by the inert gas (if the gas is assumed ideal).
Assume you have a substance (liquid or solid) in a rigid isothermal but otherwise empty box. The result is as expected evaporation or sublimation of the substance until the pressure of the vapour has increased to an equilibrium value (the saturation vapor pressure). We assume there is plenty of substance so that some remains in the condensed phase while the rest fills the box as vapour. What drives evaporation/sublimation? The fact that the chemical potential of the substance is (initially) lower in the vapour. Once equilibrium is achieved the chemical potential is the same in both phases.
Now assume an alternate scenario: in addition to the substance (solid or liquid) there is now some inert gas in the box. As suspected this gas exerts a pressure on the liquid and thereby affects its chemical potential (free energy). But the free energy of the substance in the gas phase is unaffected by the inert gas (assuming the gases behave ideally). And here the somewhat counterintuitive result: the substance again evaporates/sublimes until reaching the saturation vapour pressure, but precisely because the inert gas exerts a pressure on the condensed phase and increases its chemical potential in this phase, but does not alter the chemical potential of the vapor, it increases the saturation vapor pressure. The increased vapour pressure raises the chemical potential of the vapour until it matches the heightened potential of the condensed phase.
And the short answer to the question is that the inability of vapor A to 'feel' the total pressure including the inert gas is part of the definition of an ideal or perfect gas, which fails if the pressure is raised sufficiently.