Depending samples in ad ranking and click rate prediction
Let $X$ represent the event "a user clicks an ad" with advertiser X,
Let event $Y$ represent a new user ($\neg Y$ a returning user),


for any advertiser $X$ the probability of a user clicking an ad is:

$p(X) = p(X\wedge Y) + p(X\wedge \neg Y)$

$p(X) = p(X|Y)p(Y) + p(X| \neg Y)p(\neg Y)$

$p(X) = p(X|Y)p(Y) + p(X| \neg Y)(1 - p(Y))$

If it's possible to obtain the information "new user" (event $Y$) in the system, then obviously there should be two distinct rankings, one for $Y$ and one for $\neg Y$. Phrased in a ML context: $Y$ is a binary feature and the predicted best advertiser depends on it. In this scenario a different advertiser would be selected depending on whether the user is new or not, here A for returning users and B for new users.

If not possible, then the advertisers should be ranked according to their overall probability $p(X)$, which depends on the probability of a new user $Y$. Based only on your initial values 10% vs. 9%, this means selecting advertiser $A$ is optimal overall. 

To illustrate how this works we can calculate the detail based on the values provided. First we calculate $p(Y)$:

$p(X) = p(X|Y)p(Y) + p(X| \neg Y) - p(X| \neg Y)p(Y)$

$p(X)-p(X| \neg Y) = (p(X|Y)- p(X| \neg Y))p(Y) $

$(p(X)-p(X| \neg Y)) / (p(X|Y)- p(X| \neg Y)) = p(Y) $

We have $p(A)=0.1$, $p(A|Y) = 0.08$ and $p(A| \neg Y) = 0.3$, so we can derive $p(Y)=0.91$.

What happens in this example is that the high value $p(A| \neg Y) = 0.3$ makes up for 
the low $p(A|Y) = 0.08$ despite $p(Y)=0.91$: 


A: $0.91*0.08+0.09*0.3 = 0.1$ 
B: $0.91*0.09+0.09*0.09 = 0.09$


Of course this would be different if $p(A| \neg Y)$ was 0.15 instead of 0.3 or if $p(A|  Y)$ was 0.11 instead of 0.09, etc.



[edit based on comments by OP]

Indeed it seems that a more accurate representation of the problem would take into account the probability that a user clicks again (and again, and again...) on an ad. 

This reminds me of classical probability exercises such as the Russian roulette game, the tree of possible outcomes is somewhat similar:


user doesn't click on the ad at all: $n=0$
user clicks on the ad at least once


user never clicks again: $n=1$
user clicks on the ad at least twice 


user never clicks again: $n=2$
user clicks on the ad at least 3 times


user never clicks again: $n=3$
.......





So the goal would be to represent as accurately as possible the random variable $n$ which represents how many times a user clicks on an ad, and to use the expected value of $n$ (that is, how many times a user clicks on average) to rank the advertisers.

I'm guessing that this can get pretty sophisticated depending on how we choose to represent the problem. I'm probably not able to do anything sophisticated so I'll stick to a simple representation where I assume that there are two probabilities involved:


initial probability $F=p(n\geq 1)$ for a new user to click
probability for a user to click again after having clicked $m$ times: $G=p(n\geq m+1|n\geq m)$ for $m&gt;0$.


This representation has the advantage to be very close to the original probabilities in the question: new user clicks and returning user clicks.
Notice that the model assumes that the probability to click again is a constant, which is unlikely in reality. Also the model doesn't take into account any other features, such as for how long a user hasn't clicked. I'll  just assume that one estimate these probabilities from a training set for the sake of simplicity.

Now let's calculate the expected value [fair warning, I might get it wrong!]


$p(n=0) = 1 - F$
$p(n=1) = F \times (1-G)$
$p(n=2) = F \times G \times (1-G)$
$p(n=3) = F \times G  \times G \times (1-G)$
...
$p(n=i) = F \times G^{i-1} \times (1-G)$


Therefore the expected value is:

$\sum_{i\geq 0} (p(n=i) \times i) = \sum_{i\geq 1} (F \times G^{i-1} \times (1-G) \times i)$

[Ok so normally now we can plug in the values for advertisers A and B. The problem is that I just calculated the result for the first few values and it doesn't really work... I probably made a mistake somewhere, or maybe the values don't represent the same thing... Anyway I hope this gives you the idea, maybe I'll give it another try later]