A voting scheme where votes are scriptable
Broad context
First off, the field in which this sort of research would be carried out is called computational social choice (theory). Searching for papers or books in that field will give you exactly what you ask for: literature on mathematical models of (political) voting schemes, and related topics like resource allocation and judgement aggregation. As for a reference, I'd recommend the Handbook of Computational Social Choice. As a sidenote, if you want to ask questions about these things, the tag social-choice-theory on math.stackexchange.com might be a good idea.
Your voting rule
Now, as for your specific idea, there are some unclear parts. You consider probabilities, but are unclear on what exactly those probabilities mean. You imply that they're based on the proportion of votes any candidate gets, but the voters don't simply vote for a candidate. I'd recommend trying to really formalize your idea and see if it has some interesting properties. I'm not aware of any serious research effort in this sort of direction.
One thing to consider is the reason for such a system. That is, why would a voter want to vote for X only with a certain level of support, and Y otherwise? Is it 1.) because they prefer X but are scared that a vote for a minor candidate might be a waste, or is it 2.) because the level of support somehow changes the desirability of the candidate? In the case of 1.), the voter really just supports X, and has Y as their second favorite candidate. A solution might not involve a complicated ballot, as you propose, but perhaps just a good voting rule that fits in the existing mathematical frameworks. In the case of 2.), where a voter's true preferences might rely on other's preferences, making such a complicated ballot might indeed be a good idea.
Importantly, you must consider that every voter can submit such a ballot, and therefore changes in outcome might beget changes in preference which beget changes in outcome and so on and so forth. It might not be possible to make any decent voting rule within this system that is guaranteed to converge to some outcome. You'll even have to be careful that the outcome is not just well-defined, but computable.
Regardless of all this, we might already be able to answer one of your research questions. Unless I misunderstand things, there is not always an admissable outcome. Consider a single voter who votes X only if X has less than half of the votes, and Y otherwise. Now, if X would win, they would prefer Y, and vice versa. I don't have any direct ideas on how to relax admissibility to guarantee one exists.