A voting scheme where votes are scriptable I am looking for literature on mathematical models of (political) voting schemes. Concretely, I am interested in models that allow votes of the following form:

&quot;I am voting for $X$, but only if $&gt;40\%$ of my peers vote for them, too. Otherwise, I'd like to vote for $Y$ instead.&quot;

In that sense, I'd like to consider votes to be scripts over some language. I got this idea from Bitcoin where money isn't actually sent to specific accounts, but rather scripts are given that determine who may redeem that money.

A naive model
To give you a more precise idea, I have come up with the following (naive) model of scriptable votes:
Let $C$ be a set of candidates, e.g. $C = \{\text{rep},\text{dem}\}$ for republicans and democrates. An election outcome $O \in \mathrm{ProbDists}(C)$ is a probability distribution on $C$. For finite $C$, this is known as a normalized histogram.
A vote $v$ is a mathematical function $v\colon \mathrm{ProbDists}(C) \to C$, which – given a putative outcome – decides which candidate to vote for.
In most elections around the world, all votes are constant functions: e.g. if you mark your cross for republicans, in the above model you would cast the constant function $v \equiv \mathrm{rep}$. And if you marked democrates, you would cast $v' \equiv \mathrm{dem}$ instead.
Importantly, our model also supports votes as in the quote above: $$v^\text{example}(O = \{p_\mathrm{rep}, p_\mathrm{dem}\})=\begin{cases}
  \mathrm{rep} &amp; \text{if } p_\mathrm{rep} &gt; 0.4\\
  \mathrm{dem} &amp; \text{else}
\end{cases}$$
where the notation $O = \{p_\mathrm{rep}, p_\mathrm{dem}\}$ is syntactic sugar for extracting the probabilities assigned to $\mathrm{rep}$ and $\mathrm{dem}$, respectively. Here, you see that $v^\text{example}$ is a script over the usual mathematical language. But given some votes how do we determine the (final) outcome?
Let $V = v_1,\ldots,v_N$ be the multiset of cast votes. An admissible outcome of an election $(C, V)$ is an election outcome $O \in \mathrm{ProbDists}(C)$ such that the multiset $\{v_1(O),\ldots,v_N(O)\}$, interpreted as a normalized histogram on $C$, equals $O$. We can see this as some kind of fixpoint.
Examples. Let $C$ and $v := v^\text{example}$ as above.

For the election $(C, \{v\})$, i.e. with $v$ being the only vote, we have two admissible outcomes: $O_1 = \{p_\mathrm{rep} \mapsto 1, p_\mathrm{dem} \mapsto 0\}$ and $O_2 = \{p_\mathrm{rep} \mapsto 0, p_\mathrm{dem} \mapsto 1\}$. For $(C, \{v, v\})$, we have the same two outcomes. The same holds true for three, four, and arbitrarily many copies of $v$.

Consider a new kind of vote $w(O) \equiv \mathrm{dem}$ that always votes for democrates. The election $(C, \{v,w\})$ has two admissable outcomes: $O_1 = \{p_\mathrm{rep} \mapsto 0.5, p_\mathrm{dem} \mapsto 0.5\}$ and $O_2 = \{p_\mathrm{rep} \mapsto 0, p_\mathrm{dem} \mapsto 1\}$. In the latter case, the vote $v$ &quot;noticed&quot; that they will never be surrounded by a like-minded peer group of size $&gt; 40\%$ and hence switched its candidate it voted for.



Research Questions
Concerning scriptable votes, some broad questions I'd be interested in are:

Does every election have at least one admissible outcome? If not, can we relax the conditions on admissible outcomes to just choose the best ones?

How much influence do individual votes have? Is the function that maps $(C, V)$ to a set of admissible outcomes continuous in some sense?

Are mathematical functions for votes really the best model? While if conditions like $p_\mathrm{rep} &gt; 0.4$ sound reasonable (and interesting), should we perhaps disallow conditioning on equalities such as $p_\mathrm{rep} = 0.4578$? Perhaps votes should be scripts over a tailored domain-specific language. How do their properties such as being total (exceptions?) or deterministic (probabilistic?) affect the questions above?
Broad context
First off, the field in which this sort of research would be carried out is called computational social choice (theory). Searching for papers or books in that field will give you exactly what you ask for: literature on mathematical models of (political) voting schemes, and related topics like resource allocation and judgement aggregation. As for a reference, I'd recommend the Handbook of Computational Social Choice. As a sidenote, if you want to ask questions about these things, the tag social-choice-theory on math.stackexchange.com might be a good idea.
Your voting rule
Now, as for your specific idea, there are some unclear parts. You consider probabilities, but are unclear on what exactly those probabilities mean. You imply that they're based on the proportion of votes any candidate gets, but the voters don't simply vote for a candidate. I'd recommend trying to really formalize your idea and see if it has some interesting properties. I'm not aware of any serious research effort in this sort of direction.
One thing to consider is the reason for such a system. That is, why would a voter want to vote for X only with a certain level of support, and Y otherwise? Is it 1.) because they prefer X but are scared that a vote for a minor candidate might be a waste, or is it 2.) because the level of support somehow changes the desirability of the candidate? In the case of 1.), the voter really just supports X, and has Y as their second favorite candidate. A solution might not involve a complicated ballot, as you propose, but perhaps just a good voting rule that fits in the existing mathematical frameworks. In the case of 2.), where a voter's true preferences might rely on other's preferences, making such a complicated ballot might indeed be a good idea.
Importantly, you must consider that every voter can submit such a ballot, and therefore changes in outcome might beget changes in preference which beget changes in outcome and so on and so forth. It might not be possible to make any decent voting rule within this system that is guaranteed to converge to some outcome. You'll even have to be careful that the outcome is not just well-defined, but computable.
Regardless of all this, we might already be able to answer one of your research questions. Unless I misunderstand things, there is not always an admissable outcome. Consider a single voter who votes X only if X has less than half of the votes, and Y otherwise. Now, if X would win, they would prefer Y, and vice versa. I don't have any direct ideas on how to relax admissibility to guarantee one exists.