Adversarial Stress Testing of Lifetime Distributions
Nozer Singpurwalla
cs.AI cs.GT
  


TheTheorem





examExample
lemLemma
deDefinition
propProposition
corCorollary
hypHypothesis
remRemark

plain









“Adversarial Stress Testing” of Lifetime Distributions 






by


Nozer D. Singpurwalla 
 

The George Washington University




January, 2020
















In this paper we put forward the viewpoint that the notion of stress testing financial institutions and engineered systems can also be made viable appropos the stress testing an individual's strength of conviction in a probability distribution. The difference is interpretation and perspective. To make our case we consider a game theoretic setup entailing two players, an adversarial 𝒞, and an amicable ℳ. The underlying metrics entail a de Finetti style 2-sided bet with asymmetric payoffs as a way to give meaning to lifetime distributions, an adversarial stress testing function, and a maximization of the expected utility of betting scores via the Kullback-Liebler discrimination.  



Keywords: Cross Entropy, Discrimination Function,  Subjective Probability, 
 Utility.



section-1


§ PREAMBLE: WHAT IS “ADVERSARIAL STRESS TESTING”?


The term “stress testing” as used here is not to be interpreted in same vein as that used in banking and finance, though the intent of both is similar. Furthermore, “stress testing” is also not to be seen as another label for accelerated life testing in reliability and biometry; the two contexts are different.

During the financial crisis of the early twenties, banks and financial institutions were subjected to what is known as a “stress test”. Its aim was to assess the robustness of these institutions in withstanding disruptions, and an ability to provide their intended services for a specified timeframe. Stress tests are generally conducted by regulators, who as representatives of the public, are mandated to be adversarial. The purpose of this article is to extend the concept of a stress test of an institution to that of a probability distribution (or a survival function). The metric of discussion here is a lifetime, though the underlying idea need not be limited to such a metric.

Stress testing a probability distribution is not the same as an accelerated life test done in reliability, or a biostatistician's dose-response experiment. In these two scenarios, one assesses an item's capacity to endure a physical force by changing the conditions of the test via a systematic increase of the stress or the dose. Each stress (dose) level spawns its own lifetime distribution and the challenge is one of extrapolation based on several such distributions. By contrast, under an adversarial stress testing of a probability distribution, one assesses the strength of conviction of the individual proposing the distribution; as such there is only one distribution under discussion. This is done by changing the conditions of an underlying 2-sided bet by increasing its risk levels.

To summarize, the duality between an accelerated test and an adversarial stress test can be encapsulated via the statement that in the former one assesses an item's physical strength to endure, whereas in the latter, it is an individual's strength of belief that gets scrutinized. Furthermore, in accelerated testing one encounters a family of lifetime distributions, and any adversarial element, even if present, is not treated explicitly. By contrast, under an adversarial stress test the focus of attention is the credibility of the specifier of the distribution that gets evaluated.

There are two other comments to this preamble. The first is that the stress test of a financial entity is very much in the same spirit as an accelerated test with binary outcomes. The second is that the adversarial stress test, to be proposed here, would be a conducive instrument for validating the survivability of one of a kind items.







§ ADVERSARIAL BEHAVIOR AND SUBJECTIVE PROBABILITY


With the advent of active consumerism, demanding certification, and aggressive litigation, the need for the intensive testing of items and algorithms, under an adversarial flavor has gained increased prominence. Adversaries are individuals (or a group of individuals) whose expected utilities differ. Differences in expected utilities occur because of differences in their assessed probabilities or their innate utilities, or both. In rare circumstances, the expected utilities of adversaries could agree, even though their probabilities and utilities do not. For probabilities to be different, it is axiomatic that their interpretation be subjective. Thus the notion of subjective probabilities seems almost mandatory for any version of an adversarial set up. Neither the relative frequency, nor the propensity interpretation of probability will be meaningful for every adversarial scenario. Whereas the existence of subjective probabilities has been established by the likes of Ramsey (1931) and Savage (1954), its operationalization by de Finetti (1974) as a 2-sided monetary bet makes its meaning explicit. We shall lean on this operationalization of probability, recognizing that in doing so it is not possible to separate one's probability from one's utility for money, because the two are entangled.

Within the realm of adversarial scenarios, there are two general classes worthy of distinction. The first is where the adversaries participate in an economic or strategic conflict, with the intent of annihiliating each other. Such scenarios are best addressed by game-theoretic methods, where the actions of one adversary occur as a surprise to the other [cf. von Neumann and Morgenstern (1944)]. The second adversarial context is the one entailing the exchange of goods by buying, selling, or the certification of an entity. Here there could be genuine differences of opinion between the adversaries about the underlying probabilities, see, for example, Lindley and Singpurwalla (1991), (1993). The overall goal of both adversaries is to do common good bearing in mind the premise that both need each other to achieve the good. An example is the certification of an aircraft or a piece of software wherein one member, say a manufacturer ℳ, seeks approval and acceptance of his/her product by an adversary 𝒞, who could be a consumer (or a regulator). It is not the intent of 𝒞 to annihilate ℳ, though it is possible that 𝒞's actions may eliminate ℳ from future participation. Situation's of this type also occur in jurisprudence under courtroom settings. In the manufacturer-consumer scenario, there is technically speaking, at most one active adversary; the other player is generally amicable. Specifically, 𝒞 can be adversarial to ℳ, but not vice-versa. Indeed, 𝒞 may choose not to be adversarial at all. In this case, the scenario boils down to the classical case of acceptance sampling for quality control, typically addressed by the Fisher-Neyman-Pearson-Wald test of a hypothesis.

An adversarial scenario can also arise when there is a single 𝒞 and multiple ℳ's, the latter being adversarial to each other, with the possible goal of an ℳ annihilating the other ℳ's. Game theory enters the picture whenever the matter of annihilation comes in play, and in the context of several ℳ's the issue of coalitions among several ℳ's, each coalition endeavoring to annihilate the others, becomes germane.

In what follows, we restrict attention to the case of a single 𝒞 and a single ℳ, first articulating the case of 𝒞 not being adversarial to ℳ, and then the case when 𝒞 is an active adversary, because 𝒞's probability distribution of a lifetime does not align with that declared by ℳ, or 𝒞's is required to be demanding of the viability of ℳ's product.




§ THE DISPOSITIONS OF A PASSIVE 𝒞 AND AN AMICABLE ℳ


Consider an item whose lifetime Y takes values y≥ 0. Suppose that F̅(y)=P(Y≥ y) is absolutely continuous, with probability density f(y)=-dF̅(y)/dy. It is common for the survival function F̅(y) to be specified by ℳ, and this is what we shall assume. Of relevance to 𝒞 are lifetimes that are greater than, or equal to y^*>0; y^* is known as the mission time or a “threshold”, and its specification tends to be 𝒞's prerogative. The onus of accepting ℳ's F̅(y^*), or challenging it, is also up to 𝒞. Suppose that 𝒞 has no interest declared in challenging ℳ's F̅(y^*), other than to accept or to reject the item based on what F̅(y^*) is. How must 𝒞 make tangible sense of what F̅(y^*) really means? In other words, what is the operational import to 𝒞 of ℳ's F̅(y^*)? By operational import, we mean a system of bets between 𝒞 and ℳ entailing rewards and penalties.

de Finetti provided an operational interpretation of F̅(y^*) for y=y^* in particular, and for that matter, any y≥ 0. His notion was that of a 2-sided monetary bet with a linear state-dependent utility. Whereas de Finetti's focus was not on adversarial considerations, his operational interpretation paves a path towards how one can expand his setup to an adversarial situation wherein 𝒞's survival function for Y, G̅(y) differs from the F̅(y) of ℳ.

To appreciate this, let us first consider de Finetti's interpretation of 
F̅(y^*) def= p, for 0< p< 1. Here, the number p implies that ℳ is willing to stake p on the table in exchange for a reward of +1 from 𝒞, if Y≥ y^*, and is prepared to lose to 𝒞 the p staked should Y < y^*. The bet so placed is against 𝒞 who stands to gain p if Y < y^* (i.e. if ℳ fails requirements), and to lose 1 if Y ≥ y^* (i.e. ℳ has met requirements). The 2-sidedness of ℳ's bet also requires that ℳ also stake (1-p) in exchange of 1 if Y < y^*, and lose the (1-p) staked if Y ≥ y^*.

Under the above two bets, one for Y ≥ y^* and the other against it, the only action a passive 𝒞 need take is to choose the side of the bet. In what follows we always assume that 𝒞 chooses the first bet. Then for any choice y^* of Y, 𝒞's payoff (or reward) function S(y), y ≥ 0, is of the form shown in Figure 1. Assuming that p and 1 are monetary units, and assuming that 𝒞's utility for money is linear, the S(y) of Figure 1 is also, 𝒞's utility for realizing a lifetime Y ≥ y.


𝒞's passive disposition to ℳ's F̅(y) has resulted in a step function for 𝒞's payoff, wherein 𝒞 receives a constant payoff of +p for all unacceptable lifetimes, and also a constant payoff -1 for all acceptable lifetimes. In other words, 𝒞's utility for money is literally state independent. It was a feature like this – among others – that motivated Ramsey, Savage and others to develop a theory for the simultaneous axiomatization of probability and utility.

What must 𝒞 do if the step-function payoff is not acceptable to 𝒞, and/or if 𝒞's survival function for Y G̅(y) is different from ℳ's F̅(y)? An obvious strategy would be for 𝒞 to entice ℳ, to revise his/her F̅(y) to 𝒞's G̅(y), and repeat the 2-sided bet using G̅(y). However this approach merely translates the step-shaped payoff function; it does not change its overall character. An approach for changing the shape of the payoff to something more general is discussed later in Section 2.1. But before doing so, it is also instructive to bear in mind the shape of ℳ's payoff function, when 𝒞 chooses the first of de Finetti's 2-sided bet. Its general form is again a step-function that happens to be a 180 rotation of 𝒞's payoff function about the horizontal axis; see Figure 2.



Clearly, the payoff to ℳ is a constant +1, irrespective of how much closer, to the required minimum y^* the lifetimes are de Finetti's 2-sided bet therefore has the feature of providing an unjust utility to ℳ, when it is invoked on lifetime's, and when 𝒞 is a passive consumer.

The payoffs of Figures 1 and 2 are devoid of considerations pertaining to the costs of manufacture, the rewards of use, consequences of lost opportunities when the required minimum lifetime is not met, and other such economic matters. The focus of consideration  here is mainly the credibility of ℳ's specified F̅(y). Indeed, Figures 1 and 2 provide an interpretation of the meaning of a lifetime distribution as seen from the perspective of 𝒞 and ℳ, in terms of a de Finetti style 2-sided monetary bet. The two figures are also representative of 𝒞 not being adversarial to ℳ and ℳ being amicable to 𝒞. What would the payoff functions look like if 𝒞 requires that the payoff to ℳ encapsulate a better sense of being just, and ℳ abides with this requirement? This matter is discussed next.



 §.§ Passive 𝒞 and Amicable ℳ with Just Payoffs


A way to obviate the unjust feature of the payoffs given before is for 𝒞 to make his(her) payoff function, no more the simple step-function of Figure 2. As will be seen later, this would also be the path for an adversarial 𝒞 to express disagreement with ℳ's specified F̅(y), y≥ 0. For example, suppose that the payoff to 𝒞 is of the form indicated in Figure 3. This would correspond to a 180 rotation of Figure 4, which is ℳ's payoff function, when 𝒞 is not adversarial to ℳ. The payoff function of Figure 4 reflects the feature of being more just (to 𝒞) than that of Figure 2, because with the former, the larger the lifetime (over the minimum of y^*), the better the payoff to ℳ. Indeed, as shown in Figure 4, the payoff to ℳ for lifetimes larger than y^*, is a concave increasing function of y (≥ y^*). In the interest of simplicity, for values of y<y^*, the payoff (penalty) to 𝒞(ℳ) is assumed constant, but this too need not be so.


















In Figures 3 and 4, the bounds -1 and +1 are arbitrary and so are the bounds p and -p. Their purpose is to indicate a parallel with those of Figures 1 and 2, where, per de Finetti's set-up, the payoff functions are bounded by -1 and +1. As mentioned before, an inevitable consequence of any betting strategy based on money is the entanglement of probability and the utility for money. Thus, inherent to any choice by 𝒞 of a payoff function S(y), there is a probability distribution for Y, say G̅(y). In Section 4, we shall allude to the relationship between G̅(y) and S(y). However, before doing so it may be useful to say a few words about considerations that may go in 𝒞's specification of an S(y), y≥0 – 𝒞's state dependent utility function.

As is typical in reliability and survival analysis applications, it is common for ℳ to first specify an F̅(y), and based on this, for 𝒞 to accept or to reject ℳ's offer. The uncertain entity in question is an item's lifetime for which 𝒞 has specified a mission time y^*>0. Having specified F̅(y) and nothing more, an amicable ℳ's disposition is to necessarily abide by the payoff function of Figure 2. If 𝒞 is not adversarial to ℳ, then 𝒞 abides by the payoff function of Figure 1, which is a 180 rotation of Figure 2 around its horizontal axis. If in the interest of receiving a just payoff, ℳ prefers to use the payoff function of Figure 4, and here again 𝒞 chooses not to be adversarial to ℳ, then 𝒞's payoff function would be that of Figure 3, which is a rotation of Figure 4 around its horizontal axis. Similarly, were 𝒞 to prefer the payoff function of Figure 3, and were ℳ feel compelled to abide by 𝒞's choice, then ℳ's payoff function would be a rotation of Figure 3 around its horizontal axis. The same is true of all other possible choices for S(y), y≥0.

To summarize, adversarial behavior between 𝒞 and ℳ is characterized here in terms of the payoff functions used. Whenever the payoff function of 𝒞 is not a 180 rotation of the payoff function of ℳ, and vice-versa, an adversarial scenario arises. Alternatively put, we see adversaries as those whose payoff functions are not rotationally symmetric. Adversaries do not abide by what many would claim to be rules of fairplay.





§ THE ADVERSARIAL 𝒞 AND AMICABLE ℳ SCENARIO


Suppose that 𝒞 has specified a y^*>0, and ℳ has declared an F̅(y), y≥ 0. Based on these, and ℳ's utility for money, ℳ will arrive upon a payoff function -S(y) of the forms illustrated in Figures 2 and 4. For purpose of discussion supposed that it is the “just” payoff of Figure 4 that appeals to ℳ. Suppose that 𝒞 is adversarily dispositioned towards ℳ; then a payoff that is (a de Finetti style) symmetric rotation of Figure 4 will not be acceptable to 𝒞. Instead, 𝒞 will want to propose a payoff function that is more rewarding to 𝒞 when y<y^*, and less punitive to 𝒞 when y≥ y^*. The nature of what this payoff to 𝒞 should be like, is the topic of this section. But first some words about the possible reasons underlying 𝒞's adversarial disposition.

First and formost, 𝒞 may find ℳ's F̅(y) overly optimistic and may thus want to challenge F̅(y) via a stress test. This would especially be so if 𝒞 is a regulator who is mandated to thoroughly scrutinize F̅(y). 𝒞 may also want to prove an ℳ wrong with the intent of eliminating the ℳ. We propose that 𝒞's instrument for challenging ℳ would be a “stringent” payoff function which penalizes ℳ heavily when the observed y<y^*, and rewards ℳ sparingly when y≥ y^*. Denote this payoff function by S^*(y), and let S^*(y)=A(y)S(y), where A(y)>1 when y<y^* and A(y)≤ 1, when y ≥ y^*; see Figure 5. We call A(y) the adversarial stress function. We illustrate, via Figure 6, A(y)'s effect on ℳ's rotated payoff function S(y), to produce S^*(y) – 𝒞's adversarial payoff function. Observe that A(y) exaggerates S(y) for y<y^*, and dampens it for y≥ y^*. For reasons that will become clear in the sequel, we suppose that A(y)≥ 0, for all y≥ 0, and that A(y) is also bounded above by B>0. Thus 0≤ A(y)≤ B ≥ 1. How best must 𝒞 choose a meaningful A(y) is the topic of Section 3.1. Figure 5 illustrates an archetypal form for A(y), and Figure 6 its effect on S(y) – shown by the dashed lines of Figure 6.























 §.§ The Adversarial Stress Function and Betting Scores


The material which follows is an adaptation of some recent work by Shafer (2019) on statistical communication, adapted for the adversarial lifetime scenario considered here. It is based on the premise that when 𝒞 and ℳ agree to an exchange of goods – henceforth the (adversarial) “game” – Y is not known and thus A(Y) is a random quantity. Assume that ℳ is confident of the survivability of his (her) product and is therefore amicable, but not naive, and agrees to 𝒞's modification of S(y) to S^*(y) as a payoff function. ℳ then computes the expect value of A(Y) with respect to ℳ's F̅(y), namely E_F[A(Y)]=∑_yA(y)f(y), and offers to sell the game to 𝒞 at the price E_F[A(Y)]. If 𝒞 accepts ℳ's offer, then the amount risked by 𝒞 is E_F[A(Y)]. Once the game is played, that is Y observed as y, the quantity A(y)/E_F[A(Y)] is 𝒞's adversarial betting score. This is the factor by which 𝒞's amount risked gets multiplied. A large betting score can be seen as the best evidence 𝒞 has against F̅(y); the larger the score, the stronger the evidence against F̅(y). 𝒞's aim therefore is to maximize the betting score, subject to 𝒞's limitation on the amount 𝒞 is willing to risk. Since 𝒞's choice of A(y) influences the betting score, A(y) can be seen as the analogue of a physical stress in an accelerated stress test, or the level of dose in a dose-response test.

Since A(y) is bounded by B≥ 1, E_F[A(Y)]<∞, and thus E_F[A(Y)] can be normalized to one. When such is the case, 𝒞's adversarial betting score is simply A(y). To summarize, in the architecture describe here, there are three entities of concern to 𝒞: an adversarial stress function A(y), y≥ 0; an adversarial betting score A(y)/E_F[A(Y)], and S^*(y)-E_F[A(Y)], a payoff function adjusted for the cost to 𝒞 of subjecting F̅(y) to a stress-test; note that S^*(y)-E_F[A(Y)] ≥ -1. Whereas ℳ's payoff is not of direct concern here, the – S(y) of Figure 4 should be increased by E_F[A(Y)], this being an added reward to ℳ for being amicable to 𝒞's stress test.



 §.§ Properties of Adversarial Stress Function: Choosing A(y)



Let q(y)=A(y)f(y), and note that ∑_y q(y) = ∑_y A(y)f(y) = E_F[A(Y)] =1. Furthermore, since A(y)≥ 0, it follows that q(y)≥ 0; this means that q(y) can be seen as the probability mass function of some random variable, say Z. One can think of Z as the random variable that would have directly yielded 𝒞, the adversarial payoff S^*(y). Since A(y) is also 𝒞's betting score – when E_F[A(y)] is normalized to one, one can also see A(y) as the likelihood ratio of the probability mass functions of the random variable Z and the random variable Y. Bear in mind that the probability mass of Y is specified by ℳ, whereas that of Z is specified by 𝒞. An equivalence between a likelihood ratio and a betting score could be a noteworthy observation.

Since the betting score A(y) is the factor by which the money risked by 𝒞 gets multiplied, A(y) spawns a utility to 𝒞, say U[A(y)]. The bigger the A(y) the bigger the utility. Suppose that this utility is logarithmic, namely, U[A(y)]=log_2A(y). This utility encapsulates the satisfaction that 𝒞 derives in outfoxing ℳ, or in justifying 𝒞 as an effective regulator. The utility function need not be logarithmic, but assuming so leads to information theoretic considerations; these are articulated below.

In decision theory one aims to make choices that maximize an expected utility, namely, the expected value of log_2A(y) = log_2 (q/f). However, by the converse of Gibbs Inequality [cf. Kullback (1959)], for any utility like log_2 (q/f), taking an expectation with respect to anything other than q will not maximize the expected utility. We are thus motivated to maximize E_q[log_2q/f]=D_K-L(q:f), the Kullback-Leibler discrimination between q (which is a probability), and f (taken as reference) [see Kullback and Leibler (1951)]. Thus if 𝒞 wishes to subject f to the most severe adversarial stress-test as is possible, then 𝒞's A(y) should be such that D_K-L(q:f) is maximized. Since q(y)=A(y)f(y), D_K-L(q:f)=E_q[log_2A(y)], 𝒞's aim would be to choose that A(y) which maximizes ∑_ylog_2A(y)q(y), or equivalently minimizes -∑_ylog_2A(y)q(y), which is like the cross-entropy of A(y), the adversarial stress function, with respect to the probability q(y).

As an illustration of the workings of the above, suppose that A(y)=1, for all y≥0. Now, q=f, and E_q[log_2 1]=0; this means that 𝒞 will garner a non-zero betting score only when q≠ f. When A(y)=0, for all y≥ 0, the maximum expected utility is -∞; indeed this is so, if for any y, A(y)=0. Thus we require that A(y)>0 for all y≥0. Finally, for any B>0, the maximum expected utility assuming A(y)=B, for all y, will be log_2 B, suggesting that the larger the upper bound on A(y), the larger the expected utility. But large values of B increase the amount risked by 𝒞, namely E_F[A(Y)]. This in turn places a restriction on how much larger than one B can be. Once an A(y) is pinned down by 𝒞, with 0<A(y)≤ B>1, the cost adjusted payoff to 𝒞, S^*(y)-E_F[A(Y)] gets defined, and the game gets played. By this we mean that Y gets observed as y, and based on what y is, 𝒞 makes a choice as to whether to certify or not ℳ's specified F̅(y). Some strategies for 𝒞's operationalize of adversarial stress test are given below.




§ OPERATIONALIZING THE ADVERSARIAL STRESS TEST
 

Since 𝒞's cost adjusted payoff is S^*(y)-E_F[A(Y)] = A(y)S(y)-E_F[A(Y)], we could suppose that when y<y^*, 𝒞 should not bear the burden of having to pay ℳ the amount E_F[A(Y)] for the stress test. This of course presumes that both ℳ and 𝒞 have agreed to the stress test. However, for the case y≥ y^*, 𝒞 should subsidize the cost of the stress test as a way to compensate ℳ for 𝒞's adversarial and distrustful disposition. Thus 𝒞's risk adjusted payoff function would take the form shown in Figure 7; compare Figure 6 to Figure 7.


There can be several strategies for 𝒞 to operationalize the import of Figure 7, vis a vis certifying or not ℳ's F̅(y). Clearly, if y<y^*, 𝒞 will be reluctant to certify F̅(y). When y≥ y^*, 𝒞 is faced with a challenge, namely, for what values of y, y≥ỹ≥ y^* should F̅(y) be certified? The answer could depend 𝒞's choice of the risk adjusted payoff function. The larger the y, the larger is S^*(y)-E_F[A(Y)], which means the larger the amount that 𝒞 has to payback ℳ. This would suggest that ỹ should be as close to y^* as is meaningful. Should 𝒞 want to limit the payback to ℳ at some C>-1, then ỹ would be that y for which S^*(ỹ)-E_F[A(Y)]=C. Consequently, 𝒞 would certify F̅(y) whenever the observed lifetime y ≥ỹ.

The strategy proposed above does not take into consideration G̅(y), 𝒞's survival function of the item in question. Assuming that 𝒞 has in mind a G̅(y), it makes sense to assume that since 𝒞 is adversarial to ℳ, F̅(y) ≥G̅(y), for y≥ 0; otherwise, it does not make sense to stress test. This in turn would suggest that G̅(y^*) ≤F̅(y^*), so that certifying F̅(y) when the observed y≥ y^* will be more optimistic than what 𝒞 believes the lifetimes of the item will be. One possibility is for 𝒞 to pin down that ỹ for which G̅(ỹ) = F̅(y^*), and certify F̅(y) if the observed y≥ỹ≥ y^*. This schemata will also enable 𝒞 to certify F̅(y) with a high degree of certitude (confidence), by considering several stress tests, say n, and requiring that for at least k≤ n of these tests, the observed y ≥ỹ; here 𝒞's probability that y≥ỹ is G̅(ỹ), and n and k be chosen to assure a specified degree of certitude.




§ ACKNOWLEDGEMENTS


Supported by a grant from the City University of Hong Kong, Project Number 9380068 and the Theme Based Research Scheme Grant T 32-102/14N and T 32-101/15R. Professor Glen Shafer of Rutgers University provided a pre-print of his paper, which motivated and stimulated the work discribed here. I thank him for this gesture.



§ REFERENCES

  
 [-100pt]
4





De1974
De Finetti, B. (1974).
“Theory of Probability: A Critical Introductory Treatment.”
Vol. 2, London, New York, Wiley. .

Kullback1
Kullback, S. and Liebler, R.A. (1951).
“On Information and Sufficiency.”
Ann. Math. Stat. 22, pp. 79-86.

Kullback2
Kullback, S. (1959).
“Information Theory and Statistics,”
New York, John Wiley and Sons, Inc.

LINDLEY1
Lindley, D. V. and Singpurwalla, N. D. (1991).
“On the Evidence Needed to Reach Agreed Action Between Adversaries, with Application to Acceptance Sampling.”
Journal of the American Statistical Association. 86, 933-937.

LINDLEY2
Lindley, D. V. and Singpurwalla, N. D. (1993).
“Adversarial Life Testing.”
Journal of the Royal Statistical Society, Series B. 55(4), 837-847.

Ram
Ramsey, F. P. (1931)
“Truth and Probability”
in Studies in Subjective Probability (Ed. Kyburg, H.E. and Smokler, H.) New York, John Wiley.

savage1954foundations
Savage, L. J. (1954)
“The Foundations of Statistics”,
New York, John Wiley and Sons, Inc.

Shafer2019
Shafer, G. (2019).
“The Language of Betting as a Strategy for Statistical and Scientific Commnunication.”
Working Paper #54, Rutgers University.

Von1944
Von Neumann. J and Morgenstern, O. (1944)
“Theory of Games and Economic Behavior.”,
Princeton University Press.








