text:
Systems and methods are disclosed to assist in making decisions. The systems and methods consider situations where the user is determining which of several alternatives is the best choice. The criteria to evaluate the decisions are input into a grid, and the user rates the different alternatives against the different criteria. Given several alternative choices or options, it provides the probability each alternative is the right selection. In one embodiment, the system implements a Bayesian approach. The systems and methods may estimate the chance some important consideration was missed thereby reducing the chance of being surprised or blindsided. The system and method may also examine the inputs to a decision analysis in order to detect entries that might reflect bias, assumptions or unusual reasoning, thereby increasing the chance of obtaining the correct answer. The systems and methods may also identify the factors that were predictive. The systems and methods do not accept information as valid and correct; rather, they allow cross-checking and verification, and further employ sub-systems and sub-methods to reduce mistakes. Systems and methods so disclosed may be employed in financial analyses, intelligence analysis, etc.
1. A method of decision-making, comprising:
a. entering at least two alternative options into a database;
b. for each of the alternative options, entering at least one criterion or weighting for evaluating the options;
c. specifying the relative importance of each of the criteria or weightings;
d. specifying, for each criteria and option, a strength rating, wherein the specifying a strength rating indicates how well the criteria or weighting either supports the option or is opposed or contrary to the option; and
e. calculating a result for each alternative option based on the importance and strength ratings.
2. The method of
3. The method of
4. The method of
5. The method of
6. The method of
7. The method of
8. The method of
9. The method of
10. The method of
11. The method of
12. The method of
13. The method of
14. The method of
15. The method of
16. The method of
17. The method of
18. The method of
19. The method of
20. The method of
21. The method of
22. The method of
23. The method of
24. The method of
25. The method of
26. The method of
27. The method of
28. The method of
29. The method of
30. The method of
31. The method of
32. The method of
33. A computer program, residing on a computer readable medium, for carrying out the instructions of
34. The method of
35. The method of
36. The method of
37. The method of
38. The method of
39. The method of
40. The method of
41. The method of
42. The method of
43. The method of
44. The method of
45. The method of
46. The method of
47. The method of
48. The method of
49. The method of
50. The method of
51. The method of
52. The method of
This application claims the benefit of the following US Provisional Patent Applications: U.S. Provisional Application Ser. No. 60/758,452, filed Jan. 11, 2006, entitled �Method For Determining Bias, Assumptions Or Atypical Reasoning In Decision Making�; U.S. Provisional Application Ser. No. 60/758,453, filed Jan. 11, 2006, entitled �Method For Use When Making A Decision That Estimates The Chance That The Individual Can Do A Better Job Because He Or She Has Missed Something Important�; and U.S. Provisional Application Ser. No. 60/758,454, filed Jan. 11, 2006, entitled �Method To Visually Display The Pros and Cons of Alternative Decision Choices�; all of which are incorporated by reference herein in their entirety.
The invention relates to decision making and to reduction of deleterious influences, and in particular decision making in complex situations when one has to select which alternative is the best choice, or which alternative is most likely to occur.
As has been well-researched in various behavioral sciences and is repeatedly observed in real life, the human mind is rarely completely rational. Instead, it succumbs to hidden assumptions and has numerous biases and preconceptions that cloud and color thinking and decision making.
These deficiencies can cause dangerous problems. For example, in the area of intelligence analysis, repeatedly, over time, surprise attacks have occurred. Congressional investigations have blamed a cause of the 9/11 attacks, as well as other calamities, on bias and faulty assumptions. There is a need to reduce the potential for dangerous events such as these.
In another field, evaluating financial value and predicting financial numbers is fraught with risks. Capital budgeting, acquisitions, and sourcing decisions might end up in error by large sums of money. Experts in the field are well aware of these difficulties and there is a need to remedy these difficulties.
A variety of decision support tools have been developed to help users make better decisions in risky and uncertain situations, including: optimization programs, statistical analyses, simulations, and an array of data mining and artificial analysis techniques. These previous efforts suffer from a lack of ease of use, lack of consideration of important factors, and various other drawbacks.
In one aspect, the invention is directed to a method of decision-making, including: entering at least two alternative options into a database; for each of the alternative options, entering at least one criterion or weighting for evaluating the options; specifying the relative importance of each of the criteria or weightings; specifying, for each criteria and option, a strength rating, wherein the specifying a strength rating indicates how well the criteria or weighting either supports the option or is opposed or contrary to the option; and calculating a result for each alternative option based on the importance and strength ratings.
Implementations of the invention may include one or more of the following. The result may be selected from the group consisting of: the best choice of the options, the option most likely to occur, or a ranking of the impact of the different options. The two alternative options may be to take an action and to not take the action. The importance weightings and strength ratings may be entered using a format selected from the group consisting of: a tabular format, a grid format, matrix, cross-reference format, a spreadsheet, a database, another program and combinations of the above. The calculating may be performed using an analysis method selected from the group consisting of: a Bayesian analysis, a neural network analysis, an artificial intelligence analysis, simulations, other probabilistic algorithms, heuristics, human judgment or combinations of the above. The method may further include the step of analyzing the importance or strength ratings in combination with the alternative to determine a consistency level for the alternative-criterion pair, whereby ratings may be identified that have a relatively high chance of being due to bias, hidden assumptions, suspect origins, preconceptions, unusual reasoning or creative insights. The method may further include the step of analyzing the importance or strength ratings and identifying if any of said ratings are anomalous, where anomalous ratings may be those that are unusually higher or lower than others in its criterion. The method may further include the step of analyzing the importance or strength ratings and identifying if any of said ratings bear an unusually large or small impact on the result.
Implementations of the invention may also include one or more of the following. The method may further include the step of displaying the anomalies in a graphical or visual manner, wherein the displaying may be selected from the group consisting of: displaying a matrix including information corresponding to an anomalous criterion and alternative; displaying a diagram in which arrows, lines, or other connections reveal which criterion correspond to which alternatives. The method may further include the step of estimating the chance that a correct alternative was missed, that an unknown issue was not considered, or that a surprise or surprising event might occur. If the chance that a correct alternative was missed or that an unknown alternative was not considered exceeds a predetermined threshold, the method may further include the step of displaying a notification to the user about the exceeded threshold. The notification may indicate that additional data should be entered. The method may further include the step of positing a new alternative and positing properties about the new alternative. The new alternative may be provided with default properties, or with average or neutral properties. The calculating step may employ a methodology selected from the group consisting of: Bayesian methodology, the LaPlace plus one method, and the Good-Turing methodology. The method may further comprise estimating probabilities of at least one unknown-unknown or one known-unknown.
Implementations of the invention may also include one or more of the following. The method may further include the step of displaying the results of the calculations in a graphical user interface that also includes representations of at least the alternatives and the criteria. The method may further include the step of displaying the results of the calculations in a graphical user interface that also includes representations of at least the ratings and any unknown-unknowns or known-unknowns. The graphical user interface can have at least ratings, alternatives, or criteria modified by the user and the consequent results automatically displayed on the graphical user interface. If the chance that a correct alternative was missed or that an unknown-unknown alternative or known-unknown alternative was not considered exceeds a predetermined threshold, the method may further include the step of displaying a notification to the user about the exceeded threshold on the graphical user interface. The notification may indicate that additional data should be entered. At least one of the importance ratings and strength ratings may be entered as a non-numerical value, and further comprising converting the rating from a non-numerical value to a numerical value. At least one of the importance ratings and strength ratings may be entered as a numerical value, and further comprising re-scaling the rating from a numerical value to a probability.
Implementations of the invention may also include one or more of the following. The method may further comprise displaying pros and cons of each alternative, wherein the pros are criteria that support the alternative, and the cons are criteria that contraindicate the alternative. The pros and cons may be displayed with bar charts, and the length of a bar may be related to the strength of the criteria. The pros and cons may be displayed in such a way as to also display the degree to which the criteria distinguishes which alternative is best. The method may further include the step of identifying risks. The identifying step may include determining a set of most critical strengths and weaknesses for a given alternative. Actions designed to reduce, eliminate or mitigate the risks of any alternative may be designed and implemented. A level of error in an output result may be less than a level of error in input data. The method may further include prompting the generation of new or additional ideas, insights and perceptions by employing information relevant to the situation and/or information developed from the criteria and alternatives. The method may further include identifying critical criteria and factors that might serve to help generate pertinent new ideas and insights.
Implementations of the invention may also include one or more of the following. The method may further include permitting improved insight or forecasting of the actions, thoughts or beliefs of other parties by role-playing or simulating that the user is that other person. The method may further include prompting the input of the concerns, motivations and goals of other parties as well as the factors, considerations and alternatives those other parties might employ in making a decision or taking an action. The method may further include permitting several people to participate in making the decision or arriving at a conclusion. The method may further include creating a common framework or presentation of the different views of the several participants in a manner that facilitates discussion, promotes better understanding, helps resolve differences of opinion or prompts better decisions and results. The method may further include organizing the inputs of the participants by level of disagreement and displaying the criteria on the basis of the level of disagreement. The method may further include determining or predicting the value of a numerical variable. The method may further include estimating a range for the numerical variable, dividing that range into intervals, and then employing criteria and ratings to estimate the value of the variable.
Implementations of the invention may also include one or more of the following. The method may further include employing templates or guidelines to facilitate the entry of data where the templates or guidelines may be based upon human experience or the analysis of past information, and templates or guidelines may provide criteria to determine which rating value to employ. The method may further include identifying criteria and factors more likely to be predictive of future events as well as factors that do not predict. The method may further include examining, for a given alternative, which criteria expressly supported that alternative and which did not and, if the alternative is what actually later occurred, deeming the criteria that were supportive as possibly predictive while deeming the criteria that did not support the correct alternative not predictive, wherein criteria that consistently predict the right alternative are then identified as predictive. The method may further include forecasting the actions of at least one third party. The method may further include providing a entry form for new ideas and insights. A plurality of users may provide input about alternative options or criteria, and the method may further include collaboratively identifying at least one area of disagreement. The method may further include determining a numerical value related to an option. The method may further include identifying a risk associated with at least one option. The method may further include providing a form where an action may be entered, the action acting to mitigate or to reduce the risk.
In another aspect, the invention is directed to a computer program, residing on a computer readable medium, for carrying out the instructions of the above steps.
Advantages of the invention may include one or more of the following, depending on embodiment.
Embodiments of the invention may provide guidance to determine which of several alternative options or possibilities is the best or most likely or to rank the different alternatives. The ranking or selection would be done on the basis of certain criteria, such as facts, factors or considerations that would be believed to determine or influence which of the alternative possibilities is the best.
Embodiments of the invention may also include means of providing guidance to determine when to stop one's work in terms of making the decisions. That is, means are provided to determine if the analysis meets the needs of the decision, or whether one should continue research in the hope of arriving at a superior decision, action or analysis.
Systems are disclosed that provide means to estimate the level of unknown events in a situation, that is, events or considerations that might have been missed or unforeseen. The unknown events might include surprises or unanticipated risks, or unknown events that could be harmful or damaging or be blindsiding. Estimating the level of unknown events also facilitates the identification of those unknown events or the issues that were missed.
In this sense, the unknown events may be further divided into known-unknowns and unknown-unknowns. The known-unknowns are the issues known to be unknown. The unknown-unknowns comprise issues for which one is not even aware that they are unknown. Systems according to certain embodiments include means to estimate their likelihood and knowledge.
In many cases, it is necessary to forecast or predict what event is most likely to occur. Experience has shown that many of the criteria or factors people employ to reach a conclusion often have little predictive value and are largely irrelevant. By sorting the data that predicted from the data that did not, certain systems and methods disclosed have the advantage of helping to identify the predictive factors.
Moreover, each different alternative choice has its pros and cons, the considerations in favor of it and the considerations against it. The ones against often reflect risks, and the techniques disclosed may uncover these.
Many decisions are improved by the generation of new ideas or insights. The invention provides a system for creating new ideas that are directly pertinent to the specific issues involved in the decision. New ideas and insight can assist in many situations including:
Decision making when several individuals are involved adds a level of complexity including the possibility of disagreements, disputes, and emotional reactions. Embodiments of the invention may provide means to achieve a more objective decision that better incorporates the inputs of the individuals and that might elevate commitment or agreement with the decision.
Certain decisions involve the actions of outsiders or those who might be in opposition, such as competitors, customers or the actions of enemies. The actions of such other parties are notoriously difficult to predict since they often have different goals and motivations. Certain embodiments of the invention may permit one to role play the other party and to gain greater insight into the other party's possible thinking or actions.
In some embodiments, the invention may provide means to automatically analyze the data input and identify possible biases or hidden assumptions. More, the same may provide means to make entry of the data easy such as the use of templates, pull down menus or copy-and-paste procedures. Different criteria may also be used, each with its own scale, and yet still obtain the same result. The disclosed systems may also provide means to display the pro and cons, such as strengths and weaknesses, advantages versus risks, of each alternative, and incorporate into the pro and con ratings not just the degree of support of the factor but also the factor's ability to discriminate among the different alternatives.
After a decision is made, additional steps might be taken. Often other individuals might have to be convinced of the decision's validity. Embodiments of the invention include systems that present the rationale for the decision clearly and concisely and in color.
Even if the best decision is made, few decisions are without risks. For the decision made, the invention possesses systems that help to identify the risks of that decision and to mitigate and counter those risks.
Other advantages of the invention will be apparent from the description below, including the figures and claims.
Referring to
pA denotes the probability alternative A is the correct choice, pB denotes the probability that alternative B is the right choice and similarly for pC.
Since the probabilities must sum to 1 the following equation holds.
It is noted in this context that three choices are employed for clarity, however, an almost unlimited number of alternatives may be contemplated.
The question desired to be answered by the user may pertain to, e.g., which is the best choice among A, B, and C, or which is most likely to occur, or a ranking of the impact of the different possibilities, etc.
The next step is for the user to enter certain criteria to be employed in the calculations (step 14). Criteria generally constitute the major facts, factors, and considerations of the situation. For example, for selecting the best product, the criteria may be price, quality, performance, flexibility, ability to customize, etc.
Of course, while user determination of criteria is noted here, the criteria may be determined automatically as well by, e.g., templates, various data mining techniques, etc.
The next step is to rate the criteria for their importance on some numerical or other scale (first part of step 15). For example, if the price factor is important, it might receive a weight of High to reflect that it is a highly important consideration in the decision. The weighting might be input numerically or, in the preferred representation by symbol: Low, Medium or High. The system may then transform any symbolic entry into a numerical value via a user-generated or default table. The criteria may have other data associated with the same such as importance, date of origination, priority, and so on, and these may be used to, e.g., adjust the numerical value of the symbolic entry. For example, criteria based on more recent data may be given more weight than older data. For each criterion and its corresponding scale, the same may be re-scaled to a measure of probability, in which case the numerical values for each alternative-criterion pair may then be interpreted as a probability.
The user then rates or evaluates the different alternatives against these criteria and the resulting data is entered into the program (second part of step 15). In particular, the user provides specific ratings of how each alternative was rated or evaluated on the different criteria. The ratings might be numerical, say on a scale of 1 to 10. Or the ratings might be in symbols (or letters), as these are often more convenient to enter, especially in a drop-down menu. In this case, the system may transform the symbolic representation into an equivalent value. In one embodiment, the symbols range from −−, −, N, +, ++ (double minus, minus, neutral or not relevant, plus, double plus). A rating of double minus means the alternative performs poorly on that criterion. For instance, if alternative A has very low reliability, it might receive a double minus on the reliability criterion. If alternative B has excellent flexibility, it might receive a double plus on the criterion of flexibility. If desired, the system may be designed to translate any symbolic rating into a numerical rating.
In one embodiment, a rating of N for Neutral or �Not relevant� receives a value of 0.5. This reflects being 50-50 either way, not supportive and not against, not good and not bad, not in favor and not against. The analogy is to flipping a perfectly weighted coin, where there is nothing to lead one to believe one way or the other.
If the rating happens to be a plus, it will receive a numerical value closer to 1, indicating that it is supportive or in favor. A negative rating will be less than 0.5 and closer to zero, indicating that is it more against, not supportive or not in favor of the alternative. Clearly, the closer the rating is to 1, the more positive or in favor of the alternative. The closer it is to zero, the more negative or against it is, the less supportive or less in favor.
The ratings may be done individually, that is, separately for each alternative and each criterion, or using several together, or as part of a formal approach such as a Bayesian Network, a decision tree, a simulation, a statistical analysis, or the like.
As with the weightings, the table of transformation of the symbols into numerical values can be input by the user or a default table pre-specified and employed.
As examples, if the user rates system A high on reliability, this will increase pA, the likelihood system A is the right choice. If system B rates low on customization, that will lower pB, the probability B is the best choice. If the user rates system A higher than system B on the important criteria, system A will obtain a higher probability of being the right selection.
To illustrate further, a decision may be sought about whether to develop product A or product B, and thus there are two alternatives, develop A or develop B. Several criteria may be used to select between A or B, specifically: development cost, whether a customer may be likely to purchase the product, how competitors might respond, and the ability to produce the product at high quality.
The alternatives would then be rated on each of the criteria. For example, if product A has a high development cost, then the numerical value for that rating would be low indicating that the high development cost is unfavorable to the development of product A. If product B has a low development cost, then the numerical rating for that would be high, since low development cost is favorable to B.
In this manner, each of the alternatives would be rated on each of the criteria. The user needs to select criteria so that if an alternative gets a high rating on most of them, it is likely to be the right choice. In particular, the user rates each alternative-criterion pair on how consistent they are. In other words, the user may consider if the alternative were the right choice, whether the criterion would be likely to be seen or occur.
This rating may be done in a matrix, grid or tabular form. In one exemplary embodiment, the alternatives may be given in columns and the criteria may be given in the rows. The cells in the grid may have the ratings for each alternative-criterion pair. Weightings for the importance of the criterion might also be employed.
At the end of the data entry phase, then, the following will have been entered into the grid: the various alternatives, the criteria used to rate the alternatives and the ratings themselves, adjusted for the criterion's weighting, where any non-numerical rating will have been transformed into numbers.
Entries may optionally be checked and new insights obtained by sorting the criteria in various ways, including on: diagnosticity, weight, importance, priority, date, or other related measures as described above. The entries may then be adjusted if necessary or desired.
Referring to
Moreover, the pros and cons may be displayed in such a way as to reflect not just whether the factor is for the alternative or against the alternative, but rather may also incorporate the degree the factor distinguishes which alternative is best. For example, for a certain factor, e.g., cost, all of the alternatives may rate highly. Then the factor of cost supports all of the alternatives. But it does not help discriminate which is best. Cost is then a weak discriminator.
At this point, and referring back to
First, an estimate may be made of the likelihood of an unknown alternative occurring, such as the possibility of being surprised or �blindsided� (step 16). To help prevent surprises, certain embodiments of the invention may estimate the probability of one occurring. For a given decision, it might calculate that there is, say, an 18% chance something important was overlooked. In this way, the system alerts the user that something likely has been missed with roughly that probability.
Step 16 is now considered in more detail. Referring to
The user then specifies ratings (step 32) for the criteria for Alternative X in the same way as above for the known alternatives, or uses default values if such ratings are unknown, e.g., from a template, database, via data mining, etc. The system accepts the additional unknown alternative and considers X as an alternative (step 34). Probabilities are calculated (step 36), and the user may then examine the calculated probabilities to determine the likelihood that this unknown may occur (step 38).
In more detail, there is the original group of n possible options and to that group the system adds another possibility termed Alternative X to create a set of n+1 options. If pX depicts the probability Alternative X is the correct choice, then a new equation (2) replaces equation (1), an equation that expressly includes the possibility of Alternative X.
Equation (2) allows the estimation of the probability of Alternative X even if nothing is known about it. Although a Bayesian procedure is used, any drop in the sum (pA+pB+pC) forces px to increase. For instance, if the probability for Alternative A drops, while the probabilities for B and C remain the same, then the probability for Alternative X must rise. This constitutes one embodiment of a methodology for obtaining the probability for some unknown alternative.
As an example, the question may be as to whether to go to any one of three restaurants. Some information is received that is critical of one of them. That increases the chance of going to any of the other two, as well as to some restaurant not yet considered, Restaurant X.
As another example, it may be initially believed that A, B or C is sure to occur, meaning that the sum pA+pB+pC starts out equaling 1. But inconsistencies may be discovered in the information that drop that sum to 0.6. By equation (2), pX=0.4. This reveals that an unknown and possibly surprising event has a 40% chance of occurring.
In other words, even if almost nothing is known about X, knowledge of A, B and C provides information about X. As the probabilities for the original options are adjusted, the probability of Alternative X is adjusted.
Knowing the probability of Alternative X alerts the user to the seriousness of the possibility something might have been overlooked. If the probability of Alternative X reaches 10% or above, that is a clear warning alert in many situations. In other words, there is a relatively high probability of being surprised or blindsided. The systems and methods, in some embodiments, may also assist in discovering what has been missed, as is disclosed below.
As a corollary to this step, the embodiment also helps to determine when to stop working on a project or problem. In particular, stopping prematurely may result in some important consideration being overlooked. On the other hand, it is undesirable to continue working on a project or problem when additional effort is unlikely to improve the quality of the decision. In this embodiment, the system and method provide a signal about how much risk inures if efforts are ceased and no further examination is considered for the decision. Put another way, this method estimates the probability that something important has been missed. What was missed might be dangerous or harmful, e.g., that one will be blindsided or surprised. On the other hand, what was missed might have been a good opportunity.
As an example, consider the selection of a strategy to improve sales of a product, with the following alternatives: increase advertising, decrease price, or hold a special sale. The criteria may be: the expected number of new customers created, the cost of taking the action, the resources required in terms of having the people available for the project, the impact on profits, how quickly results can be obtained, and the possibility that action might influence the sales of other products that are being sold.
The user would rate each alternative on each of the criteria. For instance, the user would rate the alternative �increase advertising� on the criteria of �expected number of customers created�, the �cost of taking the action� and so on. As discussed above, the ratings may be done on a scale such as bad to good, weak to strong, 1-100, using probabilities, etc.
Based upon those ratings, the procedure then provides the probabilities that each of the different alternatives is the best choice. For example, �increase advertising� may receive a probability of 65% of it being the best choice, �decrease price� may receive 20%, and �hold asale� 15%.
This embodiment may assume the user already has a systematic procedure that accomplished these activities and provided these probabilities. That procedure may be capable of accepting different alternatives, and permitting the alternatives to be rated on different criteria.
The embodiment now estimates the chance that the right alternative was not even considered. It does so by postulating another alternative, such as Alternative X above, with the property that nothing is known about it, a priori. Since nothing is known about it, the best guess as to its features and qualities may be that it is average. That is, because statistically and in most circumstances, if one item is selected out of a population, the best guess about what will be selected is that its features or qualities will be average or neutral. Thus, Alternative X is assumed to be neither good nor bad on any of the criteria, just average or neutral.
The embodiment then adds �Alternative X� to the other alternatives so the list becomes: increase advertising, decrease price, hold a special sale, and Alternative X.
The procedure now recalculates the probabilities that each of the alternatives is the right choice, and yields new probabilities for all of the alternatives, including Alternative X.
In effect, the procedure reevaluates the probabilities but now with a new possibility, Alternative X. In many cases, the previously considered alternatives do well on some criteria and poorly on others. How Alternative X will emerge, relative to the others, is not clear beforehand. Nevertheless, the estimating procedure will then determine its probability of being the correct result.
One embodiment of this system and method, then, is to assume there is another alternative, Alternative X, take the best estimate of its properties, and then see how it performs relative to the other known alternatives. The result is then the probability of that other alternative, yet unknown.
The probability determined for Alternative X then represents the estimated chance that something unknown was missed and is thus the probability that one will be surprised, blindsided or will have missed an opportunity. In general, in real life, there is usually the chance something was missed, and the probability given for Alternative X provides an estimated chance of that.
As noted above, the probability of Alternative X also provides an indication of when to stop working on the decision or choice of action. If that probability is fairly low, then it is likely the user has explored the decision well. If that probability is high, then the user should continue working because it is likely something else should be considered before the decision is made.
Surprisingly, when users are confronted with a numerical probability for Alternative X, they often think of a new alternative, issue or factor that they missed. In other words, the impact of this result may be psychological, and not just statistical or mathematical. For instance, suppose that Alternative X turns out to have an 18% chance of occurring. People then see that they have an 18% chance of being blindsided since there is an 18% chance they missed something. Most people are not previously aware of this high probability of having missed something, and at this point, they often think of what they might have missed.
In more technical detail, there may be envisioned a situation where there are n criteria and m regular alternatives, excluding Alternative X.
The user inputs the n criteria (the factors listed in the yellow to the left in
Also, the user inputs the m regular alternatives. Alternative X is designated as alternative m+1. (The alternatives are along the top in blue in
Rij denotes the numerical rating of alternative j on criterion or factor i. As noted above, if symbols are employed such as in
The user may input the Rij (or their symbolic equivalent) for all of the regular alternatives, that is, for i=1, . . . ,n and j=1, . . . ,m.
The weight factor will contribute to the calculation of Rij. A weight of High will make Rij more important so it has greater impact on the outcome. A weight of Low will decrease the impact of Rij.
Alternative m+1, however, is different since it refers to Alternative X, which is unknown. As noted, the ratings for that alternative may be Neutral or Not known, since nothing is known about it. For Alternative X, the system may input the Neutral ratings on all criteria automatically.
Now that Alternative X has all of its ratings on all of the criteria, it is just like the other alternatives, since they also have all of their ratings on all of the criteria. The system can then treat Alternative X just like other alternative.
Most of the actual calculations are based upon the ratings, Rij. In a Bayesian interpretation, the Rij represent the conditional probability of the factor or criterion i, i=1, . . . ,n given the alternative j, j=1, . . . ,m.
One special rating is the initial starting value for column j (alternative j), R0,j. These provide an initial starting value or in Bayesian terms, �a priori� information.
In one embodiment, for i=0 the initial or a priori rating R0,j will have the value
That is because there are m+1 alternatives and if nothing is know about any of them a priori, they are all equally likely, at least initially. In other words, when first starting out and before doing any ratings, nothing is known about the different alternatives. Hence, they start out by being equally likely. Of course, as soon as one commences rating the different alternatives against the various criteria, that will change.
Systems and methods according to embodiments of the invention now determine, for any alternative, the probability that it is the right choice.
One of the important calculations is to obtain a value for each alternative j, i.e., Vj. In one embodiment Vj will be proportional to the chance that a given alternative is the best choice. Mathematically, Vj is a function of the ratings given for alternative j.
For any alternative, the higher its ratings Rij, the higher the value of Vj. That is because the Rij represent how much the criteria support the particular alternative. The better the alternative does on the criteria, the higher its value of Vj.
A variety of functional forms for Vj are possible. They can be a simple sum or weighted sum or a product or a more complex relationship of the various ratings Rij. In one system and method, they are a product. That is,
Often, as in
This implies
The Pj are important as they represent the probability an alternative is the best choice.
It should be noted that the above steps follow a Bayesian approach. More precisely, the Ri,j then are interpreted as conditional probabilities. Pj is then interpreted as the probability alternative j occurs, given the information provided by the factors and criteria. That means, if the preferred steps are followed, that the Pj probabilities obtained will be Bayesian estimates. Of course, the systems and methods of the invention are not limited to Bayesian approaches. Other probabilistic techniques may also be used, as are noted below.
As discussed in detail above, Alternative X thus provides an estimate about what was missed, that is, what is unknown. Systems and methods according to embodiments of the invention may also divide those unknowns into known-unknowns and unknown-unknowns. That makes the type of risks clearer.
Referring to
Thus, to develop the known-unknowns, the procedure may assume initially that the data exhibits the same behavior as the past and that a new batch of information is collected (step 42) that adds a quantity of information the same size and value as the information already collected. The procedure then recalculates the probability scores (step 44), now assuming a new batch of information is included. The initial probability of Alternative X is PX0, and the new probability is denoted PX1, and this is also the probability of the unknown-unknowns (step 46); the probability of the known-unknowns is PX0-PX1 (also step 46). This then provides statistically reasonable estimates for the probabilities of the Known-unknowns and Unknown-unknowns. The user may adjust their strategy based on this information (step 48), and finally may examine the calculated probability for Alternative X to determine the likelihood of an unknown outcome occurring (step 52).
Referring back to
In particular, the system and method detect factors that are �out of line�, herein termed �outliers� (see
Of course, not all outliers will be due to bias or faulty assumptions. But generally they are at least considered initially suspect, and the system and method highlights the same so that they can be re-inspected and double-checked.
In more technical detail, first, for each individual criterion, i=1, . . . , n, one software implementation may calculates the average rating, which may be the average rating across a row, as in
More precisely, the following is calculated for each criterion (row):
Next the system and method determines by how much the rating deviates from the average. The deviation is in the next equation:
The deviation is an indication of how much that particular rating is out-of-kilter or out-of-line with the other ratings for that criterion.
It should be noted that other measures of deviation could also be employed, including variance, absolute deviation, and so on. A further deviation measure would be a derivative, such as the derivative of the value of alternative j with respect to the rating of Ri,jj, or the derivative of the difference in the value of alternative j from other alternatives with respect to the rating, Ri,j. The gain or loss in entropy can also be employed.
Next the system and method may weight each deviation, dependent upon the alternative j. If that weight is denoted Wi,j, then if
In one representation, for i=1, . . . n
In this case, the weights are the probability that alternative j is the correct choice. In other words, the more likely an alternative is to be correct, the more weight is placed upon that deviation. The alternatives more likely to be correct receive more weight because they are likely more important in making the decision.
At this point, for a given criterion, all of the weighted deviations are given as in the next equation. It is noted that the absolute value is employed here because only the magnitude of the deviation is of interest. More precisely, the weight Wi,j is multiplied by the absolute value of the DEVi,j.
The weighted deviations, (0.10) may now be ranked or sorted. In some embodiments this may require two separate ranking (sorting) procedures.
First, for a given criterion i, the system and method may rank these weighted deviations over the regular alternatives j=1, . . . ,m. That is, for a given criterion i (row), there are m weighted deviations for j=1, . . . ,m. Those m numbers are ranked from largest to smallest.
There are n of these, one for each criterion (row). At this point the �worst� weighted deviation in each row i (that is, for each criterion i) has been obtained.
Second, these �worst� (weighted) deviation values are ranked from largest to smallest. More precisely, next the system and method finds the row with the worst, worst deviation. This is the worst of the worst. The second worst, third worst and so on down are also obtained.
In other words, two ranking are occurring. First, for each criterion, the largest deviation across a row is obtained. This provides the worst in any row. Next, the worst of the rows are ranked to determine the worst of the worst.
The higher the rank of any particular weighted deviation, the bigger the deviation, and the more likely it reflects a bias, hidden assumption or other anomaly.
The results of this ranking are then displayed. Referring to
The red highlighted cell in a row represents the worst deviations in that row. The top row is the worst of those worst deviations, and on down in rank order.
For instance, consider the red cell at the location denoted �1�. The double minus indicates a very strong negative value for that alternative. That cell is decreasing the probability that alternative is the best choice. The system and method may highlight that information and thereby invite user interaction to check that the information does not reflect a bias or erroneous assumption.
Generally, the ratings are particularly useful in revealing bias, preconception or a hidden assumption. For instance, suppose the user is biased in favor of a particular alternative. Then they might deliberately put in numbers for that alternative that are higher than they should be. Concomitantly, numbers might be inserted into the other columns that are too low, thus making the other alternatives less likely.
More precisely, if one considers a specific criterion and examines its corresponding row, if the user is based toward a particular outcome, the rating number in the corresponding column is likely to be higher than it otherwise should be. Also, the numbers in the other columns might well be lower than appropriate. Further, if the bias was effective in influencing the outcome and in creating a winner or leading contender, then if the column is a �winner� or leading contender, it will have its numbers disproportionately raised. If the column is a �loser� or a weak alternative, then its numbers would be disproportionately lower.
This leads to a rule for identifying ratings that are suspect in that they might reflect bias or a hidden assumption. Specifically, entries are examined that are disproportionately higher than others in the same row, particularly if those ratings are associated with a leading alternative. Similarly, entries may be examined that are disproportionately lower than the others in a row, particularly if they are associated with alternatives whose results are weak. To avoid the identifying entries that are irrelevant to the decision or analysis, the anomalies that are more important to the decision may be emphasized. Due to this rule, simple statistical procedures may be employed to identify ratings that might reflect bias, preconception or assumption.
This particular rule considered ratings for the same criterion, that is, along the same row. Since these rating are relative to the same criterion, they have a similar basis, so the bias it more likely to be apparent. In effect, distortions due to different criteria are eliminated, and hence the statistical analysis is not contaminated by that type of noise. Nevertheless, other statistical procedures can also be employed along these lines, namely, the identification of ratings that are anomalous in that they are somewhat distinctive from other ratings.
Once identified, anomalous or suspect ratings may be highlighted in a matrix format (
Referring back to
In particular, based upon the information and data input by the user, systems and methods according to certain embodiments of the invention may not only predict which alternative is most likely to occur, but also, after the event has occurred and the actual outcome is known, they may indicate which factors were predictive and which were not.
In more detail, experience has demonstrated that, in many cases, most of the factors employed have minimal predictive value and usually only a few factors determine the outcome. Knowing those critical factors can significantly enhance results, and certain embodiments disclosed may identify those factors. These may even identify situations where the user has a significant possibility of being in error because of inconsistencies in the data or insufficient data. By warning the user of such situations, the system helps prevent error.
In even more detail, the system may be employed to determine which of the original factors accurately predicted the correct event and which did not. This may be important information because one may wish to employ factors that predict what will occur, not ones that fail to predict. Identifying factors that were predictive for events whose outcomes are now known can help significantly with future predictions.
If one of the original alternatives turned out to occur, that is, to be the correct choice, then the factors whose ratings were positive for that alternative were the predictive factors. Conversely, factors in the original ratings whose ratings were negative for the correct event were wrong because they were against the event that actually occurred and pointed in the wrong direction. For example, and using the graphical format described above, a factor with a double plus for the correct outcome would be predictive. A factor with a double minus for the correct event would not be predictive, in fact, it predicted incorrectly.
However, just because a factor had a positive rating for the correct alternative does not mean it had much value in predicting the event. For instance, suppose a given factor has double pluses for every alternative. (In
A useful measure of this disproportionate influence is the deviation described above, and that is one measure the system and method may employ. From equations (0.6) and (0.7) the preferred manifestation of the measure is:
In row i note Ri,j pertains to alternative j, and suppose Ri,j and DEVi,j are both strictly positive. Then alternative j should receive a distinct increase versus the other alternatives, since Ri,j is distinctly above the average rating in that row.
For a given alternative j, the software may calculate these deviation values. Then it ranks them from the most positive to the most negative, where the ranking is over the criteria or factors i, i=1, . . . ,n.
Given that ranking, the factors with the most positive deviations were likely the most predictive of alternative j. The factors associated with the most negative deviations were likely the most incorrect or inaccurate in predicting alternative j.
In one embodiment, the top ten predictive factors are listed, as well as the ten that were most inaccurate.
The user clicks the �occurred� button for the alternative that actually occurred. If the event that occurred was not listed as an alternative in the original analysis, the user clicks on the �occurred� button under �Alternative X�.
The factors that were predictive (in green) and the factors that were inaccurate or incorrect (in red) appear. In the embodiment of
The event might have already occurred, but knowing which factors predicted correctly and which predicted incorrectly can help to improve predictions in the future.
Referring back to
Much of the critical information about risks may be obtained when the data are input. In one example, several alternatives may be given, say Alternative A, Alternative B and Alternative C, and one may want to determine which is best. The alternatives are rated on different criteria, say price, performance, reliability, flexibility and so on.
In one example, Alternative A does well on price and well on flexibility, and Alternative B does poorly on price and also poorly on flexibility. Depending upon the ratings, the system and method then estimate which particular alternative is likely the best choice.
The ratings information also suggests where an alternative is strong and where it is weak. That provides the risks, since the weaknesses are typically the risks. In the above example, the poor flexibility of Alternative B is a risk with that system.
It is important to note that the strengths are also often risks, and the system and method may identify these as well. For example, in one weapons system example, Alternative A may appear to be the best because it will integrate well with other systems. In this case, its integration capability is a strength.
That strength, however, is also a risk. When installing a system corresponding to Alternative A, one has to be sure that it can integrate well with other systems. In other words, Alternative A was selected for installation because it had an important strength, the ability to integrate with other systems. But now that strength must be realized and achieved, so the same becomes a risk. That is why systems and methods according to certain embodiments of the invention highlight that type of information.
Thus, the systems and methods may examine the ratings input by the user, which reveal the strengths and weaknesses. A sorting then provides a ranking and determines the most critical strengths and weaknesses for a given alternative. With the system expressly highlighting risk information, the decision maker can then take action to counter the risks. One advantage to doing this is to counter confirmation bias. Users naturally tend to adhere to their own viewpoints and opinions, sometimes to excess, and they tend to believe supportive information but to disparage and downplay information to the contrary. Confirmation bias is inherent in the human mind, and since users are generally unaware of doing it, impediments and risks are frequently disregarded or overlooked. The systems and methods disclosed here help to resist this cognitive error not only by identifying risks and dangers, but also by providing means to mitigate their potential harm.
To consider the highlighting of risks in more detail, the user may first select a particular alternative j. Then the software ranks the risks, Ri,j, for that alternative. Here the ranking is over the n criteria, i=1, . . . ,n. The result is a ranking list of the criteria for alternative j, from the ones with most positive rating to the ones with the most negative.
The ones with the most negative ratings are then displayed as risks. The most positive are also displayed in a separate list, since the strengths might also be risks as described above. In one embodiment, only the top few in each category are listed (see
Referring back to
A variety of ancillary issues and examples are now described.
One problem with prior decision support software packages is the algorithm itself. Algorithms greatly differ on how well they handle errors, and many actually make the errors worse. Technically the issue is elasticity, i.e., how much do errors in the input influence the error in the output. For example, if the input data have a 10% error, it may be desired to know if the output is in error by a similar percentage or if the output error is higher or lower. Elasticity is especially important in complex decisions since the data often have errors, are insufficient, or both. The systems and methods according to certain embodiments of the invention have an unusually low elasticity that can provide the correct answer despite a range of errors and difficulties in the data.
Two examples are provided below to clarify how different algorithms might increase or decrease error.
In this example, which is indicative of prior systems, the revenue in a business is 100 (in millions of dollars). The costs are 95. Hence, profits are 5.
But in this example there are a variety of problems so there is an estimated 10% error in the revenue figure. The real revenue, thus, is anything from 90 to 110. For simplicity, it is first assumed the costs remain at 95. Now the problem becomes
Originally, the profit was believed to be 5, but due to the error in the input information, it could be from −5 to 15. In other words, due to the procedure employed, an input error of 10% balloons to a perhaps 200% output error. The elasticity here is inordinately large since this is a very large increase.
A different procedure might treat the errors in a superior manner. To illustrate, in this example there are 60 white balls and 30 red balls. Thus the percentage of white balls is 66
Now it is assumed that an error has been discovered. There were not 60 white balls, but 70. This is a 16⅔% error in the input (10 on a base of 60).
The correct result is thus 70% white balls:
Previously it was believed that the percentage of white balls was 66⅔ when it really was 70. That is an error of 5% in the output (3⅓on a base of 66⅔).
The input data was in error by 16⅔%. However, the final result, the output, was in error only 5%. In other words, the input error of 16⅔% has been reduced to 5%, less than ⅓ as large.
The two different example procedures above treated the errors quite differently: one vastly increased the error while the other reduced it. It is a matter of the elasticity. In designing an algorithm there are many choices, and depending upon the choice the procedure employed can either increase the error or decrease the error, and sometimes dramatically so.
Systems and methods according to certain embodiments of the invention were designed to have low elasticity; their underlying algorithms as described tend to attain the correct result despite certain errors and inaccuracies in the input data.
In other words and according to research, even when the data are wrong and the corresponding probabilities contain large errors, the system and method still performs well in providing the correct results. Many common decision approaches have high elasticity and do not perform well when faced with imperfect data. By being expressly designed to have low elasticity, the system provides the correct answers when many other approaches will not.
At arrow 4 is a large white group of cells with plus and minus ratings. Blank cells default to neutral. These cells depict how each alternative rated on each criteria. Across a row are the ratings given the different alternatives for the specific criterion or factor. For example, the possibility �Acquire 100% of Indian Firm� received a double plus on the factor �Immediate growth with revenues likely doubling�. Also along the top in blue are cigarette-like bars and corresponding numbers (see arrow 5). The numbers are the probability that the corresponding alternative is the best choice. The bar height is proportional to the probability number. These numbers are calculated by the software as described above using Bayesian or any number of other probabilistic strategies.
The software permits a direct, head-to-head comparison of any two alternatives, including a check of financial information as will be described.
The direct comparison of any two alternatives considers the ratings each alternative obtained on the different factors. What is special is that only the factors where the ratings were different are displayed. If the ratings were the same, or essentially the same, on some specific factor, it has no impact on which alternative is superior. Therefore, it is not displayed. For example, both alternatives may receive a plus rating. Then that rating did not discriminate between the two alternatives, and is not displayed.
The mechanism is straightforward. Via a choice menu, the user selects which two alternatives to compare. The software then examines the ratings for the two alternatives. Only the factors where the ratings are different are displayed.
It should be noted that often only a few factors appear because the ratings are the same for most of the factors. This provides a significant simplification. Also, the difference between the ratings is determined and then ranked. In one embodiment, the factors are shown in order of the greatest difference of the two ratings. Thus, the factor where the two ratings were most different will be on top and so on down the line.
A bar proportional to the difference is displayed under the alternative that is higher on that factor. See
A financial comparison takes the above analysis a step further. The user selects which two alternatives to compare. As before, only the factors where they are reasonably different are shown. However, the software may insert an additional column for each alternative, a value column. Into that column the user may type the value of that alternative, and usually this will be the financial value, but could be other values such as cost, utility, benefit etc. The user might have to independently estimate these values, but in any case they are entered in the appropriate cells.
The software sums the value of each column and displays that at the top of each value column. The difference of the two sums is also displayed in the �Net Value Comparison�.
This feature provides a quick cross-check on the financial projections. It compares the value of any two alternatives. Only the factors where the two alternatives are different are considered. That makes obtaining the needed entries very quick.
To detect deviations, the user may click on a �Detect Deviations� button, and one or more cells become highlighted in red, as in
Value entries that differ significantly from the approximate values are then highlighted in red.
Although the disclosed embodiment employs regression, other approaches to estimate the value corresponding to the ratings may be implemented. For example, one could take the average financial value of the factors given a plus rating, and similarly for the other ratings.
It is also noted that colors may be employed extensively for convenience and for ease of display of results. Such color maps are sometimes termed �heat maps�.
The colors of the heat map are determined from the equations (0.6) and (0.7), and employ the deviation measure. The preferred manifestation of the measure is
For a given cell, the more positive the deviation value DEVi,j, the deeper and darker the green color for that cell. The more negative the deviation value, the deeper and darker the red color.
The value of the color map is that it quickly conveys the reasoning behind the ratings entries.
This situation is typical. As the weaker alternatives are eliminated, the noise is reduced and the colors clarify and simplify. It becomes easier to see the underlying argument. The colors thus convey considerable information about the analysis and which alternative is likely the best.
In this example, the range of the firm's value is from $95 (in millions) to $120. This is divided into five segments, $95, $100, $105, $110, $120 that are the alternatives and are inserted along the top in the blue section. Third, along the left side, in yellow, the user inputs the various criteria and factors that are deemed to determine the numerical value. Fourth, the user then indicates by entering plus or minus values in the cells, which alternative seems most likely for the factor being considered.
In the example of
To accomplish the ratings, for the factor �present revenues�, the exemplary user believed, for that specific factor, that the value of the firm would most likely be $105 and thus inserted the plus in that cell (see the plus sign in the cell in the $105 column and the row corresponding to �Present revenues�).
For the factor �develop and deliver a �branded� experience� the exemplary user believed that based upon that factor, the value of the firm might be $100. For that factor, the user also felt that $95 was not likely, and so inserted a double minus in that column.
The same is done for each factor on the left. That is, the most likely value of the firm value, based upon that factor alone, is determined and the corresponding pluses or minuses are entered. The stronger the performance of that factor, the higher the numerical value selected.
Conceptually, the various factors on the left are what cause the numerical value. In the example, the performance of the factors was what was believed to produce the firm's value. The ratings input are an estimation of how well a given factor will perform and contribute to the firm's value.
As discussed below in connection with
The next step is, by examining the height of the columns for each of the different alternatives, i.e., in blue along the top, the user can determine what is the most likely number for the numerical value being sought. The estimated probability each of those is correct is also presented.
In this example, the value slightly below $110 appears reasonable.
In essence, the standard features of the basic decision matrix were employed, but for an application slightly different from those previously mentioned. In this instance, they were employed to determine a numerical value.
As noted above, to help the user enter the plus or minus values, various templates or guidelines can be created.
To illustrate,
Guidelines or templates can be created for each factor and for most common situations.
In some embodiments, the invention provides systems to assist when several people are involved in a decision. In essence, each person may fill out a matrix separately. Then the system may collect each of the individual matrices and combine them into a composite. The composite can help identify areas of disagreement and assist in exploring and resolving the differences. The composite can also be used to compute averages and the other metrics the system and method provide.
More precisely, when several people are involved in the decision, the first step is for the participants to decide upon the alternatives and factors to be considered in the decision. This creates a common matrix for each person to employ. Second, each individual may complete his or her entries into the common matrix separately, meaning that each would individually enter his or her own plus or minus ratings. Third, using the web, a server or other system, each participant submits his or her matrix. Next, the system organizes the different matrices, and calculates the average entry and other metrics. Fifth, the system may organize the factors by level of disagreement and display the same, as indicated in
In the example shown in
The special format also organizes the factors by the degree of disagreement. �Market share� has the most disagreement among the four participants, and is on the top (see yellow section to left). �Entertainment� has the next most disagreement, and so on down. The entries of each individual are given on the appropriate row.
The participants can then examine where the agreements are and where the disagreements are. They can then analyze the disagreements in more depth and more objectively.
In a variety of decisions, one must consider the viewpoint or possible actions of other relevant parties such as the customer, the competitor or perhaps the enemy. It is particularly difficult to forecast the actions of a party with goals and motivations that are quite different from one's own.
Referring to
In the first window, the user inserts his or her best estimation of the other party's motivations and concerns in the situation. In this example, input are the likely motivations of a terrorist leader, and the user is role-playing that person.
Then, based upon those motivations and concerns, the user imagines the goal of that party and enters that in the second line. In the example that goal is to �Disrupt U.S. efforts in Iraq by hitting Iraqi training center or US convoy�.
In the next slots, the user enters that person's i) alternatives for achieving the goal, and
ii) the facts and factors they might employ to evaluate which alternative to select.
The concept is to first imagine the other party's motivations and goals. Then, role-playing that other party, to provide the alternatives and factors that the other party might utilize in a decision.
As noted in
The user then enters the plus or minus ratings, and the best alternative then is determined, i.e., the alternative the other party might well select in the decision.
The purpose of this system is to help the individual evaluate a decision problem from the viewpoint of a different party or individual. That is done by role-playing and considering the motivation, concerns and goals of that other party. Then, the user sets up the decision matrix that person would face and completes it.
Since this is done in a structured manner from the user's best estimation of the other party's perspective, it often provides insight into the other party's actual thinking.
In many decisions there is the need to generate new ideas and insights. One system of the invention estimates the probability something was missed, and new ideas might be needed to discern what was missed. Or perhaps a roadblock exists or impasse where there is a trade-off and one cannot improve on one factor without harming some other factor. Here, as well, some new ideas might assist.
Referring to
The Synapse feature starts as in
As the four factors were selected for their importance to the decision, that helps ensure the user is focusing on the most critical issues and factors.
Next the user, or a group, randomly generates new ideas that are triggered by the factors in the four corners. Those ideas are typed into any of the interior boxes, filling them up. The ideas generated should be anything that pops into one's mind, since the need here is for novel, creative ideas, and one should not censor one's thoughts at this point.
In this example, the important factor in the upper left corner is �wants nuclear weapons�. The users then generated ideas based upon that factor and entered them into the boxes, as in
Because the factors in the corners are pre-selected by the system to be highly important, the ideas generated tend to be focused on the critical issues. Often a new and perceptive insight is spawned.
Generally even the best possible decision has risks. Whatever the decision determined, embodiments of the invention incorporate systems for depicting the possible risks, as shown in
The system determines this information directly from entries in the basic entry matrix. Ratings that are particularly positive or supportive of an alternative will be in the �Positive Factors� section. Ratings that are particularly negative will be listed under the �Negative Factors� section.
Once the system highlights the risks, the user should enter into the software the actions that will be implemented to mitigate or minimize the risks. That information is input across a row under the three columns, Action�or what action will be taken to mitigate the risk, Ownership�the person responsible for implementing the action, and Monitoring/Performance Measure�how progress will be monitoring (see
It should be noted that, in alternative embodiments, the systems and methods of the invention may be provided as stand-alone programs, or as web- or server-enabled software which may be integrated with any database. In this way, collaborative features may be provided, such as Wiki or blog capabilities. In implementing this, a service-oriented or similar architecture may be employed, which would then permit a user to: hot-link to other programs and much more easily search for information and message traffic, saving time; and store and easily retrieve templates of standard situations, and also allow for users to create new templates. Past examples could then also be retrieved, saving the user's time and helping to ensure that important considerations are not missed.
In other alternative embodiments, e.g., for military or intelligence applications, a �future map� may be employed that presents various future options and possible enemy responses. The goal is to depict information not only about immediate options but also about the other side's possible responses and one's own counter-responses. Using a concise, visual format, this helps commanders see further into the future, to see at least a few steps beyond the other side's thinking. Senior leadership are then better able to perceive not just the different available options but also their implications.
In financial embodiments, it is noted that improved financial projections and valuations may obtain by determining the input numbers from the underlying events and actions. In other words, it is best to develop the numbers directly from their foundations of customer behavior, competitor actions, product development, operational success and the other causal factors. Starting with the foundations opens up several new capabilities. First is improved accuracy, because basing the numbers on their underlying causes improves accuracy. Second is identification of risks, because when considering the foundation factors, some of these might be favorable and benefit the firm and some might not. The ones that are not favorable are often risks. As described, the systems and methods disclosed can expressly highlight these, thereby exposing risks that usually are hidden. Third is an identification of pathways to superior results. As some of the disclosed embodiments probe �underneath� the numbers, they often discover new means to improve the results, e.g., finding that taking certain actions could significantly enhance financial results. In many of the embodiments described, a benefit may be obtained by allowing users from non-economic backgrounds, e.g., marketing, operations, etc., to take part by providing input and participating in analyses on issues of their own expertise. This enhances accuracy and the involvement of these individuals boosts their commitment. Finally, it is noted that the disclosed methods may be employed before more traditional analyses, since starting with the foundations naturally improves results. One goal in certain embodiments may be to improve the quality of decisions by 20-40%.
It should be noted that while the above description has been made with respect to specific embodiments, the scope of the invention is to be interpreted and limited only by the scope of the claims appended hereto. For example, while a Bayesian analysis has been described, any number of probabilistic methods may be employed to yield results, including: neural nets, data mining, simple probability algorithms, and various other such methods. It should also be noted that the above descriptions has used the terms �system� and �method� in an exemplary fashion, and these refer to system embodiments and method embodiments of the invention. The use of one such term does not exclude consideration of the other with respect to the described and pertaining embodiment. The term �software� is also used on occasion to mean either �system� or �method�, depending on context.
Referenced by
Original Image


text:
Fri, 7 Nov 2014
965 The Buzz
Alt-J Accoustic
Mon, 3 Nov 2014
Halloweenie Roast Night 2, October 29th 2014, featuring Alt-J. Thanks for coming out!
Thu, 23 Oct 2014
Beautiful Bodies Invincible Royals
Tue, 21 Oct 2014
Halloweenie Roast Night1 1
Fri, 21 Nov 2014
Buzz Briefs, Dogs, Scan Cards, Third Thursday, and more!
justin tuck: 7:39
Afentra had fun at Third Thursday
cold war: 0:14
Sometimes you need a scan card to get into work.
smashing pumpkins: 0:08
Your 8:00 news... kinda.
police officer: 6:54
Latest blog entries for krbz2 station
Starting at #12 Night Terrors of 1927 - When You Were Mine Walk the Moon - Shut Up And Dance Cold War Kids - All This Could Be Yours Melanie Martinez - Dollhouse Fitz and the Tantrums - Fools Gold Lorde -Yellow Flicker Beat Foo Fighters - Something From Nothing Bastille - Flaws Banks - Beggin for ...
Buzz Briefs Scan Card Third Thursday
Thu, 20 Nov 2014
Starting at #12 The Smashing Pumpkins - Being Beige Night Terrors of 1927 - When You Were Mine Misterwives - Reflections Foo Fighters - Something From Nothing Melanie Martinez - Dollhouse In the Valley Below - Peaches Hozier - Take Me to Church Alt-J - Every Other Freckle Chet Faker - Gold Night ...
In the headlines today the guys discuss The Chiefs, a gangster principal, and pride packages Kitman from SKC drops by the studio Eden Lew calls in to talk about her invention 'The Nutshell' Soren from cracked.com calls in the chat about why the NFL is evil Let's go CHIEFS! Thanks for listening ...

subset: commoncrawl


text:
As Cannes comes, so do political issues expressed in various forms. Before it begins, the press is already buzzing about films to be shown by the imprisoned filmmakers in Iran, Jafar Panahi and Mohammad Rasoulof. We wrote about this last year and will continue to do so. We'll bring up other films as they begin to make waves (if they do this year).Iranian filmmaker Jafar Panahi at his home after he was freed from jail on bail after more than two months in custody, in Tehran, Iran, Tuesday, May 25, 2010. (AP Photo)Films may be political in content covering a political subject or they could have a political affect which polarizes people aroun...Read More »
The Queer Palm is awarded by a jury of journalists and organizers of LGBT film festivals which rewards a movie for its artistic qualities and its treatments about gay, lesbian, bi or trans and gender questions. Gregg Araki's Kaboom was the first movie awarded in 2010. The Queer Palm is a collaborat...Read More »
ATO (Art Takes Over) was launched last year at Cannes by musician Dave Matthews along with Johnathan Dorfman, Sarah Lash and Temple Fennell. Their plan to release 4 to 6 films annually seems to be coming to fruition. In Sundance 2011 they acquired film The Convincer, Casino Jack was acquired in Toronto 2010, and " title="Mao's Last Dancer">Mao's Last Dancer in Cannes 2010 which was released in August. Terri was picked up from its producers on its completion September 2010 and is being sold internationally by Coach 14. 20th Century Fox has home video rights. ATO releases its films theatrically through its partnership with Samuel Goldwyn ...Read More »
Entertainment One keeps growing internationally with a rare intelligence at steady unrushed pace. As they enter Cannes to sell under the guidance of Charlotte Mickie, an international sales agent with impeccable credentials who still finds time to respond to special industry requests, they have, tucked under their belt, international distribution companies acquired with an eye toward profitability. Not relying on output deals, they now own not only the means of production but also the means of distribution itself. As a distributor in Canada, where eOne originated, it started from zero in 2007 and by 2009 had 4% of the market share (second ...Read More »
A Franco-Brazilian co-production workshop will be held in Cannes May 13. It is being offered by the French film board CNC and Brazil's Ancine cinema agency to introduce French and Brazilian producers to one another and to network with potential partners -- broadcasters, distributors and ...Read More »
Urban Distribution, a French based international sales agent founded in 2004 by Frédéric Corvez has changed its name from UMedia.Read More »
Odin's Eye is an Australia based international sales agent. Odin's Eye was launched in 2006 and represents films from all over the world. A prime example is this year's Go For It!" title="Go For It!">Go For It!, the first film of Latina filmmakers Carmen Marron ♀ which won the Audience Prize at Los...Read More »
Documentary Storytelling is Flourishing Like Never BeforeRead More »
This is one way to 'do' Cannes. Looks like a nice party space and well, if you like the water, this is the way to go. People we know who've done it think it is just the best.Read More »
Bill Cunningham New York broke all house records at Film Forum and we're still playing in 3 theatres in Manhattan. We've already opened in 30 cities and it's doing well everywhere. Our rollout will be in over 200 cities. BCNY appeals to people who love fashion because it's so much fun to watch...Read More »


text:
In my last post, I demonstrated three examples of websockets under ColdFusion 10. One thing I didn't really touch on was the JavaScript API you can use to work with websockets. These functions are available to any file making use of the cfwebsocket tag. They allow you to:
- 6
- 2882
Page Views
- 670
Posted by mitchp |
Feb 27 2012 / 10:30


text:
Home » Opinion » Editorials
New priorities: What will Concord's be?
How quickly will New Hampshire Democrats try to repay the public employee unions that helped them win the House and the Corner Office? They campaigned this fall on bringing new priorities to Concord. Now we'll get to see whose priorities those are.
In the last legislative session, Republicans passed two important reforms that public employee unions hated. One was an overhaul of the state's public pension system to put it on a more stable financial footing; the other was a tax credit for educational scholarships for lower-income children.
Pension reforms included reducing state contributions, increasing employee contributions, raising employee retirement ages, limiting the types of work that would be used to calculate pensions (overtime would be excluded, for example), and preventing employees from "spiking" their pensions by padding their pay in their final three years.
The education scholarship law allows businesses to take a tax deduction for contributing to funds that make scholarships available to lower-income children. Teachers unions perceive it as a threat to their public school monopoly.
Public employee unions are eager to repeal these reforms and get back the fat old days.
How quickly Democrats take up these issues could give us an indication of whom they really believe they were elected to represent.
- Franklin man and woman arraigned for attempted prostitution of minor - 0
- Rymes is ready to roll following Fred Fuller purchase - 0
- Police investigating death of 3-year-old girl in Nashua - 5
- Police say hit-and-run driver was observed tossing beer cans from car - 0
- City championship on the line Thanksgiving Day - 1
- Your Turn, NH -- Mike Moffett: Republicans can win in 2016 with Kasich at the top of the ticket - 6
- On being American: A point for reflection - 2
- Charles Arlinghaus: 'Tis the season for budget games and gimmicks - 0
- Rymes takes over for Fuller, deliveries start Wednesday - 1
Rymes Oil employees pull Bradford man from flaming wreck but driver succumbs to injuriesREADER COMMENTS: 0
Manchester school exec contract squeaks by
Pipes and plans: A chance to show up Mass.
Pipes and plans: A chance to show up Mass.
Manchester school exec contract squeaks by
Your Turn, NH -- Mike Moffett: Republicans can win in 2016 with Kasich at the top of the ticket
On being American: A point for reflection


text:
Added Jun 16, 2011
Ventura County, ...
This is a dress I made from vintage 70s fabric and a vintage pattern, Butterick 6170 which I bought at an estate sale. I shortened the pattern 9 inches and made a braided belt.
Justine at Sew Country Chick
Jun 17, 2011, 09.24 PMby Ooty Yaacobi
Vety cute! Looks great on you.
Jun 17, 2011, 02.38 PMby secondbedroom
So cute!
Jun 17, 2011, 08.48 AMby agucha
Soooo pretty :) I love it! And I want to thank you for inspiration. I'm making dress right now and I was wondering what belt should I make… :)
Jun 17, 2011, 08.15 AMby hdk100
It's almost more like an 80's dress, and I love the 80's! It's a really fun dress, looks great with the accessories!
Jun 16, 2011, 06.20 PMby madebymeg
Jun 16, 2011, 05.59 PMby Lady--t
Great for the summer
Jun 16, 2011, 05.55 PMby miss-t-2
This dress is really noce.. Good choice of fabric
Add to Favorites 16 people added it
Find out more about our three fabulous courses starting this week!
Backstage Report
Choose from one of 5 popular dresses.
Member Project of the Week
Kick off party dress season with this cute greenery covered dress.
Editors' Pick
See Meg's picks and get them at a special price with an extra discount for Black Friday
Sewing & Techniques
See how Meg made this edgy dress using the Janome Artistic Edge Digital Cutter!
You must allow our "request for permission" request to login to Burdastyle with Facebook.


text:
On PolicyShop, Matt Bruenig continues his series of analysis on the Federal Reserves recent Survey of Consumer Finances, this time...
An Equal Say And An Equal Chance For All
Data ByteFeedRSS Daily Data Byte
PolicyShop: The Demos BlogFeedRSS Blog
For the past four months, the people of Ferguson have been standing up for the very oldest American ideal: democracy, a say in the decisions that affect their lives. Just now, I was shocked to ...November 24, 2014 | Heather McGhee
Megan McArdle has a piece arguing that GruberGate shows an insider problem: So when I see journalists saying that Gruber's revelations don't matter because he's just kind of awkwardly saying...November 24, 2014 | Matt Bruenig
Manufacturing jobs were supposed to be the good jobs. They are, as the subtitle of a devastating new report from the National Employment Law Project points out, the jobs that "built America's middle...November 21, 2014 | Amy Traub
As Ferguson, Mo. and the nation await a grand jury decision on whether or not to indict police officer Darren Wilson for killing Mike Brown, authorities are "preparing for the worst." According to a...November 21, 2014 | Donovan X Ramsey
Key Findings The notion that all citizens have a voice in our country's governance is at the center of the American ideal of democracy. Yet the role of corporate and private money in our political...Report
Democracy has at its heart a basic promise: citizens have an equal voice in deciding who represents them. This promise went unfulfilled again in 2014. Large donors accounted for the vast majority of...Policy Brief
Karen Shanton
Meet an Expert
News & Commentary
Recently re-elected Governor Andrew Cuomo likes to complain that he's "a progressive Democrat who's broke." Here's a simple way to raise...November 20, 2014 | Vox
Nearly half of the nation's employers investigate job applicants' credit histories as a condition of employment. As a result, New Yorkers...November 14, 2014 | AM New York
Sign up for our emails to stay updated on what we're doing and how you can help.
On MSNBC, Demos President Heather McGhee explains the ways in which Texas' voter ID law is clearly...
October 24, 2014
[ 220 Fifth Ave,. 2nd Flr. New York, NY 10001 | 212.633.1405 | email@example.com ]


text:
Home Sports Tickets Other Sports American Football NFL NFL Playoffs AFC Playoffs
AFC Playoffs Tickets
Get your AFC Playoffs tickets on viagogo, your online ticket marketplace. Choose an event to see the selection of tickets available or check the calendar view for all the available dates with tickets. You can also sell your spare AFC Playoffs tickets on viagogo, and listing tickets is free. All AFC Playoffs tickets are covered by the viagogo Ticket guarantee!
Do you know of a AFC Playoffs event we can't find? Click Here to tell us about it!
Find more like this


text:
One because neither was a Han government. Jurchans/Manchurians and Mongolians were not part of China proper (and the latter isn't at all), and in fact can be seen as occupying forces. In the Yuan case especially, the Mongolians were seen as an oppressive force who allowed for little actual Chinese involvement in government. Therefore this cannot be a precedent at all.
This can be debated, especially for the Qing government, as the difference between Manchurians and the Chinese are very minute by now. The most famous of Qing emperors, Kangxi, greatly respected Han culture and customs, and was very willing to let the Manchurians and Hans become a melting pot. In fact, my mother can be considered one of these examples, she is about half Manchurian, making me 1/4 so. The Manchurians are very rarely considered separate from Hans as far as I know by now. Hell, most people probably don't even know where Manchuria (used to be) is. Furthermore, the Qing empire was very open to letting the Han people become officials (Of course they were rarely the highest ranking ones, but they were definiately satisfied) as opposed to Yuan, which completely suppressed Han peoples which caused its eventual downfall.
Secondly, there has never been credible evidence to capitulation on the Tibetan government's part to the Qing authority, and certainly never to a Han Chinese authority. Prior to the Yuan dynasty, the most influence China ever had in Tibetan was the marriage of Wen Cheng to Songtsan Gampo. Being that she was the niece of Emperor Tai-tsung (Li Shih Min) of the Tang Dynasty, this was an important marriage, but has nothing to do with Chinese authority over Tibet.
I believe that was after Tibet was recognized as a sort of client state to China, and the purpose of marrying his niece was to keep the peace, as marriage justifies the Tibetans and the Hans as a family of people (So it was thought, at least).
The PRC has every interest in controlling Tibetan politics, and by extension, religious and secular affairs. Why is really beyond me, as there is little to be gained through control of Tibet, save perhaps to keep a secure border with India.
That's probably the main reason, since China isn't on the best terms with India. It's the same reason they need North Korea as an ally; North Korea is a "barrier" for China to the North (And of course they realize this so they always know China is going to back them up, as Kim Jong-Il has indeed used to his advantage). If China allowed Tibet's freedom now, they would almost undoubtedly be allied with India, which can spell all kinds of danger for China.
The point is that China is still way behind in dealing with its modern realities, not the least of which is multiculturalism. It's not just the PRC either, most dynasties after the Tang have had a record of Han conformist policy.
China has had 100 years since the collapse of the Qing to get back on its feet after getting their asses kicked in The eight-country alliance and Opium Wars. The Sino-Japanese War and others didn't help very much, either. It's hard for anyone to expect China to come out and be a super-advanced country as their business allows them to, yet people constantly hold China by the high standards in Western Countries, when they're still licking some of the wounds they have.
Hence the issues that came to Taiwan, where now most of the island speaks Mandarin despite originally being ethnically and linguistically distinct. I do prefer the Taiwanese government (I know YOU don't, but that's a different argument), yet there are clear effects of this historical policy in action through the ROC as well as the PRC.
Whole different can there, but I sincerely doubt you can convince any non-"Taiwanese" that the Taiwanese government isn't holding up an extremely unfair elitist standard. My mother's family came from outside Taiwan, and even though she was born and raised in Taiwan, the current Taiwanese government feels the need to treat these people like outsiders, simply because their ancestors were not in Taiwan.
They are constantly ruining the lives of those that "don't belong there". A family friend of mine worked for the Taiwanese government, and they forced him to work in Spain, which was basically "You better quit or we'll keep screwing you over" in disguise. Because of this, his wife left him, and he had to return home to take care of his children, losing his job in the process.
The Taiwanese government is screwing up the economy, it's not uncommon to hear entire families commit suicide because of money issues, and kidnappings (especially those of rich travelers) is extremely common. Oftentimes, in fact, your own family members kidnap you and kill you before asking for ransom (Because obviously the kid knows who kidnapped him).
If you've kept up to date anything Chen Sui-Bian has done, you'd know he cheats, lies, and steals. For example, he won his second term by doing things like shooting himself in the leg, and then claiming Chinese spies are attempting to assassinate him, putting all policemen and such on alert, making them unable to vote (Because he knows darn well that the police force in Taiwan is mostly anti-independence). He has stolen money from the government, and accused his opposition of corruption even though the so-called corruption is a loophole through Taiwanese traditions (I can explain more on that if you don't know what I'm talking about).
America may be in support of the Taiwanese government, but they sure as hell look down upon corrupt scum like them.


text:
Hi Ruby. Please email me your address, I can send the Livingston,Montgomery,and Sansing info to you. I have a lot of Livingston stuff, a little on the Montgomery's and a few Sansing relatives, but I think I can put you in touch with someone who can help with the Sansing's. My email is email@example.com
|Home | Help | About Us | Site Index | Jobs | PRIVACY | Affiliate
|© 2007 The Generations Network


text:
Holiday travel has a way of bringing its own mini-vortex of chaos, so we like to make packing part easy.
To Grandmother's House We Go
To Grandmother's House We Go
Hey, the lady plays to win.
Holiday Sparkle
Here, our favorite subtly shimmering dressing options.
God bless DVF
DVF Gave Lady Liberty a Major Style Upgrade
Turn Your Treadmill Into a Runway
Turn Your Treadmill Into a Runway
Find detailed nutrition information, plus unique analysis tools that tell you more about how food affects your health and make it easier to choose healthy foods. Start searching now!
What We're Reading
FlashWhat's New Now
- A Thanksgiving Cranberry Compote Cocktail for Thirsty Thursday 19 hours ago
- 4 Tips for Surviving a Holiday Weekend With the Family 20 hours ago
- 10 Ways to Make Bobby Pins Look Incredibly Chic 22 hours ago
Get More from Self! sign up for our Newsletter


text:
Luma is a family-focused yoga center in downtown Santa Cruz offering products and services that nurture and support parents and children.
We are a yoga studio, family retreat, wellness center, and gathering place.
Most of all, we are yours.
Life moves fast, often compounding the stressful and complex work of raising a family. Let us be your refuge.
Visit Luma to stretch, play, and renew.
Saturday November 15, 2:00 - 4:00 pm
FREE + Open to the Public
Click here to Register.Follow @lumayoga


text:
| ----Commissions----
-CLOSED- Commission Journal:iconrequestsclosed::icontradesask::iconcollabsask::iconpointcommishesclosed::icongiftsfriendsonly::iconkiribansclosed:
|-Below here are my Stamps collections, so don't mind them~-
Motivational Stamps. And reminders.
♪~Bands/Artists that I listen to~♪
Games I <3 A LOT
Artists that I support!
Sonic FCs that I support~
And now for my Sonic FC
Le comics of epicness~
In the memory of great Artisans of our time
Not much for an ID cause I don't care~
I must say, it's been quite a while since I'd last touched this box. I thought that it'd be nice to keep it like this and all... but I guess some stuff just have to go, just like the Rivers of Time.
Anyways, my name is... I don't feel confident mentioning it aloud on the Internet, honestly. But if you really insist on calling me by at least a name, 'Axe' will do just fine. It's simple, really.
Now I'm sure that by now, many of you have seen the amount of Sonic Fan Characters in my gallery, no? Well, that's because I feel quite comfortable drawing in this style. I do draw anthro-furries and a bit of free-hand architecture (tl;dr backgrounds) from time to time, but I can't promise that I will upload them. And yes, I am a story-writer as well, and a game enthusiast, as you all can see on the left hand side of this page.
Feel free to chat with me about anything! I'm quite open to a lot of topics, honestly~
As for RolePlays (RP for short), I will be doing them via Notes from this day (26th April, 2014) onwards!
And if anyone wants my Skype contact, just Note me and we'll sort it out there.
Oh, and here's a link to my Chatroom to those who wanna bugger me.
Best of buddies that I've befriended over the years~
Favourite genre of music: Techno, the New Age, Rock-n-Roll, Classical, Symphonics, anything that has a tune that I particularly enjoy, really.
Favourite style of art: I'm quite open with art styles, really. I'm not just a Sonic fan-artist, you know?
Personal Quote: "You'll never know~ if you don't try~" – by Area 11, Moonquest: An Epic Journery