Optimal Weighting for Exam Composition

arXiv:1801.06043v1 [cs.CY] 24 Dec 2017

Sam Ganzfried and Farzana Yusuf
{sganzfri,fyusu003}@cis.fiu.edu
Florida International University
School of Computing and Information Sciences
Abstract
A problem faced by many instructors is that of designing exams that accurately assess the abilities
of the students. Typically these exams are prepared several days in advance, and generic question scores
are used based on rough approximation of the question difficulty and length. For example, for a recent
class taught by the author, there were 30 multiple choice questions worth 3 points, 15 true/false with
explanation questions worth 4 points, and 5 analytical exercises worth 10 points. We describe a novel
framework where algorithms from machine learning are used to modify the exam question weights in
order to optimize the exam scores, using the overall class grade as a proxy for a student’s true ability. We
show that significant error reduction can be obtained by our approach over standard weighting schemes,
and we make several new observations regarding the properties of the “good” and “bad” exam questions
that can have impact on the design of improved future evaluation methods.

1

Introduction and Background

Examinations have traditionally been dominant in student performance evaluation, often accompanied with
other forms of assessment such as assignments and projects. Defining standards for performance evaluation
has been studied from different perspectives [12]. Scouller documented the effectiveness of two different
methods—assignment essay vs. multiple choice test—to assess the student ability [18]; in contrast, Kirkpatrick described the negative influences of exam-oriented assessment [7]. Most relevant question evaluation
processes generally emphasize multiple choice testing for measuring students’ knowledge. Prominent scoring methods including number right scoring (NR) and negative marking (NM), along with other alternatives,
have been studied in an educational system outlining strengths and weaknesses of each method [9, 17]. But
approaches composed of diverse modules from available options, i.e., multiple choice, true/false, or explanatory analytical answers lack attention and needs to be evaluated for the effectiveness in assessment of
students’ abilities.
Effective learning models take into account students’ skills and balance the evaluation process accordingly. Question composition and establishment of difficulty levels by dynamic adjustment for scoring has
been demonstrated in different learning systems to strengthen the adaptiveness. Prior works have proposed
different approaches for student modeling and student motivation, considering the effectiveness of task difficulty and measuring the engagement level to better design adaptive educational systems [13]. The inverted
U-hypothesis depicts that increases in difficulty should generally pave the way for increase in enjoyment
up to some peak point, and afterwards further increases in difficulty lead to decreases in enjoyment. To
study the relation between difficulty and enjoyment, Abuhamdeh conducted several experiments to examine
and support the findings [1]. Learning performance curves also have been exploited in studies for adaptive
model design [11]. Generative models that explicitly capture the pairwise knowledge component (skills, procedures, concepts, or facts) relationships to produce a better fit structure reflecting subdivisions in item-type
domains with the help of learning curves has been studied [14]. Another model proposed a modified educa-

1

tional data mining system so that it attains the ability to infer individual student’s knowledge component in
an adaptive manner [15].
Designing an evaluation process which best reflects the proper assessment of each student’s ability
or effort is a crucial part of every course design. Item Response Theory, based on course structure and
appropriate topics to select the k-best questions with adequate difficulty for a particular learner to attain
adaptiveness, was brought into focus by Barla [3]. Stackelberg game theoretic model was also applied to
select effective and randomized test questions [10] for large scale, public exams i.e (driver’s license test,
Toefl iBT). This model chooses from a predefined set of questions according to the ability level of test taker
to compute the optimal test strategies when confidentiality is a concern. Also analysis have been conducted
to measure the effects of grouping student’s ability level and achievement using empirical observations [4].
Intelligent tutoring system like Cognitive tutors [5] and REDEEM authoring environment [2] model assess
students’ knowledge at different steps and allow teachers to design curricula according to individual skill
levels. It has been demonstrated that students’ ability or skill inclusion as a parameter resulted in improved
accuracy of further prediction to fit observations [8].

2

Model

We assume that there are n students and for each student i and for each exam question j there are m realnumbered scores sij for student i’s performance on question j. For each student we assume we have a real
number ai that denotes (an approximation of) their “true ability.” Ideally, the goal of the exam is to provide
as accurate an assessment of the students’ true abilities as possible. We seek to find the optimal way in which
the exam question could have been weighted in order to give scores for the exam that are as close as possible
to the true abilities ai . That is, we seek to obtain weights wj in order to minimize
n
X
i=1





ai − 

m
X

2
(wj · sij ) ,

j=0

which denotes the mean-squared error between the weighted exam score and the true ability. Note that we
can allow a “dummy” question 0 with scores of si,0 = 1 for all i in order to allow for a constant term with
weight w0 , which is useful for regression algorithms.

3

Experiments

We experimented on a dataset of graduate level class taught during Spring’17 consisting of nine students.
The course curriculum was designed using four major components: Homeworks, Midterm exam, Project,
and Final exam with equal percentage (25% for each) contributing towards the final overall scoring of the
students. Final grades were assigned using this overall score as a measure of performance. Though midterm
and final exam were considered as two different components, both exams cumulatively contributed to half
the score. Average overall score of the students was 67.92 with a standard deviation 10.18. Though the
dataset is relatively small, it contains a large degree of variance between students’ abilities, and is therefore
still representative of interesting phenomena. Distributions of grades and scores are presented in Figure 1.
The final exam was designed with 30 multiple choice question, 15 True/False questions and 5 analytical
question which are worth 3, 4 and 10 points each respectively. Each analytical question had several sub
parts which resulted in total 53 questions. The average score was 64.27 with standard deviation 27.19. The
midterm average was 49.5, which is much lower than the overall score average, and the standard deviation
was 19.43. The midterm exam consisted of 30 multiple choice, 15 T/F, and 5 analytical questions with sub

2

(a) Final grades count

(b) Overall score distribution

Figure 1: Overview of student’s performance
parts resulting in a total of 56 questions. For space we omit figures for the midterm, though qualitatively the
results were fairly similar to those for the final exam.
Both Final and Midterm exams had average scores that differed from the final overall scores of the
students. As a result, to calculate the abilities of the student in the exam we normalized the overall abilities
to conform to the final average. Both actual and normalized overall score were used in regression analysis to
compare different possibilities. Closed form least squares was implemented to predict the weights of each
questions as benchmark. Other approaches involve models i.e., Linear Regression with intercept, Huber
regression, and non-negative least squares using variants of objective functions and constraints in regression
analysis. All of these models exploit optimization as a tool to minimize the square loss and approximate the
prediction.
Closed form of ordinary least squares, denoted as normal equation, fits a line passing through the origin,
whereas linear model with y-axis intercept do not force the line to pass through the origin, which increases
the model capabilities. Huber regression checks outliers’ impact on the weights whereas non-negative least
squares enforces non-negative constraints for coefficients. Both of the exams were evaluated individually as
we are interested in each question even though the pattern of the exam was quite similar. The overall score
was normalized using the exam average to represent the exam score and then the coefficient of each question
was measured using different approaches to see the extent to which it contributed in the final prediction of
students’ scores.

3.1

Closed form Ordinary least squares

Ordinary least squares (OLS) or linear least squares attempts to estimate the unknown parameters depending
on independent explanatory variables. The main objective function is to minimize the sum of the squares of
the differences between observed value in the given sample and those predicted by a linear function of a set
−1
of features. A closed form solution in linear regression is β = (AT A) AT y where A is the independent
explanatory feature values and y is the observed response or target value. There might be cases where AT A is
singular making it non-invertible, so we used the pseudoinverse to solve the equation in our implementation.
In general the pseudoinverse is used to solve a system of linear equations where it facilitates the process to
compute a best fit solution that lacks a unique solution or to find the minimum norm solution when multiple
3

Figure 2: Question weights predicting overall score with

Figure 3: Question weights predicting normalized overall

Closed-Form OLS, for Final Exam

score with Closed-Form OLS, for Final Exam

solutions exist. Figures 2 and 3 show the weights of each final exam question using the ordinary least square
(closed-form) approach to fit the actual and normalized overall score. Since the overall score average was
close to the final exam average, the weights don’t show much deviation with two different scales.

3.2

Linear Regression with intercept

Regression through the origin enforces the y-intercept term to be zero in linear models and is used when
the line is expected to pass through origin. Linear regression attempts to describe the relationship between
a scalar dependent variable y and one or more explanatory variables, i.e., independent variables denoted X.
One of the possibilities for linear regression is to fit a line through the origin and another is to approximate the
intercept term so it passes through the center of the datapoints. We used the linear regression approach from
scikit-learn python library which uses lapack library from www.netlib.org to solve the least-squares
problem
minimize ky − Axk2
x

Although the objective functions are same, this approach produced a different weighting scheme in comparison to the closed-form method. The linear regression approach from scikit-learn offers two options for
approximation to fit the model, one with an intercept term in the equation enhancing the model capability
when the line doesn’t pass through the origin and the other without such a term. Without the intercept, the
linear regression conforms to the parameters found from OLS. Even though an bias term x0 with column
vector of all one is introduced in OLS, it favors line passing closer to the origin since the input features are
normalized. Whereas with an additional fit intercept parameter, the regressor tries to best fit the y-intercept
resulting in a better fitted line. Better approximation with intercept can be explained by the target value
which is an aggregated score of different components, i.e., homeworks and projects, along with the exam.
Since our main goal corresponds to designing an exam which best reflects students’ overall ability, the exam
scores alone can’t represent the expected outcome and sometimes overall score can introduce slightly different observation as they are weighted sum of different components. As a result, linear regression with
fit intercept seems to be more accurate choice for this experiment and follows the final observation .
Also we have only 8 rows in datasets and more than 50 questions and the problem solved by lapack
library takes into consideration the dimension of the matrix of linear equations. In case of the number of
rows being much less than the number of features and rank of A equals to number of rows, there are an
infinite number of solutions x which exactly satisfy the equation y − Ax = 0. Lapack library attempts to
find the unique solution of x which minimizes |x|2 , and the problem is referred to as finding a minimum
4

Figure 4: Question weights predicting overall score with

Figure 5: Question weights predicting normalized overall

Linear regression, Final Exam

score with Linear regression, Final Exam

Figure 6: Question weights predicting overall score with

Figure 7: Question weights predicting normalized overall

Huber regression, Final Exam

score with Huber regression, Final Exam

norm solution to an underdetermined system of linear equations. Depending on the implementation of the
pseudoinverse calculation, the two approaches can result in different optimal weights.

3.3

Huber regression

Huber regression, which is more robust to outliers, is another linear regression model which optimizes
the squared loss for the samples where |(y − A0 x)/σ| <  and the absolute loss for the samples where
|(y − A0 x)/σ| > , where x and σ are parameters to be optimized, y being the target value and A0 x is the
predicted score. The regularization parameter σ ensures that rescaling of y up to certain factor does not affect
 to obtain the same robustness. This method also confirms that the loss function is not as much influenced
by the outliers as other samples, while not totally ignoring their effects in the model. In our experiment, we
used cross validation to find out the optimal value of σ = 0.1,  = 1.8. To control the number of outliers in
the sample,  is used where smaller value of  ensures more robustness to outliers.

3.4

Non negative least squares regression

Non-negative least squares (NNLS) is a constrained version of the least squares problem in mathematical
optimization where only non-negative coefficients are allowed. That is, given a matrix A and a column
vector of response variables y, the goal is to find arg minx kAx − yk2 subject to x ≥ 0. Here x ≥ 0
means that each component of the vector x should be non-negative. As we are interested in designing an
5

Figure 8: Question weights predicting overall score with

Figure 9: Question weights predicting normalized overall

Non-negative least squares, Final Exam

score with Non-negative least squares, Final Exam

Figure 10: Overall score prediction, Final Exam

Figure 11: Normalized score prediction, Final Exam

exam, approaches defining constraints with only positive weights question can be effective, since it seems
unnatural to assign negative weight to an exam question. Figure8 and Figure 9 shows the non-zero weights
for the final exam questions. NNLS method from scipy library was used to solve the constrained optimization
problem to calculate the weights for this purpose.

3.5

Comparing the approaches

Predicted score for final exam using all four approaches produces low error as shown in Figure 10 and
Figure 11 in comparison to uniform weighting and the designed weighting used in the actual exam. As a
measurement of performance evaluation of different approaches, mean absolute error is tabulated in Table 1
for both exams with two different scale of score, normalized and actual, respectively. Uniform weighting
method where all questions are worth equal amounts is used as an benchmark to compare with the proposed
method. Also the actual weighting in the exam, to predict the overall score, is taken into consideration to
check how much they conform with the final score. For model formulation, the leave one out cross validation
approach was used and finally the weights from all the models were averaged to compute the final questions
weights, which are used to predict the overall score.

6

Table 1: Comparison of Mean absolute error
Overall Score
Final ( Normalized)
Midterm (Normalized)
Final (Actual)
Midterm (Actual)

Uniform weights
7.2368
2.9898
8.0209
17.5650

Actual Weights
6.1644
3.2856
6.3726
17.51134

Linear Regression
0.5690
0.3802
0.6013
0.5218

Huber Regression
0.5280
0.3967
0.7234
0.5094

Ordinary LS
0.6804
0.3161
0.7190
0.4338

Non-Negative LS
0.8135
0.5529
0.8597
0.7587

Figure 12: Comparison of prediction errors for Midterm

Figure 13: Comparison of prediction errors for Final,

and Overall score with and without Midterm

Overall score with and without Final Score

4

Discussion

While the approaches we use are existing techniques for linear regression, we encountered several issues of
potentially more general theoretical interest in our setting.

4.1

Overall Score Computation

One of the major concerns was how to incorporate all the components’ information for overall score computation. Since we are using overall score to compute exam questions’ weights and overall score already
contains that particular exam’s weighted score, it should be taken into consideration whether to include it in
overall score computation or not. But excluding an component from computation will result in information
loss. As a result, we experimented on both approaches of overall score computation to observe the change in
weights and prediction errors. From Figure 12 and Figure 13, it is evident that overall score which includes
all components performs better in both exams except the non-negative least square approach for Midterm
one. Also we observed that changes in weights due to exclusion of an component are trivial and almost
proportional.

4.2

Unique vs. multiplicity of solutions for linear regression, depending on the rank of the
matrix

System of linear equations or a system of polynomial equations is referred as underdetermined if number
of equations available are less than unknown parameters. Unknowns parameters in a model represents an
available degree of freedom, whereas each of the equations puts a constraint restricting the degree of freedom
along a particular axis. When the number of equations is less than the number of unknown parameters, the
system becomes underdetermined due to available degrees of freedom along some axes. As a result an
underdetermined system can have infinitely many solutions or no solution at all. Since in our case study, the
system is underdetermined and also A is singular, AT A is also singular, and the equationAT Ax = AT y had
infinitely many solutions. The pseudoinverse is a way to choose a ”best solution” x+ = A+ y.
7

4.3

Intuition for negative weights

The weights denote the relative contributions in the solution. They shows relative impact in the predictions
compared to the other features in the samples. As a result, we can think of the negative weight questions
in our sample as less important than the ones with high and positive coefficients. While for many settings
it may make perfectly good sense to include negative weights (for example in financial forecasting), it may
not be sensible for exam questions, as it would mean that students are incentivized to get those questions
wrong.

4.4

Results for truncating weights at 0, and description of algorithm for doing this

In order to limit the coefficients of linear equation to be only positive, we used non-negative least squares
where the objective function includes an additional assumption that weights are non-negative and then solves
the system [6]. NNLS from the scipy optimization library solves the system of linear equation with nonnegativity constraints which served our purpose in the experiment.

4.5

Variants of linear regression in python

The closed-form normal equation uses the dot product and inverse of matrix to solve for unknown parameters
in a system of linear equations. A bias parameter with all ones is used to introduce a constant term in the
matrix. Since it doesn’t take into consideration offset of the line from mean while fitting the intercept,
approximation error increases. However linear regression in scikit-learn [16] does not compute the inverse
of A. Instead it uses Lapack driver routine xGELS to solve least squares on the assumption that rank(A) =
min(m, n). xGELS makes use of QR or LQ factorization of A which can result in different coefficients than
the prior discussed methods. Not only that, the fit intercept term in scikit-learn represents the Y-intercept of
the regression line which is the value predicted when all the independent variables are zero at a time. On top
of that, without the intercept term, the model itself become biased and all the other parameters get affect due
to the fact that the bias term in OLS is not scaled but only an approximation with all one column vector. Also
due to normalization on dataset, when any question is answered by all the students, the matrix bias term and
that particular column become identical and is assigned same weight with OLS. As a result, including the
intercept term results in better weights which ouputs the scaled value after the coefficient calculation and
produces different results than the one with no intercept.

4.6

Closer analysis of certain notable questions

We take a closer look at several notable questions that stood out from the extreme weights they were given
in the regression output. First, multiple choice question 25 in the final exam was given a negative weight of
-1.307. The full distribution table is given in Table 2. We can see that the two students with highest overall
score got the question wrong, while several of the weaker students got the question right, which provides an
explanation for the negative weight given.
Next, multiple choice question 5 from the final exam had a very large weight of 2.376. Its distribution
is given in Table 3. We can see that the strongest three overall students got this question right, while the
weakest 6 got it wrong, which justifies the high weight. Similarly, question 6 in which the two strongest
students were the only ones to answer correctly also received a very high, but slightly lower, weight of
1.624, as shown in Table 4.
Finally, we can see the table for another question with a negative weight, where generally the weaker students in the class performed better than the stronger students—question 1a from the midterm for normalized
overall score, with weight -1.048, given in Table 5.

8

Table 2: Multiple choice 25 score distribution in Final; given lowest in linear regression
Student
1
2
3
4
5
6
7
8
9

Obtained score
1
0
0
1
1
0
1
0
0

Final Score
50.32
59.89
61.63
66.50
67.54
67.92
69.57
83.16
84.73

Table 3: Multiple choice 5 score distribution in Final; given high weight in linear regression
Student
1
2
3
4
5
6
7
8
9

Obtained score
0
0
0
0
0
0
1
1
1

Final Score
50.32
59.89
61.63
66.50
67.54
67.92
69.57
83.16
84.73

Table 4: Multiple choice 6 score distribution in Midterm normalized; given high weight linear regression
Student
1
2
3
4
5
6
7
8
9

Obtained score
0
0
0
0
0
0
0
1
1

Final Score
36.67
43.65
44.92
48.46
49.23
49.50
50.70
60.61
61.75

Table 5: Analytical question 1(a) score distribution in Midterm normalized; given low weight in linear regression
Student
1
2
3
4
5
6
7
8
9

Obtained score
0.5
1
1
0.75
1
0.75
0
0
0

9

Normalized Final Score
36.67
43.65
44.92
48.46
49.23
49.50
50.70
60.61
61.75

4.7

Effect of normalization on the approaches

We explored how it would effect the results to divide all approaches by their mean before/after applying
them. Final overall score is the factored aggregation of different components constituent of homeworks,
two exams, and project. But normalizing the overall score with their respective exam mean ratio results in
relatively better outcome since we are trying to relate the exam weights with their normalized ability. In
the final exam, the class average did not deviate much from overall average, so the mean absolute error
difference for Actual and Normalized approaches are very low. In the contrary, in midterm average score of
the class was 49.5 which is much lower than the overall average of 67.92. So multiplying the overall score
by the midterm mean ratio resulted in more precise prediction for this case.

4.8

Additional observations

We examined the results of the approaches for a question that all students answered correctly (multiple
choice 1), with results given by Table 6. The weights were zero when none of the students answered a question correctly irrespective of the approaches. In the final when only the highest scorer answered correctly,
different approaches demonstrated variations in their weighting, i.e, the linear approach put much higher
weight in comparison to the other approaches (Table 7).
In final MC 1 and MC 4 both were answered correctly by all the students (and therefore can be viewed
as a “duplicate” question). All the approaches except NNLS weighted the questions similarly, as shown by
Table 8.
Table 6: Comparison of weights for different approaches. Everyone answered MC1 correctly.
Approach
Linear
Huber
OLS
NNLS

Overall
0
0.4098
3.6673
0

Normalized
0
0.2610
2.6725
35.4471

Table 7: Comparison of weights for different approaches. Only highest scorer answered correctly AE 1(c).
Approach
Linear
Huber
OLS
NNLS

Overall
1.3431
0.7163
-0.0501
0

Normalized
1.2710
0.7295
-0.0474
0

Table 8: Comparison of weights for MC1 and MC4, which were answered correctly by everyone in the final

Approach
Linear
Huber
OLS
NNLS

MC 1
Overall Normalized
1.97E-16
0
1.4977
1.2033
3.5467
3.3563
0
0

10

MC 4
Overall Normalized
2.47E-17
0
1.4977
1.2033
3.5467
3.3563
0
0

5

Conclusion

The approaches demonstrate that, at least according to our model, novel exam question weighting policies
could lead to significantly better assessments of students’ performance. We showed that our approaches lead
to significantly lower mean squared error when optimizing weights on midterm and final exam questions in
order to most closely approximate the overall final score, which we view as a proxy for the true student’s
ability. From analyzing the optimal weights we identified several questions that stood out, and have a better
understanding of what it means for a question to be “good” and “bad.” We also described several practical
factors that would need to be taken into consideration for application of our approaches to real examinations.

References
[1] Sami Abuhamdeh and Mihaly Csikszentmihalyi. The importance of challenge for the enjoyment of intrinsically motivated, goal-directed activities. Personality and Social Psychology Bulletin, 38(3):317–
330, 2012.
[2] Shaaron Ainsworth and Shirley Grimshaw. Evaluating the redeem authoring tool: can teachers create
effective learning environments? International Journal of Artificial Intelligence in Education, 14(3,
4):279–312, 2004.
[3] Michal Barla, Mária Bieliková, Anna Bou Ezzeddinne, Tomáš Kramár, Marián Šimko, and Oto Vozár.
On the impact of adaptive test question selection for learning efficiency. Computers & Education,
55(2):846–857, 2010.
[4] L Burks. Ability group level and achievement. School Community Journal, 4(1):11–24, 1994.
[5] Hao Cen, Kenneth R Koedinger, and Brian Junker. Is over practice necessary?-improving learning
efficiency with the cognitive tutor through educational data mining. Frontiers in Artificial Intelligence
and Applications, 158:511, 2007.
[6] Donghui Chen and Robert J Plemmons. Nonnegativity constraints in numerical analysis. The birth of
numerical analysis, 10:109–140, 2009.
[7] Robert Kirkpatrick and Yuebing Zang. The negative influences of exam-oriented education on chinese
high school students: Backwash from classroom to child. Language Testing in Asia, 1(3):36, 2011.
[8] Jung In Lee and Emma Brunskill. The impact on individualizing student models on necessary practice
opportunities. International Educational Data Mining Society, 2012.
[9] Ellen Lesage, Martin Valcke, and Elien Sabbe. Scoring methods for multiple choice assessment in
higher education–is it still a matter of number right scoring or negative marking? Studies in Educational Evaluation, 39(3):188–193, 2013.
[10] Yuqian Li and Vincent Conitzer. Game-theoretic question selection for tests. In IJCAI, pages 254–262,
2013.
[11] Brent Martin, Antonija Mitrovic, Kenneth R Koedinger, and Santosh Mathan. Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction,
21(3):249–283, 2011.
[12] John J Norcini. Setting standards on educational tests. Medical education, 37(5):464–469, 2003.
11

[13] Jan Papoušek and Radek Pelánek. Impact of adaptive educational system behaviour on student motivation. In International Conference on Artificial Intelligence in Education, pages 348–357. Springer,
2015.
[14] Philip I Pavlik Jr, Hao Cen, and Kenneth R Koedinger. Learning factors transfer analysis: Using
learning curve analysis to automatically generate domain models. In Educational Data Mining 2009,
2009.
[15] Philip I Pavlik Jr, Hao Cen, and Kenneth R Koedinger. Performance factors analysis–a new alternative
to knowledge tracing. Online Submission, 2009.
[16] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and
E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research,
12:2825–2830, 2011.
[17] Eric M Scharf and Lynne P Baldwin. Assessing multiple choice question (mcq) tests-a mathematical
perspective. Active Learning in Higher Education, 8(1):31–47, 2007.
[18] Karen Scouller. The influence of assessment method on students’ learning approaches: Multiple choice
question examination versus assignment essay. Higher Education, 35(4):453–472, 1998.

12

