dependent variableor response variable (or class since this is classification problemthe set of examples the ai trains on is known as the training setand each individual example is called training instance or sample during the trainingthe ai is attempting to minimize its cost function or error rateor framed more positivelyto maximize its value function--in this casethe ratio of correctly classified emails the ai actively optimizes for minimal error rate during training its error rate is calculated by comparing the ai' predicted label with the true label howeverwhat we care about most is how well the ai generalizes its training to never-before-seen emails this will be the true test for the aican it correctly classify emails that it has never seen before using what it has learned by training on the examples in the training setthis generalization error or out-of-sample error is the main thing we use to evaluate machine learning solutions this set of never-before-seen examples is known as the test set or holdout set (because the data is held out from the trainingif we choose to have multiple holdout sets (perhaps to gauge our generalization error as we trainwhich is advisable)we may have intermediate holdout sets that we use to evaluate our progress before the final test setthese intermediate holdout sets are called validation sets to put all of this togetherthe ai trains on the training data (experienceto improve its error rate (performancein flagging spam (task)and the ultimate success criterion is how well its experience generalizes to newnever-before-seen data (generalization errorrules-based vs machine learning using rules-based approachwe can design spam filter with explicit rules to catch spam such as flag emails with "uinstead of "you," instead of "for,"buy now,etc but this system would be difficult to maintain over time as bad guys change their spam behavior to evade the rules if we used rules-based systemwe would have to frequently adjust the rules manually just to stay up-to-date alsoit would be very expensive to set up--think of all the rules we would need to create to make this well-functioning system instead of rules-based approachwe can use machine learning to train on the email data and automatically engineer rules to correctly flag malicious email as spam this machine learning-based system could be automatically adjusted over time as well this system would be much cheaper to train and maintain in this simple email problemit may be possible for us to handcraft rulesbutfor many problemshandcrafting rules is not feasible at all for exampleconsider designing self-driving car--imagine drafting rules for how the car should behave in unsupervised learning in the machine learning ecosystem