How can precision be less than one in Leave-One-Subject-Out binary classification if each subject contains only one class Say I'm trying to classify a medical condition.
Theres only two classes: Sick and Healthy.
I build a model and I can't split the data because I don't want data from the same patient being in training and test set. So I elect to use Leave-One-Subject-Out, training the model on all subject except one and testing on the left out subject.
So for each test set I have one subject and they are either healthy or sick. So the confusion matrix only contains one class where precision is technically one every time and recall equals accuracy.
I've been reading some papers that claim to use leave-subject-out training and test splits for tasks where patients either have a medical condition or do not. I've seen papers that report accuracy, recall, and precision but I don't understand how you could have precision be less than one if each subject only contains one class. I doubt these papers are lying because I've seen this more than once.
I just want to know whats going on here for them to get precision values that are less than one. Are they doing some kind of averaging or am I missing something and thinking about this in the wrong way? None of the papers explain this either.
Your reasoning is correct that the gold standard class is the same for all the instances in a single leave-one-out test set (under the assumption that the test patient cannot become sick at some point in time, thus having both healthy and sick status).
What you're missing is the aggregation across multiple test sets: a full leave-one-out experiment repeats this process for every single patient, i.e. if there are N patients then there are N unique pairs of (training set, test set). Here is a pseudo code to show this clearly:
correct,incorrect=0    
for every patient p:
    training_set = all the patients except p
    test set = patient p
    model = train(training_set)
    // to simplify I count the patient as one instance, it's easy to count instances instead
    if model.predict(test_set) == gold_standard(p):
       correct += 1
    else:
       incorrect += 1 
accuracy = correct / (correct + incorrect)

The calculation across all the patients can lead to some predictions being correct and some others being incorrect, this is why the accuracy, precision or recall can be any value between 0 and 1.