Average spam confidence
Not duplicating any of @Peilonrayz's code review points ...



Stop reading entire files into memory when you can process the file line by line in one pass, and stop creating huge lists in memory which are then iterated over exactly once.  Both of these things creates a huge unnecessary memory pressure which can be avoided by looping and/or using generator expressions.

Using a simple loop over all lines in the file:

def average_spam_confidence(filename):
    with open(filename) as file:
        count = 0
        total = 0
        for line in file:
            if line.startswith("X-DSPAM-Confidence: "):
                try:
                    total += float(line[20:])
                    count += 1
                except:
                    pass   # Line ended with garbage
        return total / count


No regex.  No reading entire file in memory.  No creating a list of individual confidence values to sum up afterwards.

Plus, we've fixed a bug!  If "X-DSPAM-Confidence:" appears in the middle of a line, instead of at the start as required by the problem text, we don't try to process it.

But perhaps you wanted a more functional way of programming, where you:


find all the confidence lines,
extract the confidence values, and
compute the average


... all as separate steps which you can compose together, and reuse to solve other problems.  Fear not!  We can do that too!  Enter generator expressions:

First, let's open the file, and read all the matching lines into a list, using list comprehension:

with open(filename) as file:
    lines = [line for line in file if line.startswith("X-DSPAM-Confidence: ")]


That second statement loops through each line in the file, checks if the line begins with the desired text, and if so, includes it in the list that is being constructed.  We can later iterator over lines to further process each line individually.

That is almost what we want to do.  Well, that is exactly what we want to do, but we don't want to do it all at once.  If we change the [...] to (...), we move from list comprehension to a generator expression.

with open(filename) as file:
    lines = (line for line in file if line.startswith("X-DSPAM-Confidence: "))


Now, the second statement has done ... nothing.  We haven't read the first character of the file yet.  What we have returned is a generator expression, which when we ask for the first value will start reading lines until it finds one that matches, and then it will pause its execution and return that value.

Ok.  Let's extract just the confidence values:

    values = [ line[20:] for line in lines ]


Whoops!  That's list comprehension.  It will loop over all the lines that the lines generator can produce, skip over the prefix and return the rest.  Again, change those [...] to (...):

    values = ( line[20:] for line in lines )


Better!  Now those are still strings, so we'll need to convert them into float point values.  Too easy.  Just map them:

    confidences = map(float, values)


confidences is a generator.  If you said list(confidences), you'd create that in-memory list of float values for all the "X-DSPAM-Confidence:" values in the file.  And you could then sum(...) and len(...) the list of values to compute the average.  But we don't want to realize the list in memory, so ...

def average(data):
    total = 0
    count = 0
    for value in data:
        total += value
        count += 1
    return total / count

average_confidence = average(confidences)


... we ask for values from the confidences generator, add them up one at a time, counting as we go.  When the generator is exhausted, the for loop ends, and we return the average.

Putting it all together:

def average(data):
    total = 0
    count = 0
    for value in data:
        total += value
        count += 1
    return total / count

def average_spam_confidence(filename):
    with open(filename) as file:
        lines = (line for line in file if line.startswith("X-DSPAM-Confidence: "))
        values = ( line[20:] for line in lines )
        confidences = map(float, values)
        return average(confidences)


Or more simply:

import statistics

def average_spam_confidence(filename):
    with open(filename) as file:
        values = (line[20:] for line in file
                            if line.startswith("X-DSPAM-Confidence: "))
        return statistics.mean(map(float, values))


Note: the non-generator solution was more robust converting the confidence value strings into floats, via a try...except block.  The generator expression solution shown above omits that.  The robustness may be improved by using a more precise matching when searching for the "X-DSPAM" lines (regex).  Alternately, a generator function could be used, which discards the non-float values.

def map_to_float(data):
    for value in data:
        try:
            yield float(value)
        except:
            pass

confidences = map_to_float(values)




Note (from @Roland Illig's comment): Because generator expressions delay the execution of their operation, any resources they use must remain available until their processing has been finished.  They cannot be used to compute the average spam confidence if the file they are reading from is closed before the average has been computed.  In the above examples, the generator expressions are fully consumed within the body of the with open(...) as file: block, so the file was kept open.

This does not mean the generator expressions must all occur within the with statement.  They can be spread out across many functions, but their execution must be constrained to the interval when the file is open:

def find_occurrences_in_file(input_file):
    """Return a generator for occurrences of X-DSPAM lines"""
    return (line for line in input_file if line.startswith("X-DSPAM-Confidence: "))

def retrieve_values(input_list):
    """Return a generator for float values from a list"""
    return map(float, (line.split()[-1] for line in input_list))

def average(input_list):
    """Compute average of a list/sequence/iterable/generator of values..."""
    return statistics.mean(input_list)

def average_spam_confidence(file_name):
    with open(file_name) as file:
        # File is open for this entire with statement.
        occurrences = find_occurrences_in_file(file)
        values = retrieve_values(occurrences)
        print(average(values))
        # File is closed here - but "average(...)" has exhaustively read
        # from all of the generators already.