Eliminating files that are completely contained within other files
It looks like you want to filter out the dupes, so let's apply that to self.files first.

noDupeFiles = self.files # no dupes at the start, so we initialize it to self.files
for a in noDupeFiles:
    for b in noDupeFiles:
        if a != b and a.isbetter(b):
            b.isDupe = a.name
            # got to remove b once we make it a dupe file
            noDupeFiles.remove(b)


This reduces the number of levels from 6 to 4 (since we removed the not x.isDupe checks). Next, we combine for a in noDupeFiles and for b in noDupeFiles with the help of itertools.permutations.

noDupeFiles = self.files
for a, b in itertools.permutations(noDupeFiles, 2):
    if a != b and a.isbetter(b):
        b.isDupe = a.name
        noDupeFiles.remove(b)


This reduces from 4 to 3 levels while keeping the original logic.



EDIT:

@marcp pointed out that removing elements from the list apparently threw an exception, so here's the new version:

for a, b in itertools.permutations(self.files, 2):
    if not (a.isDupe or b.isDupe) and a.isbetter(b):
        b.isDupe = a.name