Counting without zeroes
Well, what you want is just a product of the alphabet, with increasing numbers of elements. You can use itertools.product for this:

from itertools import product, count
from string import ascii_lowercase

def generate_labels():
    """Yields labels of the following form:
       a, b, ..., z, aa, ab, ..., zz, aaa, aab, ..., zzz, ...
    """
    for n in count(start=1):
        yield from map("".join, product(*[ascii_lowercase]*n))


Here is what it outputs:

from itertools import islice

print(list(islice(generate_labels(), 1000)))
# ['a', 'b', ..., 'z', 'aa', 'ab', ..., 'az', 'ba', 'bb', ..., 'bz', ..., 'za', ..., 'zz', 'aaa', 'aab', ..., 'all']


This has the slight disadvantage that the list being passed to product gets larger every iteration. But already with \$n=5\$ you can generate \$\sum_{k=1}^n 26^k = 12,356,630\$ labels, and the list is only about sys.getsizeof([ascii_lowercase]*5) + sys.getsizeof(ascii_lowercase) * 5 = 479 bytes large, so in practice this should not be a problem.



I also made the name a bit longer (and clearer IMO) and added a docstring to briefly describe what the function is doing.