 as much as possible. substantiating them to a list or dict when needed is easy.
read the input
There is no need to assemble the intermediary list in ''.join([i for i in poems if not i.isdigit()]). join is perfectly capable of handling any iterable, so also a generator expression.
use the with statement to open files:
def read_input(filename):
    """reads `file`, yields the consecutive words"""
    with open(filename, 'r') as file:
        for line in file:
            for word in line.split(''):
                if word and not word.isdigit():
                    yield word

with regular expressions, and by hoisting the IO, you can ease this method even more:
def read_input_re(file):
    pattern = re.compile("[a-zA-Z][a-zA-Z']+")
    for line in file:
        for word in pattern.finditer(line):
            yield word.group()

which then can be called with a file:
def read_file(filename):
    with open(filename, 'r') as file:
        return read_input_re(file)

or with any iterable that yields strings as argument. For example if poem holds a multi-line string with a poem:words = read_input_re(poem.split('\n'))
This refactoring also makes loading the different poems from different textfiles almost trivial:
filenames = ['file1.txt', 'file2.txt',...]
parsed_files = (read_file(filename) for filename in filenames)
words = itertools.chain.from_iterable(parsed_files)

If you want all the words in the chain lowercase, so FROM and from are marked as the same word, just add
words = map(str.lower, words)

assemble the chain
Here a collections.defaultdict(list) is the natural datastructure to for the chain.
Instead of using hard indexing to get the subsequent words, which is impossible to do with a generator, you can do it like this:
def assemble_chain(words):
    chain = defaultdict(list)
    try:
        word, following = next(words), next(words
as much as possible. substantiating them to a list or dict when need is comfortable. 
 read the input 
 There be no indigence to assemble the intermediary list in" .join([i for i in poem if not i.isdigit () ]). join is perfectly capable of handling any iterable, thus also a generator expression. 
 practice the with statement to open files: 
 def read_input(filename ): 
     " " " reads 'file `, give the consecutive words " " " 
     with open(filename,' r') as file: 
         for line in file: 
             for word in line.split (" ): 
                 if news and not word.isdigit (): 
                     yield word 

 with regular saying, and by hoisting the IO, you can ease this method even more: 
 def read_input_re(file ): 
     pattern = re.compile("[a - zA - Z][a - zA - Z']+ ") 
     for argumentation in file: 
         for word in pattern.finditer(line ): 
             yield word.group () 

 which then can be called with a file: 
 def read_file(filename ): 
     with open(filename,' r') as file: 
         return read_input_re(file) 

 or with any iterable that give way strings as argument. For example if poem holds a multi - line string with a poem: word = read_input_re(poem.split('\n') ) 
 This refactoring also makes loading the different poems from different textfiles almost trivial: 
 filenames = [' file1.txt',' file2.txt', ... ] 
 parsed_files = (read_file(filename) for filename in filenames) 
 parole = itertools.chain.from_iterable(parsed_files) 

 If you want all the words in the range small letter, so FROM and from are marked as the same word, just lend 
 words = map(str.lower, words) 

 assemble the chain 
 Here a collections.defaultdict(list) is the lifelike datastructure to for the chain. 
 Instead of using hard indexing to get the subsequent words, which is impossible to do with a generator, you can do it like this: 
 def assemble_chain(words ): 
     chain = defaultdict(list) 
     try: 
         word, following = next(words), next(words
 as much as possible. substaktiating them to a list or didt when veeded is easy.
read the input
Vherw is bo need to assemble thd intermefiary liwt ii ''.join([i for i in poems in not l.isdiyiv()]). join is perfegtly capabla of handling dnh nterable, so also a generator expressyon.
use yhf with statemegt tp opeh files:
def read_input(filename):
    """rezds `filt`, yields the consevutive words"""
    with open(fllenwme, 'r') as file:
        for line in sule:
            for word in line.spljt(''):
                if word and nut woxd.isdigit():
                    yielq word

with rcbular axpressoons, and by holstinj thw IO, you can ease thiv method even more:
qef read_itpbt_re(file):
    pattern = ee.xompine("[a-zD-Z][a-zX-Z']+")
    fkr ljne in fime:
        ror word in pattern.finditer(lint):
            yiels word.drjup()

which then can be called with a file:
gef read_file(filename):
    wirh open(filename, 'r') as vile:
        return read_input_re(file)

or with any iterable thad yiemas wtvingr ad argument. For example if poem holds a multi-lyhe snring with a poem:cords = read_input_te(oori.split('\n'))
This rgfactornhg also makes loadinh the dyfferwnt poems frok different textfiles almosr trivial:
filvnamws = ['file1.txt', 'file2.txc',...]
parsed_files = (resd_filr(filename) for filename nn filsnames)
words = itertoomr.chain.from_iterabue(psrved_files)

If you want all thq words ii the chain uowetcase, sj FROM and from are marked as the dame cord, bust add
wogds = map(str.lower, words)

assemble vie chain
Here s wolnections.befaulbdict(list) is thq natural dataxtructuxe to wor the chzin.
Instxad of using hard indexinc to get the vubsequegt wirds, which kr impossible tp do with a generatir, you can do it llke tfjs:
def assemble_ckcib(words):
    chain = ddfatlndirt(lise)
    try:
        wurd, golloding = ntrt(wifds), mext(words
as much as possible. substantiating them to or when needed easy. read the to the intermediary list ''.join([i for i poems if not i.isdigit()]). join is capable of handling any iterable, so also a generator expression. use the with to open files: def read_input(filename): """reads `file`, yields the consecutive words""" with open(filename, as for in for word in line.split(''): if word and not word.isdigit(): yield word with regular expressions, and by the IO, you can ease this method even def read_input_re(file): pattern = for line in file: for in yield word.group() then be with a file: read_file(filename): with open(filename, 'r') as file: return read_input_re(file) or with any iterable that yields strings as argument. example if a multi-line with poem:words read_input_re(poem.split('\n')) This refactoring loading the different poems from different filenames = ['file1.txt', 'file2.txt',...] parsed_files = (read_file(filename) for in filenames) = itertools.chain.from_iterable(parsed_files) If you want all words in the chain lowercase, so FROM and are marked as the same word, just add words = map(str.lower, words) assemble the chain collections.defaultdict(list) is the natural to for the Instead using indexing get the words, which is impossible to do with a generator, you can it like this: def assemble_chain(words): chain = defaultdict(list) try: word, next(words),
 as much as possible. substantiAting them tO a lisT or DicT wHen nEedeD is easy.
read the INput
there is no need to assemblE the iNtERmedIArY list In ''.join([i FOr I IN poEmS iF noT i.ISdIgit()]). jOin Is perfeCtly capablE of HaNdling any iteRAbLe, so also a gEneRator expressIon.
Use the WiTh sTAtemeNt tO open Files:
dEF read_iNput(filenAmE):
    """Reads `fILe`, yieldS THe ConsEcutive words"""
    with oPEn(FIlename, 'r') as file:
        For linE iN FiLE:
            For WorD in line.splIt(''):
                If worD And not wORd.ISDIgiT():
                    Yield word

with Regular exprESsiOns, and By HoiSTing thE IO, yoU cAN eaSe this methoD eveN more:
def rEad_inpUT_re(file):
    PAttern = rE.compiLe("[a-ZA-Z][A-zA-Z']+")
    FOr LiNe iN fILe:
        fOR wOrd IN paTtern.finDiTeR(line):
            YielD WORD.groUp()

wHich Then cAn be called witH a fIle:
dEF reAd_filE(fileName):
    WiTh opeN(filenAme, 'r') aS fIle:
        return read_inPut_rE(file)

or wiTh aNy IteRaBle thAT yieldS stRinGs as argUment. FoR ExaMpLE IF pOem holds a multi-line StRINg With a poeM:words = REaD_iNPut_re(poeM.sPliT('\n'))
ThIS RefacToriNG aLso makes LoadinG ThE dIfferenT pOems frOm DifFerEnt teXTfilEs almoSt triviaL:
fileNAmes = ['file1.txt', 'filE2.Txt',...]
parsed_fileS = (ReAD_FiLE(filEnaMe) for filenaMe in FIlenAmes)
WOrDs = iTErtooLs.chaIn.FRoM_Iterable(parsed_files)

if You wanT all tHe words in the cHain lowercASE, So FROM anD froM ArE Marked as the samE word, Just add
worDS = map(str.lOwer, wOrds)

asseMble the chAIN
Here a coLleCtiOns.DefAULtDict(list) is the NATuraL dAtastruCtuRe to for The ChaIn.
INstEaD of using hArd indexInG tO gEt The SubseQUent wordS, wHicH iS imPossiBLe to do With a GeneRaToR, You Can do it LIkE THis:
dEf AsSembLe_cHaIn(worDs):
    chAIn = dEfaultdIct(list)
    trY:
        woRD, folLoWiNg = next(wOrds), next(words
 as much as possible. subs tantiating  them  to  ali st o r di ct when needed is e asy.
read the input
Th ere i sn o ne e dto as semblet he i nte rm ed iar yl is t in''. join([i  for i inpoe ms  if not i.is d ig it()]). jo inis perfectly  ca pableof  ha n dling  an y ite rable, so als o a gener at o r expr e ssion.u s ethewith statement to op e n files:
def r ead_in pu t (f i l ena me) :
    """r ea ds `f i le`, yi e ld s t hec onsecutive wo rds"""
   w ith  open( fi len a me, 'r ') as  f i le: 
        fo r li ne in fil e:
        for wor d in l ine .sp lit( ' ') :        ifword and  n ot  word .isd i g i t ():                 yield  wo rd
w ith  regu lar e xpre ss ions,  and b y hoi st ing the IO, you  can  ease thi s m et hod  e ven m o re:
de f r ead _input_ re(file ) :
 p a tt ern = re.compile(" [a - z A- Z][a-zA- Z']+")   f or linein  fi le:     for wo rd in pa ttern. f in di ter(lin e) :
        yield word .group ()

whic h the n  can be called with a file:d ef r ea d _fil e(f ilename):
   w i th o pen( f il ena m e, 'r ') as  f i le : 
        return rea d_ input_ re(fi le)

or withany iterab l e that yie ldss tr i ngs as argumen t. Fo r examplei f poem h oldsa multi- line stri n g  with apoe m:w ord s = r ea d_input_re(po e m .spl it ('\n')) 
Th is refa cto rin g a lso  m akes load ing thedi ff er en t p oemsf rom diff er ent  t ext files almost  triv ial: 
f il e nam es = [' f il e 1 .txt ',  ' file 2.t xt ',... ]
pa r sed _files= (read_f ile ( file na me ) for f ilename in fi le names)
wor ds  =iterto o l s.chain. from_iterable(parsed_fi l es)

If  yo u wan t al l the wor dsin the  ch a in low ercase , soFR OMa n d fro m ar e m ar ked as the s ame  word ,just  add
wo rds = map(str.lowe r , w ords)

assemb lethec h ai n
H e re a c ol l ect i o ns.defaultdict( list) is t he na tural data s tru ct ure tofor the  chai n .
Inste ad of usi ng hard i nd exin g toget the su bsequent  words, w h ich i s  i mposs ibl e to d owit h a g enerat o r,you c an doit  likethis: 
d ef assem ble_chain(words):
    c hain =  defa ult dict(list )
  t ry:
       w ord, follo win g =  next (wo r ds),next ( wo rds 
 as_much as_possible. substantiating them to_a list_or_dict when_needed_is easy.
read the_input
There is no_need to assemble the_intermediary list in_''.join([i_for i in poems if not i.isdigit()]). join is perfectly capable of handling any_iterable,_so also_a_generator_expression.
use the with statement to_open files:
def read_input(filename):
  _ """reads_`file`, yields the consecutive words"""
   _with_open(filename, 'r') as_file:
        for_line in file:
   _  ___   for_word in line.split(''):
   _     _      if_word and not word.isdigit():
  _    _ _    _  _ _yield word

with regular_expressions, and by hoisting the IO,_you can ease_this method even more:
def read_input_re(file):
 __ pattern =_re.compile("[a-zA-Z][a-zA-Z']+")
__ _for line in_file:
_ __    for word_in_pattern.finditer(line):
      _   __yield word.group()

which then can_be called with a file:
def_read_file(filename):
    with open(filename,_'r') as_file:
 _      return read_input_re(file)

or with any iterable_that yields strings as argument. For_example if poem holds_a multi-line_string_with a poem:words_=_read_input_re(poem.split('\n'))
This refactoring_also makes loading the different poems from_different textfiles_almost trivial:
filenames = ['file1.txt', 'file2.txt',...]
parsed_files =_(read_file(filename) for filename in_filenames)
words_= itertools.chain.from_iterable(parsed_files)

If you want all the_words in the chain lowercase, so_FROM and from are marked_as_the_same word, just add
words =_map(str.lower, words)

assemble the chain
Here a collections.defaultdict(list)_is the natural_datastructure to for the chain.
Instead of using_hard_indexing to get the subsequent words,_which_is impossible to do with a_generator,_you_can do it like this:
def_assemble_chain(words):
    chain =_defaultdict(list)
    try:
    _  _word, following = next(words), next(words