Python script to download specific files from FTP and update the download directory
Question: I need some help in order to create a script to download multiple .csv files
from FTP every 24 hours, ignoring the old files and to continue downloading
the new ones to keep an update. I'm having trouble writing the pattern because
the name of the files vary from 01150728.csv, 01150904.csv to 02xxxxxx.csv,
03xxxxx.csv and currently it reached 30151007.csv. The script that I'm
currently using downloads all the files but I need a command line in order to
do what I described earlier.

    
    
    from ftplib import FTP
    import sys
    import ftplib
    import os
    import fnmatch
    os.chdir(r'______________') # Directory where the files need to be downloaded
    ftp=ftplib.FTP('xxxxxxxx', 'xxxxx', 'xxxxxx') # ftp host info
    ftp.cwd('______')
    filematch='*csv'
    for filename in ftp.nlst(filematch):
        fhandle=open(filename, 'wb')
        print 'Getting ' + filename
        ftp.retrbinary('RETR '+ filename, fhandle.write)
        fhandle.close()
    
    
    ftp.quit()
    


Answer: You should keep a list or set of the files already fetched. The following
assumes you run the code once and don't exit.

    
    
    from ftplib import FTP
    import sys
    import ftplib
    import os
    import fnmatch
    os.chdir(r'______________') # Directory where the files need to be downloaded
    ftp=ftplib.FTP('xxxxxxxx', 'xxxxx', 'xxxxxx') # ftp host info
    ftp.cwd('______')
    filematch='*csv'
    import time
    
    downloaded = []
    
    while True:  # runs forever
        skipped = 0
    
        for filename in ftp.nlst(filematch):
            if filename not in downloaded:
                fhandle=open(filename, 'wb')
                print 'Getting ' + filename
                ftp.retrbinary('RETR '+ filename, fhandle.write)
                fhandle.close()
                downloaded.append(filename)
            else:
                skipped += 1
    
        print 'Downloaded %s, skipped %d files' % (downloaded[-1], skipped)
        time.sleep(24*60*60)  # sleep 24 hours after finishing last download
    
    ftp.quit()
    

If you run the script each day, omit the while loop and use pickle or simply
write the list/set in a file, and load it at the start of the script.

