Reading set of large log files line by line and count how many hostnames appear on each line I have around 300 log files in a directory and each log file contains around 3300000 lines. I need to read through each file line by line and count how many hostnames that appear on each line. I wrote basic code for that task, but it takes more than 1 hour to run and takes lots of memory as well. How can I improve this code to make it run faster?

import pandas as pd
import gzip

directory=os.fsdecode("/home/scratch/mdsadmin/publisher_report/2018-07-25")#folder with 300 log files
listi=os.listdir(directory)#converting the logfiles into a list

for file in listi:#taking eaching log file in the list 
    tt=os.path.join(directory,file)# joining log file name along with the directory path
    with gzip.open(tt,'rt') as f: #unzipping the log file           
        rows=[]#clearing the list after every loop
        for line in f: #reading each line in the file 
            s=len(line.split('|'))
            a=line.split('|')[s-3]
            b=a.split('/')[0] #slicing just the hostname out of each line in the log file                
            if len(b.split('.'))==None:
                ''
            else:
                b=b.split('.')[0]
            rows.append(b) # appending it to a list

    df_temp= pd.DataFrame(columns=['hostname'],data=rows) #append list to the dataframe after every file is read
    df_final=df_final.append(df_temp,ignore_index=True) #appending above dataframe to a new one to avoid overwriting
    del df_temp #deleting temp dataframe to clear memory
df_final=df_final.groupby(["hostname"]).size().reset_index(name="Topic_Count") #doing the count 


Sample log lines

tx:2018-05-05T20:44:37:626 BST|rx:2018-05-05T20:44:37:626 BST|dt:0|**wokpa22**.sx.sx.com/16604/#001b0001|244/5664|2344|455
tx:2018-05-05T20:44:37:626 BST|rx:2018-05-05T20:44:37:626 BST|dt:0|**wokdd333**.sc.sc.com/16604/#001b0001|7632663/2344|342344|23244


Desired output
So I think you can improve the efficiency of your code like this.

First, as I said in one comment, you can replace:

s=len(line.split('|'))
a=line.split('|')[s-3]


by

a=line.split('|')[-3]


as no need to know the total length of a list to get the third element from the end.

Second, assigning a then b with a value take some times, you can do it in one line:

a=line.split('|')[-3]
b=a.split('/')[0]


becomes

b=line.split('|')[-3].split('/')[0]


Third, I'm not sure len can equal to None, maybe you wanted to check for 0, but if our code run like this, I would say that:

if len(b.split('.'))==None:
    ''
else:
    b=b.split('.')[0]


is not useful so you can calculate directly the final b with:

b=line.split('|')[-3].split('/')[0].split('.')[0]


Forth, because actually you don't need to assign b anymore, you can append the value into rows directly, such as:

rows=[]
for line in f:
    rows.append(line.split('|')[-3].split('/')[0].split('.')[0])


or as a list comprehension:

rows = [line.split('|')[-3].split('/')[0].split('.')[0] for line in f]


Fifth, again, you create df_temp to use it once and then delete it, you can append directly into df_final such as:

df_temp= pd.DataFrame(columns=['hostname'],data=rows)
df_final=df_final.append(df_temp,ignore_index=True) 
del df_temp


is better this way:

df_final=df_final.append(pd.DataFrame(columns=['hostname'],data=rows),
                         ignore_index=True) 


Ultimately, rows is not necessary anymore, so all the code from the line with ... until the line del ... can be written:

with gzip.open(tt,'rt') as f:
    df_final=df_final.append(pd.DataFrame(columns=['hostname'],
                                          data=[line.split('|')[-3].split('/')[0].split('.')[0] for line in f]),
                             ignore_index=True) 


So far, I think we saved some time, but I know that appening dataframe in a loop is not the best practice, especially because you need to assign again df_final each time. It's better to add all the dataframes that you want to append together in a list, and then use pd.concat outside of the loop. Your code becomes:

list_final = []
for file in listi:
    tt=os.path.join(directory,file)
    with gzip.open(tt,'rt') as f:           
        list_final.append(pd.DataFrame(columns=['hostname'],
                                       data=[line.split('|')[-3].split('/')[0].split('.')[0] 
                                             for line in f]))
df_final = (pd.concat(list_final,ignore_index=True)
              .groupby(["hostname"]).size().reset_index(name="Topic_Count"))


Timing

I create one file with around 3 millions of rows, running you method was 8.9 seconds while mine was 5.8 (a gain more than 30%). I run the code on a listi containg 10 of this file, and your method gave more than 91 seconds (a bit more than stricly 10 times the method with one file) while mine was about 57 seconds (a bit less than 10 times the method for just one file). 

I don't know about all the multiprocessing or serializing calculations in Python, but it may be a good option too.