Reading set of large log files line by line and count how many hostnames appear on each line
So I think you can improve the efficiency of your code like this.

First, as I said in one comment, you can replace:

s=len(line.split('|'))
a=line.split('|')[s-3]


by

a=line.split('|')[-3]


as no need to know the total length of a list to get the third element from the end.

Second, assigning a then b with a value take some times, you can do it in one line:

a=line.split('|')[-3]
b=a.split('/')[0]


becomes

b=line.split('|')[-3].split('/')[0]


Third, I'm not sure len can equal to None, maybe you wanted to check for 0, but if our code run like this, I would say that:

if len(b.split('.'))==None:
    ''
else:
    b=b.split('.')[0]


is not useful so you can calculate directly the final b with:

b=line.split('|')[-3].split('/')[0].split('.')[0]


Forth, because actually you don't need to assign b anymore, you can append the value into rows directly, such as:

rows=[]
for line in f:
    rows.append(line.split('|')[-3].split('/')[0].split('.')[0])


or as a list comprehension:

rows = [line.split('|')[-3].split('/')[0].split('.')[0] for line in f]


Fifth, again, you create df_temp to use it once and then delete it, you can append directly into df_final such as:

df_temp= pd.DataFrame(columns=['hostname'],data=rows)
df_final=df_final.append(df_temp,ignore_index=True) 
del df_temp


is better this way:

df_final=df_final.append(pd.DataFrame(columns=['hostname'],data=rows),
                         ignore_index=True) 


Ultimately, rows is not necessary anymore, so all the code from the line with ... until the line del ... can be written:

with gzip.open(tt,'rt') as f:
    df_final=df_final.append(pd.DataFrame(columns=['hostname'],
                                          data=[line.split('|')[-3].split('/')[0].split('.')[0] for line in f]),
                             ignore_index=True) 


So far, I think we saved some time, but I know that appening dataframe in a loop is not the best practice, especially because you need to assign again df_final each time. It's better to add all the dataframes that you want to append together in a list, and then use pd.concat outside of the loop. Your code becomes:

list_final = []
for file in listi:
    tt=os.path.join(directory,file)
    with gzip.open(tt,'rt') as f:           
        list_final.append(pd.DataFrame(columns=['hostname'],
                                       data=[line.split('|')[-3].split('/')[0].split('.')[0] 
                                             for line in f]))
df_final = (pd.concat(list_final,ignore_index=True)
              .groupby(["hostname"]).size().reset_index(name="Topic_Count"))


Timing

I create one file with around 3 millions of rows, running you method was 8.9 seconds while mine was 5.8 (a gain more than 30%). I run the code on a listi containg 10 of this file, and your method gave more than 91 seconds (a bit more than stricly 10 times the method with one file) while mine was about 57 seconds (a bit less than 10 times the method for just one file). 

I don't know about all the multiprocessing or serializing calculations in Python, but it may be a good option too.