Effective fetching data algorithm I'm fetching some data by using bs4, and want to know why access speed gets slower as my program progresses.

I'm using thread, and I know threading isn't so fast because of GIL.

My problem is that my program gets slower gradually.
When the number of word is over 300,it become much slower than that of word is about 100 words.

GIL causes this?

(Multiprocessing can solve GIL problem,however,I want to find other causes,if any.)

What is the better web-scraping code to download at once?

Thanks.

import os,gc,queue
import requests, bs4
from threading import Thread, BoundedSemaphore
class add_line:
    def __init__(self,semaphore,q):
        self.q = q # queue
        self.semaphore = semaphore

    def put_queue(self,li): #insert 
        self.q.put(li)

    def fetch_word(self,w_list):
        fetched_text = ""# targeted text
        path = "https://ejje.weblio.jp/content/"+ w_list[1]
        res = requests.get(path)
        res.raise_for_status()
        no_starch_soup = bs4.BeautifulSoup(res.text)
        fetched_text = no_starch_soup.select('.conjugateRowR &gt; table &gt; tr &gt; td')
        if len(fetched_text)== 0: #if get nothing
            fetched_text = no_starch_soup.select('.conjugateRowR &gt; a')
            if len(fetched_text) ==0:# if get nothing too
                self.put_queue(w_list)# return original conjurate
            else :
                li_ = list(map(lambda x:x.contents[0], fetched_text))
                li_.insert(0,w_list[0])
                self.put_queue(li_)
        else:
            li_=list(map(lambda x:x.contents[0], fetched_text))
            li_.insert(0,w_list[0])
            self.put_queue(li_)
        gc.collect()
        self.semaphore.release() 
if __name__ == "__main__": 
    # creating thread
    maxconnections = 5
    path = "./target_index1900.txt"
    """
    the part of this text data is below(the number of words is over 1000)

    510 abandon
    510 abandonment
    1807 abide
    1318 abolish
    1318 abolition
    1167 abortion
    1167 abortive
    1096 abound
    1683 abrupt
    1683 abruptly
    1199 absolute
    1199 absolutely
    507 absorb
    507 absorption
    899 abstract
    899 abstraction

    """
    #set semaphore 
    pool_sema = BoundedSemaphore(value=maxconnections)
    q = queue.Queue()# put word  in queue 
    al = add_line(pool_sema,q)
    path_w = "./threading_test_result.txt"
    with open(path) as f:
        while True:
            lines = f.readline()
            if not lines:
                break
            pool_sema.acquire()
            Thread(target=al.fetch_word, args=(lines.split(" "),)).start()
    with open(path_w, mode='a') as f:
        while not q.empty():
            for x in q.get():
                f.write(x)
                f.write(" ")
            f.write("\n")
        f.close()
        gc.collect()
    print("Done!")
Whitespace formatting

Apply a linter that will give you PEP8 suggestions. Among other things, it will suggest the following:

import os,gc,queue


There should be spaces after those commas. It'll also suggest that there be one line per import.

class add_line:


should be

class AddLine:


and have a couple of newlines before it.

else :


shouldn't have a space before the colon.

f-strings

"https://ejje.weblio.jp/content/"+ w_list[1]


can be

f'https://ejje.weblio.jp/content/{w_list[1]}'


Falsiness

if len(fetched_text)== 0:


can be

if not fetched_text:


Don't repeat yourself

            li_ = list(map(lambda x:x.contents[0], fetched_text))
            li_.insert(0,w_list[0])
            self.put_queue(li_)


appears twice. Put it in a function or restructure your if blocks.

File iteration

with open(path) as f:
    while True:
        lines = f.readline()
        if not lines:
            break


First of all, your variable name is confusing. This isn't lines, it's line.

Also: just change your iteration to

with open(path) as f:
    for line in f:


As for this:

with open(path_w, mode='a') as f:
    # ...
    f.close()


Get rid of the close; the file is already closed at the end of the with.

main

You do check __name__, but you don't give all of that code function scope. You should move it into a main function. Otherwise, every one of those variables will be visible from other functions.