Webcrawler for a car forum I've been experimenting more with webcrawling and hence have started to get a better understanding compared to my previous questions. Right now, my code scraps from a car forum on each page and iterates through every pages. What would you recommend to improve on?

import requests
from bs4 import BeautifulSoup, SoupStrainer
import pandas as pd

list_topic = []
list_time = []

SESSION = requests.Session()


def get_response(url):  # Gets the &lt;html&gt; structure from the website #
    response = SESSION.get(url)
    soup = BeautifulSoup(response.text, 'lxml',
                         parse_only=SoupStrainer('ul', {'class': 'posts posts-archive'}))
    return soup


def iteration(url, max_page=52):
    starting_page = 1
    while starting_page &lt;= max_page:
        ## formats the new URL etc (https://paultan.org/topics/test-drive-reviews/page/1) ##
        new_url = url + f"page/{starting_page}"
        data = get_response(new_url)
        get_reviews(data)
        ## iteration starts ##
        starting_page += 1


def get_reviews(response):
    for container in response('article'):
        title = container.h2.a.text
        time = container.time.text
        list_topic.append(title)
        list_time.append(time)
    else:
        None


def create_pdReview():
    return pd.DataFrame({'Title': list_topic, 'Time': list_time})


if __name__ == '__main__':
    URL = 'https://paultan.org/topics/test-drive-reviews/'
    print(iteration(URL))
    print(create_pdReview())


I've been wondering; would using yield improve the efficiency and simplicity of the code? How would it be done? Because I've been trying to learn from my previous inquiries that has been answered on earlier. Here is a similar question and I'm trying to put to practice what has been recommended so far.
When we are discussing performance of a particular piece of code, it's important to recognize bottlenecks and major contributors to the runtime of the program.

In your particular case, even though you've applied some optimizations like SoupStrainer speed-up for HTML parsing, the synchronous nature of the script is the biggest problem by far. The script is processing pages one by one, not getting to the next page until the processing for the current page is finished.

Switching to an asynchronous approach would be the natural next step in your optimizations. Look into using third-party frameworks like Scrapy or, if you are adventurous, things like asyncio or grequests. 



You could apply one more optimization to your current script which should help you optimize the "crawling/scraping" part - instead of using requests.get(), initialize session = requests.Session() and use session.get() to make requests (documentation). This would allow the underlying TCP connection to be re-used for subsequent requests resulting in a performance increase.