How can I do web scrapping of all the articles containing certain keywords mentioned in a dictionary, using Python? I would like to pull out the title and year of the all articles published in BBC since 2010 based on some keyword search for instance [MBA, LBS, Harvard...... and so on]. How can I do this using python? BeautifulSoup takes a link and extract some text from the link rather than searching the relevant links by itself based on some keywords.
if you want to just pull news articles from BBC news, then you could use their API.

they provide client libraries in different languages (including python)

quoting documentation:

You can search for articles with any combination of the following criteria:


Keyword or phrase. Eg: find all articles containing the word 'Microsoft'.
Date published. Eg: find all articles published yesterday.
Source name. Eg: find all articles by 'TechCrunch'.
Source domain name. Eg: find all articles published on nytimes.com.
Language. Eg: find all articles written in English.


I believe the first one is what you are looking for.