I'm looking to take all the relevant text sections of text of certain web pages and parse it into a structured format, e.g. a CSV file for later use. 
However, the web pages I want to take info from don't strictly follow the same format, for example, the pages:

`http://www.cs.bham.ac.uk/research/groupings/machine-learning/
http://www.cs.bham.ac.uk/research/groupings/robotics/
http://www.cs.bham.ac.uk/research/groupings/reasoning/`

I have been using BeautifulSoup and this has been fine for the web pages that follow a well-defined format, but these particular websites don't follow a standard format. 
How can I write my code to extract the main text from these pages? 
Could I either extract all the text and strip away the irrelevant/commonly occurring text?  

Or can I somehow select these larger text bodies even though they don't occur uniformly? 
The websites are formatted differently but not in such a convoluted way that I think this is impossible? 

Originally I had code like this for dealing with the structured pages:

```
from urllib.request import urlopen
from bs4 import BeautifulSoup
import sqlite3

conn = sqlite3.connect('/Users/tom/PycharmProjects/tmc765/Parsing/MScProject.db')
c = conn.cursor()

### Specify URL
programme_list = ["http://www.cs.bham.ac.uk/internal/programmes/2017/0144",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/9502",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/452B",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/4436",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/5914",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/9503",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/9499",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/5571",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/5955",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/4443",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/9509",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/5576",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/9501",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/4754",
              "http://www.cs.bham.ac.uk/internal/programmes/2017/5196"]

for programme_page in programme_list:
# Query page, return html to a variable
page = urlopen(programme_page)

soupPage = BeautifulSoup(page, 'html.parser')

name_box = soupPage.find('h1')
Programme_Identifier = name_box.text.strip()

Programme_Award = soupPage.find("td", text="Final Award").find_next_sibling("td").text
Interim_Award = soupPage.find("td", text="Interim Award")
if Interim_Award is not None:
    Interim_Award = Interim_Award = soupPage.find("td", text="Interim Award").find_next_sibling("td").text
Programme_Title = soupPage.find("td", text="Programme Title").find_next_sibling("td").text
School_Department = soupPage.find("td", text="School/Department").find_next_sibling("td").text
Banner_Code = soupPage.find("td", text="Banner Code").find_next_sibling("td").text
Programme_Length = soupPage.find("td", text="Length of Programme").find_next_sibling("td").text
Total_Credits = soupPage.find("td", text="Total Credits").find_next_sibling("td").text
UCAS_Code = soupPage.find("td", text="UCAS Code").find_next_sibling("td").text
Awarding_Institution = soupPage.find("td", text="Awarding Institution").find_next_sibling("td").text
QAA_Benchmarking_Groups = soupPage.find("td", text="QAA Benchmarking Groups").find_next_sibling("td").text

#SQL code for inserting into database
with conn:
    c.execute("INSERT INTO Programme_Pages VALUES (?,?,?,?,?,?,?,?,?,?,?,?)",
              (Programme_Identifier, Programme_Award, Interim_Award, Programme_Title,
               School_Department, Banner_Code, Programme_Length, Total_Credits,
               UCAS_Code, Awarding_Institution, QAA_Benchmarking_Groups, programme_page))

print("Program Title:           ", Programme_Identifier)
print("Program Award:           ", Programme_Award)
print("Interim Award:           ", Interim_Award)
print("Program Title:           ", Programme_Title)
print("School/Department:       ", School_Department)
print("Banner Code:             ", Banner_Code)
print("Length of Program:       ", Programme_Length)
print("Total Credits:           ", Total_Credits)
print("UCAS Code:               ", UCAS_Code)
print("Awarding Institution:    ", Awarding_Institution)
print("QAA Benchmarking Groups: ", QAA_Benchmarking_Groups)
print("~~~~~~~~~~\n~~~~~~~~~~")

Educational_Aims = soupPage.find('div', {"class": "programme-text-block"})
Educational_Aims_Title = Educational_Aims.find('h2')
Educational_Aims_Title = Educational_Aims_Title.text.strip()

Educational_Aims_List = Educational_Aims.findAll("li")
print(Educational_Aims_Title)
for el in Educational_Aims_List:
    text = el.text.strip()
    with conn:
        c.execute("INSERT INTO Programme_Info VALUES (?,?,?,?)", (Programme_Identifier, text,
                  Educational_Aims_Title, programme_page))
    print(el.text.strip())

```

However, I've not found a way yet to write a script to pull out the relevant text from the unstructured pages I've linked above. 
I was considering trying to pull all the sections tagged  and then processing them as they come. 
I just thought someone might have any insight on an easier way.