Python - Check if text is sentences?
I would try a semi-supervised learning technique where it passes you scraps and asks you to label them. What you're looking for will likely be kind of domain specific depending on the type of site. In the end you'll probably have a bunch of heuristics like:


If length &lt; 50 and contains "LOGOUT", "REGISTER","SIGN IN","LOGIN"
If count of "|" > 1
If count of all upper case words > 1