Retrieving data from website - Parser vs AI I'm currently working on a personal software project (as a hobby) and would like to know which kind of approach I should follow according to you.

My application scans some websites (like Amazon or eBay) to check items of a specific category and show them all together ordered, for instance, by price.

Currently I use Python with LXML to extract strings and numbers from specific positions on the page (searching by HTML element tag or CSS class).

This approach works smoothly, but I'm a little bit concerned about scalability: what if I want to add more web sources? What if Amazon and eBay change the structure of the HTML page?

So far I managed these issues by manually changing the functions I use to retrieve data from each website, but was wondering if there is a better approach, for example using AI/Machine Learning.

Finally the question: is it better to rely on "old-style" and "hard-coded" algorithms for such a scenario or Machine Learning and AI could be more helpful according to you?

Thanks a lot!
It depends what to use on what you are doing. I divide this problem in two categories.

1.Site without changing structure

Suppose, you are building a app which takes data from your national weather department site and normally which has same structure throughout decade. Here using AI for finding data is stupid thing. You should use conventional parsing technique here. and also your are getting data from number of sources but has fixed structure then you can go with this approach.

2.Site with changing structure

As per your example, I believe approach 1 is not so useful. because there are lots of site to parse and these site changes there structure periodically. So I believe using AI for parsing data from these sites is beneficial. I believe you used Optical Character Recognition. you will need similar functionality to your system. Your AI system first load page and see how it looks in browser (not code). Then finds appropriate fields from page using Named Entity Recognition from text which found using OCR

I think applying Named Entity Recognition to directly parsed code is sufficient to find different fields like item name, price, specifications, type etc