figure - sample html page as rendered in browser browsers use markup tags to understand special instructions like text formattingpositioninghyperlinksand so on but only renders the content for the end user to see for use cases where datainformation resides in html pageswe need special techniques to extract this content web scraping web scraping is technique to scrape or extract data from the webparticularly from web pages web scraping may involve manually copying the data or using automation to crawlparseand extract information from web pages in most contextsweb scraping refers to automatically crawling particular web site or portion of the web to extract and parse information that can be later on used for analytics or other use cases typical web scraping flow can be summarized as followscrawla bot or web crawler is designed to query web server using the required set of urls to fetch the web pages crawler may employ sophisticated techniques to fetch information from pages linked from the urls in question and even parse information to certain extent web sites maintain file called robots txt to employ what is called as the "robots exclusion protocolto restrict/provide access to their content more details are available at scrapeonce the raw web page has been fetchedthe next task is to extract information from it the task of scraping involves utilizing techniques like regular expressionsextraction based on xpathor specific tags and so on to narrow down to the required information on the page web scraping involves creativity from the point of view of narrowing down to the exact piece of information required with web sites changing constantly and web pages becoming dynamic (see aspjspetc )presence of access controls (username/passwordcaptchaand so oncomplicate the task even more python is very powerful programming languagewhich should be evident by nowand scraping the web is another task for which it provides multiple utilities let' begin with extracting blog post' text from the apress blog to better understand web scraping the first task is to identify the url we are interested in for our current examplewe concentrate on the first blog post of the day on apress web site' blog page at clicking on the top most blog post takes us to the main article in consideration the article is shown in the screen in figure -