Get a stock quote from a web page using F#
I have been doing my own research and here is my answer to point 3.


  An alternative way to download the price that doesn't use a web browser control


namespace ExcelVIGSCA

module getPrice = 

    open ExcelDna.Integration
    open FSharp.Data

    let saveVIGSCAjson () = 
        // Get a copy of the data shape, and save it in a file so that the compiler can type check it.
        // I used http://www.telerik.com/fiddler to find the correct data.
        let url = @"https://www.vanguard.co.uk/uk/portal/GetMFJsonForPortId.js?portId=9158" 
        let response = Http.RequestString(url)
        let json = (response.Split('=')).[1]
        System.IO.File.WriteAllText(__SOURCE_DIRECTORY__ + "\VIGSCA.json",json) |&gt; ignore

    // Clearly, VIGSCA.json has to exist at compile time, so saveVIGSCAjson() will have to be have been run during development.
    // This makes the program very fagile.
    type VIGSCAData = JsonProvider&lt;"VIGSCA.json"&gt;

    // information about attributes &amp; data types: http://exceldna.codeplex.com/wikipage?title=Reference&amp;referringTitle=Documentation 
    [&lt;ExcelFunction(Description = "Price for VIGSCA")&gt;]
    // Excel-DNA needs concrete types for input and output to work: it can't be generic.
    let VIGSCAprice () = 
        saveVIGSCAjson() // save the current data.
        let VIGSCA = VIGSCAData.GetSample()
        // The compiler will use the file it has at **compile time** to build up the type data.  This WILL lead to a runtime 
        // exception if the type has changed.
        VIGSCA.FundData.Fund.[0].NavPrice.Price.[0].Price


After some research, including this illuminating article, I realised just how complicated web scraping is.  Extracting some data from a static HTML page would have been easy.  But we are dealing with a dynamic web page built on ssl/https connection.

I know that this code is incredibly brittle.  I certainly wouldn't want to give it to anyone else to use.  Consider it a learning exercise, or a hint at what can be done.  To get the web page at all, you have to have previously been to Vanguard's web site and selected the "Individual Investor" option, which presumably saves a cookie to that effect.

My research suggests that there are 2 broad categories of getting data from a dynamic web page.


Use a web browser programmatically (as in the question) and then make sure that the whole page, or at least the bit you are interested in, has downloaded before you extract the data.  The use of a regular expressions seems OK for this.  This is "simple" in the sense that you know what the final page looks like so you know what data to extract.
Use a protocol analyser (like fiddler) and inspect the web page source to work out exactly what request has the data you want.  Then use a tool, like the wonderful FSharp.Data to extract it.  This is definitely more work but you have a better understanding of how the web site is structured so that, hopefully, it is easier to maintain.  It is definitely more focused.


Whatever approach is chosen, knowledge of how these dynamic pages are built (including some familiarity with javascript) is necessary.  Thank goodness for sites like w3schools