C# URL Crawler not getting enough links?
<p>I have the following code, however, when I launch it I only ever seam to get a few URLS returned.</p>

<pre><code>while (stopFlag != true)
{
    WebRequest request = WebRequest.Create(urlList[i]);
    using (WebResponse response = request.GetResponse())
    {
        using (StreamReader reader = new StreamReader
           (response.GetResponseStream(), Encoding.UTF8))
        {
            string sitecontent = reader.ReadToEnd();
            //add links to the list
            // process the content
            //clear the text box ready for the HTML code
            //Regex urlRx = new Regex(@"((https?|ftp|file)\://|www.)[A-Za-z0-9\.\-]+(/[A-Za-z0-9\?\&amp;\=;\+!'\(\)\*\-\._~%]*)*", RegexOptions.IgnoreCase);
            Regex urlRx = new Regex(@"(?&lt;url&gt;(http:[/][/]|www.)([a-z]|[A-Z]|[0-9]|[/.]|[~])*)", RegexOptions.IgnoreCase);

            MatchCollection matches = urlRx.Matches(sitecontent);
            foreach (Match match in matches)
            {
                string cleanMatch = cleanUP(match.Value);
                urlList.Add(cleanMatch);

                updateResults(theResults, "\"" + cleanMatch + "\",\n");

            }
        }
    }
}
</code></pre>

<p>I think the error is within the regex.</p>

<p>What I am trying to achieve is pull a webpage, then grab all the links from that page - add these to a list, then for each list item fetch the next page and repeat the process.</p>
