I am actually using PHP but such crawling can be done by any programming languages. It will be a bit difficult to cater a lot of situations. Please help me look through the problem, and please give me some suggestion on whether I am going to the right direction.

What I know is the current url address from which I can get a list of links from `<a href="` or from `<frame src="`.

What I am doing is: from current url address, I can firstly get root url, for example, from <http://www.abc.com/def>, I can get <http://www.abc.com> first. This is to cater the situation `<a href="/fff.html"`, so I have to know the root url first.

Secondly, I need to get url directory from current url, this is a little difficult and I still have no idea how to get it done perfectly. For example, from <http://www.abc.com/def/xyz.htm>, it's url directory is <http://www.abc.com/def>. This is to cater the situation `<a href="../../xyz.html">`.

The problem I am facing is, how to get the current url directory? For example, if the current url is <http://www.abc.com/def>, how can I actually know that def is a directory or a file? If def is a file, then the url directory would be <http://www.abc.com>. But if def is a directory, then the url directory would just be <http://www.abc.com/def>. 

You can say that if there is "/" at last, then it would be directory. But from my point of view, when I am crawling a webpage, I can't really ensure that the webpage builder will add "/" at the end of a directory url. A directory url is perfect valid, for example, if def is a directory, then <http://www.abc.com/def> would probably stands for <http://www.abc.com/def/index.html>.

Since it's hard to know whether <http://www.abc.com/def> is a directory or a script file, then it is hard to make full url from relative href such as `<a href="xyz.html">`.

Am I over complicating the problem? Is there any solution to this?

There are other situations for example href="# means anchor then I'll just append it to the end of current url. Is that correct and valid for any current url situation? Meaning that, is that valid for the situation where current url is <http://www.abc.com/def> (def is a directory), will <http://www.abc.com/def#xyz> be converted to <http://www.abc.com/def/index.html#xyz> ?

And for href="javascript: or href="vbscript: etc, I'll just ignore it.

And for href="xyz.???", and if ??? is an image file, exe file, or anything that is not valid html, I'll just ignore them?

Thanks.

The question might be a little messy, I hope I explained it clearly.