How do I monitor the total physical and virtual memory usage of a process and all of its descendants?
If using third-party applications is a possibility, I would suggest that you install the htop process monitor, perhaps from Macports. This tool extends top with a nice layout, and most relevant for your use case, it has an arbitrarily deep thread tree view that can be turned on with the F5 key.