Duplicate detection and reporting in a BST
General

Your code style here is good, the variable names are nice and descriptive, and the algorithm you implement is easy to identify and follow. This is good-looking code. There are a few of nit-picks:


the TreeNode class should be a 'static' class ... private static ... because it has no reason to keep a reference to the outer class.
Your TreeNode.item should be final even though it is an internal class. final variables make for more apparent logic.
this is an opinion thing, but I prefer there to be getter/setter methods even on internal classes. With modern IDE's like Eclipse, IntelliJ and NetBeans, there really is no 'lazy' excuse for it. I prefer methods like node.getRight() instead of node.right
your method print() should have a better name like printDuplicates()


Algorithm

The solution you have for this problem appears to work. The algorithm relies on passing a token around, and relies on first walking the right-edge of the tree to create a 'stop-gate'. These two items detract from the solution. There are 'simpler' ways to do this recursive problem without the overheads you have. Consider the functions:

public void printDuplicates() {
    printDuplicatesHelper(root, 0, 0);
}

private void printDuplicatesHelper(final TreeNode node, final int parentval, final int parentcount) {
    if ((node == null || node.item != parentval) &amp;&amp; parentcount &gt; 1) {
        System.out.printf("Item %d was duplicated %d times\n", parentval, parentcount);
    }
    if (node == null) {
        return;
    }
    // make sure the parentcount is 0 going left.
    printDuplicatesHelper(node.left, 0, 0);
    printDuplicatesHelper(node.right, node.item, node.item == parentval ? (parentcount + 1) : 1);
}


The above functions also do an in-order scan of the data. The difference is that the recursion is designed to go past the leaf node, and to treat the null node as if it was the end of a duplicate streak. By doing it this way, and by pushing the duplicate-detecting values down with the recursion, you can detect the end-of-duplicates in a much easier pattern.

I created a 'main method' that compares the above code with yours:

 public static void main(String[] args) {
     TreeNode left = new TreeNode(null, 1, new TreeNode(null, 1, null));
     TreeNode right = new TreeNode(
              new TreeNode(null, 5, new TreeNode(null, 5, null)), 6, new TreeNode(null, 6, new TreeNode(null, 6, null)));
    TreeNode top = new TreeNode(left, 3, right);

    DuplicateCountInBST tree = new DuplicateCountInBST();
    tree.root = top;
    tree.print();
    tree.printDuplicates();
}


And the results I get from running the two calls are:


1 : 2
5 : 2
6 : 3
Item 1 was duplicated 2 times
Item 5 was duplicated 2 times
Item 6 was duplicated 3 times