Square-tree using maps and recursion
Your direct definition of square-tree is correct.

Your definition using map calls square-tree; to make it properly recursive, call map-square-tree instead.  Further, you may recurse on the subtree itself.  This will make your code succinct.

(define (map-square-tree tree)
  (map (lambda (subtree)
         ((if (pair? subtree) map-square-tree square) subtree))
       tree))