Are there anatomical differences between male and female mammal brains before action of hormones?
Yes. Dewing et al. (20030 found that gene expression in mouse brains differs between male and female mice about 10.5 days after the mice had sex. This occurs before hormonal contributions to brain development. They identified about 35 genes that were appeared to show preferential expression in the female brain and another 16 that appeared to be show preferential expression in the male brain. Most of the genes identified were transcriptional regulators, signaling molecules, or involved with cellular differentiation. Of these 51, three were verified as showing female bias, and two of these were located on the X chromosome. Four genes were verified showing male bias, and two of these were located on the Y chromosome.
Similar results were found by Xu et al. (2002). They identified several of the same genes as Dewing et al., plus others that were linked to the sex chromosomes. They identified a total of six genes on the X chromosome that showed strong female expression bias during brain development. At least three of the genes were homologous between the X and Y chromosomes but showed different expression patterns between male and female brains at different stages of development.
The differences in genetic expression may cause some microscopic anatomical differences during brain development that is not explained by hormones (Bocklandt and Vilain 2007). They reviewed several studies that found differences in neuron growth, and density of vasopressin fibers in the lateral septum were more likely to be due to genetic expression than to hormonal expression. Bocklandt and Vilain did not identify the specific ages of the mice that showed these apparently gene-regulated differences but it is reasonable to speculate that these differences begin to appear in association with the differential gene expression.  Bocklandt and Vilain did not identify macroscopic differences between male and female brains prior to hormonal expression.

Citations
Bockland, S. and E. Vilain. 2007. Sex differences in brain and behavior: Hormones versus genes. pgs 245-266. In: Genetics of sexual differentiation and sexually dimorphic behaviors (D. Yamamoto, ed.). Academic Press.
Dewing, P. et al. 2003. Sexually dimorphic gene expression in mouse brain precedes gonadal differentiation. Molecular Brain Research 118: 82-90.
Xu, J. et al. 2002. Sex differences in sex chromosome gene expression in mouse brain. Human Molecular Genetics 11: 1409-1419.