Abstract
MRI has become an indispensable tool for brain volumetric studies, with the hippocampus an important region of interest. Automation of the MRI segmentation process has helped advance the field by facilitating the volumetric analysis of larger cohorts and more studies. FreeSurfer has emerged as the de facto standard tool for these analyses, but studies validating its output are all based on older versions. To characterize FreeSurfer's validity, we compare several versions of FreeSurfer software with traditional hand-tracing. Using MRI images of 262 males and 402 females aged 38 to 84, we directly compare estimates of hippocampal volume from multiple versions of FreeSurfer, its hippocampal subfield routines, and our manual tracing protocol. We then use those estimates to assess asymmetry and atrophy, comparing performance of different estimators with each other and with brain atrophy measures. FreeSurfer consistently reports larger volumes than manual tracing. This difference is smaller in larger hippocampi or older people, with these biases weaker in version 6.0.0 than prior versions. All methods tested agree qualitatively on rightward asymmetry and increasing atrophy in older people. FreeSurfer saves time and money, and approximates the same atrophy measures as manual tracing, but it introduces biases that could require statistical adjustments in some studies.
| I N TR ODU C TI ON
The hippocampus is a part of the medial temporal lobe, playing an important role in spatial and episodic memory (Squire, Stark, & Clark, 2004) .
The ability to measure hippocampal volume in humans via magnetic resonance imaging (MRI) has identified reduced hippocampal volume as a biomarker for numerous conditions (Geuze, Vermetten, & Bremner, 2005) , including major depressive disorder (MDD; Campbell, Marriott, Nahmias, & MacQueen, 2004; Videbech & Ravnkilde, 2004) , epilepsy (Cook, Fish, Shorvon, Straughan, & Stevens, 1992; Geuze et al., 2005; Jack et al., 1990; Jack, 1994) , post-traumatic stress disorder (Logue et al., 2017) , and Alzheimer disease (AD; Gosche, Mortimer, Smith, Markesbery, & Snowdon, 2002; Jack et al., 1998; Kesslak, Nalcioglu, & Cotman, 1991) , as well as normal aging (Allen, Bruss, Brown, & Damasio, 2005; Du et al., 2006; Raz, Rodrigue, Head, Kennedy, & Acker, 2004b; Raz & Rodrigue, 2006; Walhovd et al., 2005; Walhovd et al., 2011) . Increasing MRI availability and concurrent reductions in costs have led to a proliferation of MRI data in ever larger studies and data sets.
Determining hippocampal volume from MRI has been accomplished through varied manual (Boccardi et al., 2011; Jack, 1994; Konrad et al., 2009) and automated (Alem an-G omez, Melie-Garcia, & Valdes-Hernandez, 2006; Fischl et al., 2002; Moghaddam & SoltanianZadeh, 2009; Platero & Tobar 2016; Smith et al., 2004) methods. Manual tracing has long been considered the gold standard, allowing for flexible quantification without many of the assumptions built into algorithms. Experienced human tracers can correctly mark ambiguous boundaries, with adjustments for variation in complex or atypical anatomy and image artifacts. But this flexibility introduces variability in any manual protocol. Tracing both hippocampi can also take an expert several hours, resulting in prohibitive expense for today's large and growing MRI data sets. See (Turner, 2014) for a review on the growing size of imaging studies. In pursuit of a reliable technique that will scale up with growing studies, several algorithms have been created to automate the segmentation process. FreeSurfer was selected for its availability, ease of use, nearly full automation, and higher accuracy than alternative algorithms (Morey et al., 2009; S anchez-Benavides et al., 2010; Schoemaker et al., 2016; Tae, Kim, Lee, Nam, & Kim, 2008; Wenger et al., 2014) .
Automated techniques are often published with a brief validation of their output, but insufficient independent validations comparing manual tracing to automated replacements exist, particularly with evolving software. Existing studies of FreeSurfer's reliability use relatively small numbers of participants and older software versions (Morey et al., 2009; S anchez-Benavides et al., 2010; Schoemaker et al., 2016; Tae et al., 2008; Wenger et al., 2014) . We are unaware of any studies that assess validity of current FreeSurfer 6.0.0 software, hippocampal subfield algorithms, or assessment of handedness. To fully characterize the differences between modern FreeSurfer segmentations, prior FreeSurfer versions, and our manual tracing protocol, we compared manually traced left and right hippocampi to FreeSurfer 5.3.0 and 6.0.0 segmentations of MRIs from 664 participants from the Genetics of Microangiopathic Brain Injury (GMBI) study.
To investigate methodological similarities and differences directly, we compared raw and total intracranial volume (TIV)-adjusted hippocampal volumes by both value and voxel-wise percent agreement. To answer the more salient question of whether these methods' quantitatively different results provide accurate and comparable estimates for biologically relevant measures, we also investigated right-left asymmetry and an assessment of atrophy over the adult lifespan using data from each method. Finally, to determine the ongoing existence of previously reported age and volume biases (Wenger et al., 2014) , we investigated consistency in differences between methods relative to differences in participants' ages and hippocampal volumes.
| M E TH ODS

| Participants
The initial sample comprised 708 non-Hispanic white participants, age 37-93, 41% male, 93% right-handed, from Rochester, MN who underwent MRI between August 2001 and February 2006 as part of the GMBI study. Participants were recruited into GMBI from the Rochester Epidemiology Project (Melton, 1996) , based on at least two members of each sibship having hypertension before age 60. The sample undergoing full analyses in this study was a subset of 664 images, age 38-84, 39% male, 94% right-handed. Exclusions are shown in Table 1 . This study was approved by the Institutional Review Board of The Mayo Clinic, Rochester and The University of Mississippi Medical Center. All participants gave written informed consent prior to participation.
| MRI acquisition
MRIs were captured with the same clinical GE Signa 1.5T MRI scanner, maintained over time with recommended software updates, as a component of the GMBI Study between 2001 and 2006. The complete protocol included several sequences as described previously (Knopman et al., 2008) . This study used 0.9375 3 0.9375 3 1.6 mm voxel T1-weighted coronal 3D spoiled gradient echo (SPGR) images (256 3 192 3 124 matrix, 6-10 ms echo time, 24 ms repetition time, 25-degree flip angle, 24 3 18 3 19.8 cm field of view). TIV were previously measured manually from 2D T1-weighted sagittal images (32 slices, 5 mm thick; no gap, 256 3 192 matrix, 14-20 ms echo time, 500 ms repetition time, 24 cm field of view; Knopman et al., 2008) . To rule out the potential for bias introduced by changing scanner software, we assessed hippocampal volume, TIV, contrast to noise ratio (CNR), and signal to noise ratio (SNR) over the image acquisition dates. All four relationships, after including age in the model, were insignificant, failing to support any effect due to scanner changes. Moreover, FreeSurfer's scan-rescan reliability has been shown to remain high even across significant hardware and software upgrades (Jovicich et al., 2009 ).
| Manual tracing
708 MRIs were initially available for analysis. We followed a previously published protocol for tracing hippocampal boundaries (Jack, Theodore, Cook, & McCarthy, 1995) . Briefly, we used MIPAV (Medical Image Processing, Analysis, and Visualization) software (RRID: SCR_007371; McAuliffe et al., 2001) to orient the image coronally, then rotated the image around the left-right axis until the hippocampus aligned horizontally. We saved the reoriented image and traced the outline of each cross-sectional slice of hippocampus from the caudal end, where the Ages are reported as arithmetic mean (SD). Exclusions sum to more than the total because some participants were excluded for multiple reasons.
SCHMIDT ET AL. fornix separates from the hippocampus in the lateral ventricle, to the rostral end, where the hippocampal head diminishes beneath the amygdala. We saved the coordinates of all boundary points to calculate the area of each slice, then multiplied areas by the slice gap to approximate hippocampal volume.
MRIs were obtained in Analyze format. For 26 (3.7%) of 708 images, neurologic versus radiologic orientation could not be determined reliably. They were excluded along with three images with extensive artifacts that could not be traced, one image with a large tumor, and 5 (0.7%) images with no TIV measurements available, leaving 673 manually-traced images (Table 1) .
We distributed the workload of manually tracing hippocampi between two trained experts, MFS and KBF, both authors of this manuscript. A third tracer, BD, was also trained, but contributed tracings only for inter-rater reliability, including Figure 1 , in this study.
These tracings were periodically recalibrated by reviewing and discussing segmentations. Repeated intra-rater and inter-rater reliability studies found consistent agreement between scorers and no sign of drift over time. TIV were measured previously (Smith et al., 2009; Turner et al., 2005) .
| Automated segmentation
All 708 MRIs available for manual tracing, including those known to be mgh.harvard.edu) and has been thoroughly described elsewhere (Fischl et al., 2002; Fischl, 2012 , Iglesias et al., 2015 . Scripts used to manage this project are also freely available online (Schmidt, 2017) . We also compared two volumes assembled from FreeSurfer 6.0.0's hippocampal subfields processing stream (Iglesias et al., 2015) . The Figure S1 "subfields" or "sf1" volumes come from FreeSurfer's "Whole_hippo-campus" field in the hippoSfVolumes files. As the most obvious difference between FreeSurfer's atlas and our protocol's hippocampal boundaries is FreeSurfer's inclusion of tail tissue, we derived a "subfields-T" or "sf-"volume, by subtracting the "Hippocampal_tail"
field from the "Whole_hippocampus" field. It is intended to represent a closer approximation, and fairer comparison, to our manual traces.
| Comparisons
Comparisons between automated and manual methods were con- HVol adj 5HVol raw 2bðTIV raw 2TIV mean Þ
HVol raw and TIV raw refer to the measured values for a given participant. TIV mean refers to the mean TIV of the sample.
Second, to attain a more accurate assessment of agreement, respecting location in addition to volume, overlap of specific voxels was assessed. Dice's coefficient is the only comparison algorithm in this study able to account for agreement on the location of the hippocampus within the MRI. To generate Dice's coefficients, manually traced images were first registered onto the FreeSurfer aseg.mgz with AFNI tools (Cox, 1996; Cox & Hyde, 1997; Gold et al., 1998) to ensure coordinate consistency. Masks were created from the manual tracings using MIPAV and compared to an uncompressed copy of FreeSurfer's labeled aseg.mgz image in three dimensions with Gnu Octave (Eaton, Bateman, Hauberg, & Wehbring, 2015) . Scripts used for this process are freely available online (Schmidt, 2017) . The ratio below is expressed as Dice's coefficient (Dice, 1945; Equation 2 ) as performed in the original FreeSurfer report (Fischl et al., 2002 ; Figure 1 )
A represents voxels selected by segmentation A; B represents voxels selected by segmentation B. |A\B| represents the voxels selected by both A and B.
Third, an asymmetry index was calculated as suggested by prior literature (Hasan & Pedraza, 2009; Equation 3 ) and an additional Cohen's d value (Equation 4) was calculated for comparisons to published metaanalyses (Pedraza, Bowers, & Gilmore, 2004) . Statistical significance is reported as a one-sample t test with a null hypothesis of
Fourth, we assessed volume by each technique over age, and compared the level of atrophy calculated from each method. Change cannot be directly measured in our cross-sectional sample, so annualized atrophy was quantified with a generalized linear model regressing volume on age with a log link function and a gamma distribution. Exponentiated coefficients represent percentage annualized difference in volume, or "atrophy." A value of 0.010 implies that 1.0% of volume is being lost each year.
To assess stability and consistency of segmentation algorithms over age and volume, data from each method of segmentation, and comparison measures between methods, were compared to age and traced volumes via linear regression models. Comparison measures and log-linear relationships were also compared across age strata: <55;
55-70; >70 as assessed in a recent meta-analysis (Fraser, Shaw, & Cherbuin, 2015) .
Power analyses were performed in R with the "pwr" package.
| RE S U L TS
Of the 708 MRIs available for analysis, 664 remained after all exclusions. Our sample was 39% male, averaged 61 years of age (SD 5 9.2 years), and was exclusively non-Hispanic White from Rochester, MN. The volumes reported from each method are in Table 2 ( ( Table 4 ). In the entire sample, right hippocampi were larger than left in all methods (p < .001; Figure 2 ). Table 3 and an illustration of overlap is shown in Figure 1 . See Supporting Information Figure S1 for all slices. reports, which range from 0.28% to 6.38% per annum (Fraser et al., 2015; Raz et al., 2004a; Raz & Rodrigue, 2006) . Atrophy rates in AD patients, for comparison, have been reported at 3%-8% per annum (Barnes et al., 2009; Raz & Rodrigue, 2006) . Our low atrophy could be explained by our sample's mean age of 61.3, about a decade younger than most studies of aging and dementia. Stratifying by age reveals higher atrophy in older members of our cohort, 1.1% per annum in 129 participants over 70, similar to the 1.1% reported in controls over 70 (Fraser et al., 2015) and 1.4% reported for similarly-aged controls in AD studies (Barnes et al., 2009 ; Figure 6 ). Stratification suggests that the association between hippocampal volume and age is nonlinear However, both our manual and automated results suggest an opposite relationship prior to age 40 than that suggested by (Allen et al., 2005) , who used manual tracing. All comparison studies reported on healthy participants across the adult lifespan. FreeSurfer documentation (available online) and training prescribe manually checking several steps throughout its process to ensure reliable segmentations. In this study, 708 images were submitted to FreeSurfer, including 27 images with incorrect orientation data. 21 of the 27 misoriented images generated errors and failed, making manual checks unnecessary, but six images were registered sideways to the atlas, completed the process "successfully" and generated incorrect data. These were only excluded from our analyses after manual review of all segmentations. Two other images caused FreeSurfer to fail, but generated no error messages, leaving partial data for analysis and reinforcing the need for manual review. The necessary step of manually qualitychecking output requires under a minute per observation, comparing favorably to the hours required for manual tracing. Researchers reap many benefits from automation, but expecting complete autonomy from any software is irresponsible and increases the risk of noisy data and misleading results. These potential issues do not represent a problem with FreeSurfer, but reinforce the need for users to manually review, and even edit intermediates, before using final FreeSurfer measurements to test hypotheses.
FIG URE 4
FreeSurfer's hippocampal subfield overestimates are smaller, but also differ with volume and age. Bland-Altman plots show the difference between raw volumes from FreeSurfer 6.0.0's hippocampal subfields and manual tracing on the y-axis. Although there is little difference between subfields and manual tracing, on average, the difference is reduced over size and age similarly to FreeSurfer's subcortical segmentation stream
Manual tracing, broadly defined, is considered the gold standard in segmentation studies due largely to the human ability to deal appropriately with abnormal data. But there are at least 71 published protocols for manually tracing hippocampi (Boccardi et al., 2011; Konrad et al., 2009) , and even the same protocol can result in different volumes if performed at different angles through the image (Hasboun et al., 1996) . Defining any of these protocols, or none, as the narrowly defined gold standard is controversial (Boccardi et al., 2011; Boccardi et al., 2015b; Hasan, 2009) (Boccardi et al., 2011; Boccardi et al., 2015a; Boccardi et al., 2015b; Frisoni et al., 2015; Frisoni & Jack, 2011) . Our tracing work began in 2012 and employed a protocol (Boccardi et al., 2011; Jack, 1994) that was later merged into the EADC-ADNI harmonized protocol. It is precisely defined and well-established, but it differs from the Center for
Morphometric Analysis (CMA) guidelines (Makris et al.,) underlying
FreeSurfer's atlas. First, (Jack, 1994) and (Watson et al., 1992) define the caudal boundary of the hippocampus as "the coronal section in which the crus of the fornix is seen in full profile" while CMA recommends inclusion of further caudal regions, resulting in larger tails. Second, FreeSurfer segmentations tend to include partial voxels between hippocampus and lateral ventricle that are more often excluded in manual tracings (Cherbuin et al., 2009; Han & Fischl, 2007; Tae et al., 2008) . Both of these differences contribute to FreeSurfer's larger volumes, which were 20% larger than their manual counterparts using 6.0.0, and 24% larger using 5.3.0, consistent with previous reports of 26% with FreeSurfer 4.3 (Cherbuin et al., 2009 ) and 38% with FreeSurfer 3.04 (Tae et al., 2008) .
Regardless of manual tracing's status as the gold standard, the labor cost of hand-tracing is prohibitive for large studies, making automation the default method, and periodic re-calibration studies like this one critical. Manual hippocampal tracing costs approximately $23.00
per subject (2 hippocampi 3 1 hr each 3 $23,376 per year/2,000 paid hours per year), based on recommended graduate student stipend rates Comparisons between multiple tracings are divided into intra-rater (the same human rater re-tracing the same hippocampus) and inter-rater (different humans tracing the same hippocampus). Manual tracings and FreeSurfer version 6.0.0 were compared to all other methods. Dice's coefficients are reported with standard deviations in parentheses. Older and alternate measurement comparisons are excluded. sf1 refers to hippocampal subfields' "whole_hippocampus" field; sf-refers to sf1 with tail subtracted. See methods for caveats regarding interpretation. Table 5 for quantitative values , calculated by linear glm, and plotted in Figure 5 . The same data are also expressed as a percentage of existing volume lost each year, calculated with a gamma family log-link glm, and plotted in Figure 6 . glm 5 generalized linear model. of subcortical structures by multiple atlases and cortical metrics that would incur additional costs to replicate manually. Ideally, tools like The right hippocampus is likely to be larger than the left, although some studies have reported the opposite (Pedraza et al., 2004) . All methods used in this study support the consensus in reporting rightward (positive) asymmetry (Table 4) , which is slightly less pronounced in left-handed participants. Previous studies of automated tools occasionally mention handedness, but none have had large enough samples to assess potential effects (Cherbuin et al., 2009; Wenger et al., 2014) .
This study assessed asymmetry in 40 left-handed and 620 righthanded participants, giving us more power to detect a handedness effect than any we've found to date. The differences in asymmetry index between right-and left-handed participants were still insignifi- and sf-, were also introduced to the analyses: sf1 simply to investigate
FreeSurfer's improved hippocampal subfield routines (Iglesias et al., 2015) ; and sf-to bring FreeSurfer's segmentation boundaries closer to our manual tracing protocol. The difference between the smaller-thanexpected whole_hippocampus subfield (sf1) and aseg-reported hippocampal volumes (Figure 2 ) resulted in sf1 having higher agreement with manual tracing than any other method tested. These results, even with similar biases to the aseg-based volumes (Figures 3 and 4) , suggest the potential use of sf1 volumes rather than the aseg volumes in future studies focused on the hippocampus. This suggestion is further supported by improved Alzheimer disease discrimination using subfields' whole_hippocampus over aseg volumes (Iglesias et al., 2015) .
This study has numerous limitations. It is limited by its narrow focus on the hippocampus and its selection of a single automated segmentation pipeline. FreeSurfer reports 45 subcortical volumes in addition to cortical volumes and areas from multiple atlases for potential comparison to manual segmentation methods. This study also focused on FreeSurfer to the exclusion of multiple other automated segmentation tools. The original 1.5T MRI acquisition parameters resulted in 0.9375 3 0.9375 3 1.6 mm voxels, which were resampled to 1 mm isotropic resolution for FreeSurfer processing. The resampling has the potential to introduce biases to FreeSurfer's segmentation, relative to images originally obtained at 1mm isotropic resolution. The 1.5T field strength used here differs from the 3T strength being adopted in many present and future studies. Similar analyses with 3T MRI would provide valuable additions to this study as well as extending (Han et al., 2006) and (Iglesias et al., 2015) . Moreover, our cohort represents only normal controls aged 38-84. Investigating these biases in younger people (Wenger et al., 2014) and people with disorders previously associated with hippocampal atrophy are important future directions.
A strength of this study is the collection of 664 MRI scans, complete with manual tracings and FreeSurfer segmentations. We are unaware of any comparably sized collection of manual tracings, particularly from a community sample covering males and females ranging so widely in age. This size allows sufficient statistical power to calculate atrophy in three independent age-based subgroups, without losing statistical significance. But it still does not contain enough left-handed participants to detect significant right > left asymmetry in left-handers alone. Power analyses report that 83 participants are necessary to have a 90% chance of detecting the volume biases we reported at a p 5 .05 level for FreeSurfer 6.0.0 or 43 participants for sf1. Smaller studies may be subject to these biases, but be underpowered to detect and report them, even with both automated and manually traced volumes.
Many older population-based studies, like GMBI or its related ARIC study, would benefit from reprocessing existing images with newer versions of FreeSurfer. Each study could provide additional insights with a minimal expenditure of new funds. Having results from multiple field strengths would provide replication potential, richer information about the similarities and differences between field strengths, and a broader picture to feed meta-analyses. This report provides relevant information to investigators considering that approach and those who might seek to interpret their efforts in a larger context.
In summary, we found that FreeSurfer continues to report larger hippocampal volumes than manual tracing, in agreement with prior reports. The larger volumes correlate well with manual tracings, however, and show acceptable agreement in overlap analyses. Moreover, FreeSurfer and manual tracing report similar results in assessment of hippocampal atrophy over the lifespan, in line with prior studies (Fraser et al., 2015) . Our results indicate that FreeSurfer is an adequate, although not directly comparable, replacement for manual tracing, provided manual assessment of input images and output segmentations for exclusionary criteria relevant to each investigation.

{'paper_id': '3448593', 'title': 'A comparison of manual tracing and FreeSurfer for estimating hippocampal volume over the adult lifespan', 's2_url': 'https://api.semanticscholar.org/CorpusID:3448593', 'source': 'S2ORC'}