PMC10284584_01
Versions of the Fragile Families and Child Wellbeing Study data. There are several versions of the data which are made available to researchers depending on their particular needs. The raw data are used only by survey administrators and are stored in separate files to reduce the risk of a breach. For instance, no data file contains respondents' names and survey responses. All files given to researchers have names and other obvious identifiers removed and noise added to any data indicating place of residence. Among the files available to researchers, the restricted files provide the most information but are hardest to access. Researchers obtain restricted files through an intensive application and screening process. After this process, researchers are given only the portions of the restricted files needed for their particular projects. Most researchers' projects can be completed using only the basic files, for which one still must apply by proposing a research project. To create the Challenge files, we made modifications to the basic files.
nihms-1847715-f0019.jpg
PMC10284584_01_nihms-1847715-f0019.jpg
F4
Overall, these modifications were relatively minor because the preexisting Fragile Families Study files had already undergone an extensive process to promote privacy (Figure 4).
Fragile Families Challenge data structure. Participants built models predicting the age 15 outcomes using data collected while the focal child was age 9 and younger. We provided participants with the data represented by the white boxes. Submissions were scored on the basis of their predictive performance (mean squared error) for the observations represented by the gray boxes, which were available only to organizers. The leaderboard set contained one-eighth of all observations and was used to provide instant feedback on submissions. The holdout set contained three-eighths of observations and was used to produce final scores for all submitted models at the end of the Challenge.
nihms-1847715-f0021.jpg
PMC10284584_01_nihms-1847715-f0021.jpg
F6
These age 15 data had been collected but were not yet available to participants (Figure 6). As is typical in projects using the common task framework, we split the year 15 data for these six outcomes into three groups: (1) a training set that we provided to participants, (2) a leaderboard set that participants could access during the Challenge, and (3) a holdout set that participants could not access until the first stage of the Challenge was complete (Figure 6).
Reidentification examples with deidentified data. In both examples, the adversary succeeded because key variables were available in both the deidentified data set (blue) and an identified auxiliary data set (red). In addition to merges between two data sets, reidentification can proceed through a chain of auxiliary data sets.
nihms-1847715-f0022.jpg
PMC10284584_01_nihms-1847715-f0022.jpg
F7
Because these three variables were available in both the deidentified data and the identified auxiliary data, Sweeney was able to merge them together (Figure 7). These hospital discharge data in turn contained basic demographic information that could be used to combine them with identified voter registration records (Figure 7). Just as merged identified voting records with deidentified medical records to create identified medical records (Figure 7), we tried to find information an attacker might merge with the Fragile Families Challenge data. Thus, even though the two real attacks we described previously used ZIP code (Figure 7), we wish to highlight the fact that it is still possible to reidentify data even if it does not contain obvious geographic information.
Hypothetical example of reidentification attack of the Fragile Families Study. The Fragile Families Study does not contain information on voting, but if it did, these variables could be linked to an identified auxiliary data source: administrative voting records. After completing this linkage, an adversary could learn about potentially sensitive information, such as parental drug use and child delinquent behavior. This hypothetical attack, as well as our actual in-house attack, would be possible even if the data did not include any geographic information.
nihms-1847715-f0023.jpg
PMC10284584_01_nihms-1847715-f0023.jpg
F8
These variables are also available in identified voter registration databases, which would allow an adversary to merge them (see Figure 8).14 A key feature of this hypothetical attack, as well as our real in-house attack, was that it did not involve the use of ZIP code or any other geographic information.
Screening process for applications to the Fragile Families Challenge. Each potential participant completed an application describing his or her educational background, research experience, and motivations to participate in the study. We assessed these applications in terms of their ability to contribute to the goals of the Challenge and in terms of the risk that an applicant might try to reidentify respondents. Each application was reviewed by one of the lead organizers of the Challenge, sent to a review committee, and then approved 24 hours later if there were no objections. After approval, participants completed a set of terms and conditions, received a link to an encrypted file, and then called us for a password to open the file. For further details, see the main text. To review the application form and the terms and conditions, see the Appendix.
nihms-1847715-f0024.jpg
PMC10284584_01_nihms-1847715-f0024.jpg
F9
For the Fragile Families Challenge, we developed a process to screen applicants (Figure 9).