Abstract
Disseminating data to potential collaborators can be essential in the development of models, algorithms, and innovative research opportunities. However, it is often time consuming to get approval to access sensitive data such as health data. An alternative to sharing the real data is to use synthetic data, which has similar properties to the original data but does not disclose sensitive information. The collaborators can use the synthetic data to make preliminary models or work out bugs in their code while waiting to get approval to access the original data. A data owner can also use the synthetic data to crowdsource solutions from the public through competitions like Kaggle and then test those solutions on the original data.
Abstract
This paper implements a method that generates fully synthetic data in a way that matches the statistical moments of the true data up to a specified moment order as a SAS macro. Variables in the synthetic data set are of the same data type as the true data (for example, integer, binary, continuous). The implementation uses the linear programming solver within a column generation algorithm and the mixed integer linear programming solver from the OPTMODEL procedure in SAS/OR. The COFOR statement in PROC OPTMODEL automatically parallelizes a portion of the algorithm. This paper demonstrates the method by using the Sashelp.Heart data set to generate fully synthetic data copies.
INTRODUCTION
Data owners face barriers in sharing sensitive data with potential collaborators. They might wish to share their data to promote interdisciplinary work or increase the utility of data that was expensive to collect, but struggle to gain permission to disseminate these data among external collaborators. Potential external collaborators might also be unexperienced in the data owner's domain and therefore might become frustrated with the time and effort required to complete the protocols and procedures necessary to acquire such data. These barriers often result in foregone innovative interdisciplinary opportunities and may hinder the advancement of research within application areas that harbor extensive protections on their data. However, collaborators who primarily aim to access these data in order to develop algorithms or modeling methods might actually be able to begin work without the real data, as long as an approximate, realistic data set can be used in lieu of the real data or during the development of analytical work while waiting for real data access. This alternative data is known as synthetic data. Researchers could use synthetic data to, for example, understand the format of the real data, develop an understanding of its statistical properties, build preliminary models, or tune algorithm parameters. Analytical procedures and code developed using the synthetic data can then be passed to the data owner if the collaborators are not permitted to access the original data. Even if the collaborators are eventually granted access, synthetic data allow the work to be conducted in parallel with tasks required to access the real data, allowing an efficient path to the final analysis. This paper contains and describes an implementation of an algorithm to create synthetic data. The original algorithm can be found in A Moment Matching Approach for Generating Synthetic Data (Bogle 2016) . The method generates data that have statistical marginal and mixed raw moments similar to the real data, up to the moment order specified by the user. The data generation algorithm consists of three steps, which are implemented with the OPTMODEL procedure in SAS/OR. The preliminary step reads in the real data and computes its raw moments, which are used in the remaining steps. The linear programming (LP) step creates candidate synthetic observations by solving a series of linear programs. The integer programming (IP) step selects the desired number of synthetic observations from the candidates that most closely match the moments of the original data set.
USAGE
The macro to generate synthetic data is called %GENDATA and has parameters that can be used to control the algorithm by changing the size of the output, the time needed to run the algorithm, and the quality of the synthetic data.
Required Parameters
INPUTDATA=inputdataset Specifies the name of the input data set that the synthetic data set emulates. While it is possible to customize the macro, two key assumptions are made about INPUTDATA for using the macro as written. The first assumption is that the original data has no missing values. The SAS DATA step can be used to remove rows with missing values. Note, however, that list-wise deletion (complete case analysis) could bias any analytical results. Using an appropriate imputation technique to impute missing data before using this macro is recommended. The second assumption is that the data are all numeric; categorical data can be represented as a binary or integer variable. For example, a two factor categorical variable, such as Sex represented as "Male" and "Female", can be converted to "0" and "1" prior to using this macro. The Examples section demonstrates this.
METADATA=metadataset
Specifies the name of the user-created data set containing a column called "varName" with the names of the variables from INPUTDATA to be used and a column called "type" with the numeric type for each variable. The options are "c" for continuous, "i" for integer, and "b" for binary.
Optional Parameters
OUTPUTDATA=SyntheticData Specifies the name of the synthetic data set created by the macro. The default value is SyntheticData, which is placed in the work folder.
MOMENTORDER=2
Specifies the order up to which moments are matched. The values are typically 2, 3, or 4. Larger values increase the size of the optimization problems, and therefore the computational time. The default value is 2.
NUMOBS=0
Specifies the desired number of observations to create for the synthetic data set. The default value of 0 generates a synthetic data set with the same number of observations as INPUTDATA.
MINNUMIPCANDS=0
Specifies the minimum number of observations to create during the LP step to pass to the IP step. The procedure usually produces more than this because they are selected in batches. The default value of 0 sets it to twice the value of NUMOBS. It must not be smaller than NUMOBS, and larger values produce better synthetic data. However, note that setting NUMOBS=0 and MINNUMIPCANDS≥x is a valid operation as long as x is larger than the number of observations in INPUTDATA.
LPBATCHSIZE=10
Specifies the number of random observations to create before assessing the reduced cost of the candidate observations in order to select one to add to the model during the LP step. The default value is 10.
LPGAP=1E-3
Specifies the objective value threshold that must be achieved to stop adding observations to the LP. Once this threshold is reached, the algorithm moves on to the IP step if there are enough candidate observations, or starts with a new LP if more candidate observations are needed. The default value is 1E-3.
NUMCOFORTHREADS=1
Specifies the number of threads to be used in the COFOR loop during the LP step to create candidate observations for the IP step. The default value is 1.
MILPMAXTIME=600
Specifies the maximum time (in seconds) allowed for the MILP solver to select the final observation set. The MILP solver commonly does not reach the optimal solution in a reasonable amount of time, so this time limit stops the solver and returns the best solution found. The default value is 600 (10 minutes).
RELOBJGAP=1E-4
Specifies the largest relative difference between the best feasible solution and best lower bound for the MILP solver to declare optimality. The default value is 1E-4, which is the same as the default value for the MILP solver.
ALPHA=0.95
Specifies the confidence level used to calculate confidence upper and lower bounds on the moments of the real data. These bounds define the range for each synthetic data moment that is considered a match to its corresponding moment in the real data. For example, ALPHA=0.95 sets the upper bound to be approximately 1.96 times the standard error above the mean and the lower bound to be approximately 1.96 times the standard error below the mean. ALPHA must be strictly greater than 0 and strictly less than 1, with smaller values causing the moments to be matched more closely. The default value is 0.95.
RANDSEED=0
Specifies the random number seed to use for producing candidate observations. A value of 0 uses a random number stream based on the system clock, and values greater than 0 use a reproducible stream. This option should only be used when NUMCOFORTHREADS=1. The default value is 0.
Macro Overview
An example call of the macro is:
Macro Overview
The macro is broken into three parts within the call to PROC OPTMODEL, each a macro itself, plus setting the random number seed. Only the relevant parameters are passed into the macro for each of these parts. Note that the macro for each of these parts is not intended to be run independently, as PROC OPTMODEL objects from previous steps are expected to be available. Here is the definition of the overall macro: 
PRELIMINARY STEP
The preliminary step reads the original data into PROC OPTMODEL. It declares and computes sets and parameters to be used in the LP and IP steps. The most significant outputs from this step are the values of the statistical raw marginal and mixed moments of the real data, up to the specified order, and the desired lower and upper bounds for the moments of the synthetic data. A raw mixed moment is the mean value over all the observations of the monomials created by the variables. The acceptable ranges here are computed by using the standard error, but this can easily be customized.
PRELIMINARY STEP
For example, if there are two variables, the following list shows all monomials up to the third order:
PRELIMINARY STEP
The raw moments of a population would be the expected values of these monomials. The raw moments of a data set are the mean values of the monomials over the observations.
PRELIMINARY STEP
Two SAS functions, not specific to PROC OPTMODEL, are used in this step. The STD function computes the standard deviation. The PROBIT function computes the ALPHA quantile from the standard normal distribution. PROBIT returns the one-sided value, but it is converted in the macro to a two-sided value (for example, ALPHA=0.95 uses PROBIT(0.975)=1.96).
PRELIMINARY STEP
Here is the code for this step: 
LP STEP
The second step is the LP step, which takes the information from the preliminary step and constructs a linear programming model. For a review of linear and integer programming, see (Shrijver 1998) . The model weights the observations with a nonnegative variable value Weight, and the weights must sum to 1, as enforced by the constraint WeightEQ1. Because this step's goal is to quickly generate many candidate observations and avoid the computational burden introduced by a large integer program, observations are permitted to be unequally weighted in this step. To avoid duplicating the algebraic expression for weighted moments, an implicit variable, WeightedMoment, is used. Constraints Upper and Lower bound the weighted moment values by the bounds computed in the preliminary step. A nonnegative slack variable, Eta, represents the violation of either the upper or lower bound of the constraint (which corresponds to a moment). The objective is to minimize the sum of these violations.
LP STEP
After the above declarations, the model has been specified. Now, synthetic observations that populate the variable Weight need to be created. The process adds synthetic observations to the model until the solution has an objective value of LPGAP, which indicates that there is a way to weight the observations such that there is no violation to the desired bounds on the moments. This process is repeated until the total number of nonzero weighted observations is greater than or equal to the desired number of candidate observations for the IP step, given by the macro parameter MINNUMIPCANDS. If the parameter NUMCOFORTHREADS is used, PROC OPTMODEL can solve multiple LPs concurrently. The variables and constraints are automatically replicated across iterations, so there is no additional index needed for the iteration. The additional index is needed to store the solution for use outside the COFOR loop. Each iteration of the COFOR loop produces at least MINNUMIPCANDS/NUMCOFORTHREADS candidate observations, which guarantees at least MINNUMIPCANDS candidate observations in total. PROC OPTMODEL does not run the other statements in the loop concurrently, only the SOLVE statement. Since the LPs tend to solve quickly, the performance difference for more threads are most noticeable the LPs get large.
LP STEP
Candidate observations are created by using a uniform random variable between the least and greatest values for that variable in the original data. If the variable is to be integer or binary, it is rounded. The moment values for that observation are then computed. This is done once for the first observation to go in the LP. The LP is then iteratively solved, adding a new observation at each iteration, until the objective is less than LPGAP. After the first observation, there are LPBATCHSIZE candidate observations created in each iteration. Then the one with the best reduced cost is selected and added to the model, while the others are discarded. For more information on column generation, see (Desaulniers 2005 
IP STEP
The final step is the IP step. The candidate observations resulting from the LP step are used in an integer program that selects the desired number of observations, NUMOBS, that best fit the desired moment ranges. Unlike the LP model, which minimizes the sum of violations, in the IP step the largest scaled violation is minimized. To this end, Eta is scaled and replaced by a new variable, ScaledEta, which is equivalent to Eta divided by the acceptable moment range. The NumAssigned constraint selects exactly NUMOBS observations from the candidate observations into OUTPUTDATA, and all the selected observations are weighted equally in the expression for the moment values.
IP STEP
Because integer programs are computationally more difficult to solve than linear programs, it could take too long to solve to optimality. As long as the objective is relatively small when the time limit is reached, the synthetic data set should behave similarly to the original data set. 
EXAMPLES
This example walks through the process of using the macro on the Sashelp.Heart data set. The Sashelp.Heart data set has some categorical variables and dependent variables, as well as variables with many missing values. To simply this example, only Status, Sex, AgeAtStart, Height, Weight, Diastolic, Systolic, Smoking, and Cholesterol are considered. Two of these variables, Status and Sex, are categorical variables that must be converted into numeric values (e.g. 0 and 1) before the macro can be executed. Thus, surrogate numeric variables, Dead and Male, respectively, are created. The user can convert them back to the original form if desired after the macro execution. For simplicity, in this example rows with missing values are ignored. The other variables take integer values, except that Height has some decimals, so Metadata is set accordingly. The DATA step creates OriginalData from Sashelp.Heart, and Metadata indicates the desired numeric type for each variable: 
SMALL EXAMPLE WITH BASIC VERIFICATION
This example generates a synthetic data set with 100 observations which matches marginal and mixed moments up to the second order, use three threads, allows one minute for the IP step, and uses the default values for the other parameters. Using these specifications, you can call the macro:
SMALL EXAMPLE WITH BASIC VERIFICATION
This macro call produces a data set called Work.SyntheticData. To convert the binary variables back to the original format of categorical variables and to round Height to the nearest hundredth, you need to use another DATA step:
SMALL EXAMPLE WITH BASIC VERIFICATION
data FinalSyntheticData; set SyntheticData; label AgeAtStart = 'Age at Start'; length Status $5 Sex $6; if (Dead = 1) then Status = "Dead"; else Status = "Alive"; drop Dead; if (Male = 1) then Sex = "Male"; else Sex = "Female"; drop Male; Height = round(100*Height) / 100; run;
SMALL EXAMPLE WITH BASIC VERIFICATION
The data set FinalSyntheticData now has 100 observations with similar properties to the original data set, but nothing is identifiable from the original data set.
SMALL EXAMPLE WITH BASIC VERIFICATION
For quick validation that the first and second marginal moments of the synthetic data set are reasonable, you can call PROC MEANS to check that the mean and standard deviation of each variable in the data set is similar to the real data: Table 2 .
PROC MEANS Output for SyntheticData
The output in Table 1 and Error! Reference source not found. show that the means are close and the standard deviations are similar. Note that standard deviation and covariance use central moments, not raw moments, so their differences can be more noticeable than the mean.
PROC MEANS Output for SyntheticData
For validation that the interaction between variables is captured, you can call PROC CORR to check that the covariance matrices are similar: Table 3 and Error! Reference source not found. show the covariance matrices. In this case, most of the covariances are on the same order of magnitude, but there are noticeable differences. Using a smaller value for the ALPHA parameter, a larger MINNUMIPCANDS, or a longer MILPMAXTIME usually reduce the differences between the covariance matrices.
LARGER EXAMPLE WITH FITTED MODEL COMPARISON
If the synthetic data are truly similar to the original data, models fit to either data set should score new data similarly. This example splits OriginalData into a training data set and a testing data set, creates a synthetic data set based on the training data set, fits a logistic regression model to the training and synthetic data sets, and finally compares the fitted models and how they score against the test data set.
LARGER EXAMPLE WITH FITTED MODEL COMPARISON
The first step is to randomly partition the data into a training set (70%) and a test set (30%): The next step is to create the synthetic data based on the original data selected for the training set:
LARGER EXAMPLE WITH FITTED MODEL COMPARISON
%GENDATA(INPUTDATA=Training, METADATA=Metadata, NUMCOFORTHREADS=4, MOMENTORDER=3, MILPMAXTIME=1200, ALPHA=0.2);
LARGER EXAMPLE WITH FITTED MODEL COMPARISON
Increasing MOMENTORDER to 3 and decreasing ALPHA to 0.2 fits the original distribution more closely and is more precise. The drawback is that it takes much longer to run. Using NUMCOFORTHREADS=4 does help reduce the time by about 40% in this case compared to using NUMCOFORTHREADS=1.
LARGER EXAMPLE WITH FITTED MODEL COMPARISON
Body mass index (BMI) is generally a better independent predictor than height or weight. Since BMI is a function of height and weight, it is computed and added to the data sets by utilizing the DATA step: A large amount of output is produced, but only some of the tables and figures are included here. Table 5 and Table 6 display the odds ratio estimates and the confidence limits for each variable for TrainingWithBmi and SyntheticDataWithBmi, respectively. The ratios of four of the seven variables for the synthetic data fall within the 95% confidence intervals. The others are still relatively close. Figure 1 and Error! Reference source not found. show the ROC curves for the models scored against the test data. The area under the curve (AUC) included in the figures is a measure of the model's specificity and sensitivity. A value of 1 would be perfect, and a value of 0.5 would mean it is as good as randomly guessing. While the values of AUC for the fitted models were 0.8069 for the training data and 0.7141 for the synthetic data, the values of the models applied to the test data were very close to each hdavenport@example.com and 0.7768. This indicates that even though there are some clear differences between the fitted models, they performed similarly against new data from the original distribution. This comparison is only for one instance and one model, but demonstrates that the synthetic data produced by this implementation can be useful substitutes for the real data. An in-depth analysis of the performance and a comparison to other synthetic data methods has been previously published (Bogle 2016) .
Odds Ratio Estimates

POTENTIAL FOR CUSTOMIZATION
Numerous portions of this macro allow for customization. The lower and upper bounds on the moments can be modified to more accurately reflect the desired precision. For example, the moments could be specified to have a lower and upper bound of within one percent of the point estimate of the corresponding moment from the real data, or could be manually specified. Additionally, if only certain moments are known to be important during the matching process, these can be the sole moments specified. To emphasize lower order moments, for example, tighter bounds could be placed on first and second order moments and wider bounds on third and fourth order moments.
POTENTIAL FOR CUSTOMIZATION
There are opportunities to reduce the number of unused LP candidate observations by specifying the sampling technique used to generate values for each variable. For example, if a variable has a distribution that is thought to be very different from uniform, a different distribution could be specified based on the empirical distribution or literature. Using a multivariate distribution could help find synthetic observations that better match mixed moments, if these relationships are well-known.
POTENTIAL FOR CUSTOMIZATION
If multiple synthetic data sets are desired for one original data set, the preliminary step would only need to be run once. A loop could be put around the LP and IP steps, with care taken to prevent data sets from being overwritten.
CONCLUSION
This paper demonstrates a method for producing synthetic data using the features of the OPTMODEL procedure in SAS/OR. It leverages several strengths of PROC OPTMODEL, including the ability to easily read from and write to data sets, use statistical functions, easily run solvers in parallel using the COFOR loop, use dual information to compute reduced costs for column generation, and apply other SAS procedures to data sets produced.
CONCLUSION
Our approach to generating synthetic data can be used to create many loosely matched data sets quickly. If maintaining statistical properties is critical and with enough lead time, it can be used to produce a single data set that very closely matches the moments of the real data without disclosing any of the real data's observations. Properly specified synthetic data can streamline the research process by removing some barriers to data access and therefore is a potential mechanism to increase the utility of sensitive and expensive data. Crowdsourcing solutions through platforms like Kaggle is also easier to utilize for problems that would normally have privacy restrictions.

{'paper_id': '35641772', 'title': 'A Moment Matching Approach for Generating Synthetic Data', 's2_url': 'https://api.semanticscholar.org/CorpusID:35641772', 'source': 'S2ORC'}