Issue #142 : Added options for custom barlabel

Still need to make it print the warning in case of an error on the input

*! version 5.5 26APR2018 DIME Analytics lcardosodeandrad@worldbank.org

cap	program drop	iegraph
	program define 	iegraph, rclass

	syntax varlist, 							///
	   [noconfbars 								///
		confbarsnone(varlist) 					///
		confintval(numlist min=1 max=1 >0 <1) 	///
		BARLabel								///
		MLABColor(string)						///
		MLABPosition(numlist)					///
		MLABSize(string)						///
		BAROPTions(string)						///
		barlabelformat(string)					///
		GREYscale								///
		yzero									///
		BASICTItle(string) 						///
		VARLabels								///
		ignoredummytest 						///
		norestore								///
		save(string) *]

	if "`restore'" == "" preserve

	qui {

	version 11

/*******************************************************************************

						Identify options used

********************************************************************************/

	*Checking to see if the noconfbars option has been used and assigning 1 and 0 based
	*on that to the CONFINT_BAR variable.
	if "`confbars'" != ""  			local CONFINT_BAR 	= 0
	if "`confbars'" == ""  			local CONFINT_BAR 	= 1

	*Checking to see if the barlabel option has been used and assigning 1 and 0 based
	*on that to the LABEL_BAR variable.
	if "`barlabel'" != "" 			local LABEL_BAR 	= 1
	if "`barlabel'" == "" 			local LABEL_BAR 	= 0

	*Checking to see if the mlabposition option has been used and assigning 1 and 0 based
	*on that to the LABEL_POS variable.
	if "`mlabposition'" != "" 			local LABEL_POS 	= 1
	if "`mlabposition'" == "" 			local LABEL_POS 	= 0

	*Checking to see if the mlabcolor option has been used and assigning 1 and 0 based
	*on that to the LABEL_COL variable.
	if "`mlabcolor'" != "" 			local LABEL_COL 	= 1
	if "`mlabcolor'" == "" 			local LABEL_COL 	= 0

	*Checking to see if the mlabcolor option has been used and assigning 1 and 0 based
	*on that to the LABEL_COL variable.
	if "`mlabsize'" != "" 			local LABEL_SIZE	= 1
	if "`mlabsize'" == "" 			local LABEL_SIZE 	= 0

	*Checking to see if the barlabelformat option has been used and assigning 1 and 0 based
	*on that to the LABEL_FORMAT variable.
	if "`barlabelformat'" != "" 	local LABEL_FORMAT 	= 1
	if "`barlabelformat'" == ""		local LABEL_FORMAT 	= 0

/*******************************************************************************

							Prepare inputs

********************************************************************************/

	*Only keep the observations in the regressions
	keep if e(sample) == 1

	*Copy beta matrix to a regular matrix
	mat BETA = e(b)

	*Unabbreviate and varlists
	unab varlist : `varlist'

	*Testing to see if the variables used in the regressions are actual dummy variables as treatment vars need to be dummy variables.
	foreach var of local varlist {

		*Get the column number from this var
		local colnumber = colnumb(BETA,"`var'")

		*Test if this var was omitted from the regression
		if "`r(label`colnumber')'" == "(omitted)" {

			*Test if that dummy is not found in the estimation matrix
			noi di as error "{phang}Dummy variable `var' was not included in the regression, or was omitted from it.{p_end}"
			error 480
		}
	}

	foreach var of local varlist {
		*Assigning variable coefficient/standard errors/no of obs. to scalars with the name Coeff_(`variable name')
		*coeff_se_(`variable name'), obs_(`variable name').

		*Access and store the beta value (
		scalar coeff_`var' 		= _b[`var']

		*Access and store standard errors for the dummy
		scalar coeff_se_`var' 	= _se[`var']

		*Store the number of observations for this dummy
		count if `var' == 1 //Count one tmt group at the time
		scalar obs_`var' = r(N)

	}

	*Test if the list of dummies are valid
	if "`ignoredummytest'" == "" testDums `varlist'

	*Testing to see if the variables used in confbarsnone are
	*actually in the list of
	*variables used for the regression/graph.
	local varTest : list confbarsnone in varlist

	if `varTest' == 0 {
		*Error displayed if variables defined in confbarsnone aren't in the list of variables for the graph.
		noi display as error "{phang} Variables defined in confbarsnone(`confbarsnone') cannot be found in the graph variable list. {p_end}"
		noi display ""
		error 111
	}

	* Can only be used if the bar label is displayed
	if `LABEL_BAR' == 0 {
		if `LABEL_FORMAT' | `LABEL_POS' | `LABEL_COL' | `LABEL_SIZE' {
			noi display as error "{phang} Options barlabelformat(), mlabsize(), mlabposition() and mlabcolor() can only be specified when option barlabel is used. {p_end}"
			error 198
		}
	}
	else {
		if `LABEL_FORMAT' == 1 {
			* Check that specified format is valid
			if substr("`barlabelformat'",1,1) != "%" | !inlist(substr("`barlabelformat'",-1,1), "e", "f") | !regex("`barlabelformat'", "\.") {
				noi display as error "{phang} Option barlabelformat() was incorrectly specified. Only fixed and exponencial formats are currently allowed. See {help format} for more information on how to specify a variable format.{p_end}"
				error 198
			}
		}
	}

	*Checking to see if the save option is used what is the extension related to it.
	if "`save'" != "" {

		**Find the last . in the path name and assume that
		* the file extension is what follows. However, with names
		* that have multiple dots in it, the user has to explicitly
		* specify the file name.

		**First, will extract the file names from the combination of file
		* path and files names. We will use both backslash and forward slash
		* to account for differences in Windows/Unix file paths
		local backslash = strpos(reverse("`save'"), "\")
		local forwardslash = strpos(reverse("`save'"), "/")

		** Replacing the value of forward/back slash with the other value if one of the
		*  values is equal to zero.
        if `forwardslash' == 0  local forwardslash = `backslash'
        if `backslash' == 0     local backslash = `forwardslash'

		**Extracting the file name from the full file path by  reversing and breaking the path
		* at the first occurence of slash.
		local file_name = substr(reverse("`save'"), 1, (min(`forwardslash', `backslash')-1))
		local file_name = reverse("`file_name'")

		**If no slashes it means that there is no file path and just a file name, so the name of the file will be
		* the local save.
		if (`forwardslash' == 0 & `backslash' == 0) local file_name = "`save''"

		*Assign the full file path to the local file_suffix
		local file_suffix = "`file_name'"

        *Find index for where the file type suffix start
        local dot_index     = strpos("`file_name'",".")

		local file_suffix = substr("`file_name'", `dot_index' + 1, .)

		*If no dot in the name, then no file extension
		if `dot_index' == 0 {
			local save `"`save'.gph"'
			local file_suffix "gph"
			local save_export = 0
		}

		**If there is one or many . in the file path than loop over
		* the file path until we have found the last one.

		**Find index for where the file type suffix start. We are re-checking
		* to see if there are any more dots than the first one. If there are,
		* then there needs to be an error message saying remove the dots.
		local dot_index 	= strpos("`file_suffix'",".")

		*Extract the file index

		if (`dot_index' > 0) {
			di as error "{pstd}File names cannot have more than one dot. Please only use the dot to separate the filename and file format.{p_end}"
			error 198
		}

        *List of formats to which the file can be exported
        local nonGPH_formats png tiff gph ps eps pdf wmf emf

        *If no file format suffix is specified, use the default .gph
        if "`file_suffix'" == "gph" {
			local save_export = 0
		}

        *If a file format suffix is specified make sure that it is one of the eight formats allowed.
        else if `:list file_suffix in nonGPH_formats' != 0  {
            local save_export = 1

			if ("`file_suffix'" == "wmf" | "`file_suffix'" == "emf") & "`c(os)'" != "Windows" {
				di as error "{pstd}The file formats .wmf and .emf are only allowed when using Stata on a Windows computer.{p_end}"
                error 198
			}
		}
		*If a different extension was used then displaying an error.
        else {

            di as error "{pstd}You are not using a allowed file format in save(`save'). Only the following formats are allowed: gph `nonGPH_formats'. {p_end}"
            error 198
        }
    }
	else {

		*Save option is not used, therefore save export will not be used
		local save_export = 0

	}

	* Set default barlabel options
	if `LABEL_BAR' {
		if !`LABEL_POS'		local mlabposition 	12
		if !`LABEL_COL'	 	local mlabcolor		black
		if !`LABEL_SIZE' 	local mlabsize		medium
	}

/*******************************************************************************

						Get values from regression

*******************************************************************************/

	local count: word count `varlist' // Counting the number of total vars used as treatment.
	local graphCount = `count' + 1 // Number of vars needed for the graph is total treatment vars plus one(control).

	//Make all vars tempvars (maybe do later)
	//Make sure that missing is properly handled

	tempvar anyTMT control
	egen `anyTMT' = rowmax(`varlist')
	gen `control' = (`anyTMT' == 0) 	if !missing(`anyTMT')

	sum `e(depvar)' if `control' == 1
	scalar ctl_N		= r(N)
	scalar ctl_mean	  	= r(mean)
	scalar ctl_mean_sd 	= r(sd)

	/**
	 Calculate t-statistics
	**/

	*If not set in options, use default of 95%
	if "`confintval'" == "" {
		local confintval = .95
	}

	**Since we calculating each tail separetely we need to convert
	* the two tail % to one tail %
	local conintval_1tail = ( `confintval' + (1-`confintval' ) / 2)

	*degreeds of freedom in regression
	local df = `e(df_r)'

	*Calculate t-stats to be used
	local tstats = invt(`df' , `conintval_1tail'  )

	foreach var of local varlist {

		*Caculating confidnece interval
		scalar  conf_int_min_`var'   =	(coeff_`var'-(`tstats'*coeff_se_`var') + ctl_mean)
		scalar  conf_int_max_`var'   =	(coeff_`var'+(`tstats'*coeff_se_`var') + ctl_mean)
		**Assigning stars to the treatment vars.

		*Perform the test to get p-values
		test `var'
		local star_`var' " "

		scalar  pvalue =r(p)
		if pvalue < 0.10 {
			local star_`var' "*"
		}
		if pvalue < 0.05 {
			local star_`var' "**"
		}
		if pvalue < 0.01 {
			local star_`var' "***"
		}

		scalar tmt_mean_`var' = ctl_mean + coeff_`var'
	}

/*******************************************************************************

			Set up temp file where results are written

*******************************************************************************/

	tempfile		 newTextFile
	tempname 		 newHandle
	cap file close 	`newHandle'
	file open  		`newHandle' using "`newTextFile'", text write append

	*Write headers and control value
	file write `newHandle' ///
		"position" 	_tab "xLabeled"	_tab "mean" _tab "coeff" _tab "conf_int_min" _tab "conf_int_max" _tab "obs" _tab "star" _n 	///
		%9.3f (1) _tab "Control"  _tab %9.3f (ctl_mean)      _tab  	_tab   _tab _tab %9.3f (ctl_N) 	 _tab  _n

	tempvar newCounter
	gen `newCounter' = 2  //First tmt group starts at 2 (1 is control)

	foreach var in `varlist' {

		if "`varlabels'" == "" {

			*Default is to use the varname in legend
			local var_legend "`var'"

		}
		else {

			*Option to use the variable label in the legen instead
			local var_legend : variable label `var'

		}

		*Writing the necessary tables for the graph to list to file.
	    if `: list var in confbarsnone' {
			file write `newHandle' 	%9.3f (`newCounter') _tab `"`var_legend'"'  ///
			_tab %9.3f (tmt_mean_`var') _tab  %9.3f (coeff_`var')  _tab _tab ///
			_tab %9.3f (obs_`var') _tab "`star_`var''"  _n

			replace `newCounter' = `newCounter' + 1
		}
		else {
			file write `newHandle' %9.3f (`newCounter') _tab `"`var_legend'"' ///
			_tab %9.3f (tmt_mean_`var') _tab  %9.3f (coeff_`var')  			///
			_tab %9.3f (conf_int_min_`var') _tab %9.3f (conf_int_max_`var')	///
			_tab %9.3f (obs_`var')   _tab "`star_`var''"  _n

			replace `newCounter' = `newCounter' + 1
		}
	}

	file close `newHandle'

/*******************************************************************************

						Create the graph

*******************************************************************************/

	*Read file with results
	insheet using `newTextFile', clear

	*Defining various options to go on the graph option.

	local tmtGroupBars 	""
	local xAxisLabels 	`"xlabel( "'
	local legendLabels	""
	local legendNumbers	""

	forval tmtGroupCount = 1/`graphCount' {

		************
		*Create the bar for this group

		if "`greyscale'" == "" {
			colorPicker `tmtGroupCount' `graphCount'
		}
		else {
			greyPicker `tmtGroupCount' `graphCount'
		}

		local tmtGroupBars `"`tmtGroupBars' (bar mean position if position == `tmtGroupCount', `baroptions' color("`r(color)'") lcolor(black) ) "'

		************
		*Create labels etc. for this group

		local obs 			= obs[`tmtGroupCount']
		local stars 		= star[`tmtGroupCount']
		local legendLabel 	= xlabeled[`tmtGroupCount']

		local xAxisLabels 	`"`xAxisLabels' `tmtGroupCount' "(N = `obs') `stars'" "'
		local legendLabels 	`"`legendLabels' lab(`tmtGroupCount' "`legendLabel'") "'
		local legendNumbers	`"`legendNumbers' `tmtGroupCount'"'
	}

	*Close or comple some strings
	local xAxisLabels `"`xAxisLabels' ,noticks labsize(medsmall)) "'
	local legendOption `"legend(order(`legendNumbers') `legendLabels')"'

	*Create the confidence interval bars

	if `CONFINT_BAR' == 0 {

		local confIntGraph = ""
	}
		else if `CONFINT_BAR' == 1 {
		local confIntGraph = `"(rcap conf_int_max conf_int_min position, lc(gs)) (scatter mean position,  msym(none)  mlabsize(`mlabsize') mlabposition(`mlabposition') mlabcolor(`mlabcolor'))"'
	}

	*Create the bar label
	if `LABEL_BAR' == 0 {
		local barLabel = ""
	}
	else if `LABEL_BAR' == 1 {

		gen label = mean

		if `LABEL_FORMAT' == 1 {
			format label `barlabelformat'
		}
		else if `LABEL_FORMAT' == 0 {
			format label %9.1f
		}

		local barLabel = `"(scatter mean position,  msym(none)  mlab(label) mlabposition(`mlabposition') mlabcolor(`mlabcolor'))"'
	}

	local titleOption `" , xtitle("") ytitle("`e(depvar)'") "'

	if "`save'" != "" {
		local saveOption saving("`save'", replace)
	}

	*******************************************************************************
	*** Generating the graph axis labels for the y-zero option used..
	*******************************************************************************

	*Calculations needed if yzero used
	if  ("`yzero'" != "" ) {

		**Testing if yzero is applicable
		********************************

		**Yzero is only applicable if all values used in the graph
		* are all negative or all postive. If there is a mix, then
		* the yzero option will be ignored

		*Finding the min value for all values used in the graph
		gen row_minvalue = min(mean, conf_int_min, conf_int_max)
		sum row_minvalue
		local min_value `r(min)'

		*Finding the min value for all values used in the graph
		gen row_maxvalue = max(mean , conf_int_max, conf_int_min)
		sum row_maxvalue
		local max_value `r(max)'

		*Locals used for logic below
		noi di "local signcheck = ((`r(max)' * `r(min)') >= 0) "
		local signcheck = ((`max_value' * `min_value') >= 0) 	// dummy local for both signs the same (positive or negative)
		local negative	=  (`max_value' <= 0)				// dummy for max value still negative (including 0)

		**If yzero() is used and min and max does not have
		* the same sign, then the yzero() is not applicable.

		if (`signcheck' == 0 ) {

		**** yzero is NOT applicable and will be ignored
		*************************************************

			noi di "{pstd}{error:WARNING:} Option yzero will be ignored as the graph has values both on the the positve and negative part of the y-axis. This only affects formatting of the graph. See helpfile for more details.{p_end}"
		}
		else {

		**** yzero is applicable and will be used
		*****************************************

			*Get max value if only postive values
			if (`negative' == 0) {

				sum row_maxvalue
				local absMax = `max_value'
			}

			*Get absolute min (will convert back below) if only negative values
			else {

				sum row_minvalue
				local absMax = abs(`min_value')
			}

			*Rounded up to the nearest power of ten
			local logAbsMax = ceil(log10(`absMax'))
			local absMax = 10 ^ (`logAbsMax')

			*Generating quarter value for y-axis markers.
			local quarter = (`absMax') / 4

			**Constuct the option to be applied to
			* the graph using the values calculated
			if (`negative' == 0)  {

				local yzero_option ylabel(0(`quarter')`absMax')
			}
			else {

				local absMax = `absMax' * (-1) //Convert back to negative
				local yzero_option ylabel(`absMax'(`quarter')0)
			}
		}
	}

	*******************************************************************************
	***Graph generation based on if the option save has a export or a save feature.
	*******************************************************************************

	*Store all the options in one local
	local commandline 		`" `tmtGroupBars' `confIntGraph' `barLabel' `titleOption'  `legendOption' `xAxisLabels' title("`basictitle'") `yzero_option' `options'  "'

	*Error message used in both save-option cases below.
	local graphErrorMessage `" Something went wrong while trying to generate the graph. Click {stata di r(cmd) :display graph options } to see what graph options iegraph used. This can help in locating the source of the error in the command. "'

	if `save_export' == 0 {

		*Generate a return local with the code that will be used to generate the graph
		return local cmd `"graph twoway `commandline' `saveOption'"'

		*Generate the graph
		cap graph twoway `commandline' `saveOption'

		*If error, provide error message and then run the code again allowing the program to crash
		if _rc {

			di as error "{pstd}`graphErrorMessage'{p_end}"
            graph twoway `commandline' `saveOption'
		}
	}
	else if `save_export' == 1 {

		*Generate a return local with the code that will be used to generate the graph
		return local cmd `"graph twoway `commandline'"'

		*Generate the graph
		cap graph twoway `commandline'

		*If error, provide error message and then run the code again allowing the program to crash
		if _rc {

			di as error "{pstd}`graphErrorMessage'{p_end}"
            graph twoway `commandline'
		}

		*Export graph to preferred option
		graph export "`save'", replace

	}

	if "`restore'" == "" restore
}

end
	*******************************************
	*******************************************
		******* To pick colours based *******
		******* on the number of vars *******
	*******************************************
	*******************************************
	cap	program drop	colorPicker
		program define 	colorPicker , rclass

		args groupCount totalNumGroups

		if `totalNumGroups' == 2 {

			if `groupCount' == 1 return local color "215 25 28"
			if `groupCount' == 2 return local color "43 123 182"
		}
		else if `totalNumGroups' == 3 {

			if `groupCount' == 1 return local color "215 25 28"
			if `groupCount' == 2 return local color "255 255 191"
			if `groupCount' == 3 return local color "43 123 182"
		}
		else if `totalNumGroups' == 4 {

			if `groupCount' == 1 return local color "215 25 28"
			if `groupCount' == 2 return local color "255 255 191"
			if `groupCount' == 3 return local color "171 217 233"
			if `groupCount' == 4 return local color "43 123 182"
		}
		else {

			*For five or more colors we repeat the same pattern

			local colourNum = mod(`groupCount', 5)
			if `colourNum' == 1 return local color "215 25 28"
			if `colourNum' == 2 return local color "253 174 93"
			if `colourNum' == 3 return local color "255 255 191"
			if `colourNum' == 4 return local color "171 217 233"
			if `colourNum' == 0 return local color "43 123 182"

		}

	end

	*******************************************
	*******************************************
		******* Greyscale Option *******
		******* Colour Picker    *******
	*******************************************
	*******************************************

	cap	program drop	greyPicker
		program define 	greyPicker , rclass

		args groupCount totalNumGroups

		if `groupCount' == 1 {

			return local color "black"
		}

		else if `groupCount' == 2 & `totalNumGroups' <= 3 {

			return local color "gs14"
		}
		else {

			local grayscale =  round( (`groupCount'-1) * (100 / (`totalNumGroups'-1) ))

			return local color "`grayscale' `grayscale' `grayscale' `grayscale'"
		}

	end

	*******************************************
	*******************************************
		******* Test if valid  *******
		******* dummies 	   *******
	*******************************************
	*******************************************

	cap program drop 	testDums
		program define	testDums

		unab dumlist : `0'

		foreach dumvar of varlist `dumlist' {

			*Test: all values dummies (missing would have been excluded in regression and we keep if e(sample)
			cap assert inlist(`dumvar',0,1)
			if _rc {
				noi display as error "{phang} The variable `dumvar' is not a dummy. Treatment variable needs to be a dummy (0 or 1) variable. {p_end}"
				noi display ""
				error 149
			}
		}

		/*What we are testing for below:
			- We count the number of dummies that each obervation has the value 1 for.
			- The count numbers must either fit the case of diff-in-diff or the case of regression with one dummy for each treatment arms

			Regular regression with one dummy for each treatment arm
			- Some observations don't have 1 for any dummy - omitted control observations
			- No observation has the value 1 in more than one observation - can't be in more than one treatment group
			- No traetment group can have no observation with value 1 for that dummy

			Diff-in-Diff
			- Some observations don't have 1 for any dummy - omitted controls observations in time = 0
			- Some observation must have value 1 for only the treatment dummy - treatment observations in time = 0
			- Some observation must have value 1 for only the time dummy - control observations in time = 1
			- Some observation must have value 1 for in all three of time, treatment and interaction dummy - treatment observations in time = 1

		*/

		*Count how many dummies is 1 for each observation
		tempvar  dum_count
		egen 	`dum_count' = rowtotal(`dumlist')

		*Exactly one dummy is 1, meaning this observation is in one of the treatment arms
		count if `dum_count' == 1
		local dum_count_1 `r(N)'

		*No dummies is 1, meaning this observation is control
		count if `dum_count' == 0
		local dum_count_0 `r(N)'

		*Exactly 3 dummies are three. Only allowed in the exact case of diff-and-diff regressions
		count if `dum_count' == 3
		local dum_count_3 `r(N)'

		*Exactly 2 or more than three is never correct.
		count if `dum_count' == 2 | `dum_count' > 3
		local dum_count_2orgt3 `r(N)'

		*Test that there is at least some treatment observations
		if `dum_count_0' == 0 		noi di as error "{phang} There are no control observations. One category must be omitted and it should be the omitted category in the regression. The omitted category will be considerd the control group. See helpfile for more info. Disable this test by using option ignoredummytest.{p_end}"
		if `dum_count_0' == 0 		error 480

		*Test that there is at least some control observations (this error should be caught by dummies omitted in the regression)
		if `dum_count_1' == 0 		noi di as error "{phang} There are no treatment observations. None of the dummies have observations for which the dummy has the value 1. See helpfile for more info. Disable this test by using option ignoredummytest.{p_end}"
		if `dum_count_1' == 0 		error 480

		*Test if there are any observations that have two or more than three dummies that is 1
		if `dum_count_2orgt3' > 0 	noi di as error "{phang} There is overlap in the treatment dummies. The dummies must be mutually exclusive meaning that no observation has the value 1 in more than one treatment dummy. The exception is when you use a diff-and-diff, but this dummies is not a valid diff and diff. See helpfile for more info. Disable this test by using option ignoredummytest.{p_end}"
		if `dum_count_2orgt3' > 0 	error 480

		*After passing the previous two steps, test if there are cases that are only allowed in diff
		if `dum_count_3' > 0 {

			*Diff-and-diff must have exactly 3 dummies
			if `:list sizeof dumlist' != 3 	noi di as error "{phang} There is overlap in the treatment dummies. The dummies must be mutually exclusive meaning that no observation has the value 1 in more than one treatment dummy. The exception is when you use a diff-and-diff, but this dummies is not a valid diff and diff. See helpfile for more info. Disable this test by using option ignoredummytest.{p_end}"
			if `:list sizeof dumlist' != 3 	error 480

			* Test if valid diff-diff
			testDumsDD `dum_count' `dumlist'
		}

	end

	cap program drop 	testDumsDD
		program define	testDumsDD

		local dum_count `1'

		**Test that for only two of three dummies there are observations
		* that has only that dummy. I.e. the two that is not the
		* interaction. If the interaction is 1, all three shluld be 1.

		*Count how many dummies the condition is above applies to
		local counter 0

		*Loop over all dummies
		forvalues i = 2/4 {

			*Test the number
			count if ``i'' == 1 & `dum_count' == 1
			if `r(N)' > 0 local ++counter

		}
		*Count that exactly two dummies fullfilledthe condition
		if `counter' != 2	noi di as error "{phang} There is overlap in the treatment dummies. The dummies must be mutually exclusive meaning that no observation has the value 1 in more than one treatment dummy. The exception is when you use a diff-and-diff, but this dummies is not a valid diff and diff. See helpfile for more info. Disable this test by using option ignoredummytest.{p_end}""
		if `counter' != 2	error 480

	end

*! version 5.5 26APR2018 DIME Analytics lcardosodeandrad@worldbank.org

cap	program drop	iegraph
	program define 	iegraph, rclass

	syntax varlist, [noconfbars BASICTItle(string) save(string) ignoredummytest confbarsnone(varlist) 	///
						confintval(numlist min=1 max=1 >0 <1) VARLabels BAROPTions(string) norestore  	///
						BARLabel barlabelformat(string) GREYscale yzero *]

	if "`restore'" == "" preserve

	qui {

	version 11

	*Only keep the observations in the regressions
	keep if e(sample) == 1

	*Copy beta matrix to a regular matrix
	mat BETA = e(b)

	*Unabbriviate and varlists
	unab varlist : `varlist'

	*Testing to see if the variables used in the regressions are actual dummy variables as treatment vars need to be dummy variables.
	foreach var of local varlist {

		*Get the column number from this var
		local colnumber = colnumb(BETA,"`var'")

		*Test if this var was omitted from the regression
		if "`r(label`colnumber')'" == "(omitted)" {

			*Test if that dummy is not found in the estimation matrix
			noi di as error "{phang}Dummy variable `var' was not included in the regression, or was omitted from it.{p_end}"
			error 480
		}
	}

	foreach var of local varlist {
		*Assigning variable coefficient/standard errors/no of obs. to scalars with the name Coeff_(`variable name')
		*coeff_se_(`variable name'), obs_(`variable name').

		*Access and store the beta value (
		scalar coeff_`var' 		= _b[`var']

		*Access and store standard errors for the dummy
		scalar coeff_se_`var' 	= _se[`var']

		*Store the number of observations for this dummy
		count if `var' == 1 //Count one tmt group at the time
		scalar obs_`var' = r(N)

	}

	*Test if the list of dummies are valid
	if "`ignoredummytest'" == "" testDums `varlist'

	*Checking to see if the noconfbars option has been used and assigning 1 and 0 based
	*on that to the CONFINT_BAR variable.
	if "`confbars'" 		!= "" {
		local CONFINT_BAR 	= 0
	}
	else if "`confbars'" 	== "" {
		local CONFINT_BAR 	= 1
	}

	*Checking to see if the barlabel option has been used and assigning 1 and 0 based
	*on that to the LABEL_BAR variable.
	if "`barlabel'" 		!= "" {
		local LABEL_BAR 	= 1
	}
	else if "`barlabel'" 	== "" {
		local LABEL_BAR 	= 0
	}

	*Checking to see if the barlabelformat option has been used and assigning 1 and 0 based
	*on that to the LABEL_FORMAT variable.
	if "`barlabelformat'" 		!= "" {
		local LABEL_FORMAT 	= 1
	}
	else if "`barlabelformat'" 	== "" {
		local LABEL_FORMAT 	= 0
	}

	*Testing to see if the variables used in confbarsnone are
	*actually in the list of
	*variables used for the regression/graph.
	local varTest : list confbarsnone in varlist

	if `varTest' == 0 {
		*Error displayed if variables defined in confbarsnone aren't in the list of variables for the graph.
		noi display as error "{phang} Variables defined in confbarsnone(`confbarsnone') cannot be found in the graph variable list. {p_end}"
		noi display ""
		error 111
	}

	*Checking barlabelformat option
	if `LABEL_FORMAT' == 1 {

		* Can only be used if the bar label is displayed
		if `LABEL_BAR' == 0 {
			noi display as error "{phang} Option barlabelformat() can only be specified when option barlabel is used. {p_end}"
			error 198
		}

		* Check that specified format is valid
		if substr("`barlabelformat'",1,1) != "%" | !inlist(substr("`barlabelformat'",-1,1), "e", "f") | !regex("`barlabelformat'", "\.") {
			noi display as error "{phang} Option barlabelformat() was incorrectly specified. Only fixed and exponencial formats are currently allowed. See {help format} for more information on how to specify a variable format.{p_end}"
			error 198
		}
	}

	*Checking to see if the save option is used what is the extension related to it.
	if "`save'" != "" {

		**Find the last . in the path name and assume that
		* the file extension is what follows. However, with names
		* that have multiple dots in it, the user has to explicitly
		* specify the file name.

		**First, will extract the file names from the combination of file
		* path and files names. We will use both backslash and forward slash
		* to account for differences in Windows/Unix file paths
		local backslash = strpos(reverse("`save'"), "\")
		local forwardslash = strpos(reverse("`save'"), "/")

		** Replacing the value of forward/back slash with the other value if one of the
		*  values is equal to zero.
        if `forwardslash' == 0  local forwardslash = `backslash'
        if `backslash' == 0     local backslash = `forwardslash'

		**Extracting the file name from the full file path by  reversing and breaking the path
		* at the first occurence of slash.
		local file_name = substr(reverse("`save'"), 1, (min(`forwardslash', `backslash')-1))
		local file_name = reverse("`file_name'")

		**If no slashes it means that there is no file path and just a file name, so the name of the file will be
		* the local save.
		if (`forwardslash' == 0 & `backslash' == 0) local file_name = "`save''"

		*Assign the full file path to the local file_suffix
		local file_suffix = "`file_name'"

        *Find index for where the file type suffix start
        local dot_index     = strpos("`file_name'",".")

		local file_suffix = substr("`file_name'", `dot_index' + 1, .)

		*If no dot in the name, then no file extension
		if `dot_index' == 0 {
			local save `"`save'.gph"'
			local file_suffix "gph"
			local save_export = 0
		}

		**If there is one or many . in the file path than loop over
		* the file path until we have found the last one.

		**Find index for where the file type suffix start. We are re-checking
		* to see if there are any more dots than the first one. If there are,
		* then there needs to be an error message saying remove the dots.
		local dot_index 	= strpos("`file_suffix'",".")

		*Extract the file index

		if (`dot_index' > 0) {
			di as error "{pstd}File names cannot have more than one dot. Please only use the dot to separate the filename and file format.{p_end}"
			error 198
		}

        *List of formats to which the file can be exported
        local nonGPH_formats png tiff gph ps eps pdf wmf emf

        *If no file format suffix is specified, use the default .gph
        if "`file_suffix'" == "gph" {
			local save_export = 0
		}

        *If a file format suffix is specified make sure that it is one of the eight formats allowed.
        else if `:list file_suffix in nonGPH_formats' != 0  {
            local save_export = 1

			if ("`file_suffix'" == "wmf" | "`file_suffix'" == "emf") & "`c(os)'" != "Windows" {
				di as error "{pstd}The file formats .wmf and .emf are only allowed when using Stata on a Windows computer.{p_end}"
                error 198
			}
		}
		*If a different extension was used then displaying an error.
        else {

            di as error "{pstd}You are not using a allowed file format in save(`save'). Only the following formats are allowed: gph `nonGPH_formats'. {p_end}"
            error 198
        }
    }
	else {

		*Save option is not used, therefore save export will not be used
		local save_export = 0

	}

	local count: word count `varlist' // Counting the number of total vars used as treatment.
	local graphCount = `count' + 1 // Number of vars needed for the graph is total treatment vars plus one(control).

	//Make all vars tempvars (maybe do later)
	//Make sure that missing is properly handled

	tempvar anyTMT control
	egen `anyTMT' = rowmax(`varlist')
	gen `control' = (`anyTMT' == 0) 	if !missing(`anyTMT')

	sum `e(depvar)' if `control' == 1
	scalar ctl_N		= r(N)
	scalar ctl_mean	  	= r(mean)
	scalar ctl_mean_sd 	= r(sd)

	/**
	 Calculate t-statistics
	**/

	*If not set in options, use default of 95%
	if "`confintval'" == "" {
		local confintval = .95
	}

	**Since we calculating each tail separetely we need to convert
	* the two tail % to one tail %
	local conintval_1tail = ( `confintval' + (1-`confintval' ) / 2)

	*degreeds of freedom in regression
	local df = `e(df_r)'

	*Calculate t-stats to be used
	local tstats = invt(`df' , `conintval_1tail'  )

	foreach var of local varlist {

		*Caculating confidnece interval
		scalar  conf_int_min_`var'   =	(coeff_`var'-(`tstats'*coeff_se_`var') + ctl_mean)
		scalar  conf_int_max_`var'   =	(coeff_`var'+(`tstats'*coeff_se_`var') + ctl_mean)
		**Assigning stars to the treatment vars.

		*Perform the test to get p-values
		test `var'
		local star_`var' " "

		scalar  pvalue =r(p)
		if pvalue < 0.10 {
			local star_`var' "*"
		}
		if pvalue < 0.05 {
			local star_`var' "**"
		}
		if pvalue < 0.01 {
			local star_`var' "***"
		}

		scalar tmt_mean_`var' = ctl_mean + coeff_`var'
	}

	/*************************************

		Set up temp file where results are written

	*************************************/

	tempfile		 newTextFile
	tempname 		 newHandle
	cap file close 	`newHandle'
	file open  		`newHandle' using "`newTextFile'", text write append

	*Write headers and control value
	file write `newHandle' ///
		"position" 	_tab "xLabeled"	_tab "mean" _tab "coeff" _tab "conf_int_min" _tab "conf_int_max" _tab "obs" _tab "star" _n 	///
		%9.3f (1) _tab "Control"  _tab %9.3f (ctl_mean)      _tab  	_tab   _tab _tab %9.3f (ctl_N) 	 _tab  _n

	tempvar newCounter
	gen `newCounter' = 2  //First tmt group starts at 2 (1 is control)

	foreach var in `varlist' {

		if "`varlabels'" == "" {

			*Default is to use the varname in legend
			local var_legend "`var'"

		}
		else {

			*Option to use the variable label in the legen instead
			local var_legend : variable label `var'

		}

		*Writing the necessary tables for the graph to list to file.
	    if `: list var in confbarsnone' {
			file write `newHandle' 	%9.3f (`newCounter') _tab `"`var_legend'"'  ///
			_tab %9.3f (tmt_mean_`var') _tab  %9.3f (coeff_`var')  _tab _tab ///
			_tab %9.3f (obs_`var') _tab "`star_`var''"  _n

			replace `newCounter' = `newCounter' + 1
		}
		else {
			file write `newHandle' %9.3f (`newCounter') _tab `"`var_legend'"' ///
			_tab %9.3f (tmt_mean_`var') _tab  %9.3f (coeff_`var')  			///
			_tab %9.3f (conf_int_min_`var') _tab %9.3f (conf_int_max_`var')	///
			_tab %9.3f (obs_`var')   _tab "`star_`var''"  _n

			replace `newCounter' = `newCounter' + 1
		}
	}

	file close `newHandle'

	/*************************************

		Create the graph

	*************************************/

	*Read file with results
	insheet using `newTextFile', clear

	*Defining various options to go on the graph option.

	local tmtGroupBars 	""
	local xAxisLabels 	`"xlabel( "'
	local legendLabels	""
	local legendNumbers	""

	forval tmtGroupCount = 1/`graphCount' {

		************
		*Create the bar for this group

		if "`greyscale'" == "" {
			colorPicker `tmtGroupCount' `graphCount'
		}
		else {
			greyPicker `tmtGroupCount' `graphCount'
		}

		local tmtGroupBars `"`tmtGroupBars' (bar mean position if position == `tmtGroupCount', `baroptions' color("`r(color)'") lcolor(black) ) "'

		************
		*Create labels etc. for this group

		local obs 			= obs[`tmtGroupCount']
		local stars 		= star[`tmtGroupCount']
		local legendLabel 	= xlabeled[`tmtGroupCount']

		local xAxisLabels 	`"`xAxisLabels' `tmtGroupCount' "(N = `obs') `stars'" "'
		local legendLabels 	`"`legendLabels' lab(`tmtGroupCount' "`legendLabel'") "'
		local legendNumbers	`"`legendNumbers' `tmtGroupCount'"'
	}

	*Close or comple some strings
	local xAxisLabels `"`xAxisLabels' ,noticks labsize(medsmall)) "'
	local legendOption `"legend(order(`legendNumbers') `legendLabels')"'

	*Create the confidence interval bars

	if `CONFINT_BAR' == 0 {

		local confIntGraph = ""
	}
		else if `CONFINT_BAR' == 1 {
		local confIntGraph = `"(rcap conf_int_max conf_int_min position, lc(gs)) (scatter mean position,  msym(none)  mlabs(medium) mlabpos(10) mlabcolor(black))"'
	}

	*Create the bar label
	if `LABEL_BAR' == 0 {
		local barLabel = ""
	}
	else if `LABEL_BAR' == 1 {

		gen label = mean

		if `LABEL_FORMAT' == 1 {
			format label `barlabelformat'
		}
		else if `LABEL_FORMAT' == 0 {
			format label %9.1f
		}

		local barLabel = `"(scatter mean position,  msym(none)  mlab(label) mlabpos(12) mlabcolor(black))"'
	}

	local titleOption `" , xtitle("") ytitle("`e(depvar)'") "'

	if "`save'" != "" {
		local saveOption saving("`save'", replace)
	}

	*******************************************************************************
	*** Generating the graph axis labels for the y-zero option used..
	*******************************************************************************

	*Calculations needed if yzero used
	if  ("`yzero'" != "" ) {

		**Testing if yzero is applicable
		********************************

		**Yzero is only applicable if all values used in the graph
		* are all negative or all postive. If there is a mix, then
		* the yzero option will be ignored

		*Finding the min value for all values used in the graph
		gen row_minvalue = min(mean, conf_int_min, conf_int_max)
		sum row_minvalue
		local min_value `r(min)'

		*Finding the min value for all values used in the graph
		gen row_maxvalue = max(mean , conf_int_max, conf_int_min)
		sum row_maxvalue
		local max_value `r(max)'

		*Locals used for logic below
		noi di "local signcheck = ((`r(max)' * `r(min)') >= 0) "
		local signcheck = ((`max_value' * `min_value') >= 0) 	// dummy local for both signs the same (positive or negative)
		local negative	=  (`max_value' <= 0)				// dummy for max value still negative (including 0)

		**If yzero() is used and min and max does not have
		* the same sign, then the yzero() is not applicable.

		if (`signcheck' == 0 ) {

		**** yzero is NOT applicable and will be ignored
		*************************************************

			noi di "{pstd}{error:WARNING:} Option yzero will be ignored as the graph has values both on the the positve and negative part of the y-axis. This only affects formatting of the graph. See helpfile for more details.{p_end}"
		}
		else {

		**** yzero is applicable and will be used
		*****************************************

			*Get max value if only postive values
			if (`negative' == 0) {

				sum row_maxvalue
				local absMax = `max_value'
			}

			*Get absolute min (will convert back below) if only negative values
			else {

				sum row_minvalue
				local absMax = abs(`min_value')
			}

			*Rounded up to the nearest power of ten
			local logAbsMax = ceil(log10(`absMax'))
			local absMax = 10 ^ (`logAbsMax')

			*Generating quarter value for y-axis markers.
			local quarter = (`absMax') / 4

			**Constuct the option to be applied to
			* the graph using the values calculated
			if (`negative' == 0)  {

				local yzero_option ylabel(0(`quarter')`absMax')
			}
			else {

				local absMax = `absMax' * (-1) //Convert back to negative
				local yzero_option ylabel(`absMax'(`quarter')0)
			}
		}
	}

	*******************************************************************************
	***Graph generation based on if the option save has a export or a save feature.
	*******************************************************************************

	*Store all the options in one local
	local commandline 		`" `tmtGroupBars' `confIntGraph' `barLabel' `titleOption'  `legendOption' `xAxisLabels' title("`basictitle'") `yzero_option' `options'  "'

	*Error message used in both save-option cases below.
	local graphErrorMessage `" Something went wrong while trying to generate the graph. Click {stata di r(cmd) :display graph options } to see what graph options iegraph used. This can help in locating the source of the error in the command. "'

	if `save_export' == 0 {

		*Generate a return local with the code that will be used to generate the graph
		return local cmd `"graph twoway `commandline' `saveOption'"'

		*Generate the graph
		cap graph twoway `commandline' `saveOption'

		*If error, provide error message and then run the code again allowing the program to crash
		if _rc {

			di as error "{pstd}`graphErrorMessage'{p_end}"
            graph twoway `commandline' `saveOption'
		}
	}
	else if `save_export' == 1 {

		*Generate a return local with the code that will be used to generate the graph
		return local cmd `"graph twoway `commandline'"'

		*Generate the graph
		cap graph twoway `commandline'

		*If error, provide error message and then run the code again allowing the program to crash
		if _rc {

			di as error "{pstd}`graphErrorMessage'{p_end}"
            graph twoway `commandline'
		}

		*Export graph to preferred option
		graph export "`save'", replace

	}

	if "`restore'" == "" restore
}

end
	*******************************************
	*******************************************
		******* To pick colours based *******
		******* on the number of vars *******
	*******************************************
	*******************************************
	cap	program drop	colorPicker
		program define 	colorPicker , rclass

		args groupCount totalNumGroups

		if `totalNumGroups' == 2 {

			if `groupCount' == 1 return local color "215 25 28"
			if `groupCount' == 2 return local color "43 123 182"
		}
		else if `totalNumGroups' == 3 {

			if `groupCount' == 1 return local color "215 25 28"
			if `groupCount' == 2 return local color "255 255 191"
			if `groupCount' == 3 return local color "43 123 182"
		}
		else if `totalNumGroups' == 4 {

			if `groupCount' == 1 return local color "215 25 28"
			if `groupCount' == 2 return local color "255 255 191"
			if `groupCount' == 3 return local color "171 217 233"
			if `groupCount' == 4 return local color "43 123 182"
		}
		else {

			*For five or more colors we repeat the same pattern

			local colourNum = mod(`groupCount', 5)
			if `colourNum' == 1 return local color "215 25 28"
			if `colourNum' == 2 return local color "253 174 93"
			if `colourNum' == 3 return local color "255 255 191"
			if `colourNum' == 4 return local color "171 217 233"
			if `colourNum' == 0 return local color "43 123 182"

		}

	end

	*******************************************
	*******************************************
		******* Greyscale Option *******
		******* Colour Picker    *******
	*******************************************
	*******************************************

	cap	program drop	greyPicker
		program define 	greyPicker , rclass

		args groupCount totalNumGroups

		if `groupCount' == 1 {

			return local color "black"
		}

		else if `groupCount' == 2 & `totalNumGroups' <= 3 {

			return local color "gs14"
		}
		else {

			local grayscale =  round( (`groupCount'-1) * (100 / (`totalNumGroups'-1) ))

			return local color "`grayscale' `grayscale' `grayscale' `grayscale'"
		}

	end

	*******************************************
	*******************************************
		******* Test if valid  *******
		******* dummies 	   *******
	*******************************************
	*******************************************

	cap program drop 	testDums
		program define	testDums

		unab dumlist : `0'

		foreach dumvar of varlist `dumlist' {

			*Test: all values dummies (missing would have been excluded in regression and we keep if e(sample)
			cap assert inlist(`dumvar',0,1)
			if _rc {
				noi display as error "{phang} The variable `dumvar' is not a dummy. Treatment variable needs to be a dummy (0 or 1) variable. {p_end}"
				noi display ""
				error 149
			}
		}

		/*What we are testing for below:
			- We count the number of dummies that each obervation has the value 1 for.
			- The count numbers must either fit the case of diff-in-diff or the case of regression with one dummy for each treatment arms

			Regular regression with one dummy for each treatment arm
			- Some observations don't have 1 for any dummy - omitted control observations
			- No observation has the value 1 in more than one observation - can't be in more than one treatment group
			- No traetment group can have no observation with value 1 for that dummy

			Diff-in-Diff
			- Some observations don't have 1 for any dummy - omitted controls observations in time = 0
			- Some observation must have value 1 for only the treatment dummy - treatment observations in time = 0
			- Some observation must have value 1 for only the time dummy - control observations in time = 1
			- Some observation must have value 1 for in all three of time, treatment and interaction dummy - treatment observations in time = 1

		*/

		*Count how many dummies is 1 for each observation
		tempvar  dum_count
		egen 	`dum_count' = rowtotal(`dumlist')

		*Exactly one dummy is 1, meaning this observation is in one of the treatment arms
		count if `dum_count' == 1
		local dum_count_1 `r(N)'

		*No dummies is 1, meaning this observation is control
		count if `dum_count' == 0
		local dum_count_0 `r(N)'

		*Exactly 3 dummies are three. Only allowed in the exact case of diff-and-diff regressions
		count if `dum_count' == 3
		local dum_count_3 `r(N)'

		*Exactly 2 or more than three is never correct.
		count if `dum_count' == 2 | `dum_count' > 3
		local dum_count_2orgt3 `r(N)'

		*Test that there is at least some treatment observations
		if `dum_count_0' == 0 		noi di as error "{phang} There are no control observations. One category must be omitted and it should be the omitted category in the regression. The omitted category will be considerd the control group. See helpfile for more info. Disable this test by using option ignoredummytest.{p_end}"
		if `dum_count_0' == 0 		error 480

		*Test that there is at least some control observations (this error should be caught by dummies omitted in the regression)
		if `dum_count_1' == 0 		noi di as error "{phang} There are no treatment observations. None of the dummies have observations for which the dummy has the value 1. See helpfile for more info. Disable this test by using option ignoredummytest.{p_end}"
		if `dum_count_1' == 0 		error 480

		*Test if there are any observations that have two or more than three dummies that is 1
		if `dum_count_2orgt3' > 0 	noi di as error "{phang} There is overlap in the treatment dummies. The dummies must be mutually exclusive meaning that no observation has the value 1 in more than one treatment dummy. The exception is when you use a diff-and-diff, but this dummies is not a valid diff and diff. See helpfile for more info. Disable this test by using option ignoredummytest.{p_end}"
		if `dum_count_2orgt3' > 0 	error 480

		*After passing the previous two steps, test if there are cases that are only allowed in diff
		if `dum_count_3' > 0 {

			*Diff-and-diff must have exactly 3 dummies
			if `:list sizeof dumlist' != 3 	noi di as error "{phang} There is overlap in the treatment dummies. The dummies must be mutually exclusive meaning that no observation has the value 1 in more than one treatment dummy. The exception is when you use a diff-and-diff, but this dummies is not a valid diff and diff. See helpfile for more info. Disable this test by using option ignoredummytest.{p_end}"
			if `:list sizeof dumlist' != 3 	error 480

			* Test if valid diff-diff
			testDumsDD `dum_count' `dumlist'
		}

	end

	cap program drop 	testDumsDD
		program define	testDumsDD

		local dum_count `1'

		**Test that for only two of three dummies there are observations
		* that has only that dummy. I.e. the two that is not the
		* interaction. If the interaction is 1, all three shluld be 1.

		*Count how many dummies the condition is above applies to
		local counter 0

		*Loop over all dummies
		forvalues i = 2/4 {

			*Test the number
			count if ``i'' == 1 & `dum_count' == 1
			if `r(N)' > 0 local ++counter

		}
		*Count that exactly two dummies fullfilledthe condition
		if `counter' != 2	noi di as error "{phang} There is overlap in the treatment dummies. The dummies must be mutually exclusive meaning that no observation has the value 1 in more than one treatment dummy. The exception is when you use a diff-and-diff, but this dummies is not a valid diff and diff. See helpfile for more info. Disable this test by using option ignoredummytest.{p_end}""
		if `counter' != 2	error 480

	end


REVERT rowlab one cmd

REVERT rowlab one cmd

cap program drop 	ieddtable
	program define	ieddtable

	syntax varlist, ///
					///
		t(varname numeric) tmt(varname numeric) 			///
		[ 													///
		COVARiates(varlist numeric)							///
		STARLevels(numlist descending min=3 max=3 >0 <1)	///
		STARSNOadd											///
		firstdiff											///
		rowlabtype(string) 									///
		rowlabtext(string)									///
		]

	/*************

		Initiate the result matrix

	*************/

	*Creates the template for the result matrix in a subfunction
	templateResultMatrix
	mat startRow = r(startRow)

	*Remove this when ready for production
	noi di "Start row to see headers, remove for production"
	matlist startRow // See the default row with its column names

	*Initiate the result matrix with a place holder row that will not be used for anything. Matrices cannot be initiated empty
	mat resultMat = startRow
	mat rownames resultMat = placeholder

	/*************

		Loop over all variables and prepare the data

	*************/

	foreach var of local varlist {

		*Each variable is a row in the result matrix
		mat `var' = startRow
		mat rownames `var' = `var'
		mat colnames `var' = `colnames'

		*Local that keeps track of which column to fill
		local colindex 0

		tempvar regsample

		/*************

			Calculating 2nd difference and restict the sample
			for this output var

		*************/

		*Run the regression to get the double difference
		qui reg `var' `tmt'#`t' `covariates'
		mat resTable = r(table)

		**This is why this is done first. All other calculations
		* for this outcome var should be restricted to this sample.
		gen `regsample' = e(sample)
		local N `e(N)'

		**Test that the dummy vars creates for valid groups for each
		* combination of time and treat in the sample used for this outcome var
		testDDdums `t' `tmt' `regsample' `var'  //comment in when this is made into a command

		//Get the second differnce
		local ++colindex
		mat `var'[1,`colindex'] =  el(resTable,1,4)

		*Get the standard error of second difference
		local ++colindex
		mat `var'[1,`colindex'] =  el(resTable,2,4)

		matlist resTable

		*Get the number of stars using sub-command countStars
		local ++colindex
		local pvalue = el(resTable,4,4)
		countStars `pvalue' `starlevels'
		mat `var'[1,`colindex'] = `r(stars)'

		*Get the N of second difference regression
		local ++colindex
		mat `var'[1,`colindex'] = `N'

		/*************

			Calculating 1st differences

		*************/
		forvalues tmt01 = 0/1 {

			*Regress time against the outcome var one tmt group at the time
			qui reg `var' `t' `covariates' if `tmt' == `tmt01' & `regsample' == 1
			mat resTable = r(table)

			//Get the 1st diff
			local ++colindex
			mat `var'[1,`colindex'] =  el(resTable,1,1)

			*Get the standard error of 1st diff
			local ++colindex
			mat `var'[1,`colindex'] =  el(resTable,2,1)

			*Get the number of stars using sub-command countStars
			local ++colindex
			local pvalue = el(resTable,4,4)
			countStars `pvalue' `starlevels'
			mat `var'[1,`colindex'] = `r(stars)'

			*Get the N of first difference regression
			local ++colindex
			mat `var'[1,`colindex'] =  `e(N)'

		}

		/*************

			Calculating standard means for all groups

		*************/
		forvalues tmt01 = 0/1 {
			forvalues t01 = 0/1 {

				*Summary stats on this group
				qui mean `var' if `tmt' == `t01' & `t' == `tmt01' & `regsample' == 1

				mat resTable = r(table)

				//Get the mean
				local ++colindex
				mat `var'[1,`colindex'] =  el(resTable,1,1)

				*Get the standard error of the mean
				local ++colindex
				mat `var'[1,`colindex'] =  el(resTable,2,1)

				*Get the N of the ordinary means regressions
				local ++colindex
				mat `var'[1,`colindex'] =  `e(N)'
			}
		}

		*Append this row to the result table
		mat 	resultMat = (resultMat \ `var')

	}

	*Remove placeholder row
	matrix resultMat = resultMat[2..., 1...]

	*Show the final matrix will all data needed to start building the output
	noi di "Matlist with results"
	matlist resultMat

	/*************

		Call subcommands that prepares labels in the table

	*************/

	prepRowLabels `varlist', rowlabtype("`rowlabtype'") rowlabtext("`rowlabtext'")
	return list
	di "`r(rowlabels)'"

end

/***************************************
****************************************

	Write sub-commands for outputs

****************************************
***************************************/

	******
	*Then the result matrix can be passed into subcommands that output in either Excel, LaTeX or in the main window.

	*The results can be accessed like this, which makes the sub-commands less sensitive to changing column order in the section above.
	//mat A = resultMat[3, "C0_Mean"] // returns a 1x1 matrix with the baseline mean for the countrol grop for the thrid outcome var
	//local a = el(A,1,1)				// returns the value

	//matlist A
	//di `a'

/***************************************
****************************************

	Write sub-commands for input handling

****************************************
***************************************/
<<<<<<< HEAD

**Program to prepare row labels for each outcome var, using variable name,
* variable label or manually entered labels depedning on user input. Variable
* names are used if no input.
=======

>>>>>>> parent of 5cd8a56... combine rowlabels into one option
cap program drop 	prepRowLabels
	program define	prepRowLabels, rclass

	syntax varlist, [rowlabtype(string) rowlabtext(string)]

	*First test input
	if "`rowlabtype'" == "varlab" | "`rowlabtype'" == "" | "`rowlabtype'" == "varname" {
		//All is good do nothing
	}
	else {
		noi display as error "{phang}Row label type [`rowlabtype'] is not a valid row label type. Enter either {it:varlab} or {it:varname} (the default if not specified is {it:varname}). See option {help ieddtable:rowlabtype} for details."
		error 198
	}

	/*************
		Parse through the manually entered labels
	*************/

	*Create a local with the rowlabel input to be tokenized
	local row_labels_to_tokenize `rowlabtext'

	while "`row_labels_to_tokenize'" != "" {

		*Parsing name and label pair
		gettoken nameAndLabel row_labels_to_tokenize : row_labels_to_tokenize, parse("@@")

		*Splitting name and label
		gettoken name label : nameAndLabel

		/*
			We store two locals, one with varnames and one with label. They are
			stored in the same order, so if we find varname we know we will find
			the corresponding label in the same order in the other local.
		*/

		*** Tests

		*Checking that the variables used in rowlabels() are included in the table
		local name_correct : list name in varlist
		if `name_correct' == 0 {
			noi display as error "{phang}Variable [`name'] listed in rowlabtext(`rowlabtext') is not found among the outcome variables."
			error 111
		}

		*Testing that no label is missing
		if "`label'" == "" {
			noi display as error "{phang}For variable [`name'] listed in rowlabtext(`rowlabels') you have not specified any label. Labels are requried for all variables listed in rowlabels(). Variables omitted from rowlabtext() will be assigned labels according to the rule in rowlabtype(). See also option {help ieddtable:rowlabtext}"
			error 198
		}

		*Storing the name in local to be used to get index of corresponding label in the second local
		local rowLabelNames `"`rowLabelNames' "`name'" "'

		*Removing leading or trailing spaces and store label with the same index as in the name local above
		local label = trim("`label'")
		local rowLabelLabels `"`rowLabelLabels' "`label'" "'

		*Parse char is not removed by gettoken, so remove it and start over
		local row_labels_to_tokenize = subinstr("`row_labels_to_tokenize'" ,"@@","",1)
	}

	**Set up locals to be returned. One with row labels without varname but in
	* final order, and one local with length of longest label
	local preppedRowLabels
	local maxLen 0

	*Loop over all varaibles and prepare the labels
	foreach var of local varlist {

		**Get the index of this variable in manually entered labels. Index is
		* zero if no manually entered label for this variable
		local rowLabPos : list posof "`var'" in rowLabelNames

		if `rowLabPos' == 0 {

			*No label was manually entered for this variable, use the other rules
			if "`rowlabtype'" == "varlab" {
				local this_label : variable label `var'		//Use variable label
			}
			else {
<<<<<<< HEAD
				*Since name is the default it should always be name if it is not label, but in case there is some corner case bug
				noi display as error `"{phang}This error is never supposed to happen, please report this on {browse "https://www.github.com/worldbank/ietoolkit/issues"} or contact kbjarkefur@worldbank.org.{p_end}"'
				error 198
=======
				local this_label `var'						//Default, use varname
>>>>>>> parent of 5cd8a56... combine rowlabels into one option
			}
		}
		else {
			*Getting the manually defined label corresponding to this variable
			local this_label : word `rowLabPos' of `rowLabelLabels'
		}

		*Update the two locals that will be returned
		local preppedRowLabels "`preppedRowLabels' @@`this_label'"	//Prepare the final list
		local maxLen = max(strlen("`this_label'"),`maxLen')	//Store the length on the longest label, used on some outputs
	}

	*Return the locals
	return local rowlabels `preppedRowLabels'
	return local rowlabel_maxlen `maxLen'

end

cap program drop 	testDDdums
	program define	testDDdums

		args time treat samplevar outputvar

		*Test taht the dummies are dummies
		foreach dummy in `time' `treat' {

			cap assert inlist(`dummy',1,0) | missing(`dummy') | `samplevar' == 0
			if _rc {
				noi di as error "{phang}In the difference in differnce regression for outputvar [`outputvar'], the dummy [`dummy'] is not a real dummy as it has values differnt from 1 and 0. See tab below. The tab is restricted to the observations that are not excluded due to missing values etc. in the difference in differnce regression.{p_end}""
				noi tab `dummy' if `samplevar' == 1, missing
				error 480
			}
		}

		**Test that for only two of three dummies there are observations
		* that has only that dummy. I.e. the two that is not the
		* interaction. If the interaction is 1, all three shluld be 1.

		*Test that there is some observations in each group
		forvalues tmt01 = 0/1 {
			forvalues t01 = 0/1 {

				*Summary stats on this group
				count if `treat' == `t01' & `time' == `tmt01' & `samplevar' == 1

				//There got to be more than 1 observation in each group for the regression to make sense
				if `r(N)' < 2 {
					noi di as error "{phang}In the difference in differnce regression for outputvar [`outputvar'] there were not enough observations in each combination of treatment [`treat'] and time [`time']. The tab below is restricted to the observations that are not excluded due to missing values etc. in the difference in differnce regression.{p_end}""
					noi tab `treat' `time' if `samplevar' == 1, missing
					error 480

				}
			}
		}

	end

/***************************************
****************************************

	Write sub-commands for stats

****************************************
***************************************/

cap program drop 	templateResultMatrix
	program define	templateResultMatrix, rclass

	local 2ndDiff_cols 			2D 2D_err 2D_Stars 2D_N

	local 1stDiff_C_cols		1DC 1DC_err 1DC_stars 1DC_N
	local 1stDiff_T_cols		1DT 1DT_err 1DT_stars 1DT_N

	local basicMean_C0_cols		C0_mean C0_err C0_N
	local basicMean_T0_cols		T0_mean T0_err T0_N
	local basicMean_C1_cols		C1_mean C1_err C1_N
	local basicMean_T1_cols		T1_mean T1_err T1_N

	local colnames `2ndDiff_cols' `1stDiff_C_cols' `1stDiff_T_cols' `basicMean_C0_cols' `basicMean_T0_cols' `basicMean_C1_cols' `basicMean_T0_cols'

	*Define default row here. The results for each var will be one row that starts with all missing vlaues
	mat startRow = (.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.)
	mat colnames startRow = `colnames'

	return matrix startRow startRow

end

cap program drop 	countStars
	program define	countStars, rclass

	args pvalue star1 star2 star3

	*Test if custom star levels are used
	if "`star1'" == "" {

		*Set default levels for 1, 2 and 3 stars
		local star1 .1
		local star2 .05
		local star3 .01
	}

	local stars 0
	foreach star_p_level in `star1' `star2' `star3' {

		if `pvalue' < `star_p_level' local ++stars
	}

	return local stars `stars'

end

cap program drop 	ieddtable
	program define	ieddtable

	syntax varlist , 										///
															///
		t(varname numeric) tmt(varname numeric) 			///
		[ 													///
		COVARiates(varlist numeric)							///
		STARLevels(numlist descending min=3 max=3 >0 <1)	///
		STARSNOadd											///
		firstdiff											///
		ROWLabels(string) 									///
		]

	/***************************************
	****************************************

		Input testing

	****************************************
	***************************************/

	/*************

		Call subcommands that prepares labels in the table

	*************/

	prepRowLabels `varlist', rowlabels("`rowlabels'")

	/***************************************
	****************************************

		Calculate results and prepare
		result matrix

	****************************************
	***************************************/

	/*************

		Initiate the result matrix

	*************/

	*Creates the template for the result matrix in a subfunction
	templateResultMatrix
	mat startRow = r(startRow)

	*Remove this when ready for production
	noi di "Start row to see headers, remove for production"
	matlist startRow // See the default row with its column names

	*Initiate the result matrix with a place holder row that will not be used for anything. Matrices cannot be initiated empty
	mat resultMat = startRow
	mat rownames resultMat = placeholder

	/*************

		Loop over all variables and prepare the data

	*************/

	foreach var of local varlist {

		*Each variable is a row in the result matrix
		mat `var' = startRow
		mat rownames `var' = `var'
		mat colnames `var' = `colnames'

		*Local that keeps track of which column to fill
		local colindex 0

		tempvar regsample

		/*************

			Calculating 2nd difference and restict the sample
			for this output var

		*************/

		*Run the regression to get the double difference
		qui reg `var' `tmt'#`t' `covariates'
		mat resTable = r(table)

		**This is why this is done first. All other calculations
		* for this outcome var should be restricted to this sample.
		gen `regsample' = e(sample)
		local N `e(N)'

		**Test that the dummy vars creates for valid groups for each
		* combination of time and treat in the sample used for this outcome var
		testDDdums `t' `tmt' `regsample' `var'  //comment in when this is made into a command

		//Get the second differnce
		local ++colindex
		mat `var'[1,`colindex'] =  el(resTable,1,4)

		*Get the standard error of second difference
		local ++colindex
		mat `var'[1,`colindex'] =  el(resTable,2,4)

		//matlist resTable

		*Get the number of stars using sub-command countStars
		local ++colindex
		local pvalue = el(resTable,4,4)
		countStars `pvalue' `starlevels'
		mat `var'[1,`colindex'] = `r(stars)'

		*Get the N of second difference regression
		local ++colindex
		mat `var'[1,`colindex'] = `N'

		/*************

			Calculating 1st differences

		*************/
		forvalues tmt01 = 0/1 {

			*Regress time against the outcome var one tmt group at the time
			qui reg `var' `t' `covariates' if `tmt' == `tmt01' & `regsample' == 1
			mat resTable = r(table)

			//Get the 1st diff
			local ++colindex
			mat `var'[1,`colindex'] =  el(resTable,1,1)

			*Get the standard error of 1st diff
			local ++colindex
			mat `var'[1,`colindex'] =  el(resTable,2,1)

			*Get the number of stars using sub-command countStars
			local ++colindex
			local pvalue = el(resTable,4,4)
			countStars `pvalue' `starlevels'
			mat `var'[1,`colindex'] = `r(stars)'

			*Get the N of first difference regression
			local ++colindex
			mat `var'[1,`colindex'] =  `e(N)'

		}

		/*************

			Calculating standard means for all groups

		*************/
		forvalues tmt01 = 0/1 {
			forvalues t01 = 0/1 {

				*Summary stats on this group
				qui mean `var' if `tmt' == `t01' & `t' == `tmt01' & `regsample' == 1

				mat resTable = r(table)

				//Get the mean
				local ++colindex
				mat `var'[1,`colindex'] =  el(resTable,1,1)

				*Get the standard error of the mean
				local ++colindex
				mat `var'[1,`colindex'] =  el(resTable,2,1)

				*Get the N of the ordinary means regressions
				local ++colindex
				mat `var'[1,`colindex'] =  `e(N)'
			}
		}

		*Append this row to the result table
		mat 	resultMat = (resultMat \ `var')

	}

	*Remove placeholder row
	matrix resultMat = resultMat[2..., 1...]

	*Show the final matrix will all data needed to start building the output
	noi di "Matlist with results"
	matlist resultMat

end

/***************************************
****************************************

	Write sub-commands for outputs

****************************************
***************************************/

	******
	*Then the result matrix can be passed into subcommands that output in either Excel, LaTeX or in the main window.

	*The results can be accessed like this, which makes the sub-commands less sensitive to changing column order in the section above.
	//mat A = resultMat[3, "C0_Mean"] // returns a 1x1 matrix with the baseline mean for the countrol grop for the thrid outcome var
	//local a = el(A,1,1)				// returns the value

	//matlist A
	//di `a'

/***************************************
****************************************

	Write sub-commands for input handling

****************************************
***************************************/

**Program to prepare row labels for each outcome var, using variable name,
* variable label or manually entered labels depedning on user input. Variable
* names are used if no input.
cap program drop 	prepRowLabels
	program define	prepRowLabels, rclass

	syntax varlist, [rowlabels(string)]

	*Run subcommand that parse the sub-options in rowlabels
	local rowlabels = subinstr(`"`rowlabels'"', ",", "", 1)
	testRowlabelsinput ,`rowlabels'
	local rowlabeltype 	`s(rowlabeltype)'
	local rowlabeltext	`s(rowlabeltext)'

	/*************
		Parse through the manually entered labels
	*************/

	*Create a local with the rowlabel input to be tokenized
	local row_labels_to_tokenize `rowlabeltext'

	while "`row_labels_to_tokenize'" != "" {

		*Parsing name and label pair
		gettoken nameAndLabel row_labels_to_tokenize : row_labels_to_tokenize, parse("@@")

		*Splitting name and label
		gettoken name label : nameAndLabel

		/*
			We store two locals, one with varnames and one with label. They are
			stored in the same order, so if we find varname we know we will find
			the corresponding label in the same order in the other local.
		*/

		*** Tests

		*Checking that the variables used in rowlabels() are included in the table
		local name_correct : list name in varlist
		if `name_correct' == 0 {
			noi display as error "{phang}Variable [`name'] listed in rowlabels(`rowlabtext') is not found among the outcome variables. See option {help ieddtable:rowlabels} for more details{p_end}"
			error 111
		}

		*Testing that no label is missing
		if "`label'" == "" {
			noi display as error "{phang}For variable [`name'] listed in rowlabels(`rowlabels') you have not specified any label. Labels are requried for all variables listed in rowlabels(,text(`rowlabeltext')). See option {help ieddtable:rowlabels} for more details and for explination on what happens to variables not listed in rowlabels(,text()){p_end}"
			error 198
		}

		*Storing the name in local to be used to get index of corresponding label in the second local
		local rowLabelNames `"`rowLabelNames' "`name'" "'

		*Removing leading or trailing spaces and store label with the same index as in the name local above
		local label = trim("`label'")
		local rowLabelLabels `"`rowLabelLabels' "`label'" "'

		*Parse char is not removed by gettoken, so remove it and start over
		local row_labels_to_tokenize = subinstr("`row_labels_to_tokenize'" ,"@@","",1)
	}

	**Set up locals to be returned. One with row labels without varname but in
	* final order, and one local with length of longest label
	local preppedRowLabels
	local maxLen 0

	*Loop over all varaibles and prepare the labels
	foreach var of local varlist {

		**Get the index of this variable in manually entered labels. Index is
		* zero if no manually entered label for this variable
		local rowLabPos : list posof "`var'" in rowLabelNames

		if `rowLabPos' == 0 {
			di ":: `rowlabeltype'"
			*No label was manually entered for this variable, use the other rules
			if "`rowlabeltype'" == "label" {
				local this_label : variable label `var'		//Use variable label
			}
			else if "`rowlabeltype'" == "name" {
				local this_label `var'						//Default, use varname
			}
			else {
				*Since name is the default it should always be name if it is not label, but in case there is some corner case bug
				noi display as error `"{phang}This error is never supposed to happen, please report this on {browse "https://www.github.com/worldbank/ietoolkit/issues"} or contact kbjarkefur@worldbank.org.{p_end}"'
				error 198
			}
		}
		else {
			*Getting the manually defined label corresponding to this variable
			local this_label : word `rowLabPos' of `rowLabelLabels'
		}

		*Update the two locals that will be returned
		local preppedRowLabels "`preppedRowLabels' @@`this_label'"	//Prepare the final list
		local maxLen = max(strlen("`this_label'"),`maxLen')	//Store the length on the longest label, used on some outputs
	}

	*Return the locals
	return local rowlabels `preppedRowLabels'
	return local rowlabel_maxlen `maxLen'

end

*Parses and test the sub-options inside the option rowlabels
cap program drop 	testRowlabelsinput
	program define	testRowlabelsinput, sclass

		syntax  , [name label text(string)]

		*Test that both name and label are not used at the same time
		if "`name'" != "" & "`label'" != "" {

			noi display as error "{phang}You may not use both label and name in the option rowlabel(`rowlabel'). See option {help ieddtable:rowlabels} for details."
			error 198
		}

		*If neither is used, use name as default
		if "`name'`label'" == "" {
			local name "name"
		}

		sreturn local rowlabeltype "`name'`label'"
		sreturn local rowlabeltext "`text'"

	end

cap program drop 	testDDdums
	program define	testDDdums

		args time treat samplevar outputvar

		*Test taht the dummies are dummies
		foreach dummy in `time' `treat' {

			cap assert inlist(`dummy',1,0) | missing(`dummy') | `samplevar' == 0
			if _rc {
				noi di as error "{phang}In the difference in differnce regression for outputvar [`outputvar'], the dummy [`dummy'] is not a real dummy as it has values differnt from 1 and 0. See tab below. The tab is restricted to the observations that are not excluded due to missing values etc. in the difference in differnce regression.{p_end}""
				noi tab `dummy' if `samplevar' == 1, missing
				error 480
			}
		}

		**Test that for only two of three dummies there are observations
		* that has only that dummy. I.e. the two that is not the
		* interaction. If the interaction is 1, all three shluld be 1.

		*Test that there is some observations in each group
		forvalues tmt01 = 0/1 {
			forvalues t01 = 0/1 {

				*Summary stats on this group
				count if `treat' == `t01' & `time' == `tmt01' & `samplevar' == 1

				//There got to be more than 1 observation in each group for the regression to make sense
				if `r(N)' < 2 {
					noi di as error "{phang}In the difference in differnce regression for outputvar [`outputvar'] there were not enough observations in each combination of treatment [`treat'] and time [`time']. The tab below is restricted to the observations that are not excluded due to missing values etc. in the difference in differnce regression.{p_end}""
					noi tab `treat' `time' if `samplevar' == 1, missing
					error 480

				}
			}
		}

	end

/***************************************
****************************************

	Write sub-commands for stats

****************************************
***************************************/

cap program drop 	templateResultMatrix
	program define	templateResultMatrix, rclass

	local 2ndDiff_cols 			2D 2D_err 2D_Stars 2D_N

	local 1stDiff_C_cols		1DC 1DC_err 1DC_stars 1DC_N
	local 1stDiff_T_cols		1DT 1DT_err 1DT_stars 1DT_N

	local basicMean_C0_cols		C0_mean C0_err C0_N
	local basicMean_T0_cols		T0_mean T0_err T0_N
	local basicMean_C1_cols		C1_mean C1_err C1_N
	local basicMean_T1_cols		T1_mean T1_err T1_N

	local colnames `2ndDiff_cols' `1stDiff_C_cols' `1stDiff_T_cols' `basicMean_C0_cols' `basicMean_T0_cols' `basicMean_C1_cols' `basicMean_T0_cols'

	*Define default row here. The results for each var will be one row that starts with all missing vlaues
	mat startRow = (.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.)
	mat colnames startRow = `colnames'

	return matrix startRow startRow

end

cap program drop 	countStars
	program define	countStars, rclass

	args pvalue star1 star2 star3

	*Test if custom star levels are used
	if "`star1'" == "" {

		*Set default levels for 1, 2 and 3 stars
		local star1 .1
		local star2 .05
		local star3 .01
	}

	local stars 0
	foreach star_p_level in `star1' `star2' `star3' {

		if `pvalue' < `star_p_level' local ++stars
	}

	return local stars `stars'

end


iecompdup - syntax style edit

iecompdup - syntax style edit

The syntax description was updated to say idvaraible instead of varname

{smcl}
{* 22 Jun 2016}{...}
{hline}
help for {hi:iecompdup}
{hline}

{title:Title}

{phang2}{cmdab:iecompdup} {hline 2} Compares two duplicates and generate a list
of the variables where the duplicates are identical and a list of the variables where the
duplicates differ

{title:Syntax}

{phang2}
{cmdab:iecompdup}
{it:idvariable}
, {cmdab:id(}{it:string}{cmd:)} [{cmdab:didi:fference} {cmdab:keepdiff:erence}
{cmdab:keepoth:er(}{it:varlist}{cmd:)} {cmdab:more2ok}]

{marker opts}{...}
{synoptset 28}{...}
{synopthdr:options}
{synoptline}
{synopt :{cmdab:id(}{it:string}{cmd:)}}value in the ID variables that is duplicated{p_end}
{synopt :{cmdab:didi:fference}}outputs the list with the variables for which the two observation differs. The default is to only store them in a local{p_end}
{synopt :{cmdab:keepdiff:erence}}drops all but the variables for which the two observations differ{p_end}
{synopt :{cmdab:keepoth:er(}{it:varlist}{cmd:)}}used together with {cmdab:keepdifference}. Variables included in {it:varlist} are also kept{p_end}
{synopt :{cmdab:more2ok}}allows running the command on groups of more than two observations, although only the first two duplicates (in the order the data is sorted) are compared{p_end}
{synoptline}

{title:Description}

{pstd}{cmdab:iecompdup} compare all variables for observations that are duplicates in
ID variable {it:varname} and the duplicated value is {cmdab:id(}{it:string}{cmd:)}. Duplicates can
be identified and corrected with its sister command {help ieduplicates}. {cmdab:iecompdup}
is intended to assist in the process of investigating why two observations are duplicates in
the ID variable, and what correction is appropriate.

{pstd}{cmdab:iecompdup} returns two locals {cmd:r(matchvars)} and {cmd:r(diffvars)}. {cmd:r(matchvars)} returns
a list of the names of all variables for which the two observations
have identical values, unless both values are missing values or the empty
string. {cmd:r(diffvars)} returns a list of the names
of all variables where the two observations are not identical.

{pstd}For example, if a duplicate is found in a dataset downloaded from a data
collection server (ODK or similar) and the duplicates were due to redundant submissions
of the same data, then {cmd:r(diffvars)} would only include the submission time
variable and any unique key used by the server. In such case, one observation can be
dropped without risking losing information, since it is an identical submission of
the exact same observation. (See Examples section below for a more detailed suggestion
on how to use the command. )

{title:Options}

{phang}{cmdab:id(}{it:string}{cmd:)} is used to specify the ID value that the
duplicates share. Both text strings and numeric values are allowed.

{phang}{cmdab:didi:fference} is used to display the list of all variables for which
the ID variable duplicates differ. The default is to provide this list in a local, and only
display the number of variables that differ.

{phang}{cmdab:keepdiff:erence} is used to return the data set with only the ID
variable and variables that differs between the duplicates. This means that the
command would drop all variables where the duplicates are identical or both
missing. It also drops all observations but the two duplicates compared.

{phang}{cmdab:keepoth:er(}{it:varlist}{cmd:)} is used to keep more variables than the variables
that differs between the duplicates when {cmdab:keepdifference} is specified. The command can keep,
for example, a variable with information about who collected these data. This
option returns an error if it is specified not in conjunction with {cmdab:keepdifference}.

{phang}{cmdab:more2ok} allows running the command on groups of more than two observations,
although only the first two duplicates (in the order the data is sorted) are compared. In a group of three duplicates,
run the command three times on each combination of the three duplicates. A future update that
includes the possibility to compare more than one case is under consideration{p_end}

{title:Stored results}

{pstd}
{cmdab:iecompdup} stores the following results in {hi:r()}:

{synoptset 15 tabbed}{...}
{p2col 5 15 19 2: Locals}{p_end}
{synopt:{cmd:r(matchvars)}}a list of the variables where the duplicates has the same value{p_end}
{synopt:{cmd:r(diffvars)}}a list of the variables where the duplicates has different values{p_end}

{p2col 5 15 19 2: Scalars}{p_end}
{synopt:{cmd:r(nummatch)}}The number of variables in {cmd:r(matchvars)}{p_end}
{synopt:{cmd:r(numdiff)}}The number of variables in {cmd:r(matchvars)}{p_end}
{synopt:{cmd:r(numnomiss)}}The number of variables for which at least one of
the duplicates has a non-missing value. By definition, {cmd:r(numnomiss)} equals
 the sum of {cmd:r(nummatch)} and {cmd:r(numdiff)}{p_end}
{p2colreset}{...}

{title:Examples}

{pstd}
A series of examples on how to specify command, and how to evaluate output:

{pstd}{hi:Example 1.}

{phang2}{inp:iecompdup HH_ID , id(55424) didifference}{p_end}

{pmore}In the example above, let's say that there are two observations in the data set with the value 55424
for variable HH_ID. HH_ID holds an ID that was uniquely assigned to each household. Before continuing the analysis, one must
investigate why two observations were assigned the same ID. iecompdup is a great place to start.

{pmore}Specifying the command as above compares the two observations that both
have value 55424 in the ID variable. The output displayed will
only be number of non-missing variables for which the two observations have identical
values, and the number of non-missing variables for which the two observations
have different values. The list of those two sets of variables are stored as locals.
The data set is returned exactly as it was.

{pmore}The locals stored in {cmd:r(diffvars)} and {cmd:r(nummatch)} can be used
to provide information on why the two observations are duplicates. A suggested
method to evaluate these two lists are presented in Example 2 below.

{pstd}{hi:Example 2.}

{phang2}{inp:iecompdup HH_ID , id(55424) didifference}{p_end}

{pmore}This example makes the same assumptions as example 1 that there are two
observations in the data set with the value 55424 for variable HH_ID. The only
difference is that the option didifference is specified. The output is the same
as example 1 but with the addition that the list stored in {cmd:r(diffvars)} is
displayed in the output window. The data set is returned exactly as it was.

{pmore}The method to evaluate the output presented in this example focus on the
variables for which the duplicates are different. Therefore, start by looking at
the list of variables displayed by {inp:didifference}. Do the variables with different values across the duplicates
contain observation data like "number of household members" or "annual income", or are they
submission information such as "submission ID", "server key" or "submission time"?
The answer to this question could suggest one of the three solutions below. Note
that this method should only be used as a guiding rule of thumb, all suggested
solutions should be evaluated qualitatively as well.

{pmore}{ul:Solution 1. All variables contain submission information data.} The
far most common mistake leading to duplicates in household surveys is that the same
observation data is submitted to the server twice. If that is the case, then only submission
information variables would be outputted by the command, not any observation data.
If this is the case, then you can safely delete either of the observations.

{pmore}{ul:Solution 2. Most variables contain submission information data, but a few contain observation data.} If a
few observation data variables are displayed together with submission information
variables then it is likely that it is the same observation but some variables
were edited after the first submission. Follow up with your field team to see
why some variables were changed. See the tips in example 3 below before following up.

{pmore}{ul:Solution 3. Many variables contain observation data.} If many
observation data variables are displayed together with submission data variables,
then it is likely that two different observations have accidently been given the
same ID. That is especially likely if location variables or name variables are
different, or if the values for enumerator and/or supervisor are different. See the
tips in example 3 below before following up.

{pmore}The cases listed above will solve the vast majority of duplicates encountered in
household surveys. The appropriate correction can afterwards be applied using the command {help ieduplicates}.

{pstd}{hi:Example 3.}

{phang2}{inp:iecompdup HH_ID , id(55424) didifference keepdifference keepother(village enumerator supervisor)}{p_end}

{pmore}This example again makes the same assumptions as example 1 and example 2 that there are two
observations in the data set with the value 55424 for variable HH_ID. This
time {inp:keepdifference} and {inp:keepother()} are specified. Those two options
can be used to provide additional information to the field team when following up
based on solution 2 and solution 3 in example 2. {inp:keepdifference} drops all
variables apart from the ID variable and the variables in {cmd:r(diffvars)}. Any
variables in {inp:keepother()} are also kept. All observations apart from the
duplicates with the ID specified in {inp:id()} are also dropped. This data can be
exported to excel and sent to a field team that can see how the observations differ.
In this example the field team can also see in which village the data was collected,
as well as the name of the enumerator and the supervisor. Any other information
helpful to the field team can be entered in {inp:keepother()}.

{title:Acknowledgements}

{phang}I would like to acknowledge the help in testing and proofreading I received in relation to this command and help file from (in alphabetic order):{p_end}
{pmore}Michell Dong{break}Carlos Goes{break}Paula Gonzales

{title:Author}

{phang}Kristoffer Bjrkefur, The World Bank, DECIE{break}
         Please email bug-reports, suggestions and requests for clarifications
		 writing "ietools iecompdup" in the subject line to:{break}
		 kbjarkefur@worldbank.org

{smcl}
{* 22 Jun 2016}{...}
{hline}
help for {hi:iecompdup}
{hline}

{title:Title}

{phang2}{cmdab:iecompdup} {hline 2} Compares two duplicates and generate a list
of the variables where the duplicates are identical and a list of the variables where the
duplicates differ

{title:Syntax}

{phang2}
{cmdab:iecompdup}
{it:varname}
, {cmdab:id(}{it:string}{cmd:)} [{cmdab:didi:fference} {cmdab:keepdiff:erence}
{cmdab:keepoth:er(}{it:varlist}{cmd:)} {cmdab:more2ok}]

{marker opts}{...}
{synoptset 28}{...}
{synopthdr:options}
{synoptline}
{synopt :{cmdab:id(}{it:string}{cmd:)}}value in the ID variables that is duplicated{p_end}
{synopt :{cmdab:didi:fference}}outputs the list with the variables for which the two observation differs. The default is to only store them in a local{p_end}
{synopt :{cmdab:keepdiff:erence}}drops all but the variables for which the two observations differ{p_end}
{synopt :{cmdab:keepoth:er(}{it:varlist}{cmd:)}}used together with {cmdab:keepdifference}. Variables included in {it:varlist} are also kept{p_end}
{synopt :{cmdab:more2ok}}allows running the command on groups of more than two observations, although only the first two duplicates (in the order the data is sorted) are compared{p_end}
{synoptline}

{title:Description}

{pstd}{cmdab:iecompdup} compare all variables for observations that are duplicates in
ID variable {it:varname} and the duplicated value is {cmdab:id(}{it:string}{cmd:)}. Duplicates can
be identified and corrected with its sister command {help ieduplicates}. {cmdab:iecompdup}
is intended to assist in the process of investigating why two observations are duplicates in
the ID variable, and what correction is appropriate.

{pstd}{cmdab:iecompdup} returns two locals {cmd:r(matchvars)} and {cmd:r(diffvars)}. {cmd:r(matchvars)} returns
a list of the names of all variables for which the two observations
have identical values, unless both values are missing values or the empty
string. {cmd:r(diffvars)} returns a list of the names
of all variables where the two observations are not identical.

{pstd}For example, if a duplicate is found in a dataset downloaded from a data
collection server (ODK or similar) and the duplicates were due to redundant submissions
of the same data, then {cmd:r(diffvars)} would only include the submission time
variable and any unique key used by the server. In such case, one observation can be
dropped without risking losing information, since it is an identical submission of
the exact same observation. (See Examples section below for a more detailed suggestion
on how to use the command. )

{title:Options}

{phang}{cmdab:id(}{it:string}{cmd:)} is used to specify the ID value that the
duplicates share. Both text strings and numeric values are allowed.

{phang}{cmdab:didi:fference} is used to display the list of all variables for which
the ID variable duplicates differ. The default is to provide this list in a local, and only
display the number of variables that differ.

{phang}{cmdab:keepdiff:erence} is used to return the data set with only the ID
variable and variables that differs between the duplicates. This means that the
command would drop all variables where the duplicates are identical or both
missing. It also drops all observations but the two duplicates compared.

{phang}{cmdab:keepoth:er(}{it:varlist}{cmd:)} is used to keep more variables than the variables
that differs between the duplicates when {cmdab:keepdifference} is specified. The command can keep,
for example, a variable with information about who collected these data. This
option returns an error if it is specified not in conjunction with {cmdab:keepdifference}.

{phang}{cmdab:more2ok} allows running the command on groups of more than two observations,
although only the first two duplicates (in the order the data is sorted) are compared. In a group of three duplicates,
run the command three times on each combination of the three duplicates. A future update that
includes the possibility to compare more than one case is under consideration{p_end}

{title:Stored results}

{pstd}
{cmdab:iecompdup} stores the following results in {hi:r()}:

{synoptset 15 tabbed}{...}
{p2col 5 15 19 2: Locals}{p_end}
{synopt:{cmd:r(matchvars)}}a list of the variables where the duplicates has the same value{p_end}
{synopt:{cmd:r(diffvars)}}a list of the variables where the duplicates has different values{p_end}

{p2col 5 15 19 2: Scalars}{p_end}
{synopt:{cmd:r(nummatch)}}The number of variables in {cmd:r(matchvars)}{p_end}
{synopt:{cmd:r(numdiff)}}The number of variables in {cmd:r(matchvars)}{p_end}
{synopt:{cmd:r(numnomiss)}}The number of variables for which at least one of
the duplicates has a non-missing value. By definition, {cmd:r(numnomiss)} equals
 the sum of {cmd:r(nummatch)} and {cmd:r(numdiff)}{p_end}
{p2colreset}{...}

{title:Examples}

{pstd}
A series of examples on how to specify command, and how to evaluate output:

{pstd}{hi:Example 1.}

{phang2}{inp:iecompdup HH_ID , id(55424) didifference}{p_end}

{pmore}In the example above, let's say that there are two observations in the data set with the value 55424
for variable HH_ID. HH_ID holds an ID that was uniquely assigned to each household. Before continuing the analysis, one must
investigate why two observations were assigned the same ID. iecompdup is a great place to start.

{pmore}Specifying the command as above compares the two observations that both
have value 55424 in the ID variable. The output displayed will
only be number of non-missing variables for which the two observations have identical
values, and the number of non-missing variables for which the two observations
have different values. The list of those two sets of variables are stored as locals.
The data set is returned exactly as it was.

{pmore}The locals stored in {cmd:r(diffvars)} and {cmd:r(nummatch)} can be used
to provide information on why the two observations are duplicates. A suggested
method to evaluate these two lists are presented in Example 2 below.

{pstd}{hi:Example 2.}

{phang2}{inp:iecompdup HH_ID , id(55424) didifference}{p_end}

{pmore}This example makes the same assumptions as example 1 that there are two
observations in the data set with the value 55424 for variable HH_ID. The only
difference is that the option didifference is specified. The output is the same
as example 1 but with the addition that the list stored in {cmd:r(diffvars)} is
displayed in the output window. The data set is returned exactly as it was.

{pmore}The method to evaluate the output presented in this example focus on the
variables for which the duplicates are different. Therefore, start by looking at
the list of variables displayed by {inp:didifference}. Do the variables with different values across the duplicates
contain observation data like "number of household members" or "annual income", or are they
submission information such as "submission ID", "server key" or "submission time"?
The answer to this question could suggest one of the three solutions below. Note
that this method should only be used as a guiding rule of thumb, all suggested
solutions should be evaluated qualitatively as well.

{pmore}{ul:Solution 1. All variables contain submission information data.} The
far most common mistake leading to duplicates in household surveys is that the same
observation data is submitted to the server twice. If that is the case, then only submission
information variables would be outputted by the command, not any observation data.
If this is the case, then you can safely delete either of the observations.

{pmore}{ul:Solution 2. Most variables contain submission information data, but a few contain observation data.} If a
few observation data variables are displayed together with submission information
variables then it is likely that it is the same observation but some variables
were edited after the first submission. Follow up with your field team to see
why some variables were changed. See the tips in example 3 below before following up.

{pmore}{ul:Solution 3. Many variables contain observation data.} If many
observation data variables are displayed together with submission data variables,
then it is likely that two different observations have accidently been given the
same ID. That is especially likely if location variables or name variables are
different, or if the values for enumerator and/or supervisor are different. See the
tips in example 3 below before following up.

{pmore}The cases listed above will solve the vast majority of duplicates encountered in
household surveys. The appropriate correction can afterwards be applied using the command {help ieduplicates}.

{pstd}{hi:Example 3.}

{phang2}{inp:iecompdup HH_ID , id(55424) didifference keepdifference keepother(village enumerator supervisor)}{p_end}

{pmore}This example again makes the same assumptions as example 1 and example 2 that there are two
observations in the data set with the value 55424 for variable HH_ID. This
time {inp:keepdifference} and {inp:keepother()} are specified. Those two options
can be used to provide additional information to the field team when following up
based on solution 2 and solution 3 in example 2. {inp:keepdifference} drops all
variables apart from the ID variable and the variables in {cmd:r(diffvars)}. Any
variables in {inp:keepother()} are also kept. All observations apart from the
duplicates with the ID specified in {inp:id()} are also dropped. This data can be
exported to excel and sent to a field team that can see how the observations differ.
In this example the field team can also see in which village the data was collected,
as well as the name of the enumerator and the supervisor. Any other information
helpful to the field team can be entered in {inp:keepother()}.

{title:Acknowledgements}

{phang}I would like to acknowledge the help in testing and proofreading I received in relation to this command and help file from (in alphabetic order):{p_end}
{pmore}Michell Dong{break}Carlos Goes{break}Paula Gonzales

{title:Author}

{phang}Kristoffer Bjrkefur, The World Bank, DECIE{break}
         Please email bug-reports, suggestions and requests for clarifications
		 writing "ietools iecompdup" in the subject line to:{break}
		 kbjarkefur@worldbank.org