iedropone : link to wiki in help file

iedropone : link to wiki in help file

{smcl}
{* 15 Dec 2017}{...}
{hline}
help for {hi:iedropone}
{hline}

{title:Title}

{phang2}{cmdab:iedropone} {hline 2} Same function as {help drop} but prevents
	that additional observations are unintentionally dropped.

{phang2}For a more descriptive discussion on the intended usage and work flow of this
command please see the {browse "https://dimewiki.worldbank.org/wiki/Iedropone":DIME Wiki}.

{title:Syntax}

{phang2}
{cmdab:iedropone} [if] , {cmdab:n:umobs(}{it:integer}{cmd:)}
	{cmdab:mvar(}{it:varname}{cmd:)} {cmdab:mval(}{it:list of values}{cmd:)}
	{cmd:zerook}]

{marker opts}{...}
{synoptset 22}{...}
{synopthdr:options}
{synoptline}
{synopt :{cmdab:n:umobs(}{it:integer}{cmd:)}}Number of observations that is allowed to be dropped. Default is 1.{p_end}
{synopt :{cmd:zerook}}Allows that no observation is dropped.{p_end}
{synopt :{cmdab:mvar(}{it:varname}{cmd:)}}Variable for which multiple values should be dropped. Must be used together with {cmdab:mval()}.{p_end}
{synopt :{cmdab:mval(}{it:list of values}{cmd:)}}The list of values in {cmdab:mvar()} that should be dropped. Must be used together with {cmdab:mvar()}.{p_end}
{synoptline}

{marker desc}
{title:Description}

{pstd}This commands might be easier to understand by following
	the {help iedropone##examples:examples below} before reading the
	description or the explanations of the options.

{pstd}{cmdab:iedropone} has the identical purpose as {help drop} when dropping
	observations. However, {cmdab:iedropone} safeguards that no additional
	observations are unintentionally dropped, or that changes are made to the
	data so that the observations that are supposed to be dropped are no longer dropped.

{pstd}{cmdab:iedropone} checks that no more or fewer observations than intended are
	dropped. For example, in the case that one observation has been identified to
	be dropped, then we want to make sure that when re-running the do-file
	no other observations are dropped even if more observations are added to that data
	set or changed in any other way.

{pstd}While the default is 1, {cmdab:iedropone} allows the user to set any another number
	of observation that should be dropped. If the number of observations that fit the
	drop condition is different, then the command will throw an error.

{marker optslong}
{title:Options}

{phang}{cmdab:n:umobs(}{it:integer}{cmd:)} this allows the user to set the
	number of observation that should be dropped. The default is 1 but any
	positive integer can be used. The command throws an error if any other number
	of observations match the drop condition.

{phang}{cmd:zerook} allows that no observations are dropped. The default is that
	an error is thrown if no observations are dropped.

{phang}{cmdab:mvar(}{it:varname}{cmd:)} and {cmdab:mval(}{it:list of values}{cmd:)} allows
	that multiple values in one variable are dropped. These two options must be used together.
	If the variable in {cmd:mvar()} is a string variable and some of the values in {cmd:mval()}
	includes spaces, then the list of values in {cmd:mval()} must be listed exactly as in example 4 below. The
	command loops over the values in {cmd:mval()} and drops the observations that
	satisfy the {it:if} condition and each of the value in {cmd:mval()}. For example:{p_end}

{pmore}{inp:iedropone if village == 100 , mvar(household_id) mval(21 22 23)}

{pmore}is identical to:

{pmore}{inp:iedropone if village == 100 & household_id == 21}{break}{inp:iedropone if village == 100 & household_id == 22}{break}{inp:iedropone if village == 100 & household_id == 23}

{pmore}The default is that exactly one observation should be dropped for each
	value in {cmd:mval()} unless {cmd:numobs()} or {cmd:zerook} is used. If those
	options are used then, then they apply to all values in {cmd:mval()} separately.

{phang}{cmdab:mval(}{it:list of values}{cmd:)}, see {cmdab:mvar(}{it:varname}{cmd:)} above.

{marker examples}
{title:Examples}

{pstd} {hi:Example 1.}

{pmore}{inp:iedropone if household_id == 712047}

{pmore}Let's say that we have identified the household with the ID 712047 to be
	incorrect and it should be dropped. Identical to {inp:drop if household_id == 712047}
	but it will test that exactly one observation is dropped each time the do-file runs.
	This guarantees that we will get an error message that no observation is dropped
	if someone makes a change to the ID. Otherwise we would unknowingly keep this
	incorrect observation in our data set.

{pmore}Similarly, if a new observation is added that is the correct household with ID 712047,
	then both observation would be dropped without warning if we would have
	used {inp:drop if household_id == 712047}. {cmd:iedropone} will make sure that
	our drop condition are applied as intended even if the data set is changed.

{pstd} {hi:Example 2.}

{pmore}{inp:iedropone if household_id == 712047 & household_head == "Bob Smith"}

{pmore}Let's say we have added a new household  with the ID 712047. In order to
	drop only one of those observations we must expand the if condition to
	indicate which one of them we want to drop.

{pstd} {hi:Example 3.}

{pmore}{inp:iedropone if household_id == 712047, numobs(2)}

{pmore}Let's say we added a new household  with the ID 712047 but we want to drop
	exactly both of them, then we can use the option {cmd:numobs()} like above.
	The command will now throw an error if not exactly two observations have the
	household ID 712047.

{pstd} {hi:Example 4.}

{pmore}{inp:iedropone if village == 100, mvar(household_head) mvar(`" "Bob Smith" "Ann Davitt" "Blessing Johnson" "')}

{pmore}If the values in {cmd:mvar()} are strings with empty spaces then then each
	value have to be enclosed in double qoutes and the full list needs to start
	with {inp:`"} and end with {inp:"'}.

{title:Acknowledgements}

{phang}I would like to acknowledge the help in testing and proofreading I
	received in relation to this command and help file from (in alphabetic
	order):{p_end}
{pmore}Paula Gonzalez-Martinez{break}Seungmin Lee{break}Mrijan Rimal{break}

{title:Author}

{phang}Kristoffer Bjarkefur, The World Bank, DECIE

{phang}Please send bug-reports, suggestions and requests for clarifications
		 writing "ietoolkit iedropone" in the subject line to:{break}
		 kbjarkefur@worldbank.org

{phang}You can also see the code, make comments to the code, see the version
		 history of the code, and submit additions or edits to the code through
		 the github repository of ietoolkit:{break}
		 {browse "https://github.com/worldbank/ietoolkit"}

{smcl}
{* 15 Dec 2017}{...}
{hline}
help for {hi:iedropone}
{hline}

{title:Title}

{phang2}{cmdab:iedropone} {hline 2} Same function as {help drop} but prevents
	that additional observations are unintentionally dropped.
	See {help iedropone##examples:examples below}.

{title:Syntax}

{phang2}
{cmdab:iedropone} [if] , {cmdab:n:umobs(}{it:integer}{cmd:)}
	{cmdab:mvar(}{it:varname}{cmd:)} {cmdab:mval(}{it:list of values}{cmd:)}
	{cmd:zerook}]

{marker opts}{...}
{synoptset 22}{...}
{synopthdr:options}
{synoptline}
{synopt :{cmdab:n:umobs(}{it:integer}{cmd:)}}Number of observations that is allowed to be dropped. Default is 1.{p_end}
{synopt :{cmd:zerook}}Allows that no observation is dropped.{p_end}
{synopt :{cmdab:mvar(}{it:varname}{cmd:)}}Variable for which multiple values should be dropped. Must be used together with {cmdab:mval()}.{p_end}
{synopt :{cmdab:mval(}{it:list of values}{cmd:)}}The list of values in {cmdab:mvar()} that should be dropped. Must be used together with {cmdab:mvar()}.{p_end}
{synoptline}

{marker desc}
{title:Description}

{pstd}This commands might be easier to understand by following
	the {help iedropone##examples:examples below} before reading the
	description or the explanations of the options.

{pstd}{cmdab:iedropone} has the identical purpose as {help drop} when dropping
	observations. However, {cmdab:iedropone} safeguards that no additional
	observations are unintentionally dropped, or that changes are made to the
	data so that the observations that are supposed to be dropped are no longer dropped.

{pstd}{cmdab:iedropone} checks that no more or fewer observations than intended are
	dropped. For example, in the case that one observation has been identified to
	be dropped, then we want to make sure that when re-running the do-file
	no other observations are dropped even if more observations are added to that data
	set or changed in any other way.

{pstd}While the default is 1, {cmdab:iedropone} allows the user to set any another number
	of observation that should be dropped. If the number of observations that fit the
	drop condition is different, then the command will throw an error.

{marker optslong}
{title:Options}

{phang}{cmdab:n:umobs(}{it:integer}{cmd:)} this allows the user to set the
	number of observation that should be dropped. The default is 1 but any
	positive integer can be used. The command throws an error if any other number
	of observations match the drop condition.

{phang}{cmd:zerook} allows that no observations are dropped. The default is that
	an error is thrown if no observations are dropped.

{phang}{cmdab:mvar(}{it:varname}{cmd:)} and {cmdab:mval(}{it:list of values}{cmd:)} allows
	that multiple values in one variable are dropped. These two options must be used together.
	If the variable in {cmd:mvar()} is a string variable and some of the values in {cmd:mval()}
	includes spaces, then the list of values in {cmd:mval()} must be listed exactly as in example 4 below. The
	command loops over the values in {cmd:mval()} and drops the observations that
	satisfy the {it:if} condition and each of the value in {cmd:mval()}. For example:{p_end}

{pmore}{inp:iedropone if village == 100 , mvar(household_id) mval(21 22 23)}

{pmore}is identical to:

{pmore}{inp:iedropone if village == 100 & household_id == 21}{break}{inp:iedropone if village == 100 & household_id == 22}{break}{inp:iedropone if village == 100 & household_id == 23}

{pmore}The default is that exactly one observation should be dropped for each
	value in {cmd:mval()} unless {cmd:numobs()} or {cmd:zerook} is used. If those
	options are used then, then they apply to all values in {cmd:mval()} separately.

{phang}{cmdab:mval(}{it:list of values}{cmd:)}, see {cmdab:mvar(}{it:varname}{cmd:)} above.

{marker examples}
{title:Examples}

{pstd} {hi:Example 1.}

{pmore}{inp:iedropone if household_id == 712047}

{pmore}Let's say that we have identified the household with the ID 712047 to be
	incorrect and it should be dropped. Identical to {inp:drop if household_id == 712047}
	but it will test that exactly one observation is dropped each time the do-file runs.
	This guarantees that we will get an error message that no observation is dropped
	if someone makes a change to the ID. Otherwise we would unknowingly keep this
	incorrect observation in our data set.

{pmore}Similarly, if a new observation is added that is the correct household with ID 712047,
	then both observation would be dropped without warning if we would have
	used {inp:drop if household_id == 712047}. {cmd:iedropone} will make sure that
	our drop condition are applied as intended even if the data set is changed.

{pstd} {hi:Example 2.}

{pmore}{inp:iedropone if household_id == 712047 & household_head == "Bob Smith"}

{pmore}Let's say we have added a new household  with the ID 712047. In order to
	drop only one of those observations we must expand the if condition to
	indicate which one of them we want to drop.

{pstd} {hi:Example 3.}

{pmore}{inp:iedropone if household_id == 712047, numobs(2)}

{pmore}Let's say we added a new household  with the ID 712047 but we want to drop
	exactly both of them, then we can use the option {cmd:numobs()} like above.
	The command will now throw an error if not exactly two observations have the
	household ID 712047.

{pstd} {hi:Example 4.}

{pmore}{inp:iedropone if village == 100, mvar(household_head) mvar(`" "Bob Smith" "Ann Davitt" "Blessing Johnson" "')}

{pmore}If the values in {cmd:mvar()} are strings with empty spaces then then each
	value have to be enclosed in double qoutes and the full list needs to start
	with {inp:`"} and end with {inp:"'}.

{title:Acknowledgements}

{phang}I would like to acknowledge the help in testing and proofreading I
	received in relation to this command and help file from (in alphabetic
	order):{p_end}
{pmore}Paula Gonzalez-Martinez{break}Seungmin Lee{break}Mrijan Rimal{break}

{title:Author}

{phang}Kristoffer Bjarkefur, The World Bank, DECIE

{phang}Please send bug-reports, suggestions and requests for clarifications
		 writing "ietoolkit iedropone" in the subject line to:{break}
		 kbjarkefur@worldbank.org

{phang}You can also see the code, make comments to the code, see the version
		 history of the code, and submit additions or edits to the code through
		 the github repository of ietoolkit:{break}
		 {browse "https://github.com/worldbank/ietoolkit"}


Changing version number in parallel.ado

Changing version number in parallel.ado

*! version 1.15.8.19  19agol2015
*! PARALLEL: Stata module for parallel computing
*! by George G. Vega (gvegayon at caltech.edu)
*!
*! Project website:
*!   https://github.com/gvegayon/parallel
/*
Copyright (c) 2014  <George G. Vega>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
*/

// Syntax parser
cap program drop parallel
program def parallel
    version 11.0

	// Checks wether if is parallel prefix or not
	if  (regexm(`"`0'"', "^(do|clean|setclusters|break|version|append|printlog|viewlog)")) {
	/* If not prefix */
		*If user initiated, clean the logs
		if regexs(0)=="clean" local add_on = cond(regexm(`"`0'"',","), "logs", ", logs")
		parallel_`0' `add_on'
	}
	else if (regexm(`"`0'"', "^(bs|sim)[,]?[\s ]?")) {
	/* Prefix bootstrap or simulate */
		local cmd = regexs(1)
		local 0   = regexr(`"`0'"', "^(bs|sim)", "")
		gettoken x 0 : 0, parse(":") bind
		local 0 = regexr(`"`0'"', "^[:]", "")
		gettoken x options : x, parse(",") bind

		gettoken 0 argopt : 0, parse(",") bind
		parallel_`cmd' `0', argopt(`argopt') `options'
	}
	else if (regexm(`"`0'"',"^([,]?.*[:])")) {
	/* if prefix */
		gettoken x 0 : 0, parse(":") bind
		local 0 = regexr(`"`0'"', "^[:]", "")
		// Gets the options (if these exists) of parallel
		gettoken x options : x, parse(",") bind

		gettoken 0 argopt : 0, parse(",") bind
		parallel_do `0', `options' prefix argopt(`argopt')
	}
	else {
		local val = regexm(`"`0'"', "^([a-zA-Z0-9_]*).*")
		local subcommand = regexs(1)
		di as result `"-`subcommand'-"' as error " invalid parallel subcommand" as text " ({stata h parallel:{it:help parallel}})"
		exit 198
	}
end

/* Returns the version of parallel */
program def parallel_version, rclass
	version 11.0
	di as result "parallel" as text " Stata module for parallel computing"
	di as result "vers" as text " 1.14.7.22 (22jul2014)"
	di as result "auth" as text " George G. Vega (gvegayon at caltech.edu)"

	return local pll_vers = "1.14.7.22"
end

/* Take a look to logfiles */
program def parallel_printlog
	version 11.0
	syntax [anything(name=pll_instance)] , [Event(string)]
	parallel_checklog `pll_instance', e(`event') action(print)
end

program def parallel_viewlog
	version 11.0
	syntax [anything(name=pll_instance)] , [Event(string)]
	parallel_checklog `pll_instance', e(`event') action(view)
end

program def parallel_checklog
	version 11.0
	syntax [anything(name=pll_instance)] , [Event(string)] action(string)

	if ("`event'"=="") local event = "$LAST_PLL_ID"

	if ("`event'"!=".") {
		/* By default uses 1 */
		if ("`pll_instance'" == "") local pll_instance 1

		local pll_instance : di %04.0f `pll_instance'

		/* Does de file exists */
		local logname = "__pll`event'_do`pll_instance'.log"

		if (c(os) == "Windows") local logname = "`c(tmpdir)'`logname'"
		else local logname = "`c(tmpdir)'/`logname'"

		/* If the logfile does not exists, do nothing*/
		cap confirm file "`logname'"
		if (_rc) {
			di as result "No logfile for instance -`pll_instance'- of parallel process -`event'- found"
			exit
		}

		/* Showing the log in screen */
		if ("`action'"=="print") {
			di as result "{hline 80}"
			di as result %~80s "beginning of file -`logname'-"
			di as result "{hline 80}"
			type `"`logname'"'
			di as result "{hline 80}"
			di as result %~80s "end of file -`logname'-"
			di as result "{hline 80}"
		}
		else view `"`logname'"'
	}
	else {
		di as error "It seems that you haven't use -parallel- yet."
		exit 601
	}

end

////////////////////////////////////////////////////////////////////////////////
// Splits the dataset into clusters
cap program drop parallel_spliter
program def parallel_spliter
	version 11.0
	syntax [namelist(name=xtstructure)] [,parallelid(string) sorting(integer 0) force(integer 0) keepusing(varlist)]
	//args xtstructure parallelid Sorting Force

	if length("$PLL_CLUSTERS") == 0 {
		di as err "Number of clusters not fixed." as text " Set the number of clusters with " _newline as err "-{cmd:parallel setclusters #}-"
		exit 198
	}

	quietly {
		gen _`parallelid'cut = .

		if (length("`xtstructure'")) {

			/* Checks wheather if the data is in the correct sorting */
			if !`sorting' & !`force' {
				error 5
			}

			/* Checking the data types */
			foreach var of varlist `xtstructure' {
				capture confirm string var `var'
				if (`=_rc') local numvars `numvars' `var'
				else local strvars `strvars' `var'
			}
		}

		/* Checking types of data */
		if (length("`numvars'")) local numvars st_data(.,"`numvars'")
		else local numvars J(0,0,.)

		if (length("`strvars'")) local strvars st_data(.,"`strvars'")
		else local strvars J(0,0,"")

		/* Processing in MATA */
		mata: st_store(., "_`parallelid'cut", parallel_divide_index(`numvars', `strvars'))

		if (length("`keepusing'")) {
			keep _`parallelid'cut `keepusing'
			gen __pllnobs`parallelid' = _n
		}

		save __pll`parallelid'_dataset, replace

		drop _all
	}

end

////////////////////////////////////////////////////////////////////////////////
// MAIN PROGRAM
*cap program drop parallel_do
program def parallel_do, rclass

	version 11.0

	#delimit ;
	syntax anything(name=dofile equalok everything)
		[, by(string)
		Keep
		KEEPLast
		prefix
		Force
		PROGrams(namelist)
		Mata
		NOGlobals
		KEEPTiming
		Seeds(string)
		NOData
		Randtype(string)
		Timeout(integer 60)
		PROCessors(integer 0)
		argopt(string)
		KEEPUsing(string)
		SETparallelid(string)
		];
	#delimit cr

	if length("$PLL_CLUSTERS") == 0 {
		di "{error:You haven't set the number of clusters}" _n "{error:Please set it with: {cmd:parallel setclusters} {it:#}}"
		exit 198
	}

	// Initial checks
	foreach opt in macrolist keep keeplast prefix force mata noglobals keeptiming nodata {
		local `opt' = length("``opt''") > 0
	}

	/* Randtype */
	if ("`randtype'" == "") local randtype = "datetime"

	/* If no data parsing has to be done (because of no data!) */
	if (!(c(N)*c(k))) local nodata 1

	if (!`keeptiming') {
		timer clear 98
		timer clear 99
	}

	timer on 98

	// Delets last parallel instance ran
	if (`keeplast' & length("`r(pll_id)'")) cap parallel_clean, e(`r(pll_id)')

	// Gets some global values
	local sfn = "$S_FN"

	// Gets the directory where to work at
	if (!`prefix') {

		/* First checks if the file exists */
		mata: parallel_normalizepath(`"`dofile'"',1)
		local pll_dir = "`filedir'"
		local dofile = "`filename'"
	}
	else local pll_dir = c(pwd)+"/"

	local initialdir = c(pwd)
	qui cd "`pll_dir'"

	if length("`by'") != 0 {
		local sortlist: sortedby
		local sorting = regexm("`sortlist'","^`by'")
	}
	else local sorting = 0

	/* Creates a unique ID for the process and secures it */
	if ("`setparallelid'"=="") mata: parallel_sandbox(5)
	else local parallelid = "`setparallelid'"

	global LAST_PLL_ID = "`parallelid'"
	global LAST_PLL_N = $PLL_CLUSTERS
	global LAST_PLL_DIR = "`pll_dir'"

	/* Generates database clusters */
	if (!`nodata') parallel_spliter `by' , parallelid(`parallelid') sorting(`sorting') force(`force') keepusing(`keepusing')

	/* Starts building the files */
	quietly {

		/* Saves mata objects */
		if (`mata') {
			mata: mata mlib create l__pll`parallelid'_mlib, replace
			cap mata: mata mlib add l__pll`parallelid'_mlib *()
			cap mata: mata matsave __pll`parallelid'_mata.mmat *, replace
			if (`=_rc') local matasave = 0
			else local matasave = 1
		}
		else local matasave = 0
	}

	/* Writing the dofile */
	mata: st_local("errornum", strofreal(parallel_write_do(strtrim(`"`dofile' `argopt'"'), "`parallelid'", $PLL_CLUSTERS, `prefix', `matasave', !`noglobals', "`seeds'", "`randtype'", `nodata', "`pll_dir'", "`programs'", `processors')))

	/* Checking if every thing is ok */
	if (`errornum') {
		if (!`nodata') {
			qui use __pll`parallelid'_dataset, clear

			// Restores original S_FN (file name) value
			global S_FN = "`sfn'"

			/* Removing the cut variable */
			qui drop _`parallelid'cut
		}

		/* Removes the sandbox file (unprotect the files) */
		mata: parallel_sandbox(2, "`parallelid'")

		error `errornum'
	}

	timer off 88
	cap timer list
	if (r(t88) == .) local pll_t_setu = 0
	else local pll_t_setu = r(t88)

	/* Running the dofiles */
	timer on 99
	mata: st_local("nerrors", strofreal(parallel_run("`parallelid'",$PLL_CLUSTERS,`"$PLL_STATA_PATH"',`=`timeout'*1000')))
	timer off 99

	/* If parallel finished with an error it restores the dataset */
	if (`nerrors' & !`nodata') {
		qui use __pll`parallelid'_dataset, clear
		global S_FN = "`sfn'"
		cap drop _`parallelid'cut
	}

	cap timer list
	if (r(t99) == .) local pll_t_calc = 0
	else local pll_t_calc = r(t99)
	local pll_t_reps = r(nt99)

	timer on 97
	// Paste the databases
	if (!`nodata' & !`nerrors') {
		parallel_fusion `parallelid', clusters($PLL_CLUSTERS) keepusing(`keepusing')

		// Restores original S_FN (file name) value
		global S_FN = "`sfn'"

		cap drop _`parallelid'cut
	}

	/* Removes the sandbox file (unprotect the files) */
	if ("`setparallelid'" == "") {
		mata: parallel_sandbox(2, "`parallelid'")
		if (!`keep' & !`keeplast') parallel_clean, e("`parallelid'")
	}

	timer off 97
	cap timer list
	if (r(t97) == .) local pll_t_fini = 0
	else local pll_t_fini = r(t97)

	qui timer list
	return local  pll_seeds="`pllseeds'"
	local pllseeds ""
	return scalar pll_errs = `nerrors'
	return local  pll_dir "`pll_dir'"
	return scalar pll_t_reps = `pll_t_reps'
	return scalar pll_t_setu = `pll_t_setu'
	return scalar pll_t_calc = `pll_t_calc'
	return scalar pll_t_fini = `pll_t_fini'
	return local pll_id = "`parallelid'"
	return scalar pll_n = $PLL_CLUSTERS

	qui cd "`initialdir'"
end

////////////////////////////////////////////////////////////////////////////////
// Cleans all files generated by parallel
*cap program drop parallel_clean
program def parallel_clean
	version 11.0
	syntax [, Event(string) All Force Logs]

	if (length("`event'") != 0 & length("`all'") != 0) {
		di as error `"invalid syntax: Using -pll_id- and -all- jointly is not allowed."'
		exit 198
	}

	mata: parallel_clean("`event'", `=length("`all'") > 0', `=length("`force'") > 0', `=length("`logs'") > 0')
end

////////////////////////////////////////////////////////////////////////////////
// Sets the number of clusters as a global macro
*cap program drop parallel_setclusters
program parallel_setclusters
	version 11.0
	syntax anything(name=nclusters)  [, Force Statapath(string asis) Gateway(string) Includefile(string)]

	local nclusters = real(`"`nclusters'"')
	if (`nclusters' == .) {
		di as error `"Not allow: "#" Should be a number"'
		exit 109
	}
	local force = length("`force'")>0
	mata: parallel_setclusters(`nclusters', `force')
	mata: st_local("error", strofreal(parallel_setstatapath(`"`statapath'"', `force')))
	if (`error'){
        di as error `"Can not set Stata directory, try using -statapath()- option"'
        exit `error'
	}

	if "`c(mode)'"=="batch" & "`c(os)'"=="Windows" {
		if `"`gateway'"'==""{
			local gateway "pll_gateway.sh"
		}
		cap confirm file `"`gateway'"'
		if _rc {
			di "path: `c(pwd)'"
			di as error `"On Windows in batch-mode, parallel requires a gateway file and no such file found."'
			exit _rc
		}
		global PLL_GATEWAY_FNAME `"`gateway'"'
	}

	if `"`includefile'"'!=""{
		cap confirm file `"`includefile'"'
		if _rc {
			di as error `"Can not find the include file (`includefile')."'
			exit _rc
		}
		global PLL_INCLUDE_FILE `"`includefile'"'
	}
end

////////////////////////////////////////////////////////////////////////////////
// Exports

// Exports a copy of programs
*cap program drop program_export
program def program_export
	version 11.0
	syntax using/ [,Programlist(string) Inname(string)]

	mata: program_export("`using'", "`programlist'", "`inname'")
end

////////////////////////////////////////////////////////////////////////////////
// Appends the clusterized dataset
*cap program drop parallel_fusion
program def parallel_fusion
	version 11.0
	syntax anything(name=parallelid) , clusters(integer) [keepusing(string)]

	capture {
		cap use "__pll`parallelid'_dta0001.dta", clear
		if (_rc) di "{error:No dataset for instance 0001.}"

		forval i = 2/`clusters' {
			cap append using `"__pll`parallelid'_dta`=string(`i',"%04.0f")'.dta"'
			if (_rc) di "{error:No dataset for instance `=string(`i',"%04.0f")'.}"
		}

		/* If it just used a set of variables */
		if (length("`keepusing'")) {
			merge 1:1 __pllnobs`parallelid' using __pll`parallelid'_dataset, keep(3) nogen
			drop __pllnobs`parallelid'
		}
	}

end

////////////////////////////////////////////////////////////////////////////////
// Checks whether the user pressed break inside a loop
*cap program drop parallel_break
program def parallel_break
	version 11.0
	mata: parallel_break()
end

*! version 1.14.7.22  22jul2014
*! PARALLEL: Stata module for parallel computing
*! by George G. Vega (gvegayon at caltech.edu)
*!
*! Project website:
*!   https://github.com/gvegayon/parallel
/*
Copyright (c) 2014  <George G. Vega>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
*/

// Syntax parser
cap program drop parallel
program def parallel
    version 11.0

	// Checks wether if is parallel prefix or not
	if  (regexm(`"`0'"', "^(do|clean|setclusters|break|version|append|printlog|viewlog)")) {
	/* If not prefix */
		*If user initiated, clean the logs
		if regexs(0)=="clean" local add_on = cond(regexm(`"`0'"',","), "logs", ", logs")
		parallel_`0' `add_on'
	}
	else if (regexm(`"`0'"', "^(bs|sim)[,]?[\s ]?")) {
	/* Prefix bootstrap or simulate */
		local cmd = regexs(1)
		local 0   = regexr(`"`0'"', "^(bs|sim)", "")
		gettoken x 0 : 0, parse(":") bind
		local 0 = regexr(`"`0'"', "^[:]", "")
		gettoken x options : x, parse(",") bind

		gettoken 0 argopt : 0, parse(",") bind
		parallel_`cmd' `0', argopt(`argopt') `options'
	}
	else if (regexm(`"`0'"',"^([,]?.*[:])")) {
	/* if prefix */
		gettoken x 0 : 0, parse(":") bind
		local 0 = regexr(`"`0'"', "^[:]", "")
		// Gets the options (if these exists) of parallel
		gettoken x options : x, parse(",") bind

		gettoken 0 argopt : 0, parse(",") bind
		parallel_do `0', `options' prefix argopt(`argopt')
	}
	else {
		local val = regexm(`"`0'"', "^([a-zA-Z0-9_]*).*")
		local subcommand = regexs(1)
		di as result `"-`subcommand'-"' as error " invalid parallel subcommand" as text " ({stata h parallel:{it:help parallel}})"
		exit 198
	}
end

/* Returns the version of parallel */
program def parallel_version, rclass
	version 11.0
	di as result "parallel" as text " Stata module for parallel computing"
	di as result "vers" as text " 1.14.7.22 (22jul2014)"
	di as result "auth" as text " George G. Vega (gvegayon at caltech.edu)"

	return local pll_vers = "1.14.7.22"
end

/* Take a look to logfiles */
program def parallel_printlog
	version 11.0
	syntax [anything(name=pll_instance)] , [Event(string)]
	parallel_checklog `pll_instance', e(`event') action(print)
end

program def parallel_viewlog
	version 11.0
	syntax [anything(name=pll_instance)] , [Event(string)]
	parallel_checklog `pll_instance', e(`event') action(view)
end

program def parallel_checklog
	version 11.0
	syntax [anything(name=pll_instance)] , [Event(string)] action(string)

	if ("`event'"=="") local event = "$LAST_PLL_ID"

	if ("`event'"!=".") {
		/* By default uses 1 */
		if ("`pll_instance'" == "") local pll_instance 1

		local pll_instance : di %04.0f `pll_instance'

		/* Does de file exists */
		local logname = "__pll`event'_do`pll_instance'.log"

		if (c(os) == "Windows") local logname = "`c(tmpdir)'`logname'"
		else local logname = "`c(tmpdir)'/`logname'"

		/* If the logfile does not exists, do nothing*/
		cap confirm file "`logname'"
		if (_rc) {
			di as result "No logfile for instance -`pll_instance'- of parallel process -`event'- found"
			exit
		}

		/* Showing the log in screen */
		if ("`action'"=="print") {
			di as result "{hline 80}"
			di as result %~80s "beginning of file -`logname'-"
			di as result "{hline 80}"
			type `"`logname'"'
			di as result "{hline 80}"
			di as result %~80s "end of file -`logname'-"
			di as result "{hline 80}"
		}
		else view `"`logname'"'
	}
	else {
		di as error "It seems that you haven't use -parallel- yet."
		exit 601
	}

end

////////////////////////////////////////////////////////////////////////////////
// Splits the dataset into clusters
cap program drop parallel_spliter
program def parallel_spliter
	version 11.0
	syntax [namelist(name=xtstructure)] [,parallelid(string) sorting(integer 0) force(integer 0) keepusing(varlist)]
	//args xtstructure parallelid Sorting Force

	if length("$PLL_CLUSTERS") == 0 {
		di as err "Number of clusters not fixed." as text " Set the number of clusters with " _newline as err "-{cmd:parallel setclusters #}-"
		exit 198
	}

	quietly {
		gen _`parallelid'cut = .

		if (length("`xtstructure'")) {

			/* Checks wheather if the data is in the correct sorting */
			if !`sorting' & !`force' {
				error 5
			}

			/* Checking the data types */
			foreach var of varlist `xtstructure' {
				capture confirm string var `var'
				if (`=_rc') local numvars `numvars' `var'
				else local strvars `strvars' `var'
			}
		}

		/* Checking types of data */
		if (length("`numvars'")) local numvars st_data(.,"`numvars'")
		else local numvars J(0,0,.)

		if (length("`strvars'")) local strvars st_data(.,"`strvars'")
		else local strvars J(0,0,"")

		/* Processing in MATA */
		mata: st_store(., "_`parallelid'cut", parallel_divide_index(`numvars', `strvars'))

		if (length("`keepusing'")) {
			keep _`parallelid'cut `keepusing'
			gen __pllnobs`parallelid' = _n
		}

		save __pll`parallelid'_dataset, replace

		drop _all
	}

end

////////////////////////////////////////////////////////////////////////////////
// MAIN PROGRAM
*cap program drop parallel_do
program def parallel_do, rclass

	version 11.0

	#delimit ;
	syntax anything(name=dofile equalok everything)
		[, by(string)
		Keep
		KEEPLast
		prefix
		Force
		PROGrams(namelist)
		Mata
		NOGlobals
		KEEPTiming
		Seeds(string)
		NOData
		Randtype(string)
		Timeout(integer 60)
		PROCessors(integer 0)
		argopt(string)
		KEEPUsing(string)
		SETparallelid(string)
		];
	#delimit cr

	if length("$PLL_CLUSTERS") == 0 {
		di "{error:You haven't set the number of clusters}" _n "{error:Please set it with: {cmd:parallel setclusters} {it:#}}"
		exit 198
	}

	// Initial checks
	foreach opt in macrolist keep keeplast prefix force mata noglobals keeptiming nodata {
		local `opt' = length("``opt''") > 0
	}

	/* Randtype */
	if ("`randtype'" == "") local randtype = "datetime"

	/* If no data parsing has to be done (because of no data!) */
	if (!(c(N)*c(k))) local nodata 1

	if (!`keeptiming') {
		timer clear 98
		timer clear 99
	}

	timer on 98

	// Delets last parallel instance ran
	if (`keeplast' & length("`r(pll_id)'")) cap parallel_clean, e(`r(pll_id)')

	// Gets some global values
	local sfn = "$S_FN"

	// Gets the directory where to work at
	if (!`prefix') {

		/* First checks if the file exists */
		mata: parallel_normalizepath(`"`dofile'"',1)
		local pll_dir = "`filedir'"
		local dofile = "`filename'"
	}
	else local pll_dir = c(pwd)+"/"

	local initialdir = c(pwd)
	qui cd "`pll_dir'"

	if length("`by'") != 0 {
		local sortlist: sortedby
		local sorting = regexm("`sortlist'","^`by'")
	}
	else local sorting = 0

	/* Creates a unique ID for the process and secures it */
	if ("`setparallelid'"=="") mata: parallel_sandbox(5)
	else local parallelid = "`setparallelid'"

	global LAST_PLL_ID = "`parallelid'"
	global LAST_PLL_N = $PLL_CLUSTERS
	global LAST_PLL_DIR = "`pll_dir'"

	/* Generates database clusters */
	if (!`nodata') parallel_spliter `by' , parallelid(`parallelid') sorting(`sorting') force(`force') keepusing(`keepusing')

	/* Starts building the files */
	quietly {

		/* Saves mata objects */
		if (`mata') {
			mata: mata mlib create l__pll`parallelid'_mlib, replace
			cap mata: mata mlib add l__pll`parallelid'_mlib *()
			cap mata: mata matsave __pll`parallelid'_mata.mmat *, replace
			if (`=_rc') local matasave = 0
			else local matasave = 1
		}
		else local matasave = 0
	}

	/* Writing the dofile */
	mata: st_local("errornum", strofreal(parallel_write_do(strtrim(`"`dofile' `argopt'"'), "`parallelid'", $PLL_CLUSTERS, `prefix', `matasave', !`noglobals', "`seeds'", "`randtype'", `nodata', "`pll_dir'", "`programs'", `processors')))

	/* Checking if every thing is ok */
	if (`errornum') {
		if (!`nodata') {
			qui use __pll`parallelid'_dataset, clear

			// Restores original S_FN (file name) value
			global S_FN = "`sfn'"

			/* Removing the cut variable */
			qui drop _`parallelid'cut
		}

		/* Removes the sandbox file (unprotect the files) */
		mata: parallel_sandbox(2, "`parallelid'")

		error `errornum'
	}

	timer off 88
	cap timer list
	if (r(t88) == .) local pll_t_setu = 0
	else local pll_t_setu = r(t88)

	/* Running the dofiles */
	timer on 99
	mata: st_local("nerrors", strofreal(parallel_run("`parallelid'",$PLL_CLUSTERS,`"$PLL_STATA_PATH"',`=`timeout'*1000')))
	timer off 99

	/* If parallel finished with an error it restores the dataset */
	if (`nerrors' & !`nodata') {
		qui use __pll`parallelid'_dataset, clear
		global S_FN = "`sfn'"
		cap drop _`parallelid'cut
	}

	cap timer list
	if (r(t99) == .) local pll_t_calc = 0
	else local pll_t_calc = r(t99)
	local pll_t_reps = r(nt99)

	timer on 97
	// Paste the databases
	if (!`nodata' & !`nerrors') {
		parallel_fusion `parallelid', clusters($PLL_CLUSTERS) keepusing(`keepusing')

		// Restores original S_FN (file name) value
		global S_FN = "`sfn'"

		cap drop _`parallelid'cut
	}

	/* Removes the sandbox file (unprotect the files) */
	if ("`setparallelid'" == "") {
		mata: parallel_sandbox(2, "`parallelid'")
		if (!`keep' & !`keeplast') parallel_clean, e("`parallelid'")
	}

	timer off 97
	cap timer list
	if (r(t97) == .) local pll_t_fini = 0
	else local pll_t_fini = r(t97)

	qui timer list
	return local  pll_seeds="`pllseeds'"
	local pllseeds ""
	return scalar pll_errs = `nerrors'
	return local  pll_dir "`pll_dir'"
	return scalar pll_t_reps = `pll_t_reps'
	return scalar pll_t_setu = `pll_t_setu'
	return scalar pll_t_calc = `pll_t_calc'
	return scalar pll_t_fini = `pll_t_fini'
	return local pll_id = "`parallelid'"
	return scalar pll_n = $PLL_CLUSTERS

	qui cd "`initialdir'"
end

////////////////////////////////////////////////////////////////////////////////
// Cleans all files generated by parallel
*cap program drop parallel_clean
program def parallel_clean
	version 11.0
	syntax [, Event(string) All Force Logs]

	if (length("`event'") != 0 & length("`all'") != 0) {
		di as error `"invalid syntax: Using -pll_id- and -all- jointly is not allowed."'
		exit 198
	}

	mata: parallel_clean("`event'", `=length("`all'") > 0', `=length("`force'") > 0', `=length("`logs'") > 0')
end

////////////////////////////////////////////////////////////////////////////////
// Sets the number of clusters as a global macro
*cap program drop parallel_setclusters
program parallel_setclusters
	version 11.0
	syntax anything(name=nclusters)  [, Force Statapath(string asis) Gateway(string) Includefile(string)]

	local nclusters = real(`"`nclusters'"')
	if (`nclusters' == .) {
		di as error `"Not allow: "#" Should be a number"'
		exit 109
	}
	local force = length("`force'")>0
	mata: parallel_setclusters(`nclusters', `force')
	mata: st_local("error", strofreal(parallel_setstatapath(`"`statapath'"', `force')))
	if (`error'){
        di as error `"Can not set Stata directory, try using -statapath()- option"'
        exit `error'
	}

	if "`c(mode)'"=="batch" & "`c(os)'"=="Windows" {
		if `"`gateway'"'==""{
			local gateway "pll_gateway.sh"
		}
		cap confirm file `"`gateway'"'
		if _rc {
			di "path: `c(pwd)'"
			di as error `"On Windows in batch-mode, parallel requires a gateway file and no such file found."'
			exit _rc
		}
		global PLL_GATEWAY_FNAME `"`gateway'"'
	}

	if `"`includefile'"'!=""{
		cap confirm file `"`includefile'"'
		if _rc {
			di as error `"Can not find the include file (`includefile')."'
			exit _rc
		}
		global PLL_INCLUDE_FILE `"`includefile'"'
	}
end

////////////////////////////////////////////////////////////////////////////////
// Exports

// Exports a copy of programs
*cap program drop program_export
program def program_export
	version 11.0
	syntax using/ [,Programlist(string) Inname(string)]

	mata: program_export("`using'", "`programlist'", "`inname'")
end

////////////////////////////////////////////////////////////////////////////////
// Appends the clusterized dataset
*cap program drop parallel_fusion
program def parallel_fusion
	version 11.0
	syntax anything(name=parallelid) , clusters(integer) [keepusing(string)]

	capture {
		cap use "__pll`parallelid'_dta0001.dta", clear
		if (_rc) di "{error:No dataset for instance 0001.}"

		forval i = 2/`clusters' {
			cap append using `"__pll`parallelid'_dta`=string(`i',"%04.0f")'.dta"'
			if (_rc) di "{error:No dataset for instance `=string(`i',"%04.0f")'.}"
		}

		/* If it just used a set of variables */
		if (length("`keepusing'")) {
			merge 1:1 __pllnobs`parallelid' using __pll`parallelid'_dataset, keep(3) nogen
			drop __pllnobs`parallelid'
		}
	}

end

////////////////////////////////////////////////////////////////////////////////
// Checks whether the user pressed break inside a loop
*cap program drop parallel_break
program def parallel_break
	version 11.0
	mata: parallel_break()
end


add nosample option (doesnt make much diff)

*! version 4.0.6 dev - 21feb2017

program reghdfe, eclass
	* Intercept old+version
	cap syntax, version old
	if !c(rc) {
		reghdfe_old, version
		exit
	}

	* Intercept old
	cap syntax anything(everything) [fw aw pw/], [*] old
	if !c(rc) {
		di as error "(running historical version of reghdfe)"
		if ("`weight'"!="") local weightexp [`weight'=`exp']
		reghdfe_old `anything' `weightexp', `options'
		exit
	}

	* Aux. subcommands
	cap syntax, [*]
	if inlist("`options'", "check", "compile", "reload", "update", "version", "requirements") {
		if ("`options'"=="compile") loc args force
		if ("`options'"=="check") loc options compile
		if ("`options'"=="update") {
			loc args 1
			loc options reload
		}
		loc subcmd = proper("`options'")
		`subcmd' `args'
	}
	else if replay() {
		Replay `0'
	}
	else {
		Cleanup 0
		Compile // takes 0.01s to run this useful check (ensures .mlib exists)
		cap noi Estimate `0'
		Cleanup `c(rc)'
	}
end

program Compile
	args force
	ftools, check // in case lftools.mlib does not exist or is outdated
	ms_get_version reghdfe // from moresyntax package; save local package_version

	loc list_objects "FixedEffects() fixed_effects() BipartiteGraph()"
	loc list_functions "reghdfe_*() transform_*() accelerate_*() panelmean() panelsolve_*()"
	loc list_misc "weighted_quadcolsum() safe_divide() check_convergence()"
	// TODO: prefix everything with reghdfe_*

	ms_compile_mata, ///
		package(reghdfe) ///
		version(`package_version') ///
		fun("`list_objects' `list_functions' `list_misc'") ///
		verbose ///
		`force'
end

program Reload
	* Internal debugging tool.
	* Updates dependencies and reghdfe from local path or from github
	* Usage:
	* 	reghdfe, update // from c:\git\..
	* 	reghdfe, reload // from github

	args online
	if ("`online'" == "") loc online 0

	di as text _n "{bf:reghdfe: updating required packages}"
	di as text "{hline 64}"

	* -moresyntax- https://github.com/sergiocorreia/moresyntax/
	cap ado uninstall moresyntax
	if (`online') net install moresyntax, from("https://github.com/sergiocorreia/moresyntax/raw/master/src/")
	if (!`online') net install moresyntax, from("c:\git\moresyntax\src")
	di as text "{hline 64}"

	* -ftools- https://github.com/sergiocorreia/ftools/
	cap ado uninstall ftools
	if (`online') net install ftools, from("https://github.com/sergiocorreia/ftools/raw/master/src/")
	if (!`online') net install ftools, from("c:\git\ftools\src")
	di as text "{hline 64}"
	ftools, compile // requires moresyntax
	di as text "{hline 64}"

	* Update -reghdfe-
	di as text _n  _n "{bf:reghdfe: updating self}"
	di as text "{hline 64}"
	qui ado uninstall reghdfe
	if (`online') net install reghdfe, from("https://github.com/sergiocorreia/reghdfe/raw/version-4/src/")
	if (!`online') net install reghdfe, from("c:\git\reghdfe\src")
	qui which reghdfe
	di as text "{hline 64}"
	reghdfe, compile
	di as text "{hline 64}"

	* Cleaning up
	di as text _n "{bf:Note:} You need to run {stata program drop _all} now."
end

program Version
	which reghdfe
	Requirements
end

program Requirements
	di as text _n "Required packages installed?"
	loc reqs moresyntax ftools
	// ivreg2 avar tuples group3hdfe
	if (c(version)<13) loc reqs `reqs' boottest

	loc ftools_github "https://github.com/sergiocorreia/ftools/raw/master/src/"
	loc moresyntax_github "https://github.com/sergiocorreia/moresyntax/raw/master/src/"

	loc error 0

	foreach req of local reqs {
		loc fn `req'.ado
		if ("`req'"=="moresyntax") loc fn ms_get_version.ado
		cap findfile `fn'
		if (_rc) {
			loc error 1
			di as text "{lalign 20:- `req'}" as error "not" _c
			di as text "    {stata ssc install `req':install from SSC}" _c
			if inlist("`req'", "ftools", "moresyntax") {
				loc github ``req'_github'
				di as text `"    {stata `"net install `req', from(`"`github'"')"':install from github}"'
			}
			else {
				di as text // newline
			}
		}
		else {
			di as text "{lalign 20:- `req'}" as text "yes"
		}
	}

	if (`error') exit 601
end

program Cleanup
	args rc
	cap mata: mata drop HDFE
	cap mata: mata drop hdfe_*
	cap drop __temp_reghdfe_resid__
	cap matrix drop reghdfe_statsmatrix
	if (`rc') exit `rc'
end

program Parse
	* Trim whitespace (caused by "///" line continuations; aesthetic only)
	mata: st_local("0", stritrim(st_local("0")))

	* Main syntax
	#d;
	syntax anything(id=varlist equalok) [if] [in] [aw pw fw/] , [

		/* Model */
		Absorb(string) NOAbsorb
		RESiduals(name) RESiduals2 /* use _reghdfe_resid */
		SUmmarize SUmmarize2(string asis) /* simulate implicit options */

		/* Standard Errors */
		VCE(string) CLuster(string)

		/* Diagnostic */
		Verbose(numlist min=1 max=1 >=-1 <=5 integer)
		TIMEit

		/* Speedup and memory Tricks */
		NOSAMPle /* do not save e(sample) */

		/* Undocumented */
		KEEPSINgletons
		OLD /* use latest v3 */
		NOTES(string) /* NOTES(key=value ...), will be stored on e() */

		] [*] /* capture optimization options, display options, etc. */
		;
	#d cr

	* Unused
	* SAVEcache
	* USEcache
	* CLEARcache
	* COMPACT /* use as little memory as possible but is slower */

	* Convert options to boolean
	if ("`verbose'" == "") loc verbose 0
	loc timeit = ("`timeit'"!="")
	loc drop_singletons = ("`keepsingletons'"=="")

	* Sanity checks
	if (`verbose'>-1 & "`keepsingletons'"!="") {
		loc url "http://scorreia.com/reghdfe/nested_within_cluster.pdf"
		loc msg "WARNING: Singleton observations not dropped; statistical significance is biased"
		di as error `"`msg' {browse "`url'":(link)}"'
	}
	if ("`cluster'"!="") {
		_assert ("`vce'"==""), msg("cannot specify both cluster() and vce()")
		loc vce cluster `cluster'
		loc cluster // clear it to avoid bugs in subsequent lines
	}

	* Parse Varlist
	ms_fvunab `anything'
	ms_parse_varlist `s(varlist)'
	loc base_varlist "`s(basevars)'"
	foreach cat in depvar indepvars endogvars instruments {
		loc original_`cat' "`s(`cat')'"
	}
	loc model = cond("`s(instruments)'" == "", "ols", "iv")
	loc original_varlist = "`s(varlist)'" // no parens or equal

	* Parse Weights
	if ("`weight'"!="") {
		unab exp : `exp', min(1) max(1) // simple weights only
	}

	* Parse VCE
	ms_parse_vce, vce(`vce') weighttype(`weight')
	loc vcetype = "`s(vcetype)'"
	loc num_clusters = `s(num_clusters)'
	loc clustervars = "`s(clustervars)'"
	loc base_clustervars = "`s(base_clustervars)'"
	loc vceextra = "`s(vceextra)'"

	* Select sample (except for absvars)
	loc varlist `original_varlist' `base_clustervars'
	tempvar touse
	marksample touse, strok // based on varlist + cluster + if + in + weight

	* Parse noabsorb
	_assert  ("`absorb'`noabsorb'" != ""), msg("option {bf:absorb()} or {bf:noabsorb} required")
	if ("`noabsorb'" != "") {
		_assert ("`absorb'" == ""), msg("{bf:absorb()} and {bf:noabsorb} are mutually exclusive")
		tempvar c
		gen byte `c' = 1
		loc absorb `c'
	}

	* Construct HDFE object
	// SYNTAX: fixed_effects(absvars | , touse, wtype, wtvar, dropsing, verbose)
	mata: st_local("comma", strpos(`"`absorb'"', ",") ? "" : ",")
	mata: HDFE = fixed_effects(`"`absorb' `comma' `options'"', "`touse'", "`weight'", "`exp'", `drop_singletons', `verbose')
	mata: HDFE.cmdline = "reghdfe " + st_local("0")
	loc options `s(options)'

	mata: st_local("N", strofreal(HDFE.N))
	if (`N' == 0) error 2000

	* Fill out HDFE object
	mata: HDFE.clustervars = tokens("`clustervars'")
	mata: HDFE.timeit = `timeit'

	mata: HDFE.varlist = "`base_varlist'"
	mata: HDFE.original_depvar = "`original_depvar'"
	mata: HDFE.original_indepvars = "`original_indepvars'"
	mata: HDFE.original_endogvars = "`original_endogvars'"
	mata: HDFE.original_instruments = "`original_instruments'"
	mata: HDFE.original_varlist = "`original_varlist'"
	mata: HDFE.model = "`model'"

	mata: HDFE.vcetype = "`vcetype'"
	mata: HDFE.num_clusters = `num_clusters'
	mata: HDFE.clustervars = tokens("`clustervars'")
	mata: HDFE.base_clustervars = tokens("`base_clustervars'")
	mata: HDFE.vceextra = "`vceextra'"

	* Parse summarize
	if ("`summarize'" != "") {
		_assert ("`summarize2'" == ""), msg("summarize() syntax error")
		loc summarize2 mean min max  // default values
	}
	ParseSummarize `summarize2'
	mata: HDFE.summarize_stats = "`s(stats)'"
	mata: HDFE.summarize_quietly = `s(quietly)'

	* Parse residuals
	if ("`residuals2'" != "") {
		_assert ("`residuals'" == ""), msg("residuals() syntax error")
		loc residuals _reghdfe_resid
		cap drop `residuals' // destructive!
	}
	else if ("`residuals'"!="") {
		conf new var `residuals'
	}
	mata: HDFE.residuals = "`residuals'"

	* Parse misc options
	mata: HDFE.notes = `"`notes'"'
	mata: HDFE.store_sample = ("`nosample'"=="")

	* Parse Coef Table Options (do this last!)
	_get_diopts diopts options, `options' // store in `diopts', and the rest back to `options'
	_assert (`"`options'"'==""), msg(`"invalid options: `options'"')
	if ("`hascons'"!="") di in ye "(option ignored: `hascons')"
	if ("`tsscons'"!="") di in ye "(option ignored: `tsscons')"
	mata: HDFE.diopts = `"`diopts'"'
end

program ParseSummarize, sclass
	sreturn clear
	syntax [namelist(name=stats)] , [QUIetly]
	local quietly = ("`quietly'"!="")
	sreturn loc stats "`stats'"
	sreturn loc quietly = `quietly'
end

// --------------------------------------------------------------------------

program Estimate, eclass
	ereturn clear

	* Parse and fill out HDFE object
	Parse `0'

	* Compute degrees-of-freedom
	mata: HDFE.estimate_dof()

	* Save updated e(sample) (singletons reduce sample);
	* required to parse factor variables to partial out
	tempvar touse
	mata: HDFE.save_touse("`touse'")

	* Expand varlists
	foreach cat in varlist depvar indepvars endogvars instruments {
		mata: st_local("vars", HDFE.original_`cat')
		if ("`vars'" == "") continue
		// HACK: addbn replaces 0.foreign with 0bn.foreign , to prevent st_data() from loading a bunch of zeros
		ms_fvstrip `vars' if `touse', expand dropomit addbn onebyone
		// If we don't use onebyone, then 1.x 2.x ends up as 2.x
		loc vars "`r(varlist)'"
		mata: HDFE.`cat' = "`vars'"
	}

	* Stats
	mata: st_local("stats", HDFE.summarize_stats)
	if ("`stats'" != "") Stats

	* Partial out; save TSS of depvar
	mata: hdfe_variables = HDFE.partial_out(HDFE.varlist, 1) // 1=Save TSS of first var if HDFE.tss is missing

	* Regress
	mata: assert(HDFE.model=="ols")
	RegressOLS `touse'

	* (optional) Store FEs
	mata: st_local("save_any_fe", strofreal(HDFE.save_any_fe))
	assert inlist(`save_any_fe', 0, 1)
	if (`save_any_fe') {
		_assert e(depvar) != "", msg("e(depvar) is empty")
		_assert e(resid) != "", msg("e(resid) is empty")
		confirm numeric var `e(depvar)', exact
		confirm numeric var `e(resid)', exact
		tempvar d
		if (e(rank)) {
			qui _predict double `d' if e(sample), xb
		}
		else {
			gen double `d' = 0
		}
		qui replace `d' = `e(depvar)' - `d' - `e(resid)' if e(sample)
		mata: HDFE.store_alphas("`d'")
		drop `d'
		cap drop __temp_reghdfe_resid__ // in case we didn't intend to save resids
	}

	* View estimation tables
	mata: st_local("diopts", HDFE.diopts)
	Replay, `diopts'

	// ~~ Preserve relevant dataset ~~-
	//
	// if (`compact') {
	// 	preserve
	// 	keep `varlist' `absorb'
	// 	loc N = c(N)
	// }
	// else {
	// 	qui cou if `touse'
	// 	loc N = r(N)
	// }
end

program RegressOLS, eclass
	args touse

	tempname b V N rank df_r
	// Carefully split y from X, without duplicating the memory used
	mata: hdfe_y = hdfe_variables[., 1]
	mata: hdfe_variables = cols(hdfe_variables)==1 ? J(rows(hdfe_variables), 0, .) : hdfe_variables[., 2..cols(hdfe_variables)]
	mata: reghdfe_post_ols(HDFE, hdfe_y, hdfe_variables, "`b'", "`V'", "`N'", "`rank'", "`df_r'")
	mata: st_local("indepvars", HDFE.indepvars)
	mata: hdfe_y = hdfe_variables = .

	loc esample
	mata: st_local("store_sample", strofreal(HDFE.store_sample))
	if (`store_sample') loc esample "esample(`touse')"

	if ("`indepvars'" != "") {
		matrix colnames `b' = `indepvars'
		matrix colnames `V' = `indepvars'
		matrix rownames `V' = `indepvars'
		ereturn post `b' `V', `esample' buildfvinfo depname(`depvar')
	}
	else {
		ereturn post, `esample' buildfvinfo depname(`depvar')
	}

	ereturn scalar N       = `N'
	ereturn scalar rank    = `rank'
	ereturn scalar df_r    = `df_r'
	ereturn local  cmd     "reghdfe"
	mata: HDFE.post(HDFE)

	* Post stats
	cap conf matrix reghdfe_statsmatrix
	if (!c(rc)) {
		ereturn matrix summarize = reghdfe_statsmatrix
		mata: st_local("summarize_quietly", strofreal(HDFE.summarize_quietly))
		ereturn scalar summarize_quietly = `summarize_quietly'
	}
end

program Replay, rclass
	syntax [, *]
	_get_diopts options, `options'
	reghdfe_header // _coef_table_header
	di ""
	_coef_table, `options' // ereturn display, `options'
	return add // adds r(level), r(table), etc. to ereturn (before the footnote deletes them)
	reghdfe_footnote

	* Replay stats
	if (e(summarize_quietly)==0) {
		di as text _n "{sf:Regression Summary Statistics:}" _c
		matlist e(summarize)', border(top bottom) rowtitle(Variable) // twidth(18)
	}
end

program Stats
	* Optional weights
	mata: st_local("weight", sprintf("[%s=%s]", HDFE.weight_type, HDFE.weight_var))
	assert "`weight'" != ""
	if ("`weight'" == "[=]") loc weight
	loc weight : subinstr local weight "[pweight" "[aweight"

	mata: st_local("stats", HDFE.summarize_stats)
	mata: st_local("varlist", HDFE.varlist)
	mata: st_local("cvars", invtokens(HDFE.cvars))
	loc full_varlist `varlist' `cvars'

	qui tabstat `full_varlist' `weight' , stat(`stats') col(stat) save
	matrix reghdfe_statsmatrix = r(StatTotal)
end

*! version 4.0.6 dev - 21feb2017

program reghdfe, eclass
	* Intercept old+version
	cap syntax, version old
	if !c(rc) {
		reghdfe_old, version
		exit
	}

	* Intercept old
	cap syntax anything(everything) [fw aw pw/], [*] old
	if !c(rc) {
		di as error "(running historical version of reghdfe)"
		if ("`weight'"!="") local weightexp [`weight'=`exp']
		reghdfe_old `anything' `weightexp', `options'
		exit
	}

	* Aux. subcommands
	cap syntax, [*]
	if inlist("`options'", "check", "compile", "reload", "update", "version", "requirements") {
		if ("`options'"=="compile") loc args force
		if ("`options'"=="check") loc options compile
		if ("`options'"=="update") {
			loc args 1
			loc options reload
		}
		loc subcmd = proper("`options'")
		`subcmd' `args'
	}
	else if replay() {
		Replay `0'
	}
	else {
		Cleanup 0
		Compile // takes 0.01s to run this useful check (ensures .mlib exists)
		cap noi Estimate `0'
		Cleanup `c(rc)'
	}
end

program Compile
	args force
	ftools, check // in case lftools.mlib does not exist or is outdated
	ms_get_version reghdfe // from moresyntax package; save local package_version

	loc list_objects "FixedEffects() fixed_effects() BipartiteGraph()"
	loc list_functions "reghdfe_*() transform_*() accelerate_*() panelmean() panelsolve_*()"
	loc list_misc "weighted_quadcolsum() safe_divide() check_convergence()"
	// TODO: prefix everything with reghdfe_*

	ms_compile_mata, ///
		package(reghdfe) ///
		version(`package_version') ///
		fun("`list_objects' `list_functions' `list_misc'") ///
		verbose ///
		`force'
end

program Reload
	* Internal debugging tool.
	* Updates dependencies and reghdfe from local path or from github
	* Usage:
	* 	reghdfe, update // from c:\git\..
	* 	reghdfe, reload // from github

	args online
	if ("`online'" == "") loc online 0

	di as text _n "{bf:reghdfe: updating required packages}"
	di as text "{hline 64}"

	* -moresyntax- https://github.com/sergiocorreia/moresyntax/
	cap ado uninstall moresyntax
	if (`online') net install moresyntax, from("https://github.com/sergiocorreia/moresyntax/raw/master/src/")
	if (!`online') net install moresyntax, from("c:\git\moresyntax\src")
	di as text "{hline 64}"

	* -ftools- https://github.com/sergiocorreia/ftools/
	cap ado uninstall ftools
	if (`online') net install ftools, from("https://github.com/sergiocorreia/ftools/raw/master/src/")
	if (!`online') net install ftools, from("c:\git\ftools\src")
	di as text "{hline 64}"
	ftools, compile // requires moresyntax
	di as text "{hline 64}"

	* Update -reghdfe-
	di as text _n  _n "{bf:reghdfe: updating self}"
	di as text "{hline 64}"
	qui ado uninstall reghdfe
	if (`online') net install reghdfe, from("https://github.com/sergiocorreia/reghdfe/raw/version-4/src/")
	if (!`online') net install reghdfe, from("c:\git\reghdfe\src")
	qui which reghdfe
	di as text "{hline 64}"
	reghdfe, compile
	di as text "{hline 64}"

	* Cleaning up
	di as text _n "{bf:Note:} You need to run {stata program drop _all} now."
end

program Version
	which reghdfe
	Requirements
end

program Requirements
	di as text _n "Required packages installed?"
	loc reqs moresyntax ftools
	// ivreg2 avar tuples group3hdfe
	if (c(version)<13) loc reqs `reqs' boottest

	loc ftools_github "https://github.com/sergiocorreia/ftools/raw/master/src/"
	loc moresyntax_github "https://github.com/sergiocorreia/moresyntax/raw/master/src/"

	loc error 0

	foreach req of local reqs {
		loc fn `req'.ado
		if ("`req'"=="moresyntax") loc fn ms_get_version.ado
		cap findfile `fn'
		if (_rc) {
			loc error 1
			di as text "{lalign 20:- `req'}" as error "not" _c
			di as text "    {stata ssc install `req':install from SSC}" _c
			if inlist("`req'", "ftools", "moresyntax") {
				loc github ``req'_github'
				di as text `"    {stata `"net install `req', from(`"`github'"')"':install from github}"'
			}
			else {
				di as text // newline
			}
		}
		else {
			di as text "{lalign 20:- `req'}" as text "yes"
		}
	}

	if (`error') exit 601
end

program Cleanup
	args rc
	cap mata: mata drop HDFE
	cap mata: mata drop hdfe_*
	cap drop __temp_reghdfe_resid__
	cap matrix drop reghdfe_statsmatrix
	if (`rc') exit `rc'
end

program Parse
	* Trim whitespace (caused by "///" line continuations; aesthetic only)
	mata: st_local("0", stritrim(st_local("0")))

	* Main syntax
	#d;
	syntax anything(id=varlist equalok) [if] [in] [aw pw fw/] , [

		/* Model */
		Absorb(string) NOAbsorb
		RESiduals(name) RESiduals2 /* use _reghdfe_resid */
		SUmmarize SUmmarize2(string asis) /* simulate implicit options */

		/* Standard Errors */
		VCE(string) CLuster(string)

		/* Diagnostic */
		Verbose(numlist min=1 max=1 >=-1 <=5 integer)
		TIMEit

		/* Undocumented */
		KEEPSINgletons
		OLD /* use latest v3 */
		NOTES(string) /* NOTES(key=value ...), will be stored on e() */

		] [*] /* capture optimization options, display options, etc. */
		;
	#d cr

	* Unused
			/* Speedup and memory Tricks */
	* SAVEcache
	* USEcache
	* CLEARcache
	* COMPACT /* use as little memory as possible but is slower */
	* NOSAMPle /* do not save e(sample) */

	* Convert options to boolean
	if ("`verbose'" == "") loc verbose 0
	loc timeit = ("`timeit'"!="")
	loc drop_singletons = ("`keepsingletons'"=="")

	* Sanity checks
	if (`verbose'>-1 & "`keepsingletons'"!="") {
		loc url "http://scorreia.com/reghdfe/nested_within_cluster.pdf"
		loc msg "WARNING: Singleton observations not dropped; statistical significance is biased"
		di as error `"`msg' {browse "`url'":(link)}"'
	}
	if ("`cluster'"!="") {
		_assert ("`vce'"==""), msg("cannot specify both cluster() and vce()")
		loc vce cluster `cluster'
		loc cluster // clear it to avoid bugs in subsequent lines
	}

	* Parse Varlist
	ms_fvunab `anything'
	ms_parse_varlist `s(varlist)'
	loc base_varlist "`s(basevars)'"
	foreach cat in depvar indepvars endogvars instruments {
		loc original_`cat' "`s(`cat')'"
	}
	loc model = cond("`s(instruments)'" == "", "ols", "iv")
	loc original_varlist = "`s(varlist)'" // no parens or equal

	* Parse Weights
	if ("`weight'"!="") {
		unab exp : `exp', min(1) max(1) // simple weights only
	}

	* Parse VCE
	ms_parse_vce, vce(`vce') weighttype(`weight')
	loc vcetype = "`s(vcetype)'"
	loc num_clusters = `s(num_clusters)'
	loc clustervars = "`s(clustervars)'"
	loc base_clustervars = "`s(base_clustervars)'"
	loc vceextra = "`s(vceextra)'"

	* Select sample (except for absvars)
	loc varlist `original_varlist' `base_clustervars'
	tempvar touse
	marksample touse, strok // based on varlist + cluster + if + in + weight

	* Parse noabsorb
	_assert  ("`absorb'`noabsorb'" != ""), msg("option {bf:absorb()} or {bf:noabsorb} required")
	if ("`noabsorb'" != "") {
		_assert ("`absorb'" == ""), msg("{bf:absorb()} and {bf:noabsorb} are mutually exclusive")
		tempvar c
		gen byte `c' = 1
		loc absorb `c'
	}

	* Construct HDFE object
	// SYNTAX: fixed_effects(absvars | , touse, wtype, wtvar, dropsing, verbose)
	mata: st_local("comma", strpos(`"`absorb'"', ",") ? "" : ",")
	mata: HDFE = fixed_effects(`"`absorb' `comma' `options'"', "`touse'", "`weight'", "`exp'", `drop_singletons', `verbose')
	mata: HDFE.cmdline = "reghdfe " + st_local("0")
	loc options `s(options)'

	mata: st_local("N", strofreal(HDFE.N))
	if (`N' == 0) error 2000

	* Fill out HDFE object
	mata: HDFE.clustervars = tokens("`clustervars'")
	mata: HDFE.timeit = `timeit'

	mata: HDFE.varlist = "`base_varlist'"
	mata: HDFE.original_depvar = "`original_depvar'"
	mata: HDFE.original_indepvars = "`original_indepvars'"
	mata: HDFE.original_endogvars = "`original_endogvars'"
	mata: HDFE.original_instruments = "`original_instruments'"
	mata: HDFE.original_varlist = "`original_varlist'"
	mata: HDFE.model = "`model'"

	mata: HDFE.vcetype = "`vcetype'"
	mata: HDFE.num_clusters = `num_clusters'
	mata: HDFE.clustervars = tokens("`clustervars'")
	mata: HDFE.base_clustervars = tokens("`base_clustervars'")
	mata: HDFE.vceextra = "`vceextra'"

	* Parse summarize
	if ("`summarize'" != "") {
		_assert ("`summarize2'" == ""), msg("summarize() syntax error")
		loc summarize2 mean min max  // default values
	}
	ParseSummarize `summarize2'
	mata: HDFE.summarize_stats = "`s(stats)'"
	mata: HDFE.summarize_quietly = `s(quietly)'

	* Parse residuals
	if ("`residuals2'" != "") {
		_assert ("`residuals'" == ""), msg("residuals() syntax error")
		loc residuals _reghdfe_resid
		cap drop `residuals' // destructive!
	}
	else if ("`residuals'"!="") {
		conf new var `residuals'
	}
	mata: HDFE.residuals = "`residuals'"

	* Parse misc options
	mata: HDFE.notes = `"`notes'"'

	* Parse Coef Table Options (do this last!)
	_get_diopts diopts options, `options' // store in `diopts', and the rest back to `options'
	_assert (`"`options'"'==""), msg(`"invalid options: `options'"')
	if ("`hascons'"!="") di in ye "(option ignored: `hascons')"
	if ("`tsscons'"!="") di in ye "(option ignored: `tsscons')"
	mata: HDFE.diopts = `"`diopts'"'
end

program ParseSummarize, sclass
	sreturn clear
	syntax [namelist(name=stats)] , [QUIetly]
	local quietly = ("`quietly'"!="")
	sreturn loc stats "`stats'"
	sreturn loc quietly = `quietly'
end

// --------------------------------------------------------------------------

program Estimate, eclass
	ereturn clear

	* Parse and fill out HDFE object
	Parse `0'

	* Compute degrees-of-freedom
	mata: HDFE.estimate_dof()

	* Save updated e(sample) (singletons reduce sample);
	* required to parse factor variables to partial out
	tempvar touse
	mata: HDFE.save_touse("`touse'")

	* Expand varlists
	foreach cat in varlist depvar indepvars endogvars instruments {
		mata: st_local("vars", HDFE.original_`cat')
		if ("`vars'" == "") continue
		// HACK: addbn replaces 0.foreign with 0bn.foreign , to prevent st_data() from loading a bunch of zeros
		ms_fvstrip `vars' if `touse', expand dropomit addbn onebyone
		// If we don't use onebyone, then 1.x 2.x ends up as 2.x
		loc vars "`r(varlist)'"
		mata: HDFE.`cat' = "`vars'"
	}

	* Stats
	mata: st_local("stats", HDFE.summarize_stats)
	if ("`stats'" != "") Stats

	* Partial out; save TSS of depvar
	mata: hdfe_variables = HDFE.partial_out(HDFE.varlist, 1) // 1=Save TSS of first var if HDFE.tss is missing

	* Regress
	mata: assert(HDFE.model=="ols")
	RegressOLS `touse'

	* (optional) Store FEs
	mata: st_local("save_any_fe", strofreal(HDFE.save_any_fe))
	assert inlist(`save_any_fe', 0, 1)
	if (`save_any_fe') {
		_assert e(depvar) != "", msg("e(depvar) is empty")
		_assert e(resid) != "", msg("e(resid) is empty")
		confirm numeric var `e(depvar)', exact
		confirm numeric var `e(resid)', exact
		tempvar d
		if (e(rank)) {
			qui _predict double `d' if e(sample), xb
		}
		else {
			gen double `d' = 0
		}
		qui replace `d' = `e(depvar)' - `d' - `e(resid)' if e(sample)
		mata: HDFE.store_alphas("`d'")
		drop `d'
		cap drop __temp_reghdfe_resid__ // in case we didn't intend to save resids
	}

	* View estimation tables
	mata: st_local("diopts", HDFE.diopts)
	Replay, `diopts'

	// ~~ Preserve relevant dataset ~~-
	//
	// if (`compact') {
	// 	preserve
	// 	keep `varlist' `absorb'
	// 	loc N = c(N)
	// }
	// else {
	// 	qui cou if `touse'
	// 	loc N = r(N)
	// }
end

program RegressOLS, eclass
	args touse

	tempname b V N rank df_r
	// Carefully split y from X, without duplicating the memory used
	mata: hdfe_y = hdfe_variables[., 1]
	mata: hdfe_variables = cols(hdfe_variables)==1 ? J(rows(hdfe_variables), 0, .) : hdfe_variables[., 2..cols(hdfe_variables)]
	mata: reghdfe_post_ols(HDFE, hdfe_y, hdfe_variables, "`b'", "`V'", "`N'", "`rank'", "`df_r'")
	mata: st_local("indepvars", HDFE.indepvars)
	mata: hdfe_y = hdfe_variables = .

	if ("`indepvars'" != "") {
		matrix colnames `b' = `indepvars'
		matrix colnames `V' = `indepvars'
		matrix rownames `V' = `indepvars'
		ereturn post `b' `V', esample(`touse') buildfvinfo depname(`depvar')
	}
	else {
		ereturn post, esample(`touse') buildfvinfo depname(`depvar')
	}

	ereturn scalar N       = `N'
	ereturn scalar rank    = `rank'
	ereturn scalar df_r    = `df_r'
	ereturn local  cmd     "reghdfe"
	mata: HDFE.post(HDFE)

	* Post stats
	cap conf matrix reghdfe_statsmatrix
	if (!c(rc)) {
		ereturn matrix summarize = reghdfe_statsmatrix
		mata: st_local("summarize_quietly", strofreal(HDFE.summarize_quietly))
		ereturn scalar summarize_quietly = `summarize_quietly'
	}
end

program Replay, rclass
	syntax [, *]
	_get_diopts options, `options'
	reghdfe_header // _coef_table_header
	di ""
	_coef_table, `options' // ereturn display, `options'
	return add // adds r(level), r(table), etc. to ereturn (before the footnote deletes them)
	reghdfe_footnote

	* Replay stats
	if (e(summarize_quietly)==0) {
		di as text _n "{sf:Regression Summary Statistics:}" _c
		matlist e(summarize)', border(top bottom) rowtitle(Variable) // twidth(18)
	}
end

program Stats
	* Optional weights
	mata: st_local("weight", sprintf("[%s=%s]", HDFE.weight_type, HDFE.weight_var))
	assert "`weight'" != ""
	if ("`weight'" == "[=]") loc weight
	loc weight : subinstr local weight "[pweight" "[aweight"

	mata: st_local("stats", HDFE.summarize_stats)
	mata: st_local("varlist", HDFE.varlist)
	mata: st_local("cvars", invtokens(HDFE.cvars))
	loc full_varlist `varlist' `cvars'

	qui tabstat `full_varlist' `weight' , stat(`stats') col(stat) save
	matrix reghdfe_statsmatrix = r(StatTotal)
end


Fixed a few minor mistakes.

Fixed a few minor mistakes.

Also removed the $output as that might confuse some people and just put the filename "graph.gph" instead. Since, we haven't shown where we have declared the $Output part, i felt that it could be confusing to some.

{smcl}
{* 18 Apr 2017}{...}
{hline}
help for {hi:iegraph}
{hline}

{title:Title}

{phang2}{cmdab:iegraph} {hline 2} Generates graphs based on regressions with treatment dummies common in impact evaluations.

{title:Syntax}

{phang2}
{cmdab:iegraph} {varlist}
, [ {cmdab:ti:tle(}{it:string}{cmd:)} {cmdab:varl:abels}
{cmdab:save(}{it:string}{cmd:)} {cmdab:grey:scale} {cmdab:yzero}
{cmd:noconfbars} {cmdab:confbarsnone(}{it:varlist}{cmd:)}
{it:any twoway graph options}
]

{marker opts}{...}
{synoptset 22}{...}
{synopthdr:options}
{synoptline}
{synopt :{cmdab:ti:tle(}{it:string}{cmd:)}} Manually sets the title of the graph.{p_end}
{synopt :{cmdab:varl:abels}} Uses variable labels for legends instead of variable names.{p_end}
{synopt :{cmdab:save(}{it:string}{cmd:)}} Sets the filename and the directory to which the graph will be set/exported.{p_end}
{synopt :{cmdab:grey:scale}} Uses greyscales for the bars instead of colors.{p_end}
{synopt :{cmdab:yzero}} Forces y-axis on the graph to start at 0.{p_end}
{synopt :{cmd:noconfbars}} Removes the confidence interval bars from graphs for all treatments.{p_end}
{synopt :{cmdab:confbarsnone(}{it:varlist}{cmd:)}} Removes confidence interval bars from only the {it:varlist} listed.{p_end}
{synoptline}

{pstd}Any twoway graph options that can be used with normal twoway graph commands can
	also be used. If any of these commands conflict with any of the built in options,
	then the user specified settings have precedence. See example 2 for details.{p_end}

{marker desc}
{title:Description}

{pstd}{cmdab:iegraph} This command creates bar graphs on the basis of the coefficients
	of treatment dummies in regression results. This command is developed for reading
	stored results from two types of impact evaluation regression models, but there
	are countless of other examples where the command also can be used. {cmd:iegraph} must
	be used immediately after running the regression or as long as the regression result is
	still stored in or restored to Stata's {help ereturn} results.

{pstd}{bf:Model 1: OLS with Treatment Dummies}{break}The most typical impact evaluation regression is
	to have the outcome variable as the dependent variable and one dummy for each
	treatment arm where control is the omitted category. These regressions can also include
	covariates, fixed effects etc., but as long as the treatment status is defined by
	mutually exclusive dummy variables. See especially examples 1 and 2 below. This command
	works with any number of treatment arms but works best from two arms (treatment
	and control) to five treatment arms (4 different treatments and control). More
	arms than that may result in a still correct but perhaps cluttered graph.

{pstd}{bf:Model 2: Difference-in-Differences}{break}Another typical regression model in impact
	evaluations are difference-in-difference (Diff-in-Diff) models with two treatment arms (treatment
	and control) and two time periods. If the Diff-in-Diff regression is specified as having the
	outcome variable as the dependent variable and three dummy variables (time, treatment
	and timeXtreatment) as the independent variables, then this command will produce a nice
	graph. Controls, treatment effects etc. may be added to the regression model. See especially example 3.

{pstd}{bf:Graph Output}{break}The graph generated by this command is created using the following values. The
	control bar is the mean of the outcome variable for the control group. It is not
	the constant from the regression as those are not identical if, for example, fixed effects
	and covariates were used. For each treatment group the bar is the sum of the value
	of the control bar and the beta coefficient in the regression of the corresponding
	treatment dummy. The confidence intervals are calculated from the variance in the
	beta coefficients in the regression.{p_end}

{pstd}The graph also includes the N for each treatment arm in the regression and uses
	that value as labels on the x-axis. Stars are added to this value if the corresponding
	coefficient is statistically different from zero in the regression{p_end}

{marker optslong}
{title:Options}

{phang}{cmdab:ti:tle(}{it:string}{cmd:)} Manually sets the title of the graph.{p_end}

{phang}{cmdab:varl:abels} Sets the legends to the variable labels for the
	variables instead of the variable names.{p_end}

{phang}{cmdab:save(}{it:string}{cmd:)} Sets the filename and the directory to which
	the graph will be set. If the filename ends with no extension or '.gph', it will be
	saved in the graph format. Extensions(png,pdf,tif,wmf,emf,ps,and eps) will be exported.
	Wmf and Emf are only available for the Windows version of Stata. Png and Tif for all
	versions of Stata except Console.{p_end}

{phang}{cmdab:grey:scale} Uses greyscales for the bars instead of colors. The color
	of the control bar will be black and the treatment bar will run in equal shade
	differences from light grey to dark grey.{p_end}

{phang}{cmd:noconfbars} Removes the confidence interval bars from graphs for all
	treatments. The default value for the confidence interval bars is 95%. {p_end}

{phang}{cmdab:confbarsnone(}{it:varlist}{cmd:)} Removes confidence interval bars
	from only the {it:varlist} listed. The remaining variables in the graphs which
	have not been specified in {cmdab:confbarsnone} will still have the confidence
	interval bars. {p_end}

{phang}{cmdab:yzero} Manually sets the y-axis of the graph to start at zero
	instead of the Stata default.{p_end}

{marker optslong}
{title:Examples}

{pstd} {hi:Example 1.}

{pmore} {inp:regress} {it:outcomevar treatment_dummy}{break}
		{inp:iegraph} {it:treatment_dummy} , {inp:title({it:"Treatment Effect on Outcome"})}

{pmore}In the example above, there are only two treatment arms (treatment and
		control). {it:treatment_dummy} has a 1 for all treatment observations and
		a 0 for all control observations. The graph will have one bar for control and
		it shows the mean for {it:outcomevar} for all observations in control. The
		second bar in the graph will be the sum of that mean and the coefficient
		for {it:treatment_dummy} in the regression. The graph will also have the
		title: Treatment Effect on Outcome.

{pstd} {hi:Example 2.}

{pmore} {inp:regress} {it:income tmt_1 tmt_2 age education}{inp:, cluster(}{it:district}{inp:)}{break}
		{inp:iegraph} {it:tmt_1 tmt_2}{inp:, noconfbars yzero title({it:"Treatment effect on income"}) }

{pmore}In the example above, the treatment effect on income in researched. There
		are three treatment arms; control, treatment 1 ({it:tmt_1}) and treatment
		2 ({it:tmt_2}). It is important that no observation has the value 1 in
		both {it:tmt_1} and {it:tmt_2} (i.e. no observation is in more than one
		treatment) and some observations must have the value 0 in both {it:tmt_1}
		and {it:tmt_2} (i.e. control observations). The variables {it:age} and
		{it:education} are covariates (control variables) and are not included
		in {cmd:iegraph}. {inp:noconfbars} omits the confidence interval bars
		, and {inp:yzero} sets the y-axis to start at 0.

{pstd} {hi:Example 3.}

{pmore} {inp:regress} {it:chld_wght time treat timeXtreat}{break}
		{inp:iegraph} {it:time treat timeXtreat} {inp:, title({it:"Treatment effect on Child Weight (Diff-in-Diff)"})}

{pmore}In the example above, the data set is a panel data set with two time
		periods and the regression estimates the treatment effect on child weight
		using a Difference-in-Differences model.
		The dummy variable {it:time} indicates if it is time period 0 or 1.
		The dummy variable {it:treat} indicates if the observation is treatment
		or control. {it:timeXtreat} is the interaction term of {it:time}
		and {it:treat}. This the standard way to set up a Difference-in-Differences
		regression model.

{pstd} {hi:Example 4.}

{pmore} {inp:regress} {it:harvest T1 T2 T3 } {break}
		{inp:iegraph} {it:T1 T2 T3} {inp:, title({it:"Treatment effect on harvest"})
		xlabel(,angle(45)) yzero ylabel(minmax) save({it:"Graph1.gph"})}

{pmore}The example above shows how to save a graph to disk. It also shows that
	most two-way graph options can be used. In this example the {cmd:iegraph}
	option {cdm:yzero} conflicts with the two-way option {cmd:ylabel(minmax)}.
	In such a case the user specified option takes precedence over {cmd:iegraph}
	options like {cdm:yzero}.

{title:Acknowledgements}

{phang}We would like to acknowledge the help in testing and proofreading we received in relation to this command and help file from (in alphabetic order):{p_end}
{pmore}Luiza Cardoso De Andrade{break}Michael Orevba{break}Ahmad Zia Wahdat{break}

{title:Authors}

{phang}Kristoffer Bjarkefur & Mrijan Rimal, The World Bank, DECIE

{phang}Please send bug-reports, suggestions and requests for clarifications
		 writing "ietoolkit iegraph" in the subject line to:{break}
		 kbjarkefur@worldbank.org

{phang}You can also see the code, make comments to the code, see the version
		 history of the code, and submit additions or edits to the code through
		 the github repository of ietoolkit:{break}
		 {browse "https://github.com/worldbank/ietoolkit"}

{smcl}
{* 18 Apr 2017}{...}
{hline}
help for {hi:iegraph}
{hline}

{title:Title}

{phang2}{cmdab:iegraph} {hline 2} Generates graphs based on regressions with treatment dummies common in impact evaluations.

{title:Syntax}

{phang2}
{cmdab:iegraph} {varlist}
, [ {cmdab:ti:tle(}{it:string}{cmd:)} {cmdab:varl:abels}
{cmdab:save(}{it:string}{cmd:)} {cmdab:grey:scale} {cmdab:yzero}
{cmd:noconfbars} {cmdab:confbarsnone(}{it:varlist}{cmd:)}
{it:any towway graph options}
]

{marker opts}{...}
{synoptset 22}{...}
{synopthdr:options}
{synoptline}
{synopt :{cmdab:ti:tle(}{it:string}{cmd:)}} Manually sets the title of the graph.{p_end}
{synopt :{cmdab:varl:abels}} Uses variable labels for legends instead of variable names.{p_end}
{synopt :{cmdab:save(}{it:string}{cmd:)}} Sets the filename and the directory to which the graph will be set/exported.{p_end}
{synopt :{cmdab:grey:scale}} Uses greyscales for the bars instead of colors.{p_end}
{synopt :{cmdab:yzero}} Forces y-axis on the graph to start at 0.{p_end}
{synopt :{cmd:noconfbars}} Removes the confidence interval bars from graphs for all treatments.{p_end}
{synopt :{cmdab:confbarsnone(}{it:varlist}{cmd:)}} Removes confidence interval bars from only the {it:varlist} listed.{p_end}
{synoptline}

{pstd}Any towway graph options that can be used with normal twoway graph commands can
	also be used. If any of these commands conflict with any of the built in options,
	then the user specified settings have precedence. See example 2 for details.{p_end}

{marker desc}
{title:Description}

{pstd}{cmdab:iegraph} This command creates bar graphs on the based on the coefficients
	of treatment dummies in regression results. This command is developed for reading
	stored results of from two types of impact evaluation regression models, but there
	are countless of other examples where the command also can be used. {cmd:iegraph} must
	be used immediately after running the regression or as long as the regression result is
	still stored in or restored to Stata's {help ereturn} results.

{pstd}{bf:Model 1: OLS with Treatment Dummies}{break}The most typical impact evaluation regression is
	to have the outcome variable as the dependent variable and one dummy for each
	treatment arm where control is the omitted category. These regressions can also include
	covariates, fixed effects etc., but as long as the treatment status is defined by
	mutually exclusive dummy variables. See especially example 1 and 2 below. This command
	works with any number of treatment arms but works best from two arms (treatment
	and control) to five treatment arms (4 different treatments and control). More
	arms than that may result in a still correct but perhaps cluttered graph.

{pstd}{bf:Model 2: Difference-in-Differences}{break}Another typical regression model in impact
	evaluations are difference-in-difference (Diff-in-Diff) models with two treatment arms (treatment
	and control) and two time periods. If the Diff-in-Diff regression is specified as having the
	outcome variable as the dependent variable and three dummy variables (time, treatment
	and timeXtreatment) as the independent variables, then this command will produce a nice
	graph. Controls, treatment effects etc. may be added to the regression model. See especially example 3.

{pstd}{bf:Graph Output}{break}The graph generated by this command is created using the following values. The
	control bar is the mean of the outcome variable for the control group. It is not
	the constant from the regression as those are not identical if, for example, fixed effects
	and covariates were used. For each treatment group the bar is the sum of the value
	of the control bar and the beta coefficient in the regression of the corresponding
	treatment dummy. The confidence intervals are calculated from the variance in the
	beta coefficients in the regression.{p_end}

{pstd}The graph also includes the N for each treatment arm in the regression and uses
	that value as labels on the x-axis. Stars are added to this value if the corresponding
	coefficient is statistically different from zero in the regression{p_end}

{marker optslong}
{title:Options}

{phang}{cmdab:ti:tle(}{it:string}{cmd:)} Manually sets the title of the graph.{p_end}

{phang}{cmdab:varl:abels} Sets the legends to the variable labels for the
	variables instead of the variable names.{p_end}

{phang}{cmdab:save(}{it:string}{cmd:)} Sets the filename and the directory to which
	the graph will be set. If the filename ends with no extension or '.gph', it will be
	saved in the graph format. Extensions(png,pdf,tif,wmf,emf,ps,and eps) will be exported.
	Wmf and Emf are only available for the Windows version of Stata. Png and Tif for all
	versions of Stata except Console.{p_end}

{phang}{cmdab:grey:scale} Uses greyscales for the bars instead of colors. The color
	of the control bar will be black and the treatment bar will run in equal shade
	differnces from light grey to dark grey.{p_end}

{phang}{cmd:noconfbars} Removes the confidence interval bars from graphs for all
	treatments. The default value for the confidence interval bars ar 95%. {p_end}

{phang}{cmdab:confbarsnone(}{it:varlist}{cmd:)} Removes confidence interval bars
	from only the {it:varlist} listed. The remaining variables in the graphs which
	have not been specified in {cmdab:confbarsnone} will still have the confidence
	interval bars. {p_end}

{phang}{cmdab:yzero} Manually sets the y-axis of the graph to start at zero
	instead of the Stata default.{p_end}

{marker optslong}
{title:Examples}

{pstd} {hi:Example 1.}

{pmore} {inp:regress} {it:outcomevar treatment_dummy}{break}
		{inp:iegraph} {it:treatment_dummy} , {inp:title({it:"Treatment Effect on Outcome"})}

{pmore}In the example above, there is only two treatment arms (treatment and
		control). {it:treatment_dummy} has a 1 for all treatment observations and
		a 0 for all control observations. The graph will have one bar for control and
		it shows the mean for {it:outcomevar} for all observations in control. The
		second bar in the graph will be the sum of that mean and the coefficient
		for {it:treatment_dummy} in the regression. The graph will also have the
		title: Treatment Effect on Outcome.

{pstd} {hi:Example 2.}

{pmore} {inp:regress} {it:income tmt_1 tmt_2 age education}{inp:, cluster(}{it:district}{inp:)}{break}
		{inp:iegraph} {it:tmt_1 tmt_2}{inp:, noconfbars yzero title({it:"Treatment effect on income"}) }

{pmore}In the example above, the treatment effect on income in researched. There
		are three treatment arms; control, treatment 1 ({it:tmt_1}) and treatment
		2 ({it:tmt_2}). It is important that no observation has the value 1 in
		both {it:tmt_1} and {it:tmt_2} (i.e. no observation is in more than one
		treatment) and some observations must have the value 0 in both {it:tmt_1}
		and {it:tmt_2} (i.e. control observations). The variables {it:age} and
		{it:education} are covariates (control variables) and are not included
		in {cmd:iegraph}. {inp:noconfbars} omits the confidence interval bars
		, and {inp:yzero} sets the y-axis to start at 0.

{pstd} {hi:Example 3.}

{pmore} {inp:regress} {it:chld_wght time treat timeXtreat}{break}
		{inp:iegraph} {it:time treat timeXtreat} {inp:, title({it:"Treatment effect on Child Weight (Diff-in-Diff)"})}

{pmore}In the example above, the data set is a panel data set with two time
		periods and the regression estimates the treatment effect on child weight
		using a Difference-in-Differences model.
		The dummy variable {it:time} indicates if it is time period 0 or 1.
		The dummy variable {it:treat} indicates if the observation is treatment
		or control. {it:timeXtreat} is the interaction term of {it:time}
		and {it:treat}. This the standard way to set up a Difference-in-Differences
		regression model.

{pstd} {hi:Example 4.}

{pmore} {inp:regress} {it:harvest T1 T2 T3 } {break}
		{inp:iegraph} {it:T1 T2 T3} {inp:, title({it:"Treatment effect on harvest"})
		xlabel(,angle(45)) yzero ylabel(minmax) save({it:"$Output/Graph1.gph"})}

{pmore}The example above shows how to save a graph to disk. It also shows that
	most two-way graph options can be used. In this example the {cmd:iegraph}
	option {cdm:yzero} conflicts with the two-way option {cmd:ylabel(minmax)}.
	In such a case the user specified option takes precedence over {cmd:iegraph}
	options like {cdm:yzero}.

{title:Acknowledgements}

{phang}We would like to acknowledge the help in testing and proofreading we received in relation to this command and help file from (in alphabetic order):{p_end}
{pmore}Luiza Cardoso De Andrade{break}Michael Orevba{break}Ahmad Zia Wahdat{break}

{title:Authors}

{phang}Kristoffer Bjarkefur & Mrijan Rimal, The World Bank, DECIE

{phang}Please send bug-reports, suggestions and requests for clarifications
		 writing "ietoolkit iegraph" in the subject line to:{break}
		 kbjarkefur@worldbank.org

{phang}You can also see the code, make comments to the code, see the version
		 history of the code, and submit additions or edits to the code through
		 the github repository of ietoolkit:{break}
		 {browse "https://github.com/worldbank/ietoolkit"}