Estimate quadratic model with demographics

clear
*2000:
insheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_frame_energy_2000.csv", delimiter(",")
quaids w1-w3, anot(5) prices(p1-p3) expenditure(depenses_par_uc) noquadratic
estat expenditure mu*
estat uncompensated ce*
outsheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_quaids_2000.csv", delimiter(",") replace

*2005:
clear
insheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_frame_energy_2005.csv", delimiter(",")
quaids w1-w3, anot(5) prices(p1-p3) expenditure(depenses_par_uc) noquadratic
estat expenditure mu*
estat uncompensated ce*
outsheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_quaids_2005.csv", delimiter(",") replace

*2011:
clear
insheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_frame_energy_2011.csv", delimiter(",")
quaids w1-w3, anot(5) prices(p1-p3) expenditure(depenses_par_uc) noquadratic
estat expenditure mu*
estat uncompensated ce*
outsheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_quaids_2011.csv", delimiter(",") replace

*If we introduce some selection, i.e. if we focus on those with w1>0:

clear
*2000:
insheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_frame_energy_2000.csv", delimiter(",")
replace w1 = . if w1 == 0
quaids w1-w3, anot(5) prices(p1-p3) expenditure(depenses_par_uc) noquadratic
estat expenditure mu*
estat uncompensated ce*
outsheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_quaids_2000.csv", delimiter(",") replace

*2005:
clear
insheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_frame_energy_2005.csv", delimiter(",")
replace w1 = . if w1 == 0
quaids w1-w3, anot(5) prices(p1-p3) expenditure(depenses_par_uc) noquadratic
estat expenditure mu*
estat uncompensated ce*
outsheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_quaids_2005.csv", delimiter(",") replace

*2011:
clear
insheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_frame_energy_2011.csv", delimiter(",")
replace w1 = . if w1 == 0
quaids w1-w3, anot(5) prices(p1-p3) expenditure(depenses_par_uc) noquadratic
estat expenditure mu*
estat uncompensated ce*
outsheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_quaids_2011.csv", delimiter(",") replace

*Still with selection, if we use demographics and the quadratic specification:

clear
*2000:
insheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_frame_energy_2000.csv", delimiter(",")
replace w1 = . if w1 == 0
quaids w1-w3, anot(5) prices(p1-p3) expenditure(depenses_par_uc) demographics(strate stalog nactifs nenfants dip14pr)
estat uncompensated, atmeans
matrix list r(uncompelas)
estat expenditure, atmeans
matrix list r(expelas)

*2005:
clear
insheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_frame_energy_2005.csv", delimiter(",")
replace w1 = . if w1 == 0
quaids w1-w3, anot(5) prices(p1-p3) expenditure(depenses_par_uc) demographics(strate stalog nactifs nenfants dip14pr) vce(bootstrap)
estat uncompensated, atmeans
matrix list r(uncompelas)
estat expenditure, atmeans
matrix list r(expelas)

*2011:
clear
insheet using "C:\Users\thomas.douenne\Documents\GitHub\openfisca-france-indirect-taxation\openfisca_france_indirect_taxation\almost_ideal_demand_system\data_frame_energy_2011.csv", delimiter(",")
replace w1 = . if w1 == 0
quaids w1-w3, anot(5) prices(p1-p3) expenditure(depenses_par_uc) demographics(strate stalog nactifs nenfants dip14pr)
estat uncompensated, atmeans
matrix list r(uncompelas)
estat expenditure, atmeans
matrix list r(expelas)

*To compute bootstrap standard errors, add vce(bootstrap) at the end of the quaids line


migrating to stata_script

migrating to stata_script

clear
*Cargar base de datos;
cd "C:\Users\rsanchez\Dropbox\Tesis_ITAM_RS"
use "tesisDB_anexo2"

*Establecer temporalidad
keep if ao>=2008 & ao<=2013
tsset year, format(%tmCY)

*Creacin de variables
gen lnhom=ln(homicidio)
gen lnrob=ln(robo)
gen Dhomicidio=D.homicidio
gen Dlnhom=D.lnhom

*Pre-anlisis grfico
	twoway (tsline homicidio) if ao>=2007
	qui tsline homicidio, name(f, replace)
	qui tsline D.homicidio, name(df, replace) yline(0)
	qui tsline robo, name(b, replace)
	qui tsline D.robo, name(db, replace) yline(0)
	qui tsline DD.homicidio, name(df, replace) yline(0)
	qui tsline DD.robo, name(db, replace) yline(0)
	graph combine f df b db, cols(2)
	graph combine f b, cols(1)

*i) Pruebas de raiz unitaria (I. Verificar que las variables son I(d) (Integrada de orden d) con d>0)
	dfuller homicidio
	dfuller robo
*ii) no podemos rechazar la hiptesis de raz unitaria. Verificamos que su orden de integracin sea el mismo:
	dfuller D.homicidio
	dfuller D.robo
		*La prueba Dickey-Fuller nos indica que ambas series son integradas de orden uno

* iii) Modelo Engle Granger de cointegracin
	regress robo homicidio
	outreg2 using cointegracion, replace tex(fragment)
	predict Rhat, residual
*iv)  prueba Dickey-Fuller en residuales
		*Para asegurarnos que las variables estn co-integradas y que el modelo es apropiado se debe probar que los residuales son I(1)
	dfuller Rhat

	regress robo L.homicidio
	predict ehat, residual
	dfuller ehat

*v) ECM  Modelo de Correccin de Errore
	regress D.robo L.Rhat D.homicidio
	predict Recm, residual
	dfuller Recm
	twoway (tsline Recm)

clear
*Cargar base de datos;
cd "C:\Users\rsanchez\Dropbox\Tesis_ITAM_RS"
use "tesisDB_anexo2"

*Establecer temporalidad
keep if ao>=2008 & ao<=2013
tsset year, format(%tmCY)

*Creacin de variables
gen lnhom=ln(homicidio)
gen lnrob=ln(robo)
gen Dhomicidio=D.homicidio
gen Dlnhom=D.lnhom

*Pre-anlisis grfico
	twoway (tsline homicidio) if ao>=2007
	qui tsline homicidio, name(f, replace)
	qui tsline D.homicidio, name(df, replace) yline(0)
	qui tsline robo, name(b, replace)
	qui tsline D.robo, name(db, replace) yline(0)
	qui tsline DD.homicidio, name(df, replace) yline(0)
	qui tsline DD.robo, name(db, replace) yline(0)
	graph combine f df b db, cols(2)
	graph combine f b, cols(1)

*i) Pruebas de raiz unitaria (I. Verificar que las variables son I(d) (Integrada de orden d) con d>0)
	dfuller homicidio
	dfuller robo
*ii) no podemos rechazar la hiptesis de raz unitaria. Verificamos que su orden de integracin sea el mismo:
	dfuller D.homicidio
	dfuller D.robo
		*La prueba Dickey-Fuller nos indica que ambas series son integradas de orden uno

* iii) Modelo Engle Granger de cointegracin
	regress robo homicidio
	predict Rhat, residual
*iv)  prueba Dickey-Fuller en residuales
		*Para asegurarnos que las variables estn co-integradas y que el modelo es apropiado se debe probar que los residuales son I(1)
	dfuller Rhat

	regress robo L.homicidio
	predict ehat, residual
	dfuller ehat

*v) ECM  Modelo de Correccin de Errore
	regress D.robo L.Rhat D.homicidio
	predict Recm, residual
	dfuller Recm
	twoway (tsline Recm)


iegitaddmd - always make recursive call on subfolder

*! version 5.4 15DEC2017  DIME Analytics lcardosodeandrad@worldbank.org

cap program drop   iegitaddmd
	program define iegitaddmd

	qui {

		syntax , folder(string) [all file(string) skip replace]

		/******************************

			Test user input

		******************************/

		*Test that folder exist
		mata : st_numscalar("r(dirExist)", direxists("`folder'"))

		if `r(dirExist)' == 0 {

			noi di as error `"{phang}The "`folder'" folder does not exist. You must enter the full path. For example, on most Windows computers it starts with {it:C:} and on most Mac computers with {it:/user/}. Important: Specify the whole file path to the repository folder, not just {it:C:} or {it:/user/} as that would creaet the placeholder file in every empty folder on your computer.{p_end}"'
			error 693
			exit
		}

		* File paths can have both forward and/or back slash. We'll standardize them so they're easier to handle
		local folderStd	= subinstr("`folder'","\","/",.)

		*Test that file exist if not using default file
		if "`file'" != "" {
			*File option used test if file exists
			capture confirm file "`file'"
			if _rc {
				*File does not exist, throw error
				noi di as error `"{phang}File "`file'" was not found. Remember that you must enter the full path. For example, on most Windows computers it starts with {it:C:} and on most Mac computers with {it:/user/}.{p_end}"'
				error 693
				exit
			}

			* Get the name of file (include path to keep )

			* File paths can have both forward and/or back slash. We'll standardize them so they're easier to handle
			local fileStd	= subinstr("`file'","\","/",.)

			*File path can have both forward and back slash
			local slash = strpos(strreverse("`fileStd'"),"/")

			*Use that position to get the file name. Multiply with minus one as we count from the back
			local userfilename = substr("`fileStd'", (-1 * `slash') ,.)

			*Used for the recursive call of the command when there is a subfolder
			local fileRecurse `"file(`fileStd')"'
		}

		*Options skip and replace cannot be used together
		if "`skip'" != "" & "`replace'" != "" {

			noi di as error `"{phang}The options skip and replace may not be used together.{p_end}"'
			error 198
			exit
		}

		/******************************

			Execute command

		******************************/

		*List files, directories and other files
		local flist : dir `"`folderStd'"' files "*"	, respectcase
		local dlist : dir `"`folderStd'"' dirs "*" 	, respectcase
		local olist : dir `"`folderStd'"' other "*"	, respectcase

		*Test if all of those lists are empty.
		if `"`flist'`dlist'`olist'"' == "" | ("`all'" == "all") {

			** If all those lists are empty then we are in an
			*  empty folder or al is used and should write the placeholder file

			if "`file'" == "" {

				*Write default README.md file
				writeGitKeep `"`folderStd'"' `skip' `replace'
			}
			else {

				*Use user provided file. First test if it already exist

				cap confirm file `"`folderStd'`userfilename'"'
				if _rc == 0 & "`skip'" == "" & "`replace'" == "" {

					*File exist and neither skip or replace is used
					local fileNameNoSlash = substr("`userfilename'",2,.)

					noi di as error `"{phang}A file with name `fileNameNoSlash' already exist in `folder'. Either remove option {it:all} or use either option {it:skip} or {it:replace}. See help file before using either of these options.{p_end}"'
					error 602
					exit
				}
				else if _rc == 0 & "`skip'" == "skip" {

					*File exist but option skip is used so do nothing, move to next folder

				}
				else if _rc == 0 & "`fileStd'" == `"`folderStd'`userfilename'"' {

					*If the custom template is inside the specified folder, we won't replace it

				}
				else{

					**Either file does not exist or replace is used, so write
					* placeholder file provided by user in folder location
					copy "`fileStd'" `"`folderStd'`userfilename'"', replace

				}
			}
		}

		*Use the command on each subfolder to this folder (if any)
		foreach dir of local dlist {

			*Recursive call on each subfolder
			iegitaddmd , folder(`"`folderStd'/`dir'"') `all' `fileRecurse' `skip' `replace'

		}
	}

end

*Write a README.md file iegitaddmd have found an empty folder
cap program drop   writeGitKeep
	program define writeGitKeep

		args folder skipreplace

		* Test if README.md already exist, and if it does test if skip or replace is used
		cap confirm file `"`folder'/README.md"'
		if _rc == 0 & "`skipreplace'" == "" {

			noi di as error `"{phang}A file with name README.md already exist in `folder'. Either remove option {it:all} or use either option {it:skip} or {it:replace}. See help file before using either of these options{p_end}"'
			error 602
			exit

		}
		else if _rc == 0 & "`skipreplace'" == "skip" {

			*Do nothing as option skip is used, move to next folder

		}
		else {

			*If file does not exist or replace is used, then write the file in this location

			*Create file
			tempname 	newHandle
			cap file close 	`newHandle'
				file open  	`newHandle' using "`folder'/README.md", text write replace

			*Add some text to the file
			file write  `newHandle' ///
				"# Placeholder file" _n _n ///
				"This file has been created automatically by the command **iegitaddmd** from Stata package [**ietoolkit**](https://worldbank.github.io/ietoolkit) to make GitHub sync this folder. GitHub does not sync empty folders or folders that only contain ignored files, but in research projects it is often important to share the full standardized folder structure along with the actual files. This command is intended to be used with **iefolder**, but it can be used in any folder structure intended to be shared on GitHub." _n _n ///
				"In recently started projects, it is typical to create data folders, script folders (do-files, r-files) and output folders, among others. The output folders are initially empty, but the script files might include a file path to them for later use. If an output folder is empty and another collaborator clones the repository, then the output folder expected by the scripts will not be included by GitHub in the cloned repository, and the script will not run properly." _n _n ///
				"## You should replace the content of this placeholder file" _n _n ///
				"The text in this file should be replaced with text that describes the intended use of this folder. This file is written in markdown (.md) format, which is suitable for GitHub syncing (unlike .doc/.docx). If the file is named *README.md*, GitHub automatically displays its contents when someone opens the containing folder on GitHub.com using a web browser." _n _n ///
				"If you are new to markup languages (markdown, html etc.) then this [Markdown Tutorial](https://www.markdowntutorial.com/) is a great place to start. If you have some experience with markup languages, then this [Markdown Cheat Sheet](https://guides.github.com/pdfs/markdown-cheatsheet-online.pdf) is great place to start." _n _n ///
				"## Add similar files in other folders" _n _n ///
				"The Stata command **iegitaddmd** does not add anything to folders that already have content unless this is explicitly requested using the option `all`. It is best practice to create a *README.md* file in any folders that have content, for example, in folders whose purpose might not obvious to someone using the repository for the first time. Again, if the file is named *README.md*, then the content of the file will be shown in the browser when someone explores the repository on GitHub.com." _n _n ///
				"Another great use of a *README.md* file is to use it as a documentation on how the folder it sits in and its subfolders are organized,  where its content can be found, and where new content is meant to be saved. For example, if you have a folder called `/Baseline/`, then you can give a short description of the activities conducted during the baseline and where data, scripts and outputs related to it can be found." _n _n ///
				"## Removing this file" _n _n ///
				"Removing this file is not recommended, as GitHub may stop syncing the parent folder unless the folder now has other content. However, even when content is added and the file can be removed without breaking the GitHub functionality, our recommendation is to replace the content with more relevant content rather than deleting it." _n _n

			*Closing the file
			file close 		`newHandle'

		}

end

*! version 5.4 15DEC2017  DIME Analytics lcardosodeandrad@worldbank.org

cap program drop   iegitaddmd
	program define iegitaddmd

	qui {

		syntax , folder(string) [all file(string) skip replace]

		/******************************

			Test user input

		******************************/

		*Test that folder exist
		mata : st_numscalar("r(dirExist)", direxists("`folder'"))

		if `r(dirExist)' == 0 {

			noi di as error `"{phang}The "`folder'" folder does not exist. You must enter the full path. For example, on most Windows computers it starts with {it:C:} and on most Mac computers with {it:/user/}. Important: Specify the whole file path to the repository folder, not just {it:C:} or {it:/user/} as that would creaet the placeholder file in every empty folder on your computer.{p_end}"'
			error 693
			exit
		}

		* File paths can have both forward and/or back slash. We'll standardize them so they're easier to handle
		local folderStd	= subinstr("`folder'","\","/",.)

		*Test that file exist if not using default file
		if "`file'" != "" {
			*File option used test if file exists
			capture confirm file "`file'"
			if _rc {
				*File does not exist, throw error
				noi di as error `"{phang}File "`file'" was not found. Remember that you must enter the full path. For example, on most Windows computers it starts with {it:C:} and on most Mac computers with {it:/user/}.{p_end}"'
				error 693
				exit
			}

			* Get the name of file (include path to keep )

			* File paths can have both forward and/or back slash. We'll standardize them so they're easier to handle
			local fileStd	= subinstr("`file'","\","/",.)

			*File path can have both forward and back slash
			local slash = strpos(strreverse("`fileStd'"),"/")

			*Use that position to get the file name. Multiply with minus one as we count from the back
			local userfilename = substr("`fileStd'", (-1 * `slash') ,.)

			*Used for the recursive call of the command when there is a subfolder
			local fileRecurse `"file(`fileStd')"'
		}

		*Options skip and replace cannot be used together
		if "`skip'" != "" & "`replace'" != "" {

			noi di as error `"{phang}The options skip and replace may not be used together.{p_end}"'
			error 198
			exit
		}

		/******************************

			Execute command

		******************************/

		*List files, directories and other files
		local flist : dir `"`folderStd'"' files "*"	, respectcase
		local dlist : dir `"`folderStd'"' dirs "*" 	, respectcase
		local olist : dir `"`folderStd'"' other "*"	, respectcase

		*Test if all of those lists are empty.
		if `"`flist'`dlist'`olist'"' == "" | ("`all'" == "all") {

			** If all those lists are empty then we are in an
			*  empty folder or al is used and should write the placeholder file

			if "`file'" == "" {

				*Write default README.md file
				writeGitKeep `"`folderStd'"' `skip' `replace'
			}
			else {

				*Use user provided file. First test if it already exist

				cap confirm file `"`folderStd'`userfilename'"'
				if _rc == 0 & "`skip'" == "" & "`replace'" == "" {

					*File exist and neither skip or replace is used
					local fileNameNoSlash = substr("`userfilename'",2,.)

					noi di as error `"{phang}A file with name `fileNameNoSlash' already exist in `folder'. Either remove option {it:all} or use either option {it:skip} or {it:replace}. See help file before using either of these options.{p_end}"'
					error 602
					exit
				}
				else if _rc == 0 & "`skip'" == "skip" {

					*File exist but option skip is used so do nothing, move to next folder

				}
				else if _rc == 0 & "`fileStd'" == `"`folderStd'`userfilename'"' {

					*If the custom template is inside the specified folder, we won't replace it

				}
				else{

					**Either file does not exist or replace is used, so write
					* placeholder file provided by user in folder location
					copy "`fileStd'" `"`folderStd'`userfilename'"', replace

				}
			}
		}

		*Use the command on each subfolder to this folder (if any)
		foreach dir of local dlist {

			* If the custom template is inside the specified folder, we won't replace it
			if "`fileStd'" != `"`folderStd'/`dir'`userfilename'"' {

				*Recursive call on each subfolder
				iegitaddmd , folder(`"`folderStd'/`dir'"') `all' `fileRecurse' `skip' `replace'
			}
		}
	}

end

*Write a README.md file iegitaddmd have found an empty folder
cap program drop   writeGitKeep
	program define writeGitKeep

		args folder skipreplace

		* Test if README.md already exist, and if it does test if skip or replace is used
		cap confirm file `"`folder'/README.md"'
		if _rc == 0 & "`skipreplace'" == "" {

			noi di as error `"{phang}A file with name README.md already exist in `folder'. Either remove option {it:all} or use either option {it:skip} or {it:replace}. See help file before using either of these options{p_end}"'
			error 602
			exit

		}
		else if _rc == 0 & "`skipreplace'" == "skip" {

			*Do nothing as option skip is used, move to next folder

		}
		else {

			*If file does not exist or replace is used, then write the file in this location

			*Create file
			tempname 	newHandle
			cap file close 	`newHandle'
				file open  	`newHandle' using "`folder'/README.md", text write replace

			*Add some text to the file
			file write  `newHandle' ///
				"# Placeholder file" _n _n ///
				"This file has been created automatically by the command **iegitaddmd** from Stata package [**ietoolkit**](https://worldbank.github.io/ietoolkit) to make GitHub sync this folder. GitHub does not sync empty folders or folders that only contain ignored files, but in research projects it is often important to share the full standardized folder structure along with the actual files. This command is intended to be used with **iefolder**, but it can be used in any folder structure intended to be shared on GitHub." _n _n ///
				"In recently started projects, it is typical to create data folders, script folders (do-files, r-files) and output folders, among others. The output folders are initially empty, but the script files might include a file path to them for later use. If an output folder is empty and another collaborator clones the repository, then the output folder expected by the scripts will not be included by GitHub in the cloned repository, and the script will not run properly." _n _n ///
				"## You should replace the content of this placeholder file" _n _n ///
				"The text in this file should be replaced with text that describes the intended use of this folder. This file is written in markdown (.md) format, which is suitable for GitHub syncing (unlike .doc/.docx). If the file is named *README.md*, GitHub automatically displays its contents when someone opens the containing folder on GitHub.com using a web browser." _n _n ///
				"If you are new to markup languages (markdown, html etc.) then this [Markdown Tutorial](https://www.markdowntutorial.com/) is a great place to start. If you have some experience with markup languages, then this [Markdown Cheat Sheet](https://guides.github.com/pdfs/markdown-cheatsheet-online.pdf) is great place to start." _n _n ///
				"## Add similar files in other folders" _n _n ///
				"The Stata command **iegitaddmd** does not add anything to folders that already have content unless this is explicitly requested using the option `all`. It is best practice to create a *README.md* file in any folders that have content, for example, in folders whose purpose might not obvious to someone using the repository for the first time. Again, if the file is named *README.md*, then the content of the file will be shown in the browser when someone explores the repository on GitHub.com." _n _n ///
				"Another great use of a *README.md* file is to use it as a documentation on how the folder it sits in and its subfolders are organized,  where its content can be found, and where new content is meant to be saved. For example, if you have a folder called `/Baseline/`, then you can give a short description of the activities conducted during the baseline and where data, scripts and outputs related to it can be found." _n _n ///
				"## Removing this file" _n _n ///
				"Removing this file is not recommended, as GitHub may stop syncing the parent folder unless the folder now has other content. However, even when content is added and the file can be removed without breaking the GitHub functionality, our recommendation is to replace the content with more relevant content rather than deleting it." _n _n

			*Closing the file
			file close 		`newHandle'

		}

end


iegrpah help typo

iegrpah help typo

{smcl}
{* 31 May 2017}{...}
{hline}
help for {hi:iegraph}
{hline}

{title:Title}

{phang2}{cmdab:iegraph} {hline 2} Generates graphs based on regressions with treatment dummies common in impact evaluations.

{title:Syntax}

{phang2}
{cmdab:iegraph} {varlist}
, [ {cmdab:basicti:tle(}{it:string}{cmd:)} {cmdab:varl:abels}
{cmdab:save(}{it:string}{cmd:)} {cmdab:grey:scale} {cmdab:yzero}
{cmd:noconfbars} {cmdab:confbarsnone(}{it:varlist}{cmd:)}
{cmd:norestore} {cmdab:baropt:ions(}{it:string}{cmd:)}
{it:{help scatter##twoway_options:twoway_scatter_options}}
]

{marker opts}{...}
{synoptset 22}{...}
{synopthdr:options}
{synoptline}
{synopt :{cmdab:basicti:tle(}{it:string}{cmd:)}} Manually sets the title of the graph.{p_end}
{synopt :{cmdab:varl:abels}} Uses variable labels for legends instead of variable names.{p_end}
{synopt :{cmdab:save(}{it:string}{cmd:)}} Sets the filename and the directory to which the graph will be set/exported.{p_end}
{synopt :{cmdab:grey:scale}} Uses greyscales for the bars instead of colors.{p_end}
{synopt :{cmdab:yzero}} Forces y-axis on the graph to start at 0.{p_end}
{synopt :{cmd:noconfbars}} Removes the confidence interval bars from graphs for all treatments.{p_end}
{synopt :{cmdab:confbarsnone(}{it:varlist}{cmd:)}} Removes confidence interval bars from only the {it:varlist} listed.{p_end}
{synopt :{cmd:norestore}} Allows you to debug your two way graph settings on the data set prepared by iegraph. To be used with {it:r(cmd)}.{p_end}
{synoptline}

{pstd}Any twoway graph scatter options that can be used with normal twoway graph scatter commands can
	also be used. If any of these commands conflict with any of the built in options,
	then the user specified settings have precedence. See example 2 for details.{p_end}

{marker desc}
{title:Description}

{pstd}{cmdab:iegraph} This command creates bar graphs on the basis of the coefficients
	of treatment dummies in regression results. This command is developed for reading
	stored results from two types of impact evaluation regression models, but there
	are countless of other examples where the command also can be used. {cmd:iegraph} must
	be used immediately after running the regression or as long as the regression result is
	still stored in or restored to Stata's {help ereturn} results.

{pstd}{bf:Model 1: OLS with Treatment Dummies}{break}The most typical impact evaluation regression is
	to have the outcome variable as the dependent variable and one dummy for each
	treatment arm where control is the omitted category. These regressions can also include
	covariates, fixed effects etc., but as long as the treatment status is defined by
	mutually exclusive dummy variables. See especially examples 1 and 2 below. This command
	works with any number of treatment arms but works best from two arms (treatment
	and control) to five treatment arms (4 different treatments and control). More
	arms than that may result in a still correct but perhaps cluttered graph.

{pstd}{bf:Model 2: Difference-in-Differences}{break}Another typical regression model in impact
	evaluations are difference-in-difference (Diff-in-Diff) models with two treatment arms (treatment
	and control) and two time periods. If the Diff-in-Diff regression is specified as having the
	outcome variable as the dependent variable and three dummy variables (time, treatment
	and timeXtreatment) as the independent variables, then this command will produce a nice
	graph. Controls, treatment effects etc. may be added to the regression model. See especially example 3.

{pstd}{bf:Graph Output}{break}The graph generated by this command is created using the following values. The
	control bar is the mean of the outcome variable for the control group. It is not
	the constant from the regression as those are not identical if, for example, fixed effects
	and covariates were used. For each treatment group the bar is the sum of the value
	of the control bar and the beta coefficient in the regression of the corresponding
	treatment dummy. The confidence intervals are calculated from the variance in the
	beta coefficients in the regression.{p_end}

{pstd}The graph also includes the N for each treatment arm in the regression and uses
	that value as labels on the x-axis. Stars are added to this value if the corresponding
	coefficient is statistically different from zero in the regression{p_end}

{marker optslong}
{title:Options}

{phang}{cmdab:basicti:tle(}{it:string}{cmd:)} Manually sets the title of the graph. To
	apply formatting like title size, position, etc., use Stata's built
	in {help title_options:title() option} instead.{p_end}

{phang}{cmdab:varl:abels} Sets the legends to the variable labels for the
	variables instead of the variable names.{p_end}

{phang}{cmdab:save(}{it:string}{cmd:)} Sets the filename and the directory to which
	the graph will be set. If the filename ends with no extension or '.gph', it will be
	saved in the graph format. Extensions(png,pdf,tif,wmf,emf,ps,and eps) will be exported.
	Wmf and Emf are only available for the Windows version of Stata. Png and Tif for all
	versions of Stata except Console.{p_end}

{phang}{cmdab:grey:scale} Uses greyscales for the bars instead of colors. The color
	of the control bar will be black and the treatment bar will run in equal shade
	differences from light grey to dark grey.{p_end}

{phang}{cmdab:yzero} Manually sets the y-axis of the graph to start at zero
	instead of the Stata default. In many cases, we expect that neither the default
	settings nor this option will make the axes look perfect, but you may use Stata's built
	in {help axis_option:axis options} that allow you to set the axes to perfectly fit
	your data.{p_end}

{phang}{cmd:noconfbars} Removes the confidence interval bars from graphs for all
	treatments. The default value for the confidence interval bars is 95%. {p_end}

{phang}{cmdab:confbarsnone(}{it:varlist}{cmd:)} Removes confidence interval bars
	from only the {it:varlist} listed. The remaining variables in the graphs which
	have not been specified in {cmdab:confbarsnone} will still have the confidence
	interval bars. {p_end}

{phang}{cmdab:norestore} Returns the data set that iegraph prepares to create
	the graph. This is helpful when de-bugging how one of Stata's many graph
	options can be applied to an iegraph graph. This option is meant to be
	used in combination with the {help return:returned result} in {it:r(cmd)}. {it:r(cmd)}
	gives you the line of code iegraph prepares to create the graph and {cmdab:norestore} gives you
	access to the data that code is meant to be used on. This approach will help
	you de-bug how to apply Stata's built in graph options to an iegraph graph.
	Note that this option deletes any unsaved changes made to your data.{p_end}

{marker optslong}
{title:Examples}

{pstd} {hi:Example 1.}

{pmore} {inp:regress} {it:outcomevar treatment_dummy}{break}
		{inp:iegraph} {it:treatment_dummy} , {inp:basictitle({it:"Treatment Effect on Outcome"})}

{pmore}In the example above, there are only two treatment arms (treatment and
		control). {it:treatment_dummy} has a 1 for all treatment observations and
		a 0 for all control observations. The graph will have one bar for control and
		it shows the mean for {it:outcomevar} for all observations in control. The
		second bar in the graph will be the sum of that mean and the coefficient
		for {it:treatment_dummy} in the regression. The graph will also have the
		title: Treatment Effect on Outcome.

{pstd} {hi:Example 2.}

{pmore} {inp:regress} {it:income tmt_1 tmt_2 age education}{inp:, cluster(}{it:district}{inp:)}{break}
		{inp:iegraph} {it:tmt_1 tmt_2}{inp:, noconfbars yzero basictitle({it:"Treatment effect on income"}) }

{pmore}In the example above, the treatment effect on income in researched. There
		are three treatment arms; control, treatment 1 ({it:tmt_1}) and treatment
		2 ({it:tmt_2}). It is important that no observation has the value 1 in
		both {it:tmt_1} and {it:tmt_2} (i.e. no observation is in more than one
		treatment) and some observations must have the value 0 in both {it:tmt_1}
		and {it:tmt_2} (i.e. control observations). The variables {it:age} and
		{it:education} are covariates (control variables) and are not included
		in {cmd:iegraph}. {inp:noconfbars} omits the confidence interval bars
		, and {inp:yzero} sets the y-axis to start at 0.

{pstd} {hi:Example 3.}

{pmore} {inp:regress} {it:chld_wght time treat timeXtreat}{break}
		{inp:iegraph} {it:time treat timeXtreat} {inp:, basictitle({it:"Treatment effect on Child Weight (Diff-in-Diff)"})}

{pmore}In the example above, the data set is a panel data set with two time
		periods and the regression estimates the treatment effect on child weight
		using a Difference-in-Differences model.
		The dummy variable {it:time} indicates if it is time period 0 or 1.
		The dummy variable {it:treat} indicates if the observation is treatment
		or control. {it:timeXtreat} is the interaction term of {it:time}
		and {it:treat}. This the standard way to set up a Difference-in-Differences
		regression model.

{pstd} {hi:Example 4.}

{pmore} {inp:regress} {it:harvest T1 T2 T3 } {break}
		{inp:iegraph} {it:T1 T2 T3} {inp:, basictitle({it:"Treatment effect on harvest"})
		xlabel(,angle(45)) yzero ylabel(minmax) save({it:"Graph1.gph"})}

{pmore}The example above shows how to save a graph to disk. It also shows that
	most two-way graph options can be used. In this example the {cmd:iegraph}
	option {cdm:yzero} conflicts with the two-way option {cmd:ylabel(minmax)}.
	In such a case the user specified option takes precedence over {cmd:iegraph}
	options like {cdm:yzero}.

{title:Acknowledgements}

{phang}We would like to acknowledge the help in testing and proofreading we received in relation to this command and help file from (in alphabetic order):{p_end}
{pmore}Luiza Cardoso De Andrade{break}Michael Orevba{break}Ahmad Zia Wahdat{break}

{title:Authors}

{phang}Kristoffer Bjarkefur & Mrijan Rimal, The World Bank, DECIE

{phang}Please send bug-reports, suggestions and requests for clarifications
		 writing "ietoolkit iegraph" in the subject line to:{break}
		 kbjarkefur@worldbank.org

{phang}You can also see the code, make comments to the code, see the version
		 history of the code, and submit additions or edits to the code through
		 the github repository of ietoolkit:{break}
		 {browse "https://github.com/worldbank/ietoolkit"}

{smcl}
{* 31 May 2017}{...}
{hline}
help for {hi:iegraph}
{hline}

{title:Title}

{phang2}{cmdab:iegraph} {hline 2} Generates graphs based on regressions with treatment dummies common in impact evaluations.

{title:Syntax}

{phang2}
{cmdab:iegraph} {varlist}
, [ {cmdab:basicti:tle(}{it:string}{cmd:)} {cmdab:varl:abels}
{cmdab:save(}{it:string}{cmd:)} {cmdab:grey:scale} {cmdab:yzero}
{cmd:noconfbars} {cmdab:confbarsnone(}{it:varlist}{cmd:)}
{cmd:norestore} {cmdab:baropt:ions(}{it:string}{cmd:)}
{it:{help scatter##twoway_options:twoway_scatter_options}}
]

{marker opts}{...}
{synoptset 22}{...}
{synopthdr:options}
{synoptline}
{synopt :{cmdab:basicti:tle(}{it:string}{cmd:)}} Manually sets the title of the graph.{p_end}
{synopt :{cmdab:varl:abels}} Uses variable labels for legends instead of variable names.{p_end}
{synopt :{cmdab:save(}{it:string}{cmd:)}} Sets the filename and the directory to which the graph will be set/exported.{p_end}
{synopt :{cmdab:grey:scale}} Uses greyscales for the bars instead of colors.{p_end}
{synopt :{cmdab:yzero}} Forces y-axis on the graph to start at 0.{p_end}
{synopt :{cmd:noconfbars}} Removes the confidence interval bars from graphs for all treatments.{p_end}
{synopt :{cmdab:confbarsnone(}{it:varlist}{cmd:)}} Removes confidence interval bars from only the {it:varlist} listed.{p_end}
{synopt :{cmd:norestore}} Allows you to debug your two way graph settings on the data set prepared by iegraph. To be used with {it:r(cmd)}.{p_end}
{synoptline}

{pstd}Any twoway graph scatter options that can be used with normal twoway graph scatter commands can
	also be used. If any of these commands conflict with any of the built in options,
	then the user specified settings have precedence. See example 2 for details.{p_end}

{marker desc}
{title:Description}

{pstd}{cmdab:iegraph} This command creates bar graphs on the basis of the coefficients
	of treatment dummies in regression results. This command is developed for reading
	stored results from two types of impact evaluation regression models, but there
	are countless of other examples where the command also can be used. {cmd:iegraph} must
	be used immediately after running the regression or as long as the regression result is
	still stored in or restored to Stata's {help ereturn} results.

{pstd}{bf:Model 1: OLS with Treatment Dummies}{break}The most typical impact evaluation regression is
	to have the outcome variable as the dependent variable and one dummy for each
	treatment arm where control is the omitted category. These regressions can also include
	covariates, fixed effects etc., but as long as the treatment status is defined by
	mutually exclusive dummy variables. See especially examples 1 and 2 below. This command
	works with any number of treatment arms but works best from two arms (treatment
	and control) to five treatment arms (4 different treatments and control). More
	arms than that may result in a still correct but perhaps cluttered graph.

{pstd}{bf:Model 2: Difference-in-Differences}{break}Another typical regression model in impact
	evaluations are difference-in-difference (Diff-in-Diff) models with two treatment arms (treatment
	and control) and two time periods. If the Diff-in-Diff regression is specified as having the
	outcome variable as the dependent variable and three dummy variables (time, treatment
	and timeXtreatment) as the independent variables, then this command will produce a nice
	graph. Controls, treatment effects etc. may be added to the regression model. See especially example 3.

{pstd}{bf:Graph Output}{break}The graph generated by this command is created using the following values. The
	control bar is the mean of the outcome variable for the control group. It is not
	the constant from the regression as those are not identical if, for example, fixed effects
	and covariates were used. For each treatment group the bar is the sum of the value
	of the control bar and the beta coefficient in the regression of the corresponding
	treatment dummy. The confidence intervals are calculated from the variance in the
	beta coefficients in the regression.{p_end}

{pstd}The graph also includes the N for each treatment arm in the regression and uses
	that value as labels on the x-axis. Stars are added to this value if the corresponding
	coefficient is statistically different from zero in the regression{p_end}

{marker optslong}
{title:Options}

{phang}{cmdab:basicti:tle(}{it:string}{cmd:)} Manually sets the title of the graph. To
	apply formatting like title size, position, etc., use Stata's buil
	in {help title_options:title() option} instead.{p_end}

{phang}{cmdab:varl:abels} Sets the legends to the variable labels for the
	variables instead of the variable names.{p_end}

{phang}{cmdab:save(}{it:string}{cmd:)} Sets the filename and the directory to which
	the graph will be set. If the filename ends with no extension or '.gph', it will be
	saved in the graph format. Extensions(png,pdf,tif,wmf,emf,ps,and eps) will be exported.
	Wmf and Emf are only available for the Windows version of Stata. Png and Tif for all
	versions of Stata except Console.{p_end}

{phang}{cmdab:grey:scale} Uses greyscales for the bars instead of colors. The color
	of the control bar will be black and the treatment bar will run in equal shade
	differences from light grey to dark grey.{p_end}

{phang}{cmdab:yzero} Manually sets the y-axis of the graph to start at zero
	instead of the Stata default. In many cases, we expect that neither the default
	settings nor this option will make the axes look perfect, but you may use Stata's built
	in {help axis_option:axis options} that allow you to set the axes to perfectly fit
	your data.{p_end}

{phang}{cmd:noconfbars} Removes the confidence interval bars from graphs for all
	treatments. The default value for the confidence interval bars is 95%. {p_end}

{phang}{cmdab:confbarsnone(}{it:varlist}{cmd:)} Removes confidence interval bars
	from only the {it:varlist} listed. The remaining variables in the graphs which
	have not been specified in {cmdab:confbarsnone} will still have the confidence
	interval bars. {p_end}

{phang}{cmdab:norestore} Returns the data set that iegraph prepares to create
	the graph. This is helpful when de-bugging how one of Stata's many graph
	options can be applied to an iegraph graph. This option is meant to be
	used in combination with the {help return:returned result} in {it:r(cmd)}. {it:r(cmd)}
	gives you the line of code iegraph prepares to create the graph and {cmdab:norestore} gives you
	access to the data that code is meant to be used on. This approach will help
	you de-bug how to apply Stata's built in graph options to an iegraph graph.
	Note that this option deletes any unsaved changes made to your data.{p_end}

{marker optslong}
{title:Examples}

{pstd} {hi:Example 1.}

{pmore} {inp:regress} {it:outcomevar treatment_dummy}{break}
		{inp:iegraph} {it:treatment_dummy} , {inp:basictitle({it:"Treatment Effect on Outcome"})}

{pmore}In the example above, there are only two treatment arms (treatment and
		control). {it:treatment_dummy} has a 1 for all treatment observations and
		a 0 for all control observations. The graph will have one bar for control and
		it shows the mean for {it:outcomevar} for all observations in control. The
		second bar in the graph will be the sum of that mean and the coefficient
		for {it:treatment_dummy} in the regression. The graph will also have the
		title: Treatment Effect on Outcome.

{pstd} {hi:Example 2.}

{pmore} {inp:regress} {it:income tmt_1 tmt_2 age education}{inp:, cluster(}{it:district}{inp:)}{break}
		{inp:iegraph} {it:tmt_1 tmt_2}{inp:, noconfbars yzero basictitle({it:"Treatment effect on income"}) }

{pmore}In the example above, the treatment effect on income in researched. There
		are three treatment arms; control, treatment 1 ({it:tmt_1}) and treatment
		2 ({it:tmt_2}). It is important that no observation has the value 1 in
		both {it:tmt_1} and {it:tmt_2} (i.e. no observation is in more than one
		treatment) and some observations must have the value 0 in both {it:tmt_1}
		and {it:tmt_2} (i.e. control observations). The variables {it:age} and
		{it:education} are covariates (control variables) and are not included
		in {cmd:iegraph}. {inp:noconfbars} omits the confidence interval bars
		, and {inp:yzero} sets the y-axis to start at 0.

{pstd} {hi:Example 3.}

{pmore} {inp:regress} {it:chld_wght time treat timeXtreat}{break}
		{inp:iegraph} {it:time treat timeXtreat} {inp:, basictitle({it:"Treatment effect on Child Weight (Diff-in-Diff)"})}

{pmore}In the example above, the data set is a panel data set with two time
		periods and the regression estimates the treatment effect on child weight
		using a Difference-in-Differences model.
		The dummy variable {it:time} indicates if it is time period 0 or 1.
		The dummy variable {it:treat} indicates if the observation is treatment
		or control. {it:timeXtreat} is the interaction term of {it:time}
		and {it:treat}. This the standard way to set up a Difference-in-Differences
		regression model.

{pstd} {hi:Example 4.}

{pmore} {inp:regress} {it:harvest T1 T2 T3 } {break}
		{inp:iegraph} {it:T1 T2 T3} {inp:, basictitle({it:"Treatment effect on harvest"})
		xlabel(,angle(45)) yzero ylabel(minmax) save({it:"Graph1.gph"})}

{pmore}The example above shows how to save a graph to disk. It also shows that
	most two-way graph options can be used. In this example the {cmd:iegraph}
	option {cdm:yzero} conflicts with the two-way option {cmd:ylabel(minmax)}.
	In such a case the user specified option takes precedence over {cmd:iegraph}
	options like {cdm:yzero}.

{title:Acknowledgements}

{phang}We would like to acknowledge the help in testing and proofreading we received in relation to this command and help file from (in alphabetic order):{p_end}
{pmore}Luiza Cardoso De Andrade{break}Michael Orevba{break}Ahmad Zia Wahdat{break}

{title:Authors}

{phang}Kristoffer Bjarkefur & Mrijan Rimal, The World Bank, DECIE

{phang}Please send bug-reports, suggestions and requests for clarifications
		 writing "ietoolkit iegraph" in the subject line to:{break}
		 kbjarkefur@worldbank.org

{phang}You can also see the code, make comments to the code, see the version
		 history of the code, and submit additions or edits to the code through
		 the github repository of ietoolkit:{break}
		 {browse "https://github.com/worldbank/ietoolkit"}


Include kernel tests in test suite

Include kernel tests in test suite

BIND=../../bin

roms="numb/test/eitherHexOrDecimal.rom"
roms+=" numb/test/tryConversion.rom"
roms+=" numb/test/tryUnsignedNumber.rom"

roms+=" dict/test/compareNames.rom"
roms+=" dict/test/compareNamesUpToLength.rom"
roms+=" dict/test/findWord.rom"

roms+=" kern/test/enqueue.rom"

roms+=" test/char.rom"
roms+=" test/dstk.rom"
roms+=" test/math.rom"
roms+=" test/oint.rom"
roms+=" test/scan.rom"

redo-ifchange $BIND/e
redo-ifchange $roms

for rom in $roms; do (e romfile $rom >&2); done

BIND=../../bin

roms="numb/test/eitherHexOrDecimal.rom"
roms+=" numb/test/tryConversion.rom"
roms+=" numb/test/tryUnsignedNumber.rom"

roms+=" dict/test/compareNames.rom"
roms+=" dict/test/compareNamesUpToLength.rom"
roms+=" dict/test/findWord.rom"

roms+=" test/char.rom"
roms+=" test/dstk.rom"
roms+=" test/math.rom"
roms+=" test/oint.rom"
roms+=" test/scan.rom"

redo-ifchange $BIND/e
redo-ifchange $roms

for rom in $roms; do (e romfile $rom >&2); done


Add transitivity cluster error analysis

Add transitivity cluster error analysis

#!/usr/bin/env python3
from collections import OrderedDict, defaultdict
from pathlib import Path
import os
import yaml
import pandas as pd
import pymysql
import redo
redo.ifchange('comap.py'); import comap

project, variation_id = redo.base.split('.')
project_path = Path('projects') / project

with redo.ifchange(
        evaluation_config=project_path / 'evaluation-config.yaml',
        evaluation=project + '.evaluation.yaml'
) as files:
    evaluation_config = yaml.load(files['evaluation_config'])
    evaluations = yaml.load(files['evaluation'])

databases = OrderedDict(evaluation_config['databases'])
outcome_ids = evaluation_config['outcome_ids']

def get_cuis_for_codes(coding_system, codes):
    res = defaultdict(lambda: set())
    if codes:
        query = """
            select distinct cui, code from MRCONSO
            where sab = %s and code in %s
        """
        with comap.umls_db.cursor() as cursor:
            cursor.execute(query, tuple(coding_system, codes))
            for cui, code in cursor.fetchall():
                res[code].add(cui)
    return res

def get_terms_for_cuis(cuis):
    res = defaultdict(lambda: [None, set()])
    if cuis:
        query = """
            select distinct cui, str, ispref = 'Y' and ts = 'P' and stt = 'PF' as preferred
            from MRCONSO where cui in %s
        """
        with comap.umls_db.cursor() as cursor:
            cursor.execute(query, (cuis))
            for cui, term, preferred in cursor.fetchall():
                if preferred:
                    res[cui][0] = term
                res[cui][1].add(term)
    return res

class Analysis(object):

    def in_database(self, outcome_id, database_id, coding_system, cm):
        return None

    def in_all_databases(self, outcome_id, cms):
        return None

class Terms(Analysis):

    def __init__(self, *positions):
        self.positions = list(positions)

    def in_database(outcome_id, database_id, coding_system, cm):
        """
        { code: { cui: {"name": str, "terms": { str } } }
        """
        codes = set()
        for position in self.positions:
            codes.update(cm[position])

        original_coding_system = coding_system
        if original_coding_system == 'RCD2':
            coding_system = 'RCD'
            translations = comap.translation_read_2to3(codes)
            codes = {
                code0
                for code in codes
                for code0 in translations.get(code, [])
            }

        cuis_for_codes = get_cuis_for_codes(coding_system, codes)
        terms_for_cuis = get_terms_for_cuis({
            cui
            for cuis in cuis_for_codes.values()
            for cui in cuis
        })

        res = OrderedDict()
        if original_coding_system == 'RCD2':
            backtranslation = comap.translation_read_3to2(codes)
            code_pairs = [
                (code, code2)
                for code in codes
                for code2 in backtranslation.get(code, [])
            ]
        else:
            code_pairs = [ (code, code) for code in codes ]

        for code, code2 in code_pairs:
            res[code2] = OrderedDict()
            for cui in sorted(cuis_for_codes[code]):
                name, terms = [term for term in terms_for_cuis[cui]]
                res[code2][cui] = sorted(terms)
        return res

class Clusters(Analysis):

    def __init__(self, positions, relations):
        self.positions = positions
        self.relations = relations

    def in_all_databases(self, outcome_id, cms):

        mapping = {}
        for database_id in databases.keys():
            mapping[database_id] = set()
            if database_id in cms:
                for position in self.positions:
                    mapping[database_id].update(cms[database_id][position])

        cuis = comap.mapping_to_max_recall_cuis(databases, mapping)
        relations = comap.relations_for_cuis(cuis, self.relations)
        connected_sets = comap.connected_sets(cuis, relations)
        # terms_for_cuis = get_terms_for_cuis(cuis)

        res = OrderedDict([
            ('max-sensitivity-cuis', len(cuis)),
            ('transitivity-sets', len(connected_sets)),
        ])
        # res = OrderedDict()
        # for root, connected in connected_sets.items():
            # res[root] = list()
            # for cui in [root] + [cui0 for cui0 in connected if cui0 != root]:
            #     res[root].append(OrderedDict([
            #         ('cui', cui),
            #         ('name', terms_for_cuis[cui][0] or next(terms_for_cuis[cui])),
            #     ]))
        return res

def run_analyses(analyses):
    res = OrderedDict()
    for outcome_id in outcome_ids:
        res[outcome_id] = OrderedDict()

        cms = {}
        for database_id in databases.keys():
            for_database = evaluations[outcome_id][database_id]
            if 'variations' in for_database:
                cms[database_id] = for_database['variations'][variation_id]['comparison']['codes']

        in_all_databases = OrderedDict()
        for analysis_id, analysis in analyses:
            analysis_result = analysis.in_all_databases(outcome_id, cms)
            if analysis_result is not None:
                in_all_databases[analysis_id] = analysis_result
        if in_all_databases:
            res[outcome_id]['analysis'] = in_all_databases

        for database_id, coding_system in databases.items():
            res[outcome_id][database_id] = OrderedDict()
            evaluation = evaluations[outcome_id][database_id]
            if 'variations' in evaluation:
                this_variation_id, variation = next(
                    (this_variation_id, variation)
                    for this_variation_id, variation in  evaluation['variations'].items()
                    if this_variation_id == variation_id
                )
                cm = variation['comparison']['codes']
                for analysis_id, analysis in analyses:
                    analysis_result = analysis.in_database(outcome_id, database_id, coding_system, cm)
                    if analysis_result is not None:
                        res[outcome_id][database_id]['analysis'][analysis_id] = analysis_result
    return res

if os.getenv('DOCTEST'):
    import doctest
    doctest.testmod(comap, report=True)
    doctest.testmod(report=True)

analyses = [
#    ('FN-terms', Terms('FN')),
    ('FN-clusters-RN-CHD-RB-PAR', Clusters(['FN'], ['RN', 'CHD', 'RB', 'PAR'])),
]

res = run_analyses(analyses)

with redo.output() as f:
    yaml.dump(res, f)

#!/usr/bin/env python3
from collections import OrderedDict, defaultdict
import yaml
from pathlib import Path
import pandas as pd
import pymysql
import redo
import comap # For YAML loader/constructor

project, variation_id = redo.base.split('.')
project_path = Path('projects') / project

with redo.ifchange(
        evaluation_config=project_path / 'evaluation-config.yaml',
        evaluation=project + '.evaluation.yaml'
) as files:
    evaluation_config = yaml.load(files['evaluation_config'])
    evaluation = yaml.load(files['evaluation'])

databases = OrderedDict(evaluation_config['databases'])
outcome_ids = evaluation_config['outcome_ids']

def get_cuis_for_codes(coding_system, codes):
    res = defaultdict(lambda: set())
    if codes:
        query = "select distinct code, cui from MRCONSO where sab = %s and code in ({})" \
            .format(', '.join(['%s'] * len(codes)))
        with comap.umls_db.cursor() as cursor:
            cursor.execute(query, tuple([coding_system] + list(codes)))
            for row in cursor.fetchall():
                code, cui = row
                res[code].add(cui)
    return res

def get_terms_for_cuis(cuis):
    res = defaultdict(lambda: [None, set()])
    if cuis:
        query = "select distinct cui, str, ispref = 'Y' and ts = 'p' and stt = 'PF' as preferred from MRCONSO where cui in ({})" \
            .format(', '.join(['%s'] * len(cuis)))
        with comap.umls_db.cursor() as cursor:
            cursor.execute(query, tuple(cuis))
            for row in cursor.fetchall():
                cui, term, preferred = row
                if preferred:
                    res[cui][0] = term
                res[cui][1].add(term)
    return res

def error_analysis(coding_system, codes):
    """
    { code: { cui: {"name": str, "terms": { str } } }
    """

    original_coding_system = coding_system
    if original_coding_system == 'RCD2':
        coding_system = 'RCD'
        translations = comap.translation_read_2to3(codes)
        codes = set(code0
                    for code in codes
                    for code0 in translations.get(code, []))

    cuis_for_codes = get_cuis_for_codes(coding_system, codes)
    terms_for_cuis = get_terms_for_cuis([
        cui for cuis in cuis_for_codes.values() for cui in cuis
    ])

    if original_coding_system == 'RCD2':
        backtranslation = comap.translation_read_3to2(codes)
        # codes = set(code0
        #             for code in codes
        #             for code0 in backtranslation[code])

    res = OrderedDict()
    if original_coding_system == 'RCD2':
        code_pairs = [
            (code, code2)
            for code in codes
            for code2 in backtranslation.get(code, [])
        ]
    else:
        code_pairs = [ (code, code) for code in codes ]

    for code, code2 in code_pairs:
        res[code2] = OrderedDict()
        for cui in sorted(cuis_for_codes[code]):
            name, terms = [term for term in terms_for_cuis[cui]]
            res[code2][cui] = sorted(terms)
    return res

error_analyses = OrderedDict()
for outcome_id in outcome_ids:
    error_analyses[outcome_id] = OrderedDict()
    for database_name, _ in databases.items():
        evaluation_for_database = evaluation[outcome_id][database_name]
        error_analyses[outcome_id][database_name] = OrderedDict()
        if 'variations' in evaluation_for_database:
            variation = next(
                variation
                for id, variation in  evaluation_for_database['variations'].items()
                if id == variation_id
            )
            codes = variation['comparison']['codes']
            res = OrderedDict([
                ('FN', error_analysis(databases[database_name], codes['FN']))
            ])
            error_analyses[outcome_id][database_name] = res

with redo.output() as f:
    yaml.dump(error_analyses, f)


last changes

last changes

*insheet using "C:\Users\rsanchez\Dropbox\Tesis_ITAM_RS\Base_Tesis.csv", comma case
*save "C:\Users\rsanchez\Dropbox\tesis\Cuerpo\FINAL\BDTesis.dta", replace
*Clear and set directory;
	clear
	cd "C:\Users\rsanchez\Dropbox\tesis\Cuerpo\FINAL\"
*database query;
	use "BDTesis.dta"
	xtset id year
*Clean code

	*keep id code MunName km2 dens_mar year rivalry_homicide hom_rate_riv lnPIB poverty ind_marginacion niv_pobreza  mean_edu niv_rezago_edu prop_sinprim prop_desempleo coef_gini PrP15PRIM lnUnemployment tomasC Laboratorios lab tomas  Border1 Border2 Border3 Border4 route port PortGolfo PortPacifico coca mar opi e_mar e_pop erad_mar erad_pop Imar Icoca Iopi  erad_marijuana co_count conflicto_12 alianza_sinconf violentconflict estructura 	tratamientosequiamar ttratamientosequiamar 	PR_Secuestro S_Extorsion PR_Extorsion  S_Robo  PR_Robo  S_Allanamiento  PR_Allanamiento  S_Secuestro  eficiencia  hom_rate_riv SPATIALLAG WEFICI LWhom_rate_riv Lhom_rate_riv Leficiencia  LWeficiencia agresion_a_militares

*Variable construction;
	gen coef_gini_2 = coef_gini*coef_gini
	gen lnPIB_2 = lnPIB*lnPIB

	*Muerte de Capo por Ubicacin de Crtel
	drop capo
	gen capo=0
	replace capo=1 if year==2007 & (beltran==1)
	replace capo=1 if year==2008 & (sinaloa==1 | zetas==1)
	replace capo=1 if year==2009 & (golfo==1 | zetas==1 | sinaloa==1 | beltran==1)
	replace capo=1 if year==2010 & (golfo==1 | sinaloa==1 | beltran==1 | familia==1 | tijuana==1)
	replace capo=0 if capo==.

*fe not used because of time invariant variables
*re
*The random effects assumption (made in a random effects model) is that the individual specific effects are uncorrelated with the independent variables. The fixed effect assumption is that the individual specific effect is correlated with the independent variables. If the random effects assumption holds, the random effects model is more efficient than the fixed effects model. However, if this assumption does not hold (i.e., if the DurbinWatson test fails), the random effects model is not consistent.

*Tabla 1) Parametrizacin de la violencia (Nivel Municipal)
	*Modelo 1
		xtpoisson hom_rate_riv lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim, pa corr(unstructured)
		outreg2 using tabla1, replace title(Parametrizacin de violencia - Poisson) ctitle(Tasa Homicidios Relacionados)  tex(fragment)
	*Modelo 2
		xtpoisson hom_rate_riv tomasC Laboratorios erad_mar erad_pop Imar Icoca Iopi Border1 Border2 Border3 Border4 route PortGolfo PortPacifico, pa corr(unstructured)
		outreg2 using tabla1, append ctitle(tasa Homicidios Relacionados)  tex(fragment)
	*Modelo 3
		xtpoisson hom_rate_riv  L.eficiencia L.WEFICI L.hom_rate_riv L.SPATIALLAG, pa corr(unstructured)
		outreg2 using tabla1, append ctitle(tasa Homicidios Relacionados)  tex(fragment)
	*Modelo 4
		xtpoisson hom_rate_riv lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim tomasC Laboratorios erad_mar erad_pop  Imar Icoca Iopi Border1 Border2 Border3 Border4 route PortGolfo PortPacifico, pa corr(unstructured)
		outreg2 using tabla1, append ctitle(tasa Homicidios Relacionados)  tex(fragment)
	*Modelo 5
		xtpoisson hom_rate_riv lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim tomasC Laboratorios erad_mar erad_pop  Imar Icoca Iopi Border1 Border2 Border3 Border4 route PortGolfo PortPacifico L.eficiencia L.WEFICI L.hom_rate_riv L.SPATIALLAG, pa corr(unstructured)
		outreg2 using tabla1, append ctitle(tasa Homicidios Relacionados)  tex(fragment)
	*Modelo 6
		xtpoisson hom_rate_riv lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim tomasC Laboratorios erad_mar erad_pop  Imar Icoca Iopi Border1 Border2 Border3 Border4 route PortGolfo PortPacifico L.eficiencia L.WEFICI L.hom_rate_riv L.SPATIALLAG capo, pa corr(unstructured)
		outreg2 using tabla1, append ctitle(tasa Homicidios Relacionados Control Capos)  tex(fragment)

*Elegir si mantenemos pa o no
*Tabla 2) Anlisis de la competencia - *estructura control cluster municipal socioeconmico usarlo?
	*co_count
		xtpoisson co_count Border1 Border2 Border3 Border4 route PortGolfo PortPacifico coca mar opi lab tomas e_mar e_pop i.estructura
		outreg2 using tabla2, replace title(competencia) ctitle(Nmero_Crteles) tex(fragment)

		xtlogit violentconflict Border1 Border2 Border3 Border4 route PortGolfo PortPacifico coca mar opi lab tomas e_mar e_pop i.estructura
		outreg2 using tabla2, append ctitle(contrarios) eq(auto) tex(fragment)

		xtpoisson Ejecucionconflicto Border1 Border2 Border3 Border4 route PortGolfo PortPacifico coca mar opi lab tomas e_mar e_pop i.estructura
		outreg2 using tabla2, append ctitle(contrarios) eq(auto) tex(fragment)

		xtpoisson agresion_a_militares Border1 Border2 Border3 Border4 route PortGolfo PortPacifico coca mar opi lab tomas e_mar e_pop i.estructura, pa corr(unstructured)
		outreg2 using tabla2, append ctitle(militares) tex(fragment)

		*****************************
		*****************************
		*****************************

***********************************;
********Anlisis de cluster********;
	clear
	cd "C:\Users\rsanchez\Dropbox\tesis\Cuerpo\FINAL\"
*database query;
	use "BDTesis.dta"
	xtset id year
	keep if year==2008

*1) Clusterizacin Socioeconmica -Wardslinkage-;

	egen std_mean_edu = std(mean_edu)
	*sutex2 lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim std_mean_edu LWhom_rate_riv Lhom_rate_riv Leficiencia LWeficiencia , minmax

	sum lnPIB coef_gini prop_desempleo std_mean_edu prop_sinprim
	cluster wardslinkage lnPIB coef_gini prop_desempleo std_mean_edu prop_sinprim, measure(L2) name(cluster1)

	cluster dendrogram cluster1, cutnumber(5) showcount
	cluster generate estructura_= groups(3), name(cluster1) ties(error)
	drop cluster1_id cluster1_ord cluster1_hgt

*2) Clusterizacin sobre violencia e impunidad -Wardslinkage-;Leficiencia
	cluster  wardslinkage  LWhom_rate_riv Lhom_rate_riv LWeficiencia , measure(L2) name(cluster2)
	cluster dendrogram cluster2, cutnumber(5)
	cluster generate categoriaviolencia2= groups(3), name(cluster2) ties(error)
	table categoriaviolencia2

*3) Categora industria
	gen industria=0
	replace industria=3 if (lab==1 |Border1==1 | Border2==1 | Border3==1 | Border4==1 | PortPacifico==1)
	replace industria=2 if (PortGolfo==1 | e_mar==1 | e_pop==1 | mar==1 | coca==1 | opi==1)
	replace industria=1 if (route==1 | tomas==1)
tabstat hom_rate_riv , statistics( mean var median ) by(industria) columns(variables)

*COSTO DE OPORTUNIDAD -> re-ordenamiento por nivel de violencia;
tabstat hom_rate_riv , statistics( mean var median ) by(estructura_) columns(variables)
	gen costo_oportunidad = 0
	replace costo_oportunidad=3 if estructura_==1
	replace costo_oportunidad=2 if estructura_==2
	replace costo_oportunidad=1 if estructura_==3

tabstat hom_rate_riv , statistics( mean var median ) by(costo_oportunidad) columns(variables)
tabstat PIB ind_marginacion prop_desempleo  mean_edu if hom_rate_riv>0, statistics( mean var median ) by(costo_oportunidad) columns(variables)
tabstat hom_rate_riv , statistics( mean median var count) by(costo_oportunidad) columns(variables)

*CATEGORIA VIOLENCIA -> re-ordenamiento por nivel de violencia;
tabstat hom_rate_riv , statistics( mean var median ) by(categoriaviolencia2) columns(variables)
	gen categoriaviolencia = 0
	replace categoriaviolencia=1 if categoriaviolencia2==1
	replace categoriaviolencia=2 if categoriaviolencia2==2
	replace categoriaviolencia=3 if categoriaviolencia2==3
	tab categoriaviolencia

tabstat hom_rate_riv , statistics( mean var median count) by(categoriaviolencia) columns(variables)
tabstat LWhom_rate_riv Lhom_rate_riv Leficiencia LWeficiencia, statistics( mean var median ) by(categoriaviolencia) columns(variables)

sum hom_rate_riv, detail
centile (hom_rate_riv) , centile(75, 90,100)

gen riesgo_violencia = 0
	replace riesgo_violencia = 1 if  hom_rate_riv <=   .9479672
	replace riesgo_violencia = 2 if  hom_rate_riv <=   10.16147 & riesgo_violencia  !=1
	replace riesgo_violencia = 3 if  hom_rate_riv > 10.16147 & riesgo_violencia  !=1 & riesgo_violencia  !=2

tabstat hom_rate_riv , statistics( mean var median count) by(riesgo_violencia) columns(variables)

drop categoriaviolencia2
drop estructura_

save "C:\Users\rsanchez\Dropbox\tesis\Cuerpo\FINAL\shape_bd_full.dta", replace

keep costo_oportunidad categoriaviolencia  industria riesgo_violencia hom_rate_riv
outsheet using "C:\Users\rsanchez\Dropbox\tesis\decisiontree_test.csv", comma replace

***********************************;
***************Anexo 1*************;
clear
cd "C:\Users\rsanchez\Dropbox\tesis\Cuerpo\FINAL\"
use "BDTesis.dta"
xtset id year

	*Sequia (Quasi Experimental Method)
	******************
		*Anlsis de Diferencia en Diferencias (grupo tratamiento: sequia)-> con kernel Maching
		*Relies on the panel structure of the data (usually two periods: based line and follow up).
		*Control for unobservable and time invariant characteristics. Control for observable characteristics if available.
		*psm
		*Stata module
		ssc install diff
	******************
		global breps 100
		*global xlist estructura
		*global xlist lnPIB PrP15PRIM lnUnemployment  coef_gini  mean_edu  Border1 Border2 Border3 route PortPacifico
		global xlist lnPIB PrP15PRIM lnUnemployment ind_marginacion prop_sinprim niv_pobreza niv_rezago_edu coef_gini
		global xlist lnPIB PrP15PRIM mean_edu lnUnemployment coef_gini

	*Efecto de la sequia en la erradicacin de mariguana
		*Quantile diff in diff
		diff erad_marijuana, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist) kernel rcs qdid(.5)
		diff erad_marijuana, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist) id(code) kernel rcs
		diff erad_marijuana, t(tratamientosequiamar) p(ttratamientosequiamar)
			*Balancing - t test
			diff erad_marijuana, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist)  test

		*Efecto de la sequia en la violencia
		diff hom_rate_riv, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist) id(code) kernel rcs qdid(.5)
		diff hom_rate_riv, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist) id(code) kernel rcs
		diff hom_rate_riv, t(tratamientosequiamar) p(ttratamientosequiamar)
			*Balancing - t test
			diff hom_rate_riv, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist)  test

***********************************;
***************Anexo 3*************;
	*Efecto de la violencia en otros crmenes  (Mtodo de Variables instrumentales)
	******************
		*Se instrumentaliza utilizando las principales variables de posicin geogrfica del municipio
		*Para obtener un estimador consistente mediante el mtodo de variables instrumentales se deben cumplir dos condiciones.
		*1) Relevancia del instrumento con nuestra variable de inters (Cov(Homicidios Relacionados, Zi)).
		*2) Exogeneidad o restriccin de exclusin: que el instrumento no est correlacionado con el error (Cov(Ui,Zi)=0).
	******************
	xtivreg S_Secuestro lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, replace title(Instrumentos) ctitle(S_Secuestro) tex(fragment)
	xtivreg PR_Secuestro lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(PR_Secuestro)  tex(fragment)
	xtivreg S_Extorsion  lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append  ctitle(S_Extorsion) tex(fragment)
	xtivreg PR_Extorsion  lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(PR_Extorsion) tex(fragment)
	xtivreg S_Robo lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(S_Robo) tex(fragment)
	xtivreg PR_Robo lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(PR_Robo) tex(fragment)
	xtivreg S_Allanamiento lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(S_Allanamiento) tex(fragment)
	xtivreg PR_Allanamiento lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(PR_Allanamiento) tex(fragment)

*insheet using "C:\Users\rsanchez\Dropbox\Tesis_ITAM_RS\Base_Tesis.csv", comma case
*save "C:\Users\rsanchez\Dropbox\tesis\Cuerpo\FINAL\BDTesis.dta", replace
*Clear and set directory;
	clear
	cd "C:\Users\rsanchez\Dropbox\tesis\Cuerpo\FINAL\"
*database query;
	use "BDTesis.dta"
	xtset id year
*Clean code

	*keep id code MunName km2 dens_mar year rivalry_homicide hom_rate_riv lnPIB poverty ind_marginacion niv_pobreza  mean_edu niv_rezago_edu prop_sinprim prop_desempleo coef_gini PrP15PRIM lnUnemployment tomasC Laboratorios lab tomas  Border1 Border2 Border3 Border4 route port PortGolfo PortPacifico coca mar opi e_mar e_pop erad_mar erad_pop Imar Icoca Iopi  erad_marijuana co_count conflicto_12 alianza_sinconf violentconflict estructura 	tratamientosequiamar ttratamientosequiamar 	PR_Secuestro S_Extorsion PR_Extorsion  S_Robo  PR_Robo  S_Allanamiento  PR_Allanamiento  S_Secuestro  eficiencia  hom_rate_riv SPATIALLAG WEFICI LWhom_rate_riv Lhom_rate_riv Leficiencia  LWeficiencia agresion_a_militares

*Variable construction;
	gen coef_gini_2 = coef_gini*coef_gini
	gen lnPIB_2 = lnPIB*lnPIB

	*Muerte de Capo por Ubicacin de Crtel
	drop capo
	gen capo=0
	replace capo=1 if year==2007 & (beltran==1)
	replace capo=1 if year==2008 & (sinaloa==1 | zetas==1)
	replace capo=1 if year==2009 & (golfo==1 | zetas==1 | sinaloa==1 | beltran==1)
	replace capo=1 if year==2010 & (golfo==1 | sinaloa==1 | beltran==1 | familia==1 | tijuana==1)
	replace capo=0 if capo==.

*fe not used because of time invariant variables
*re
*The random effects assumption (made in a random effects model) is that the individual specific effects are uncorrelated with the independent variables. The fixed effect assumption is that the individual specific effect is correlated with the independent variables. If the random effects assumption holds, the random effects model is more efficient than the fixed effects model. However, if this assumption does not hold (i.e., if the DurbinWatson test fails), the random effects model is not consistent.

*Tabla 1) Parametrizacin de la violencia (Nivel Municipal)
	*Modelo 1
		xtpoisson hom_rate_riv lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim, pa corr(unstructured)
		outreg2 using tabla1, replace title(Parametrizacin de violencia - Poisson) ctitle(Tasa Homicidios Relacionados)  tex(fragment)
	*Modelo 2
		xtpoisson hom_rate_riv tomasC Laboratorios erad_mar erad_pop Imar Icoca Iopi Border1 Border2 Border3 Border4 route PortGolfo PortPacifico, pa corr(unstructured)
		outreg2 using tabla1, append ctitle(tasa Homicidios Relacionados)  tex(fragment)
	*Modelo 3
		xtpoisson hom_rate_riv  L.eficiencia L.WEFICI L.hom_rate_riv L.SPATIALLAG, pa corr(unstructured)
		outreg2 using tabla1, append ctitle(tasa Homicidios Relacionados)  tex(fragment)
	*Modelo 4
		xtpoisson hom_rate_riv lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim tomasC Laboratorios erad_mar erad_pop  Imar Icoca Iopi Border1 Border2 Border3 Border4 route PortGolfo PortPacifico, pa corr(unstructured)
		outreg2 using tabla1, append ctitle(tasa Homicidios Relacionados)  tex(fragment)
	*Modelo 5
		xtpoisson hom_rate_riv lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim tomasC Laboratorios erad_mar erad_pop  Imar Icoca Iopi Border1 Border2 Border3 Border4 route PortGolfo PortPacifico L.eficiencia L.WEFICI L.hom_rate_riv L.SPATIALLAG, pa corr(unstructured)
		outreg2 using tabla1, append ctitle(tasa Homicidios Relacionados)  tex(fragment)
	*Modelo 6
		xtpoisson hom_rate_riv lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim tomasC Laboratorios erad_mar erad_pop  Imar Icoca Iopi Border1 Border2 Border3 Border4 route PortGolfo PortPacifico L.eficiencia L.WEFICI L.hom_rate_riv L.SPATIALLAG capo, pa corr(unstructured)
		outreg2 using tabla1, append ctitle(tasa Homicidios Relacionados Control Capos)  tex(fragment)

*Elegir si mantenemos pa o no
*Tabla 2) Anlisis de la competencia - *estructura control cluster municipal socioeconmico usarlo?
	*co_count
		xtpoisson co_count Border1 Border2 Border3 Border4 route PortGolfo PortPacifico coca mar opi lab tomas e_mar e_pop i.estructura
		outreg2 using tabla2, replace title(competencia) ctitle(Nmero_Crteles) tex(fragment)

		xtlogit violentconflict Border1 Border2 Border3 Border4 route PortGolfo PortPacifico coca mar opi lab tomas e_mar e_pop i.estructura
		outreg2 using tabla2, append ctitle(contrarios) eq(auto) tex(fragment)

		xtpoisson Ejecucionconflicto Border1 Border2 Border3 Border4 route PortGolfo PortPacifico coca mar opi lab tomas e_mar e_pop i.estructura
		outreg2 using tabla2, append ctitle(contrarios) eq(auto) tex(fragment)

		xtpoisson agresion_a_militares Border1 Border2 Border3 Border4 route PortGolfo PortPacifico coca mar opi lab tomas e_mar e_pop i.estructura, pa corr(unstructured)
		outreg2 using tabla2, append ctitle(militares) tex(fragment)

		*****************************
		*****************************
		*****************************

***********************************;
********Anlisis de cluster********;
	clear
	cd "C:\Users\rsanchez\Dropbox\tesis\Cuerpo\FINAL\"
*database query;
	use "BDTesis.dta"
	xtset id year
	*keep if year==2008 & hom_rate_riv > 0
	keep if year==2008

*1) Clusterizacin Socioeconmica -Wardslinkage-;

	egen std_mean_edu = std(mean_edu)
	*sutex2 lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim std_mean_edu LWhom_rate_riv Lhom_rate_riv Leficiencia LWeficiencia , minmax

	sum lnPIB coef_gini prop_desempleo std_mean_edu prop_sinprim
	*subset
	cluster wardslinkage lnPIB coef_gini prop_desempleo mean_edu prop_sinprim, measure(L2) name(cluster1)
	*full
	cluster wardslinkage lnPIB coef_gini prop_desempleo std_mean_edu prop_sinprim, measure(L2) name(cluster1)

	cluster dendrogram cluster1, cutnumber(5) showcount
	cluster generate estructura_= groups(3), name(cluster1) ties(error)
	drop cluster1_id cluster1_ord cluster1_hgt

*2) Clusterizacin sobre violencia e impunidad -Wardslinkage-;Leficiencia
	cluster  wardslinkage  LWhom_rate_riv Lhom_rate_riv LWeficiencia , measure(L2) name(cluster2)
	cluster dendrogram cluster2, cutnumber(5)
	cluster generate categoriaviolencia2= groups(3), name(cluster2) ties(error)
	table categoriaviolencia2

*3) Categora industria
	gen industria=0
	replace industria=3 if (lab==1 |Border1==1 | Border2==1 | Border3==1 | Border4==1 | PortPacifico==1)
	replace industria=2 if (PortGolfo==1 | e_mar==1 | e_pop==1 | mar==1 | coca==1 | opi==1)
	replace industria=1 if (route==1 | tomas==1)
tabstat hom_rate_riv , statistics( mean var median ) by(industria) columns(variables)

*COSTO DE OPORTUNIDAD -> re-ordenamiento por nivel de violencia;
tabstat hom_rate_riv , statistics( mean var median ) by(estructura_) columns(variables)
	gen costo_oportunidad = 0
	*subset
	replace costo_oportunidad=1 if estructura_==1
	replace costo_oportunidad=2 if estructura_==2
	replace costo_oportunidad=3 if estructura_==3
	*full
	replace costo_oportunidad=3 if estructura_==1
	replace costo_oportunidad=2 if estructura_==2
	replace costo_oportunidad=1 if estructura_==3

tabstat hom_rate_riv , statistics( mean var median ) by(costo_oportunidad) columns(variables)
*full - ms violencia en municipios ricos-
*subset ms violencia en municipios pobres

tabstat PIB ind_marginacion prop_desempleo  mean_edu if hom_rate_riv>0, statistics( mean var median ) by(costo_oportunidad) columns(variables)
tabstat hom_rate_riv , statistics( mean median var count) by(costo_oportunidad) columns(variables)

*CATEGORIA VIOLENCIA -> re-ordenamiento por nivel de violencia;
tabstat hom_rate_riv , statistics( mean var median ) by(categoriaviolencia2) columns(variables)
	gen categoriaviolencia = 0
	*subset
	replace categoriaviolencia=2 if categoriaviolencia2==1
	replace categoriaviolencia=1 if categoriaviolencia2==2
	replace categoriaviolencia=3 if categoriaviolencia2==3
	*full
	replace categoriaviolencia=1 if categoriaviolencia2==1
	replace categoriaviolencia=2 if categoriaviolencia2==2
	replace categoriaviolencia=3 if categoriaviolencia2==3
	tab categoriaviolencia

tabstat hom_rate_riv , statistics( mean var median count) by(categoriaviolencia) columns(variables)
tabstat LWhom_rate_riv Lhom_rate_riv Leficiencia LWeficiencia, statistics( mean var median ) by(categoriaviolencia) columns(variables)

sum hom_rate_riv, detail
centile (hom_rate_riv) , centile(75, 90,100)

gen riesgo_violencia = 0
*full
	replace riesgo_violencia = 1 if  hom_rate_riv <=   .9479672
	replace riesgo_violencia = 2 if  hom_rate_riv <=   10.16147 & riesgo_violencia  !=1
	replace riesgo_violencia = 3 if  hom_rate_riv > 10.16147 & riesgo_violencia  !=1 & riesgo_violencia  !=2
*subset? checar
	replace riesgo_violencia = 1 if  hom_rate_riv <=   25.81986
	replace riesgo_violencia = 2 if  hom_rate_riv <=   66.885  & riesgo_violencia  !=1
	replace riesgo_violencia = 3 if  hom_rate_riv > 799.7218  & riesgo_violencia  !=1 & riesgo_violencia  !=2

tabstat hom_rate_riv , statistics( mean var median count) by(riesgo_violencia) columns(variables)

save "C:\Users\rsanchez\Dropbox\tesis\Cuerpo\FINAL\shape_bd.dta", replace

*subset

*full
gen riesgo_violencia = 0
replace riesgo_violencia = 1 if  hom_rate_riv <=   25.81986
replace riesgo_violencia = 2 if  hom_rate_riv <=   66.885  & riesgo_violencia  !=1
replace riesgo_violencia = 3 if  hom_rate_riv > 799.7218  & riesgo_violencia  !=1 & riesgo_violencia  !=2
tabstat hom_rate_riv , statistics( mean var median count) by(riesgo_violencia) columns(variables)

keep costo_oportunidad categoriaviolencia  industria riesgo_violencia
outsheet using "C:\Users\rsanchez\Dropbox\tesis\decisiontree_test_subset.csv", comma replace

***********************************;
***************Anexo 1*************;
clear
cd "C:\Users\rsanchez\Dropbox\tesis\Cuerpo\FINAL\"
use "BDTesis.dta"
xtset id year

	*Sequia (Quasi Experimental Method)
	******************
		*Anlsis de Diferencia en Diferencias (grupo tratamiento: sequia)-> con kernel Maching
		*Relies on the panel structure of the data (usually two periods: based line and follow up).
		*Control for unobservable and time invariant characteristics. Control for observable characteristics if available.
		*psm
		*Stata module
		ssc install diff
	******************
		global breps 100
		*global xlist estructura
		*global xlist lnPIB PrP15PRIM lnUnemployment  coef_gini  mean_edu  Border1 Border2 Border3 route PortPacifico
		global xlist lnPIB PrP15PRIM lnUnemployment ind_marginacion prop_sinprim niv_pobreza niv_rezago_edu coef_gini
		global xlist lnPIB PrP15PRIM mean_edu lnUnemployment coef_gini

	*Efecto de la sequia en la erradicacin de mariguana
		*Quantile diff in diff
		diff erad_marijuana, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist) kernel rcs qdid(.5)
		diff erad_marijuana, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist) id(code) kernel rcs
		diff erad_marijuana, t(tratamientosequiamar) p(ttratamientosequiamar)
			*Balancing - t test
			diff erad_marijuana, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist)  test

		*Efecto de la sequia en la violencia
		diff hom_rate_riv, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist) id(code) kernel rcs qdid(.5)
		diff hom_rate_riv, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist) id(code) kernel rcs
		diff hom_rate_riv, t(tratamientosequiamar) p(ttratamientosequiamar)
			*Balancing - t test
			diff hom_rate_riv, t(tratamientosequiamar) p(ttratamientosequiamar) cov($xlist)  test

***********************************;
***************Anexo 3*************;
	*Efecto de la violencia en otros crmenes  (Mtodo de Variables instrumentales)
	******************
		*Se instrumentaliza utilizando las principales variables de posicin geogrfica del municipio
		*Para obtener un estimador consistente mediante el mtodo de variables instrumentales se deben cumplir dos condiciones.
		*1) Relevancia del instrumento con nuestra variable de inters (Cov(Homicidios Relacionados, Zi)).
		*2) Exogeneidad o restriccin de exclusin: que el instrumento no est correlacionado con el error (Cov(Ui,Zi)=0).
	******************
	xtivreg S_Secuestro lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, replace title(Instrumentos) ctitle(S_Secuestro) tex(fragment)
	xtivreg PR_Secuestro lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(PR_Secuestro)  tex(fragment)
	xtivreg S_Extorsion  lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append  ctitle(S_Extorsion) tex(fragment)
	xtivreg PR_Extorsion  lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(PR_Extorsion) tex(fragment)
	xtivreg S_Robo lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(S_Robo) tex(fragment)
	xtivreg PR_Robo lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(PR_Robo) tex(fragment)
	xtivreg S_Allanamiento lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(S_Allanamiento) tex(fragment)
	xtivreg PR_Allanamiento lnPIB  ind_marginacion niv_pobreza  coef_gini prop_desempleo mean_edu prop_sinprim (rivalry_homicide = Border1 Border2 Border3 route port), re
	outreg2 using instrumento.doc, append ctitle(PR_Allanamiento) tex(fragment)


meanline done

meanline done

clear all
set more off
set matsize 11000
program drop _all

qui adopath + "/mnt/nfs6/wikipedia.proj/jmp/jmpscripts/stata/ado"
cd ${path}

// this declares global vars
declare_global
global time "quarter"

// 0. Data

// 0.1 make cleancnty, changesets data
clear
preparebasic

// 0.2 make relevant dependent variables
use ${stash}cleanchangeset1, clear
makedv_uid fips
balancepanel_uid fips
maketime
mergebasic
qui destring, replace
save ${stash}panelfips, replace

// 0.3 make node and way datasets
preparenode

use ${stash}mergemaster_node, clear
makedv fips node
balancepanel fips node
save ${stash}panelnode, replace

// 0.3 make way dataset
prepareway

use ${stash}way_stash, clear
makedv fips way
save ${stash}mergemaster_way, replace

use ${stash}mergemaster_way, clear
balancepanel fips way
merge m:1 fips using ${stash}cleancnty, keep(master match) nogen
save ${stash}panelway, replace

// 1. Summary Stats
// Panel A: By county Level
program drop _all
use ${stash}panelfips, clear
bysort fips: drop if _n > 1
global tabname "summary_county"
local vars "treat aland_sqmi emp_earnings age_median educ_college emp_computer"
makesummary "`vars'"

// Panel B: By county/time Level
use ${stash}panelfips, clear
global tabname "summary_panel"
local vars "year pop_year numcontrib numusers numnewusers numserious18 numnewusers_t2 numnewusers_c18"
makesummary "`vars'"

// 2. Treatment Control Balance
use ${stash}panelfips, clear
makehist cntypop kdensity "Population"
makehist emp_earnings kdensity "Earnings"
makehist age_median kdensity "Median Age"
makehist educ_college kdensity "Num. College Educated"
** makehist region hist

// 2. TODO: TTest or regressions treatment / control

//  3.1 Meanline charts (raw data)
makemeanline numcontrib time 2011 "Contributions"
makemeanline numusers time 2011 "Users"
makemeanline numserious18 time 2011 "Super Users"

// 3.2 Cross Sectional Regressions
crossreg

// 3.3 Diff in Diff --- Baseline
local outcomes "numcontrib numuser numserious90"
dd_simple "`outcomes'" run xtreg
dd_simple "`outcomes'" run xtpoisson
dd_simple "`outcomes'" write ddsimple_baseline

// 3.4 Diff in Diff --- Additional
local outcomes "numnewusers numnewusers6 numnewusers90"
dd_simple "`outcomes'" run xtreg
dd_simple "`outcomes'" run xtpoisson
dd_simple "`outcomes'" write ddsimple_additional

// 3.5 Diff in Diff Picture
ddchart numuser
ddchart numcontrib
ddchart numserious90

// 4. How do results vary?
// takes a while
makehetero

// 5. Individual Level Data

// 5.0 create data
makeperson

//5.1 Summary
use ${stash}paneluid, clear

label variable treat "TIGER"
label variable year "YEAR"
label variable numcontrib "Contribs"
label variable numcontrib_home "Home County"
label variable numcontrib_treat "TIGER"
label variable numcontrib_notreat "Non-TIGER"
label variable numcontrib_statenotreat "State(Control)"
label variable numcounties "Num. Counties"

local vars "treat year numcontrib numcontrib_home numcontrib_treat numcontrib_notreat numcontrib_statenotreat numcounties"
global tabname "summary_uid"
makesummary "`vars'"

//Regressions
ddperson maketables
ddperson makechart

// 6. Map Quality

//6.0 summary
use ${stash}panelotherlayers, clear

label variable treat "TIGER"
label variable year "Year"
label variable otherlayer "NonStreet Layers"
label variable amenities "Amenities"
label variable bld_addr "Building/Addresses"
label variable numclass4 "Trails/Bikepaths"

label variable numattrib1 "Attributes:Major Highway"
label variable numattrib2 "Attributes:Class2"
label variable numattrib3 "Attributes:Class3"
label variable numattrib4 "Attributes:Trails etc."

local vars "treat year otherlayer amenities bld_addr numclass4 numattrib1 numattrib2 numattrib3 numattrib4"
global tabname "summary_otherl"
makesummary "`vars'"

// 6.1 Street data completeness

// 6.2 Other layers
reg_layers

// 6.3 Highway level regressions

// 7. Craiglist
reg_cl

////////// OLD //
 ** Analysis

// 0. Data

// 0.1 make fips dataset
use ${stash}mergemaster1, clear

balancepanel fips
save ${stash}panelfips, replace

// 0.2 make node dataset
preparenode

use ${stash}mergemaster_node, clear
makedv fips node
balancepanel fips node
save ${stash}panelnode, replace

// 0.3 make way dataset
prepareway
makedv fips way
balancepanel fips way
save ${stash}panelway, replace

// 3. Baseline effects (xtpoisson, cluster at unit level)

// 3.2 FIPS Sample -- DD

/// 3.2.1 -- contrib, users, superu
program drop _all

clear
local dv "numcontrib numuser numserious90"
local unit fips
rundd panelfips `unit' "`dv'"
diffindiff xtpoisson "`dv'" `unit' 2014 write tab_3.2.1 "Contributions" "Users" "Super Users"

/// 3.2.2 -- newusers, newusers6, newuserssuper
clear
local dv "numnewusers numnewusers6 numnewusers90"
local unit fips
rundd panelfips `unit' "`dv'"
diffindiff xtpoisson "`dv'" `unit' 2014 write tab_3.2.2 "New Users" "New Users(6+)" "New Super Users"

/// 3.2.3 -- streetcontrib, nonstreet contrib, amenities
// TODO

use ${stash}panelnode, clear
program drop _all

diffindiff xtpoisson fips run tab_test numcontrib numuser

/// 3.2.4 -- completeness
// TODO

// 3.2.5 -- Diff in Diff charts
// TODO : make this more flexible
// TODO : this works only for fips
ddchart

// 3.3 Robustness : Drop empty counties
// TODO
//  e. Repeat (a) by dropping empty counties
//  f. Repeat (a) by dropping west
//  g. Repeat (a) for Tile Sample, MSA sample
//  h. Repeat (a) with matched estimators
//  i. Repeat (a) with spatially clustered se, Region X Time trends

// 4 Heterogenous Effects

// 1. Large vs. small
use ${stash}panelfips, clear
gen large = (cntypop > 100000)

diffindiff2 large LARGE tab_4.1 run numcontrib numuser numserious90 numnewusers6

diffindiff2 large LARGE tab_4.1 write numcontrib numuser numserious90 numnewusers6

// 2. engaged vs. non-engaged county

use ${stash}panelfips, clear

bysort fips post: egen tmp = total(numcontrib > 0)
replace tmp = . if post == 1
bysort fips: egen precontrib = max(tmp)
bysort fips: egen totcontrib = total(numcontrib)
drop tmp

drop if precontrib == 0
gen active = (prec > 1)

diffindiff2 active ACTIVE tab_4.2 run numcontrib numuser numserious90 numnewusers6

diffindiff2 active ACTIVE tab_4.2 write numcontrib numuser numserious90 numnewusers6

// 3. populated vs. not

use ${stash}panelfips, clear

gen iseast = (region==1 | region == 2)

diffindiff2 iseast EAST tab_4.3 run numcontrib numuser numserious90 numnewusers6

diffindiff2 iseast EAST tab_4.3 write numcontrib numuser numserious90 numnewusers6

// 3. urban vs rural
// 4. university vs non university
// 5. rich vs poor

// 5 Street Level Regressions

// 6 Person Level Regressions

// [A] outcome vars: County sample -- DD spec

   // a. Contributions
   // b. Street, Non Street
   // b. TIGER vs. Non Tiger
   // d. Home county, Num Counties

// [B] heterogenous

   // a. Junior vs Senior

// 7 Impacts on Craigslist

preparecl

// PENDING

// GEOID Level

// TASKS for tomorrow

// STASH

 // 1.1 Mean Charts

 // 1.2 Produce Diff in diff Latex tables
 // rundd -> batchreg.sh -> runreg.do -> diffindiff.ado
 program drop _all

 // TODO: fix manual process
 // have to do this manually

 // 1.3 Produce Diff in diff Pictures

 ** person level regressions
 // person level

// TODOs for Tuesday Jun 3
// GEOID dataset
// TILEID diff in diff
// What happened to previous people?
// What happened at the street/amenity level?
// Urban vs Rural (framework for third level diff)

// 1.4 Robust: product diff in diff for GEOID

use ${stash}panelfips_geoid10, clear

** Analysis Stage 2
//

// PROGRAMS

clear all
set more off
set matsize 11000
program drop _all

qui adopath + "/mnt/nfs6/wikipedia.proj/jmp/jmpscripts/stata/ado"
cd ${path}

// this declares global vars
declare_global

// 0. Data

// 0.1 make cleancnty, changesets data
clear
preparebasic

// 0.2 make relevant dependent variables
use ${stash}cleanchangeset1, clear
makedv_uid fips quarter
balancepanel_uid fips
mergebasic
qui destring, replace
save ${stash}panelfips, replace

// 0.3 make node and way datasets
preparenode

use ${stash}mergemaster_node, clear
makedv fips node
balancepanel fips node
save ${stash}panelnode, replace

// 0.3 make way dataset
prepareway

use ${stash}way_stash, clear
makedv fips way
save ${stash}mergemaster_way, replace

use ${stash}mergemaster_way, clear
balancepanel fips way
merge m:1 fips using ${stash}cleancnty, keep(master match) nogen
save ${stash}panelway, replace

// 1. Summary Stats
program drop _all
use ${stash}panelfips, clear

global tabname "summary"
local vars "treat aland_sqmi emp_earnings age_median num_households educ_college emp_computer"

local vars "treat year cntypop emp_earnings age_median num_households numcontrib numuser numserious18 numnewusers numnewusers6 numnewusers90"

makesummary "`vars'"

// 2. Treatment Control Balance
use ${stash}panelfips, clear
makehist cntypop kdensity "Population"
makehist emp_earnings kdensity "Earnings"
makehist age_median kdensity "Median Age"
makehist educ_college kdensity "Num. College Educated"
** makehist region hist

// 2. TODO: TTest or regressions treatment / control

//  3.1 Meanline charts (raw data)
makemeanline numcontrib quarter 2011 "Contributions"
makemeanline numuser quarter 2011 "Users"
makemeanline numserious90 quarter 2011 "Super Users"

* makemeanline numnewusers quarter 2011 "New Users"
* makemeanline numnewusers6 quarter 2011 "New Users (Stay for 6+ Months)"
* makemeanline numnewusers90 quarter 2011 "New Users (who become super users)"
* makemeanline numchanges quarter 2011 "Changes"

// 3.2 Cross Sectional Regressions
crossreg

// 3.3 Diff in Diff --- Baseline
local outcomes "numcontrib numuser numserious90"
dd_simple "`outcomes'" run xtreg
dd_simple "`outcomes'" run xtpoisson
dd_simple "`outcomes'" write ddsimple_baseline

// 3.4 Diff in Diff --- Additional
local outcomes "numnewusers numnewusers6 numnewusers90"
dd_simple "`outcomes'" run xtreg
dd_simple "`outcomes'" run xtpoisson
dd_simple "`outcomes'" write ddsimple_additional

// 3.5 Diff in Diff Picture
ddchart numuser
ddchart numcontrib
ddchart numserious90

// 4. How do results vary?
// takes a while
makehetero

// 5. Individual Level Data

// 5.0 create data
makeperson

//5.1 Summary
use ${stash}paneluid, clear

label variable treat "TIGER"
label variable year "YEAR"
label variable numcontrib "Contribs"
label variable numcontrib_home "Home County"
label variable numcontrib_treat "TIGER"
label variable numcontrib_notreat "Non-TIGER"
label variable numcontrib_statenotreat "State(Control)"
label variable numcounties "Num. Counties"

local vars "treat year numcontrib numcontrib_home numcontrib_treat numcontrib_notreat numcontrib_statenotreat numcounties"
global tabname "summary_uid"
makesummary "`vars'"

//Regressions
ddperson maketables
ddperson makechart

// 6. Map Quality

//6.0 summary
use ${stash}panelotherlayers, clear

label variable treat "TIGER"
label variable year "Year"
label variable otherlayer "NonStreet Layers"
label variable amenities "Amenities"
label variable bld_addr "Building/Addresses"
label variable numclass4 "Trails/Bikepaths"

label variable numattrib1 "Attributes:Major Highway"
label variable numattrib2 "Attributes:Class2"
label variable numattrib3 "Attributes:Class3"
label variable numattrib4 "Attributes:Trails etc."

local vars "treat year otherlayer amenities bld_addr numclass4 numattrib1 numattrib2 numattrib3 numattrib4"
global tabname "summary_otherl"
makesummary "`vars'"

// 6.1 Street data completeness

// 6.2 Other layers
reg_layers

// 6.3 Highway level regressions

// 7. Craiglist
reg_cl

////////// OLD //
 ** Analysis

// 0. Data

// 0.1 make fips dataset
use ${stash}mergemaster1, clear

balancepanel fips
save ${stash}panelfips, replace

// 0.2 make node dataset
preparenode

use ${stash}mergemaster_node, clear
makedv fips node
balancepanel fips node
save ${stash}panelnode, replace

// 0.3 make way dataset
prepareway
makedv fips way
balancepanel fips way
save ${stash}panelway, replace

// 3. Baseline effects (xtpoisson, cluster at unit level)

// 3.2 FIPS Sample -- DD

/// 3.2.1 -- contrib, users, superu
program drop _all

clear
local dv "numcontrib numuser numserious90"
local unit fips
rundd panelfips `unit' "`dv'"
diffindiff xtpoisson "`dv'" `unit' 2014 write tab_3.2.1 "Contributions" "Users" "Super Users"

/// 3.2.2 -- newusers, newusers6, newuserssuper
clear
local dv "numnewusers numnewusers6 numnewusers90"
local unit fips
rundd panelfips `unit' "`dv'"
diffindiff xtpoisson "`dv'" `unit' 2014 write tab_3.2.2 "New Users" "New Users(6+)" "New Super Users"

/// 3.2.3 -- streetcontrib, nonstreet contrib, amenities
// TODO

use ${stash}panelnode, clear
program drop _all

diffindiff xtpoisson fips run tab_test numcontrib numuser

/// 3.2.4 -- completeness
// TODO

// 3.2.5 -- Diff in Diff charts
// TODO : make this more flexible
// TODO : this works only for fips
ddchart

// 3.3 Robustness : Drop empty counties
// TODO
//  e. Repeat (a) by dropping empty counties
//  f. Repeat (a) by dropping west
//  g. Repeat (a) for Tile Sample, MSA sample
//  h. Repeat (a) with matched estimators
//  i. Repeat (a) with spatially clustered se, Region X Time trends

// 4 Heterogenous Effects

// 1. Large vs. small
use ${stash}panelfips, clear
gen large = (cntypop > 100000)

diffindiff2 large LARGE tab_4.1 run numcontrib numuser numserious90 numnewusers6

diffindiff2 large LARGE tab_4.1 write numcontrib numuser numserious90 numnewusers6

// 2. engaged vs. non-engaged county

use ${stash}panelfips, clear

bysort fips post: egen tmp = total(numcontrib > 0)
replace tmp = . if post == 1
bysort fips: egen precontrib = max(tmp)
bysort fips: egen totcontrib = total(numcontrib)
drop tmp

drop if precontrib == 0
gen active = (prec > 1)

diffindiff2 active ACTIVE tab_4.2 run numcontrib numuser numserious90 numnewusers6

diffindiff2 active ACTIVE tab_4.2 write numcontrib numuser numserious90 numnewusers6

// 3. populated vs. not

use ${stash}panelfips, clear

gen iseast = (region==1 | region == 2)

diffindiff2 iseast EAST tab_4.3 run numcontrib numuser numserious90 numnewusers6

diffindiff2 iseast EAST tab_4.3 write numcontrib numuser numserious90 numnewusers6

// 3. urban vs rural
// 4. university vs non university
// 5. rich vs poor

// 5 Street Level Regressions

// 6 Person Level Regressions

// [A] outcome vars: County sample -- DD spec

   // a. Contributions
   // b. Street, Non Street
   // b. TIGER vs. Non Tiger
   // d. Home county, Num Counties

// [B] heterogenous

   // a. Junior vs Senior

// 7 Impacts on Craigslist

preparecl

// PENDING

// GEOID Level

// TASKS for tomorrow

// STASH

 // 1.1 Mean Charts

 // 1.2 Produce Diff in diff Latex tables
 // rundd -> batchreg.sh -> runreg.do -> diffindiff.ado
 program drop _all

 // TODO: fix manual process
 // have to do this manually

 // 1.3 Produce Diff in diff Pictures

 ** person level regressions
 // person level

// TODOs for Tuesday Jun 3
// GEOID dataset
// TILEID diff in diff
// What happened to previous people?
// What happened at the street/amenity level?
// Urban vs Rural (framework for third level diff)

// 1.4 Robust: product diff in diff for GEOID

use ${stash}panelfips_geoid10, clear

** Analysis Stage 2
//

// PROGRAMS