[iegitaddmd] improved language

[iegitaddmd] improved language

Co-authored-by: Luiza Andrade <3e25dce60d6313c358f2ba3c7da4f9a53677fa7f@worldbank.org>

{smcl}
{* 5 Nov 2019}{...}
{hline}
help for {hi:iegitaddmd}
{hline}

{title:Title}

{cmd:iegitaddmd} {hline 2} Creates a placeholder file in subfolders of a GitHub repository folder, which allows committing folder structures with empty folders.

{phang2}For a more descriptive discussion on the intended usage and work flow of this
command please see the {browse "https://dimewiki.worldbank.org/wiki/Iegitaddmd":DIME Wiki}.

{title:Syntax}

{phang} {cmdab:iegitaddmd} , {opt folder(full_file_path)} [{opt comparefolder(full_file_path)} {cmd:customfile(}{it:{help filename}}{cmd:)} {opt all} {opt skip} {opt replace} {opt skipfolders(folder_name)}  {opt auto:matic} {opt dry:run}]

{marker opts}{...}
{synoptset 28}{...}
{synopthdr:options}
{synoptline}
{synopt :{opt folder(full_file_path)}}Specifies the folder path to the project folder clone where placeholder files will be created{p_end}
{synopt :{opt comparefolder(full_file_path)}}Specifies the top folder of a folder tree to be compared with {opt folder()}{p_end}
{synopt :{cmd:customfile(}{it:{help filename}}{cmd:)}}Specifies a file saved on disk that is used instead of the default file as placeholder{p_end}
{synopt :{cmd:all}}Creates the placeholder file in every subfolder of {opt folder()}, whether empty or not{p_end}
{synopt :{opt skip}}If option {opt all} is used and a folder has a file with same name as placeholder file, then nothing is done{p_end}
{synopt :{opt replace}}If option {opt all} is used and a folder has a file with same name as placeholder file, then the file is overwritten{p_end}
{synopt :{opt skipfolders(folder_name)}}List of folders to be skipped. The folder {it:.git} is always added to this list, even when option is not used{p_end}
{synopt :{opt auto:matic}}Makes the command create placeholder files without prompting the user for each file{p_end}
{synopt :{opt dry:run}}Makes the command list all the files that would have been created without this option.{p_end}
{synoptline}

{marker desc}
{title:Description}

{pstd}GitHub does not sync empty folders, or folder that only contain ignored files. However, it is common in research projects that these types of folders needs to be shared anyways. {cmd:iegitaddmd} provides a solution to this by creating placeholder files in in these folders. This placeholder file can then be shared to GitHub and that way the folder is also shared. {cmd:iegitaddmd} have two ways to identify which those folders are that are developed with the two use cases described below in mind.{p_end}

{pstd}{bf:Use case 1.} It is common in the beginning of a research project that a standardized folder structure is set up (for example with {help iefolder}) where some folders are created that are not yet needed. Examples of folders like that could be the output folders. If these folders are not shared at the time they are created, there is a risk that different team members name them differently which will be confusing and could lead to errors. {cmd:iegitaddmd} can solve this by scanning a folder and all its sub-folders for completely empty folders, and create a placeholder file in that folder.{p_end}

{pstd}The solution to use case 1 in {cmd:iegitaddmd} is a Stata adaptation of {it:Solution B} in {browse "http://bytefreaks.net/gnulinux/bash/how-to-add-automatically-all-empty-folders-in-git-repository" :this post}.{p_end}

{pstd}{bf:Use case 2.} A common method when transferring an already ongoing research project to GitHub is to copy all of the data work folder from wherever the files and folders where shared before (for example DropBox) to a clone of the repository. Then only the files that are meant to be shared on GitHub are uploaded to the cloud using a {it:.gitignore file}. Empty folders or folders where all files are ignored is then not shared through GitHub. {cmd:iegitaddmd} can solve this by scanning the DropBox folder (or wherever the project folder was shared before) for sub-folders that exist in that folder but not in the clone and crate that folder in the clone and put a placeholder in it.{p_end}

{pstd}The way you differentiate between these two use cases when using {cmd:iegitaddmd} is whether the option {opt comparefolder()} is used or not. If that option is used, then {cmd:iegitaddmd} will identify all folders that are in {opt comparefolder()} that is not in {opt folder()}, and create a placeholder file there. If {opt comparefolder()} is not used, then {cmd:iegitaddmd} will identify all completely empty folders (no files or subfolders) and create a placeholder file there.{p_end}

{pstd}The default placeholder file used if no file is specified using option {opt customfile()} is called README.md, which is a name and format recognized by GitHub.com so that the content of the file is displayed on GitHub.com when navigating to that folder using a browser. The placeholder file may be removed as	soon as files that are not ignored have been added to the folder, or even better is to keep the file but edit the content of it so it documents the content of that folder for other and future team members.{p_end}

{marker optslong}
{title:Options}

{phang}{opt folder(file_path)} is the folder path to the project folder clone where the placeholder files will be created. Unless option {opt comparefolder()} or {opt all} is used, then placeholder files will be created in all empty folders. See option {opt manual} for a way to manually confirm each folder before a placeholder file is created in it.{p_end}

{phang}{opt comparefolder(file_path)} is used to compare a file tree in a different location and create the folders with placeholder files in all folders that exist in {opt comparefolder()} but not in {opt folder()}. If a folder that does no exist in {opt folder()} has a subfolder, then both the folder and the subfolder will be created and the placeholder file is only crated in the sub-folder. This is enough to share both folder and the subfolder in GitHub. If the {opt comparefolder()} and {opt folder()} ends with folders with different names, for example {opt folder(${project}/datawork/baseline)} and {opt comparefolder(${project}/datawork)}, then the command will throw an error as it is likely that the paths do not point to corresponding folders in the two folder trees. See option {opt manual} for a way to manually confirm each folder before a placeholder file is created in it.{p_end}

{phang}{cmd:file(}{it:{help filename}}{cmd:)} allows the user to specify a file saved on
	the computer to be used as the placeholder file	instead of the default README.md file. This
	allows people and organizations to write their own template placeholder files according to their own
	preferences. We recommend that a file of type .md (markdown) and name README.md is used as
	GitHub.com recognizes this name and format and will display the content of the file when someone
	browse to that folder on GitHub.com. But this is not a technical requirement, any file type
	and name can be used as placeholder file.{p_end}

{phang}{opt all} creates the placeholder file in {opt folder()} and in every subfolder,
	regardless if they are empty or not. This allows the user to create a placeholder file in every
	folder that can be edited with documentation and instructions to the purpose and usage of
	each subfolder. This option can also be important when .{it:gitignore} is used, as {cmd:iegitaddmd} will not
	create files to subfolders that only has ignored file -- in which case the folder will not
	be synced by GitHub.{p_end}

{phang}{opt skip} and {opt replace} tells {cmd:iegitaddmd} what to do if the option {opt all} is used
	and if any of the folders contain a file with the same name as the placeholder file the command
	is trying to create. If a file with the same name as the placeholder file exist and neither of these options
	are used, then the command will throw an error. If {opt skip} is used, then nothing is done in the
	folder where the file with the same name exists and the command proceeds to the next folder.
	If {opt replace} is used then the file with the same name is overwritten with the new placeholder
	file before the command proceeds to the next folder.{p_end}

{phang}{opt skipfolders(folder_name)} can be used to tell {cmd:iegitaddmd} which folders
in which a placeholder file should never be created. The best example of this is the {it:.git}
folder in which placeholders never should be created. That name is always skipped regardless of
this option being used or not. Use this option to list additional folders to be skipped. You
should not list the full folder path in this option, just the folder name. All folders with
that name will be skipped regardless of their location in the project folder. The folder names may
be listed with or without quotation marks as long as there are no spaces in the names. If any of
the folder names has spaces, then quotation marks must be used for all folder names.

{phang}{opt auto:matic} can be used to speed up the creation of placeholders by telling the
command to not prompt the users for confirmation for each file before it is created. The
default is that the command is asking the user before creating each place holder file.
This option should only be used when you are confident you have specified the correct folder
paths. We recommend that you use the {opt dryrun} with this option to make sure that the
folder paths are correct.{p_end}

{phang}{opt dry:run} can be used to safely test that the folder paths are specified correctly
before any placeholder files are created. When this option it used the command simply lists
the file that would have been created if this option were not used. Once you are confident
that list is correct, you can remove this option and re-run the command and the files will
be created.{p_end}

{title:Example}

{pstd}{inp:global github_folder "C:\Users\JohnSmith\Documents\GitHub\ProjectA"}{break}{inp:iegitaddmd , folder({it:"$github_folder"})}{p_end}

{pstd}In the example above, there is a GitHub repository in the folder ProjectA. This
	repository has a folder structure where some folders are still empty but will later
	be populated with files. In order to have all folders, even the empty ones, synced on all
	collaborators' cloned local copies of the repository, the folders need to contain at least
	one file, which is being created by the command.{p_end}

{title:Acknowledgements}

{pstd}I would like to acknowledge the help in testing and proofreading I received in relation to this command and help file from (in alphabetic order):{p_end}
{pmore}Guadalupe Bedoya{break}Luiza Cardoso de Andrade{break}Mrijan Rimal{break}Benjamin Daniels{break}

{title:Author}

{phang}All commands in ietoolkit is developed by DIME Analytics at DECIE, The World Bank's unit for Development Impact Evaluations.

{phang}Main author: Kristoffer Bjarkefur, DIME Analytics, The World Bank Group

{phang}Please send bug-reports, suggestions and requests for clarifications
		 writing "ietoolkit iegitaddmd" in the subject line to:{break}
		 dimeanalytics@worldbank.org

{phang}You can also see the code, make comments to the code, see the version
		 history of the code, and submit additions or edits to the code through {browse "https://github.com/worldbank/ietoolkit":the GitHub repository of ietoolkit}.{p_end}

{smcl}
{* 5 Nov 2019}{...}
{hline}
help for {hi:iegitaddmd}
{hline}

{title:Title}

{cmd:iegitaddmd} {hline 2} Creates a placeholder file in subfolders of a GitHub repository folder, which allows committing folder structures with empty folders.

{phang2}For a more descriptive discussion on the intended usage and work flow of this
command please see the {browse "https://dimewiki.worldbank.org/wiki/Iegitaddmd":DIME Wiki}.

{title:Syntax}

{phang} {cmdab:iegitaddmd} , {opt folder(full_file_path)} [{opt comparefolder(full_file_path)} {cmd:customfile(}{it:{help filename}}{cmd:)} {opt all} {opt skip} {opt replace} {opt skipfolders(folder_name)}  {opt auto:matic} {opt dry:run}]

{marker opts}{...}
{synoptset 28}{...}
{synopthdr:options}
{synoptline}
{synopt :{opt folder(full_file_path)}}Specifies the folder path to the project folder clone where placeholder files will be created{p_end}
{synopt :{opt comparefolder(full_file_path)}}Specifies the top folder of a folder tree to be compared with {opt folder()}{p_end}
{synopt :{cmd:customfile(}{it:{help filename}}{cmd:)}}Specifies a file saved on disk that is used instead of the default file as placeholder{p_end}
{synopt :{cmd:all}}Creates the placeholder file in every subfolder of {opt folder()}, whether empty or not{p_end}
{synopt :{opt skip}}If option {opt all} is used and a folder has a file with same name as placeholder file, then nothing is done{p_end}
{synopt :{opt replace}}If option {opt all} is used and a folder has a file with same name as placeholder file, then the file is overwritten{p_end}
{synopt :{opt skipfolders(folder_name)}}List of folders to be skipped. The folder {it:.git} is always added to this list, even when option is not used{p_end}
{synopt :{opt auto:matic}}Makes the command create placeholder files without prompting the user for each file{p_end}
{synopt :{opt dry:run}}Makes the command list all the files that would have been created without this option.{p_end}
{synoptline}

{marker desc}
{title:Description}

{pstd}GitHub does not sync empty folders, or folder that only contain ignored files. However, it is common in research projects that these types of folders needs to be shared anyways. {cmd:iegitaddmd} provides a solution to this by creating placeholder files in in these folders. This placeholder file can then be shared to GitHub and that way the folder is also shared. {cmd:iegitaddmd} have two ways to identify which those folders are that are developed with the two use cases described below in mind.{p_end}

{pstd}{bf:Use case 1.} It is common in the beginning of a research project that a standardized folder structure is set up (for example with {help iefolder}) where some folders are created that are not yet needed. Examples of folders like that could be the output folders. If these folders are not shared at the time they are created, there is a risk that different team members name them differently which will be confusing and could lead to errors. {cmd:iegitaddmd} can solve this by scanning a folder and all its sub-folders for completely empty folders, and create a placeholder file in that folder.{p_end}

{pstd}The solution to use case 1 in {cmd:iegitaddmd} is a Stata adaptation of {it:Solution B} in {browse "http://bytefreaks.net/gnulinux/bash/how-to-add-automatically-all-empty-folders-in-git-repository" :this post}.{p_end}

{pstd}{bf:Use case 2.} A common method when transferring an already ongoing research project to GitHub is to copy all of the data work folder from wherever the files and folders where shared before (for example DropBox) to a clone of the repository. Then only the files that are meant to be shared on GitHub are uploaded to the cloud using a {it:.gitignore file}. Empty folders or folders where all files are ignored is then not shared through GitHub. {cmd:iegitaddmd} can solve this by scanning the DropBox folder (or wherever the project folder was shared before) for sub-folders that exist in that folder but not in the clone and crate that folder in the clone and put a placeholder in it.{p_end}

{pstd}The way you differentiate between these two use cases when using {cmd:iegitaddmd} is whether the option {opt comparefolder()} is used or not. If that option is used, then {cmd:iegitaddmd} will identify all folders that are in {opt comparefolder()} that is not in {opt folder()}, and create a placeholder file there. If {opt comparefolder()} is not used, then {cmd:iegitaddmd} will identify all completely empty folders (no files or subfolders) and create a placeholder file there.{p_end}

{pstd}The default placeholder file used if no file is specified using option {opt customfile()} is called README.md, which is a name and format recognized by GitHub.com so that the content of the file is displayed on GitHub.com when navigating to that folder using a browser. The placeholder file may be removed as	soon as files that are not ignored have been added to the folder, or even better is to keep the file but edit the content of it so it documents the content of that folder for other and future team members.{p_end}

{marker optslong}
{title:Options}

{phang}{opt folder(file_path)} is the folder path to the project folder clone where the placeholder files will be created. Unless option {opt comparefolder()} or {opt all} is used, then placeholder files will be created in all empty folders. See option {opt manual} for a way to manually confirm each folder before a placeholder file is created in it.{p_end}

{phang}{opt comparefolder(file_path)} is used to compare a file tree in a different location and create the folders with placeholder files in all folders that exist in {opt comparefolder()} but not in {opt folder()}. If a folder that does no exist in {opt folder()} has a subfolder, then both the folder and the subfolder will be created and the placeholder file is only crated in the sub-folder. This is enough to share both folder and the subfolder in GitHub. If the {opt comparefolder()} and {opt folder()} ends with folders with different names, for example {opt folder(${project}/datawork/baseline)} and {opt comparefolder(${project}/datawork)}, then the command will throw an error as it is likely that the paths do not point to corresponding folders in the two folder trees. See option {opt manual} for a way to manually confirm each folder before a placeholder file is created in it.{p_end}

{phang}{cmd:file(}{it:{help filename}}{cmd:)} allows the user to specify a file saved on
	the computer to be used as the placeholder file	instead of the default README.md file. This
	allows people and organizations to write their own template placeholder files according to their own
	preferences. We recommend that a file of type .md (markdown) and name README.md is used as
	GitHub.com recognizes this name and format and will display the content of the file when someone
	browse to that folder on GitHub.com. But this is not a technical requirement, any file type
	and name can be used as placeholder file.{p_end}

{phang}{opt all} creates the placeholder file in {opt folder()} and in every subfolder,
	regardless if they are empty or not. This allows the user to create a placeholder file in every
	folder that can be edited with documentation and instructions to the purpose and usage of
	each subfolder. This option can also be important when .{it:gitignore} is used, as {cmd:iegitaddmd} will not
	create files to subfolders that only has ignored file -- in which case the folder will not
	be synced by GitHub.{p_end}

{phang}{opt skip} and {opt replace} tells {cmd:iegitaddmd} what to do if the option {opt all} is used
	and if any of the folders contain a file with the same name as the placeholder file the command
	is trying to create. If a file with the same name as the placeholder file exist and neither of these options
	are used, then the command will throw an error. If {opt skip} is used, then nothing is done in the
	folder where the file with the same name exists and the command proceeds to the next folder.
	If {opt replace} is used then the file with the same name is overwritten with the new placeholder
	file before the command proceeds to the next folder.{p_end}

{phang}{opt skipfolders(folder_name)} can be used to tell {cmd:iegitaddmd} which folders
in which a placeholder file should never be created. The best example of this is the {it:.git}
folder in which placeholders never should be created. That name is always skipped regardless of
this option being used or not. Use this option to list additional folders to be skipped. You
should not list the full folder path in this option, just the folder name. All folders with
that name will be skipped regardless of location in the project folder. The folder names may
be listed with or without quotation marks as long as there are no spaces in the names. If any of
the folder names has spaces, then quotation marks must be used for all folder names.

{phang}{opt auto:matic} can be used to speed up the creation of placeholders by telling the
command to not prompt the users for confirmation for each file before it is created. The
default is that the command is asking the user before creating each place holder file.
This option should only be used when you are confident you have specified the correct folder
paths. We recommend that you use the {opt dryrun} with this option to make sure that the
folder paths are correct.{p_end}

{phang}{opt dry:run} can be used to safely test that the folder paths are specified correctly
before any placeholder files are created. When this option it used the command simply lists
the file that would have been created if this option were not used. Once you are confident
that list is correct, you can remove this option and re-run the command and the files will
be created.{p_end}

{title:Example}

{pstd}{inp:global github_folder "C:\Users\JohnSmith\Documents\GitHub\ProjectA"}{break}{inp:iegitaddmd , folder({it:"$github_folder"})}{p_end}

{pstd}In the example above, there is a GitHub repository in the folder ProjectA. This
	repository has a folder structure where some folders are still empty but will later
	be populated with files. In order to have all folders, even the empty ones, synced on all
	collaborators' cloned local copies of the repository, the folders need to contain at least
	one file, which is being created by the command.{p_end}

{title:Acknowledgements}

{pstd}I would like to acknowledge the help in testing and proofreading I received in relation to this command and help file from (in alphabetic order):{p_end}
{pmore}Guadalupe Bedoya{break}Luiza Cardoso de Andrade{break}Mrijan Rimal{break}Benjamin Daniels{break}

{title:Author}

{phang}All commands in ietoolkit is developed by DIME Analytics at DECIE, The World Bank's unit for Development Impact Evaluations.

{phang}Main author: Kristoffer Bjarkefur, DIME Analytics, The World Bank Group

{phang}Please send bug-reports, suggestions and requests for clarifications
		 writing "ietoolkit iegitaddmd" in the subject line to:{break}
		 dimeanalytics@worldbank.org

{phang}You can also see the code, make comments to the code, see the version
		 history of the code, and submit additions or edits to the code through {browse "https://github.com/worldbank/ietoolkit":the GitHub repository of ietoolkit}.{p_end}


Option to start graph at 0

Option to start graph at 0

Option to start graph at 0

cap	program drop	iegraph
	program define 	iegraph
	preserve
	syntax varlist, [noconfbars TItle(string) save(string) confbarsnone(varlist) yzero *]

	mat beta_ = e(b)

	local counter = 0
	//local textfile = "Mrijan"

	qui{
		if "`confbars'" 			!= "" {
		local CONFINT_BAR 	= 0
		}
		else if "`confbars'" 			== "" {
		local CONFINT_BAR 	= 1
		}

		local varTest : list confbarsnone in varlist

		if `varTest' == 0 {
					noi display as error "{phang} Variables defined in confbarsnone cannot be found in the graph variable list. {p_end}"
					noi display ""
					error 111
					 }

		foreach var of local varlist{
			cap assert inlist(`var',0,1) | missing(`var')
				if _rc {
					noi display as error "{phang} The variable `var' is not a dummy. Treatment variable needs to be a dummy(0 or 1) variable. {p_end}"
					noi display ""
					error 149
					}
				else {
					}

		local counter = `counter' + 1
		di `counter'

		scalar coeff_`var' = _b[`var']
		//noi display coeff_`var'

		scalar coeff_se_`var' = _se[`var']
		//noi display coeff_se_`var'

		count if e(sample) == 1 & `var' == 1 //Count one tmt group at the time
		scalar obs_`var' = r(N)
		//noi display obs_`var'

	}

	local count: word count `varlist'
	local graphCount = `count' + 1
	//tokenize "`varlist'"
	//noi di `count'

	//Make all vars tempvars (maybe do later)
	//Make sure that missing is properly handled
	tempvar anyTMT
	egen `anyTMT' = rowmax(`varlist') if e(sample) == 1
	tempvar control
	gen `control' = (`anyTMT' == 0) if !missing(`anyTMT')

	sum `e(depvar)' if e(sample) == 1 & `control' == 1
	scalar ctl_N		= r(N)
	scalar ctl_mean	  	= r(mean)
	scalar ctl_mean_sd 	= r(sd)

		foreach var of local varlist {
			scalar  conf_int_min_`var'   =	coeff_`var'-(1.96*coeff_se_`var') + ctl_mean
			scalar  conf_int_max_`var'   =	coeff_`var'+(1.96*coeff_se_`var') + ctl_mean

			test `var'
			local star_`var' " "

			scalar  pvalue =r(p)
			if pvalue < 0.10 {
				local star_`var' "*"
			}
			if pvalue < 0.05 {
				local star_`var' "**"
			}
			if pvalue < 0.01 {
				local star_`var' "***"
			}
			*noi display "`star_`var''"
			scalar tmt_mean_`var' = ctl_mean + coeff_`var'
		}

	tempname 	newHandle
	tempfile	newTextFile
	cap file close 	`newHandle'
	file open  	`newHandle' using "`newTextFile'", text write append

	*Write headers and control value
	file write `newHandle' ///
		"order" 	_tab "xLabeled"	_tab "mean" _tab "coeff" _tab "conf_int_min" _tab "conf_int_max" _tab "obs" _tab "star" _n 	///
		%9.3f (1) _tab "Control"  _tab %9.3f (ctl_mean)      _tab  		 			_tab 		 			  	  _tab 			 				_tab %9.3f (ctl_N) 	   _tab			  	  _n

	tempvar newCounter
	gen `newCounter' = 2  //First tmt group starts at 2 (1 is control)

	foreach var in `varlist' {
	     if `: list var in confbarsnone' {
		file write `newHandle' ///
			%9.3f (`newCounter') _tab "`var'" _tab %9.3f (tmt_mean_`var') _tab  %9.3f (coeff_`var')  _tab %9.3f  _tab _tab %9.3f (obs_`var')   _tab "`star_`var''"  _n

		replace `newCounter' = `newCounter' + 1
			}
		else {
		file write `newHandle' ///
			%9.3f (`newCounter') _tab "`var'" _tab %9.3f (tmt_mean_`var') _tab  %9.3f (coeff_`var')  _tab %9.3f (conf_int_min_`var') _tab %9.3f (conf_int_max_`var') _tab %9.3f (obs_`var')   _tab "`star_`var''"  _n

		replace `newCounter' = `newCounter' + 1
		}
	}

	file close `newHandle'

	copy "`newTextFile'"  "mainMasterDofile.txt" , replace

	/*************************************

		Create the graph

	*************************************/

	insheet using mainMasterDofile.txt, clear

	local tmtGroupBars 	""
	local xAxisLabels 	`"xlabel( "'
	local legendLabels	`"lab(1 "`obsLabelControl'")"'
	local legendNumbers	""

	forval tmtGroupCount = 1/`graphCount' {

		************
		*Create the bar for this group

		colorPicker `tmtGroupCount' `graphCount'

		local tmtGroupBars `"`tmtGroupBars' (bar mean order if order == `tmtGroupCount', color("`r(color)'")) "'

		************
		*Create labels etc. for this group

		local obs 			= obs[`tmtGroupCount']
		local stars 		= star[`tmtGroupCount']
		local legendLabel 	= xlabeled[`tmtGroupCount']

		local xAxisLabels 	`"`xAxisLabels' `tmtGroupCount' "(N = `obs') `stars'" "'
		local legendLabels 	`"`legendLabels' lab(`tmtGroupCount' "`legendLabel'") "'
		local legendNumbers	`"`legendNumbers' `tmtGroupCount'"'
	}

	*Close or comple some strings
	local xAxisLabels `"`xAxisLabels' ,noticks labsize(medsmall)) "'
	local legendOption `"legend(order(`legendNumbers') `legendLabels')"'

	*Create the confidence interval bars
	if `CONFINT_BAR' == 0 {
		//local confIntGraph == `"(scatter mean order,  msym(none)  mlabs(medium) mlabpos(10) mlabcolor(black)), xtitle("") ytitle("`e(depvar)'") "'
		local confIntGraph = ""
	}
		else if `CONFINT_BAR' == 1 {
		local confIntGraph = `"(rcap conf_int_max conf_int_min order, lc(gs)) (scatter mean order,  msym(none)  mlabs(medium) mlabpos(10) mlabcolor(black))"'
	}

	local titleOption `" , xtitle("") ytitle("`e(depvar)'") "'

	if "`save'" != "" {
		local saveOption saving("`save'", replace)
	}

	if "`yzero'" != "" {

		gen maxvalue = max(mean , conf_int_max)

		sum maxvalue
		local max = `r(max)'

		local logmax = log10(`max')

		local logmax = round(`logmax') - 1

		local tenpower = 10 ^ (`logmax')

		local up = round(`max', `tenpower')

		local half = (`up') / 2

		noi di "up: `up'"
		noi di "half: `half'"

		local yzero_option ylabel(0(`half')`up')
	}

	noi di `" graph twoway `tmtGroupBars' `confIntGraph' `titleOption' `options' `legendOption' `xAxisLabels' `saveOption' title("`title'") `yzero_option'  "'
	graph twoway `tmtGroupBars' `confIntGraph' `titleOption' `options' `legendOption' `xAxisLabels' `saveOption' title("`title'") `yzero_option'
	//noi di 	`"graph twoway `tmtGroupBars' `legendOption' (scatter mean order,  msym(none)  mlabs(medium) mlabpos(10) mlabcolor(black))), xtitle("") ytitle("`e(depvar)'") `xAxisLabels' `saveOption' title("`title'")  "'

	restore
}

end

cap	program drop	colorPicker
	program define 	colorPicker , rclass

	args groupCount totalNumGroups

	if `totalNumGroups' == 2 {

		if `groupCount' == 1 return local color "215 25 28"
		if `groupCount' == 2 return local color "43 123 182"
	}
	else if `totalNumGroups' == 3 {

		if `groupCount' == 1 return local color "215 25 28"
		if `groupCount' == 2 return local color "255 255 191"
		if `groupCount' == 3 return local color "43 123 182"
	}
	else if `totalNumGroups' == 4 {

		if `groupCount' == 1 return local color "215 25 28"
		if `groupCount' == 2 return local color "255 255 191"
		if `groupCount' == 3 return local color "171 217 233"
		if `groupCount' == 4 return local color "43 123 182"
	}
	else {

		*For five or more colors we repeat the same pattern

		local colourNum = mod(`groupCount', 5)
		if `colourNum' == 1 return local color "215 25 28"
		if `colourNum' == 2 return local color "253 174 93"
		if `colourNum' == 3 return local color "255 255 191"
		if `colourNum' == 4 return local color "171 217 233"
		if `colourNum' == 0 return local color "43 123 182"

	}

end

cap	program drop	iegraph
	program define 	iegraph
	preserve
	syntax varlist, [noconfbars TItle(string) save(string) confbarsnone(varlist) yzero *]

	mat beta_ = e(b)

	local counter = 0
	//local textfile = "Mrijan"

	qui{
		if "`confbars'" 			!= "" {
		local CONFINT_BAR 	= 0
		}
		else if "`confbars'" 			== "" {
		local CONFINT_BAR 	= 1
		}

		local varTest : list confbarsnone in varlist

		if `varTest' == 0 {
					noi display as error "{phang} Variables defined in confbarsnone cannot be found in the graph variable list. {p_end}"
					noi display ""
					error 111
					 }

		foreach var of local varlist{
			cap assert inlist(`var',0,1) | missing(`var')
				if _rc {
					noi display as error "{phang} The variable `var' is not a dummy. Treatment variable needs to be a dummy(0 or 1) variable. {p_end}"
					noi display ""
					error 149
					}
				else {
					}

		local counter = `counter' + 1
		di `counter'

		scalar coeff_`var' = _b[`var']
		//noi display coeff_`var'

		scalar coeff_se_`var' = _se[`var']
		//noi display coeff_se_`var'

		count if e(sample) == 1 & `var' == 1 //Count one tmt group at the time
		scalar obs_`var' = r(N)
		//noi display obs_`var'

	}

	local count: word count `varlist'
	local graphCount = `count' + 1
	//tokenize "`varlist'"
	//noi di `count'

	//Make all vars tempvars (maybe do later)
	//Make sure that missing is properly handled
	tempvar anyTMT
	egen `anyTMT' = rowmax(`varlist') if e(sample) == 1
	tempvar control
	gen `control' = (`anyTMT' == 0) if !missing(`anyTMT')

	sum `e(depvar)' if e(sample) == 1 & `control' == 1
	scalar ctl_N		= r(N)
	scalar ctl_mean	  	= r(mean)
	scalar ctl_mean_sd 	= r(sd)

		foreach var of local varlist {
			scalar  conf_int_min_`var'   =	coeff_`var'-(1.96*coeff_se_`var') + ctl_mean
			scalar  conf_int_max_`var'   =	coeff_`var'+(1.96*coeff_se_`var') + ctl_mean

			test `var'
			local star_`var' " "

			scalar  pvalue =r(p)
			if pvalue < 0.10 {
				local star_`var' "*"
			}
			if pvalue < 0.05 {
				local star_`var' "**"
			}
			if pvalue < 0.01 {
				local star_`var' "***"
			}
			*noi display "`star_`var''"
			scalar tmt_mean_`var' = ctl_mean + coeff_`var'
		}

	tempname 	newHandle
	tempfile	newTextFile
	cap file close 	`newHandle'
	file open  	`newHandle' using "`newTextFile'", text write append

	*Write headers and control value
	file write `newHandle' ///
		"order" 	_tab "xLabeled"	_tab "mean" _tab "coeff" _tab "conf_int_min" _tab "conf_int_max" _tab "obs" _tab "star" _n 	///
		%9.3f (1) _tab "Control"  _tab %9.3f (ctl_mean)      _tab  		 			_tab 		 			  	  _tab 			 				_tab %9.3f (ctl_N) 	   _tab			  	  _n

	tempvar newCounter
	gen `newCounter' = 2  //First tmt group starts at 2 (1 is control)

	foreach var in `varlist' {
	     if `: list var in confbarsnone' {
		file write `newHandle' ///
			%9.3f (`newCounter') _tab "`var'" _tab %9.3f (tmt_mean_`var') _tab  %9.3f (coeff_`var')  _tab %9.3f  _tab _tab %9.3f (obs_`var')   _tab "`star_`var''"  _n

		replace `newCounter' = `newCounter' + 1
			}
		else {
		file write `newHandle' ///
			%9.3f (`newCounter') _tab "`var'" _tab %9.3f (tmt_mean_`var') _tab  %9.3f (coeff_`var')  _tab %9.3f (conf_int_min_`var') _tab %9.3f (conf_int_max_`var') _tab %9.3f (obs_`var')   _tab "`star_`var''"  _n

		replace `newCounter' = `newCounter' + 1
		}
	}

	file close `newHandle'

	copy "`newTextFile'"  "mainMasterDofile.txt" , replace

	/*************************************

		Create the graph

	*************************************/

	insheet using mainMasterDofile.txt, clear

	pause

	local tmtGroupBars 	""
	local xAxisLabels 	`"xlabel( "'
	local legendLabels	`"lab(1 "`obsLabelControl'")"'
	local legendNumbers	""

	forval tmtGroupCount = 1/`graphCount' {

		************
		*Create the bar for this group

		colorPicker `tmtGroupCount' `graphCount'

		local tmtGroupBars `"`tmtGroupBars' (bar mean order if order == `tmtGroupCount', color("`r(color)'")) "'

		************
		*Create labels etc. for this group

		local obs 			= obs[`tmtGroupCount']
		local stars 		= star[`tmtGroupCount']
		local legendLabel 	= xlabeled[`tmtGroupCount']

		local xAxisLabels 	`"`xAxisLabels' `tmtGroupCount' "(N = `obs') `stars'" "'
		local legendLabels 	`"`legendLabels' lab(`tmtGroupCount' "`legendLabel'") "'
		local legendNumbers	`"`legendNumbers' `tmtGroupCount'"'
	}

	*Close or comple some strings
	local xAxisLabels `"`xAxisLabels' ,noticks labsize(medsmall)) "'
	local legendOption `"legend(order(`legendNumbers') `legendLabels')"'

	*Create the confidence interval bars
	if `CONFINT_BAR' == 0 {
		//local confIntGraph == `"(scatter mean order,  msym(none)  mlabs(medium) mlabpos(10) mlabcolor(black)), xtitle("") ytitle("`e(depvar)'") "'
		local confIntGraph = ""
	}
		else if `CONFINT_BAR' == 1 {
		local confIntGraph = `"(rcap conf_int_max conf_int_min order, lc(gs)) (scatter mean order,  msym(none)  mlabs(medium) mlabpos(10) mlabcolor(black))"'
	}

	local titleOption `" , xtitle("") ytitle("`e(depvar)'") "'

	if "`save'" != "" {
		local saveOption saving("`save'", replace)
	}

	if "`yzero'" != "" {

		gen maxvalue = max(mean , conf_int_max)

		sum maxvalue
		local max = `r(max)'

		local magnitude = 10^(round(log(`max'),1)-2) //round up only

		local up = round(`max', `magnitude')

		local half = (`max' - `magnitude') / 2

		noi di "up: `up'"
		noi di "half: `half'"

		local yzero_option ylabel(0(`half')`up')
	}

	noi di `" graph twoway `tmtGroupBars' `confIntGraph' `titleOption' `options' `legendOption' `xAxisLabels' `saveOption' title("`title'") `yzero_option'  "'
	graph twoway `tmtGroupBars' `confIntGraph' `titleOption' `options' `legendOption' `xAxisLabels' `saveOption' title("`title'") `yzero_option'
	//noi di 	`"graph twoway `tmtGroupBars' `legendOption' (scatter mean order,  msym(none)  mlabs(medium) mlabpos(10) mlabcolor(black))), xtitle("") ytitle("`e(depvar)'") `xAxisLabels' `saveOption' title("`title'")  "'

	restore
}

end

cap	program drop	colorPicker
	program define 	colorPicker , rclass

	args groupCount totalNumGroups

	if `totalNumGroups' == 2 {

		if `groupCount' == 1 return local color "215 25 28"
		if `groupCount' == 2 return local color "43 123 182"
	}
	else if `totalNumGroups' == 3 {

		if `groupCount' == 1 return local color "215 25 28"
		if `groupCount' == 2 return local color "255 255 191"
		if `groupCount' == 3 return local color "43 123 182"
	}
	else if `totalNumGroups' == 4 {

		if `groupCount' == 1 return local color "215 25 28"
		if `groupCount' == 2 return local color "255 255 191"
		if `groupCount' == 3 return local color "171 217 233"
		if `groupCount' == 4 return local color "43 123 182"
	}
	else {

		*For five or more colors we repeat the same pattern

		local colourNum = mod(`groupCount', 5)
		if `colourNum' == 1 return local color "215 25 28"
		if `colourNum' == 2 return local color "253 174 93"
		if `colourNum' == 3 return local color "255 255 191"
		if `colourNum' == 4 return local color "171 217 233"
		if `colourNum' == 0 return local color "43 123 182"

	}

end


fix 9999999 income errors

fix 9999999 income errors

* the world is crumbling. solutions are being introduced. they wont pass without radical political change
* 1 billion divided equally between states
* 1 billion distributed on the basis of relative 16-25 y/o compared to total in all states
* 1 billion on relative unemployed compared to toal in all states
* 1 billion on the basis of the rleative # disadvantaged youth compared to total in all states

*STEPS:
* # of 16-25 y/o's per state
* total # of 16-25 y/os
* # of unemployed per state
* total # unemployed
* # disdvantaged youth
* total # disadvantaged youth

*then by puma w/in state
*
*tabulate age if statefip == 06 & puma == 3703 [fweight=perwt]
drop if year == 2013

gen youth = (age <= 25 & age >= 16)
gen disadvantaged = (poverty <= 100)
gen disadvantaged_youth = (youth & disadvantaged)
gen unemployed = (empstat == 2)
gen unemployedyouth = youth & unemployed

egen youth_in_hh = max(youth), by(serial)
egen unemployed_in_hh = max(unemployed), by(serial)
egen disadvantaged_hh = max(disadvantaged_youth), by(serial)

egen youth_state = total(perwt * youth), by(statefip)
egen youth_puma = total(perwt * youth), by(statefip puma)
*sum youth_puma if statefip == 06 & puma == 3703
*25371 youth in lancaster
gen youth_percentage = youth_puma / youth_state

egen unemployed_state = total(perwt * unemployed), by(statefip)
egen unemployed_puma = total(perwt * unemployed), by(statefip puma)

gen unemployed_percentage = unemployed_puma / unemployed_state

egen unemployed_youth_state = total(perwt * unemployedyouth), by(statefip)

*sum youth_puma if statefip == 06 & puma == 3703
*5797 youth in lancaster

egen disadvantaged_state = total(perwt * disadvantaged_youth), by(statefip)
egen disadvantaged_puma = total(perwt * disadvantaged_youth), by(statefip puma)

gen disadvantaged_percentage = disadvantaged_puma / disadvantaged_state

egen youth_total = total(perwt * youth)
egen unemployed_total = total(perwt * unemployed)
egen disadvantaged_total = total(perwt * disadvantaged_youth)

gen youth_percentage_state = youth_state/youth_total
gen unemployed_percentage_state = unemployed_state/unemployed_total
gen disadvantaged_percentage_state = disadvantaged_state/disadvantaged_total

gen state_money = 1000 * .02 + 1000 * youth_percentage_state + 1000 * unemployed_percentage_state + 1000 * disadvantaged_percentage_state

egen npumas = nvals(puma), by(statefip)

gen puma_money = (state_money/3) * (youth_percentage + unemployed_percentage + disadvantaged_percentage)
gen numjobspuma = puma_money / .0105 //.0105 is a 10,500 dollar per yr job ($15*30hrs*12weeks in summer and $15*10hrs*34weeks in schoolyr)
//alternatively, could be doled out as 48 weeks of $15/hr pay 15 hrs a week
gen numjobsstate = state_money / .0105

*figure out households w/ youth in them
egen householdwithyouth = max(youth), by(serial)
*youth hh's living in poverty
gen youthinpov = (householdwithyouth & (poverty <= 100))
tab youthinpov if pernum == 1 [fweight = perwt]
*162 jobs for 5287 households

gen job_money = 0
replace job_money = 10500 if youthinpov & pernum == 1
gen newincome = inctot+job_money if inctot < 9999999
replace newincome = ftotinc+job_money if ftotinc < 9999999

sgini newincome if pernum == 1 & newincome < 9999999 [fweight=hhwt]
sgini ftotinc if pernum == 1 & ftotinc < 9999999 [fweight=hhwt]

gen povline = 11670 + (numprec-1)*4060
gen outofpov = poverty & (newincome > povline) & (newincome < 9999999)

/* collapse (mean) state_money, by(statefip)
. export delimited using "/Users/braddv/Desktop/BERNIE/state_money.csv",
how to collapse without losing data? or i guess just use file again.
*/
//tab occ2010 if statefip==06 & youth [fweight=perwt] (california young jobs distribution)
//
/*collapse (mean) youth_state, by(statefip)
export delimited using "/Users/braddv/Desktop/BERNIE/youth_state.csv"*/
/*collapse (mean) disadvantaged_state, by(statefip)
export delimited using "/Users/braddv/Desktop/BERNIE/disadvantaged_state.csv"*/
/*collapse (mean) unemployed_state, by(statefip)
export delimited using "/Users/braddv/Desktop/BERNIE/unemployed_state.csv"*/
/*collapse (mean) unemployed_youth_state, by(statefip)
export delimited using "/Users/braddv/Desktop/BERNIE/unemployed_youth_state.csv"*/
/*collapse (mean) numjobsstate, by(statefip)
export delimited using "/Users/braddv/Desktop/BERNIE/numjobs_state.csv"*/

* the world is crumbling. solutions are being introduced. they wont pass without radical political change
* 1 billion divided equally between states
* 1 billion distributed on the basis of relative 16-25 y/o compared to total in all states
* 1 billion on relative unemployed compared to toal in all states
* 1 billion on the basis of the rleative # disadvantaged youth compared to total in all states

*STEPS:
* # of 16-25 y/o's per state
* total # of 16-25 y/os
* # of unemployed per state
* total # unemployed
* # disdvantaged youth
* total # disadvantaged youth

*then by puma w/in state
*
*tabulate age if statefip == 06 & puma == 3703 [fweight=perwt]
drop if year == 2013
gen youth = (age <= 25 & age >= 16)
gen disadvantaged_youth = (youth & (poverty <= 125))
gen unemployed = (empstat == 2)
gen unemployedyouth = youth & unemployed

egen youth_in_hh = max(youth), by(serial)
egen unemployed_in_hh = max(unemployed), by(serial)
egen disadvantaged_hh = max(disadvantaged_youth), by(serial)

egen youth_state = total(perwt * youth), by(statefip)
egen youth_puma = total(perwt * youth), by(statefip puma)
*sum youth_puma if statefip == 06 & puma == 3703
*25371 youth in lancaster
gen youth_percentage = youth_puma / youth_state

egen unemployed_state = total(perwt * unemployed), by(statefip)
egen unemployed_puma = total(perwt * unemployed), by(statefip puma)

gen unemployed_percentage = unemployed_puma / unemployed_state

egen unemployed_youth_state = total(perwt * unemployedyouth), by(statefip)

*sum youth_puma if statefip == 06 & puma == 3703
*5797 youth in lancaster

egen disadvantaged_state = total(perwt * disadvantaged_youth), by(statefip)
egen disadvantaged_puma = total(perwt * disadvantaged_youth), by(statefip puma)

gen disadvantaged_percentage = disadvantaged_puma / disadvantaged_state

egen youth_total = total(perwt * youth)
egen unemployed_total = total(perwt * unemployed)
egen disadvantaged_total = total(perwt * disadvantaged_youth)

gen youth_percentage_state = youth_state/youth_total
gen unemployed_percentage_state = unemployed_state/unemployed_total
gen disadvantaged_percentage_state = disadvantaged_state/disadvantaged_total

gen state_money = 1000 * .02 + 1000 * youth_percentage_state + 1000 * unemployed_percentage_state + 1000 * disadvantaged_percentage_state

egen npumas = nvals(puma), by(statefip)

gen puma_money = (state_money/3) * (youth_percentage + unemployed_percentage + disadvantaged_percentage)
gen numjobspuma = puma_money / .0105 //.0105 is a 10,500 dollar per yr job ($15*30hrs*12weeks in summer and $15*10hrs*34weeks in schoolyr)
//alternatively, could be doled out as 48 weeks of $15/hr pay 15 hrs a week
gen numjobsstate = state_money / .0105

*figure out households w/ youth in them
egen householdwithyouth = max(youth), by(serial)
*youth hh's living in poverty
gen youthinpov = (householdwithyouth & (poverty <= 100))
tab youthinpov if pernum == 1 [fweight = perwt]
*162 jobs for 5287 households

gen job_money = 0
replace job_money = 10500 if youthinpov & pernum == 1
gen newincome = ftotinc+job_money

sgini newincome if pernum == 1 & newincome < 9999999 [fweight=hhwt]
sgini ftotinc if pernum == 1 & newincome < 9999999 [fweight=hhwt]

gen povline = 11670 + (numprec-1)*4060

/* collapse (mean) state_money, by(statefip)
. export delimited using "/Users/braddv/Desktop/BERNIE/state_money.csv",
how to collapse without losing data? or i guess just use file again.
*/
//tab occ2010 if statefip==06 & youth [fweight=perwt] (california young jobs distribution)
//
/*collapse (mean) youth_state, by(statefip)
export delimited using "/Users/braddv/Desktop/BERNIE/youth_state.csv"*/
/*collapse (mean) disadvantaged_state, by(statefip)
export delimited using "/Users/braddv/Desktop/BERNIE/disadvantaged_state.csv"*/
/*collapse (mean) unemployed_state, by(statefip)
export delimited using "/Users/braddv/Desktop/BERNIE/unemployed_state.csv"*/
/*collapse (mean) unemployed_youth_state, by(statefip)
export delimited using "/Users/braddv/Desktop/BERNIE/unemployed_youth_state.csv"*/
/*collapse (mean) numjobsstate, by(statefip)
export delimited using "/Users/braddv/Desktop/BERNIE/numjobs_state.csv"*/


(misc)

(misc)

mata:

// --------------------------------------------------------------------------
// Transformations: Compute RESIDUALS, not projections
// --------------------------------------------------------------------------

`Void' function transform_cimmino(`FixedEffects' S, `Variables' y, `Variables' ans,| `Boolean' get_proj) {
	`Integer' 	g
	if (args()<4 | get_proj==.) get_proj = 0
	ans = S.project_one_fe(y, 1)
	for (g=2; g<=S.G; g++) {
		ans = ans + S.project_one_fe(y, g)
	}
	ans = get_proj ? ans / S.G : y - ans / S.G
}

// --------------------------------------------------------------------------

`Void' function transform_kaczmarz(`FixedEffects' S, `Variables' y, `Variables' ans,| `Boolean' get_proj) {
	`Integer' 	g

	if (args()<4 | get_proj==.) get_proj = 0

	ans = y - S.project_one_fe(y, 1)
	for (g=2; g<=S.G; g++) {
		ans = ans - S.project_one_fe(ans, g)
	}
	if (get_proj) ans = y - ans
}

// --------------------------------------------------------------------------
// This seems slower than kaczmarz (sym kaczmarz!); not used currently
`Void' function transform_rand_kaczmarz(`FixedEffects' S, `Variables' y, `Variables' ans,| `Boolean' get_proj) {
	`Integer' 	g
	`Vector' rand
	if (args()<4 | get_proj==.) get_proj = 0
	rand = sort( ( (1::S.G) , uniform(S.G,1) ) , 2 )[.,1]
	ans = y - S.project_one_fe(y, rand[1])
	for (g=2; g<=S.G; g++) {
		ans = ans - S.project_one_fe(ans, rand[g])
	}
	for (g=S.G-1; g>=1; g--) {
		ans = ans - S.project_one_fe(ans, rand[g])
	}
	if (get_proj) ans = y - ans
}

// --------------------------------------------------------------------------

 `Void' function transform_sym_kaczmarz(`FixedEffects' S, `Variables' y, `Variables' ans,| `Boolean' get_proj) {
	`Integer' 	g
	if (args()<4 | get_proj==.) get_proj = 0
	if (S.timeit) timer_on(72)
	ans = y - S.project_one_fe(y, 1)
	if (S.timeit) timer_off(72)
	for (g=2; g<=S.G; g++) {
		if (S.timeit) timer_on(72)
		ans = ans - S.project_one_fe(ans, g)
		if (S.timeit) timer_off(72)
	}
	for (g=S.G-1; g>=1; g--) {
		if (S.timeit) timer_on(72)
		ans = ans - S.project_one_fe(ans, g)
		if (S.timeit) timer_off(72)
	}
	if (get_proj) ans = y - ans
}
end

mata:

// --------------------------------------------------------------------------
// Transformations: Compute RESIDUALS, not projections
// --------------------------------------------------------------------------

`Void' function transform_cimmino(`FixedEffects' S, `Variables' y, `Variables' ans,| `Boolean' get_proj) {
	`Integer' 	g
	if (args()<4 | get_proj==.) get_proj = 0
	ans = S.project_one_fe(y, 1)
	for (g=2; g<=S.G; g++) {
		ans = ans + S.project_one_fe(y, g)
	}
	ans = get_proj ? ans / S.G : y - ans / S.G
}

// --------------------------------------------------------------------------

`Void' function transform_kaczmarz(`FixedEffects' S, `Variables' y, `Variables' ans,| `Boolean' get_proj) {
	`Integer' 	g

	if (args()<4 | get_proj==.) get_proj = 0

	ans = y - S.project_one_fe(y, 1)
	for (g=2; g<=S.G; g++) {
		ans = ans - S.project_one_fe(ans, g)
	}
	if (get_proj) ans = y - ans
}

// --------------------------------------------------------------------------
// This seems slower than kaczmarz (sym kaczmarz!); not used currently
`Void' function transform_rand_kaczmarz(`FixedEffects' S, `Variables' y, `Variables' ans,| `Boolean' get_proj) {
	`Integer' 	g
	`Vector' rand
	if (args()<4 | get_proj==.) get_proj = 0
	rand = sort( ( (1::S.G) , uniform(S.G,1) ) , 2 )[.,1]
	ans = y - S.project_one_fe(y, rand[1])
	for (g=2; g<=S.G; g++) {
		ans = ans - S.project_one_fe(ans, rand[g])
	}
	for (g=S.G-1; g>=1; g--) {
		ans = ans - S.project_one_fe(ans, rand[g])
	}
	if (get_proj) ans = y - ans
}

// --------------------------------------------------------------------------

 `Void' function transform_sym_kaczmarz(`FixedEffects' S, `Variables' y, `Variables' ans,| `Boolean' get_proj) {
	`Integer' 	g
	if (args()<4 | get_proj==.) get_proj = 0
	if (S.timeit) timer_on(62)
	ans = y - S.project_one_fe(y, 1)
	if (S.timeit) timer_off(62)
	for (g=2; g<=S.G; g++) {
		if (S.timeit) timer_on(62)
		ans = ans - S.project_one_fe(ans, g)
		if (S.timeit) timer_off(62)
	}
	for (g=S.G-1; g>=1; g--) {
		if (S.timeit) timer_on(62)
		ans = ans - S.project_one_fe(ans, g)
		if (S.timeit) timer_off(62)
	}
	if (get_proj) ans = y - ans
}
end


fix bug that pumas dont all get a minimum amount

* the world is crumbling. solutions are being introduced. they wont pass without radical political change
* 1 billion divided equally between states
* 1 billion distributed on the basis of relative 16-25 y/o compared to total in all states
* 1 billion on relative unemployed compared to toal in all states
* 1 billion on the basis of the rleative # disadvantaged youth compared to total in all states

*STEPS:
* # of 16-25 y/o's per state
* total # of 16-25 y/os
* # of unemployed per state
* total # unemployed
* # disdvantaged youth
* total # disadvantaged youth

*then by puma w/in state
*
*tabulate age if statefip == 06 & puma == 3703 [fweight=perwt]
egen youth_state = total(perwt * (age <= 25 & age >= 16)), by(statefip)
egen youth_puma = total(perwt * (age <= 25 & age >= 16)), by(statefip puma)
*sum youth_puma if statefip == 06 & puma == 3703
*25371 youth in lancaster
gen youth_percentage = youth_puma / youth_state

egen unemployed_state = total(perwt * (empstat == 2)), by(statefip)
egen unemployed_puma = total(perwt * (empstat == 2)), by(statefip puma)

gen unemployed_percentage = unemployed_puma / unemployed_state

*sum youth_puma if statefip == 06 & puma == 3703
*5797 youth in lancaster

egen disadvantaged_state = total(perwt * ((age <= 25 & age >= 16) & poverty <= 125)), by(statefip)
egen disadvantaged_puma = total(perwt * ((age <= 25 & age >= 16) & poverty <= 125)), by(statefip puma)

gen disadvantaged_percentage = disadvantaged_puma / disadvantaged_state

egen youth_total = total(perwt * (age <= 25 & age >= 16))
egen unemployed_total = total(perwt * (empstat == 2))
egen disadvantaged_total = total(perwt * ((age <= 25 & age >= 16) & poverty <= 125))

gen youth_percentage_state = youth_state/youth_total
gen unemployed_percentage_state = unemployed_state/unemployed_total
gen disadvantaged_percentage_state = disadvantaged_state/disadvantaged_total

gen state_money = 1000 * .02 + 1000 * youth_percentage_state + 1000 * unemployed_percentage_state + 1000 * disadvantaged_percentage_state

egen npumas = nvals(puma), by(statefip)

gen puma_money = (state_money/3) * (youth_percentage + unemployed_percentage + disadvantaged_percentage)
gen numjobs = puma_money / .0105 //.0105 is a 10,500 dollar per yr job

*figure out households w/ youth in them
egen householdwithyouth = max(age >= 16 & age <= 26), by(serial)
*youth hh's living in poverty
gen youthinpov = (householdwithyouth & (poverty <= 100))
tab youthinpov if pernum == 1 [fweight = perwt]
*158 jobs for 5287 households

gen job_money = 0
replace job_money = 10500 if youthinpov & pernum == 1
gen newincome = hhinc+job_money

sgini newincome if pernum == 1 [fweight=hhwt]
sgini hhinc if pernum == 1 [fweight=hhwt]

* the world is crumbling. solutions are being introduced. they wont pass without radical political change
* 1 billion divided equally between states
* 1 billion distributed on the basis of relative 16-25 y/o compared to total in all states
* 1 billion on relative unemployed compared to toal in all states
* 1 billion on the basis of the rleative # disadvantaged youth compared to total in all states

*STEPS:
* # of 16-25 y/o's per state
* total # of 16-25 y/os
* # of unemployed per state
* total # unemployed
* # disdvantaged youth
* total # disadvantaged youth

*then by puma w/in state
*
*tabulate age if statefip == 06 & puma == 3703 [fweight=perwt]
egen youth_state = total(perwt * (age <= 25 & age >= 16)), by(statefip)
egen youth_puma = total(perwt * (age <= 25 & age >= 16)), by(statefip puma)
*sum youth_puma if statefip == 06 & puma == 3703
*25371 youth in lancaster
gen youth_percentage = youth_puma / youth_state

egen unemployed_state = total(perwt * (empstat == 2)), by(statefip)
egen unemployed_puma = total(perwt * (empstat == 2)), by(statefip puma)

gen unemployed_percentage = unemployed_puma / unemployed_state

*sum youth_puma if statefip == 06 & puma == 3703
*5797 youth in lancaster

egen disadvantaged_state = total(perwt * ((age <= 25 & age >= 16) & poverty <= 125)), by(statefip)
egen disadvantaged_puma = total(perwt * ((age <= 25 & age >= 16) & poverty <= 125)), by(statefip puma)

gen disadvantaged_percentage = disadvantaged_puma / disadvantaged_state

egen youth_total = total(perwt * (age <= 25 & age >= 16))
egen unemployed_total = total(perwt * (empstat == 2))
egen disadvantaged_total = total(perwt * ((age <= 25 & age >= 16) & poverty <= 125))

gen youth_percentage_state = youth_state/youth_total
gen unemployed_percentage_state = unemployed_state/unemployed_total
gen disadvantaged_percentage_state = disadvantaged_state/disadvantaged_total

gen state_money = 1000 * .02 + 1000 * youth_percentage_state + 1000 * unemployed_percentage_state + 1000 * disadvantaged_percentage_state

egen npumas = nvals(puma), by(statefip)

gen puma_money = (state_money/4) * (1/npumas + youth_percentage + unemployed_percentage + disadvantaged_percentage)
gen numjobs = puma_money / .0105 //.0105 is a 10,500 dollar per yr job

*figure out households w/ youth in them
egen householdwithyouth = max(age >= 16 & age <= 26), by(serial)
*youth hh's living in poverty
gen youthinpov = (householdwithyouth & (poverty <= 100))
tab youthinpov if pernum == 1 [fweight = perwt]
*158 jobs for 5287 households

gen job_money = 0
replace job_money = 10500 if youthinpov & pernum == 1
gen newincome = hhinc+job_money

sgini newincome if pernum == 1 [fweight=hhwt]
sgini hhinc if pernum == 1 [fweight=hhwt]


added alternate quality measure

added alternate quality measure

clear all
set more off
set matsize 11000
program drop _all

/////////// SET PARAMETERS //////////////////

// enter path of the wikibaseball directory here
local home "/home/nagaraj/Dropbox/research/wikibaseball/"

qui adopath + "`home'scripts/stata/ado"
declare_global `home'
cd ${path}

/////////// DATA CREATION //////////////////

// STEP 1: Make Sample A
make_cite

// STEP 2 : Make Sample B

// 2a. create three separate files

make_baseball

make_basketball

make_traf

// 2b. Merge data

use ${lahman}bb_master, clear
qui append using ${basketball}bk_master
gen isbaseball = (minutesrank==.)
save ${stash}bbk_master, replace

make_merge

// we now have two main datasets
// sample A -- ${stash}citelines
// sample B -- ${stash}master

/////////// ANALYSIS //////////////////

// TABLES
//////////////////////////////

// 1. Summary Stats (Sample A and B)
summary

// 2. T-Test Table (Sample A and B)
t_test

// 3. Impact of copyright DD (Sample A and B)
reg_dd_combined numcite bd 2004 2012 2009

// 4. Traffic Sample B
reg_traf

// 5. Images vs. Text (Samples A and B)
reg_dd_compare numimg numtext

// FIGURES
//////////////////////////////

// 1. schematic figure -- no data required

// 2. Time-series impact of digitization (Sample A)
mean_digit

// 3. DD chart-- impact on citations (Sample A and B)
cite_ddpic cites
ddpic bd 1

// 4. Killer chart (Sample A)
cite_killer_pic

// 5. Killer pic (Sample A) -- for images and text
cite_killer_pic

// 6. DD Pic (Sample A)
cite_ddpic img
cite_ddpic text

// 7. Quality-wise impacts
chart_qual img numallstar
chart_qual traf numallstar

/////////////////////////////////
// APPENDIX

//1. Digitization effect using Basketball data (Sample B)
reg_dd_appendix

// 2. Leads and Lags Regression (Sample A)
ddpic_table

// 3. alternate years
reg_dd_combined numcite bd 2004 2009 2007

// 4. this shortens the panel length
reg_dd_combined numcite bd 2005 2011 2009
reg_dd_combined numcite bd 2006 2010 2009

// 5. alternate specs
reg_robust copy

// 6. Out of copyright exposure index
robust_traf

// 7. differential effects images vs. test using Sample B
reg_dd_compare_sampleb img text

// 8. table for chart quality
reg_chart_qual

// 9. robustness figure with alternate quality measure
chart_qual img gp
chart_qual traf gp

clear all
set more off
set matsize 11000
program drop _all

/////////// SET PARAMETERS //////////////////

// enter path of the wikibaseball directory here
local home "/home/nagaraj/Dropbox/research/wikibaseball/"

qui adopath + "`home'scripts/stata/ado"
declare_global `home'
cd ${path}

/////////// DATA CREATION //////////////////

// STEP 1: Make Sample A
make_cite

// STEP 2 : Make Sample B

// 2a. create three separate files

make_baseball

make_basketball

make_traf

// 2b. Merge data

use ${lahman}bb_master, clear
qui append using ${basketball}bk_master
gen isbaseball = (minutesrank==.)
save ${stash}bbk_master, replace

make_merge

// we now have two main datasets
// sample A -- ${stash}citelines
// sample B -- ${stash}master

/////////// ANALYSIS //////////////////

// TABLES
//////////////////////////////

// 1. Summary Stats (Sample A and B)
summary

// 2. T-Test Table (Sample A and B)
t_test

// 3. Impact of copyright DD (Sample A and B)
reg_dd_combined numcite bd 2004 2012 2009

// 4. Traffic Sample B
reg_traf

// 5. Images vs. Text (Samples A and B)
reg_dd_compare numimg numtext

// FIGURES
//////////////////////////////

// 1. schematic figure -- no data required

// 2. Time-series impact of digitization (Sample A)
mean_digit

// 3. DD chart-- impact on citations (Sample A and B)
cite_ddpic cites
ddpic bd 1

// 4. Killer chart (Sample A)
cite_killer_pic

// 5. Killer pic (Sample A) -- for images and text
cite_killer_pic

// 6. DD Pic (Sample A)
cite_ddpic img
cite_ddpic text

// 7. Quality-wise impacts
chart_qual img
chart_qual traf

/////////////////////////////////
// APPENDIX

//1. Digitization effect using Basketball data (Sample B)
reg_dd_appendix

// 2. Leads and Lags Regression (Sample A)
ddpic_table

// 3. alternate years
reg_dd_combined numcite bd 2004 2009 2007

// 4. this shortens the panel length
reg_dd_combined numcite bd 2005 2011 2009
reg_dd_combined numcite bd 2006 2010 2009

// 5. alternate specs
reg_robust copy

// 6. Out of copyright exposure index
robust_traf

// 7. differential effects images vs. test using Sample B
reg_dd_compare_sampleb img text

// 8. table for chart quality
reg_chart_qual


md-to-man.do: redo-ifcreate on possible paths to pandoc

This way, if you run

  ./redo
  $PKGMGR install pandoc
  ./redo

you'll still wind up with man pages.

if pandoc </dev/null 2>/dev/null; then
	echo 'pandoc -s -r markdown -w man -o $3 $1.md'
else
	(IFS=:; for DIR in $PATH; do redo-ifcreate "$DIR/pandoc"; done)
	echo "Warning: pandoc not installed; can't generate manpages." >&2
	echo 'echo Skipping: $1.1 >&2'
fi

if pandoc </dev/null 2>/dev/null; then
	echo 'pandoc -s -r markdown -w man -o $3 $1.md'
else
	echo "Warning: pandoc not installed; can't generate manpages." >&2
	echo 'echo Skipping: $1.1 >&2'
fi


do file

do file

/*******************************************************************************
AUTHOR: Leonel Fernandez
ORGANIZATION: Mexico Evalua
URL: github.com/mexicoevalua/incidenciaDelictiva/blob/master/descarga_de_datos.do
DATE:January, 2015
PURPOSE: Create clean database and indicators of official crime data in Mexico
DATA IN:
			-IncidenciaDelictiva_FueroComun_Estatal_1997-{MONTH_YEAR}.xls
			(Raw monthly data of 66 crime types in the 32 federative entities
			of Mxico, 1997-actual [Updates in the 20th of each month])

			-Incidencia Delictiva FC Municipal 2011 -{YEAR}.xlsx"(Raw monthly
			data of 66 crime types in all the municipalities
			of Mxico, 1997-actual [Updates in the 20th of each month])

			-state-population.csv (Cleaned estimated mid-year population data at
			the state level from the CONAPO (2010) by Diego Valle
			https://github.com/diegovalle/conapo-2010

			-municipio-population2010-2030.csv (Cleaned estimated mid-year
			population data at the municipal level from the CONAPO (2010)
			by Diego Valle.  https://github.com/diegovalle/conapo-2010

DATA OUT:
			fuero-comun-estados.csv
			fuero-comun-municipios.csv
*******************************************************************************/
version 13.1
clear all
set more off

* Defining globals. Change working directory HERE*
global micrositio "~/Desktop" /*Change the '~' simbol with the directory where the
						Micrositio is stored */
global files "$micrositio/Micrositio/incidenciaDelictiva/data"

*Downloading datasets. The program needs at least 2.5 GB of free Hard Disk space*
cd "$files"
/*IMPORTANT:After the 20th of each month check the new URL in SESNSP site and
			1)Change updated URLs in /*1*/ & /*3*/ (lines 44 & 48)
			2)Chamge updated files names in /*2*/ & /*4*/(line 46 & 50)		*/
/*1*/ copy "http://www.secretariadoejecutivo.gob.mx/docs/pdfs/incidencia%20delictiva%20del%20fuero%20comun/IncidenciaDelictiva_FueroComun_Estatal_1997-Diciembre2014.zip" incidencia_e.zip, replace
		unzipfile "incidencia_e.zip", replace
/*2*/ global state "$files/IncidenciaDelictiva_FueroComun_Estatal_1997-Diciembre2014.xlsx"
		copy "$state" IncidenciaDelictivaEstatal.xlsx, replace
/*3*/ copy "http://secretariadoejecutivo.gob.mx/Incidencia%20Municipal%20Diciembre/IncidenciaDelictiva-Municipal2011-2014.zip" incidencia_m.zip, replace
		unzipfile "incidencia_m.zip", replace
/*4*/ global municip "$files/Incidencia Delictiva FC Municipal 2011 - 2014.xlsx"
		copy "$municip" IncidenciaDelictivaMunicipal.xlsx, replace
copy "https://raw.githubusercontent.com/diegovalle/conapo-2010/master/clean-data/state-population.csv" statepop.csv
copy "https://raw.githubusercontent.com/diegovalle/conapo-2010/master/clean-data/municipio-population2010-2030.csv" munpop.csv, replace

*STATES***********************************************************************
*Cleaning population data set*
clear
import delimited "$files/statepop.csv"
drop if year > 2014 | year <1997
rename statecode state_code
save statepop.dta, replace

*Cleaning SESNSP's crime data set*

clear
import excel using "IncidenciaDelictivaEstatal.xlsx", firstrow
drop if AO == .

*Erase leading, trailing and intermediate blank spaces in all string variables
replace  ENTIDAD = trim(itrim(ENTIDAD))
replace  MODALIDAD = trim(itrim(MODALIDAD))
replace  TIPO = trim(itrim(TIPO))
replace  SUBTIPO = trim(itrim(SUBTIPO))

*Encoding state codes*
encode ENTIDAD, gen(state_code)
drop ENTIDAD TOTALAO

recode state_code (5=45) (6=46)
recode state_code (7=5) (8=6)
recode state_code (45=7) (46=8)

label define state_code 5 "COAHUILA", modify
label define state_code 6 "COLIMA", modify
label define state_code 7 "CHIAPAS", modify
label define state_code 8 "CHIHUAHUA", modify

*Renaming variables*
rename (AO MODALIDAD TIPO SUBTIPO ENERO FEBRERO MARZO ABRIL MAYO JUNIO JULIO ///
		AGOSTO SEPTIEMBRE OCTUBRE NOVIEMBRE DICIEMBRE) 						 ///
       (year category type subtype count1 count2 count3 count4 count5 count6 ///
		count7 count8 count9 count10 count11 count12)

*Reshaping to long format*
reshape long count, i(year category type subtype state_code) j(month)
destring count, replace

*Creating crime variable wich groups all categories of "ROBO" in just one*
gen crime = category
replace crime = "ROBO" if word(category,1) == "ROBO"

*Merging with population data base*
merge m:1 year state_code using "$files/statepop.dta", keepusing(total) ///
			nogenerate
rename total population

*Ordering variables and sorting cases*
order state_code year month crime category type subtype count population
sort state_code year crime category type subtype month

*Exporting to .cvs and saveing in .dta*
export delimited using "$files/fuero-comun-estados.csv", nolabel replace
save fuero-comun-estados.dta, replace

*Stop here if you want to keep all files
*deleting files*
rm incidencia_e.zip
rm "$state"
rm statepop.dta
rm fuero-comun-estados.dta
rm IncidenciaDelictivaEstatal.xlsx

*MUNICIPALITIES****************************************************************
*Cleaning population data set*
clear
import delimited "$files/munpop.csv"
drop if sex == "Males" | sex == "Females"
drop sex
drop if year == 2010 | year >2014
nsplit code, d(2 3) gen(state_code mun_code)
format code %05.0f
rename code id

save munpop.dta, replace

*Cleaning SESNSP's crime data set*

clear
set excelxlsxlargefile on
import excel using "IncidenciaDelictivaMunicipal.xlsx", firstrow
drop if AO == .
save fuero-comun-municipios.dta, replace

**Erase leading, trailing and intermediate blank spaces in all string variables
replace  ENTIDAD = trim(itrim(ENTIDAD))
replace  MUNICIPIO = trim(itrim(MUNICIPIO))
replace  MODALIDAD = trim(itrim(MODALIDAD))
replace  TIPO = trim(itrim(TIPO))
replace  SUBTIPO = trim(itrim(SUBTIPO))

replace SUBTIPO ="CON ARMA DE FUEGO" if SUBTIPO == "POR ARMA DE FUEGO"
replace SUBTIPO ="CON ARMA BLANCA" if SUBTIPO == "POR ARMA BLANCA"

*Encoding state codes*

drop ENTIDAD MUNICIPIO

*renaming variables*
rename (AO INEGI MODALIDAD TIPO SUBTIPO ENERO FEBRERO MARZO ABRIL MAYO JUNIO JULIO ///
		AGOSTO SEPTIEMBRE OCTUBRE NOVIEMBRE DICIEMBRE) 						 ///
       (year id category type subtype count1 count2 count3 count4 count5 count6 ///
		count7 count8 count9 count10 count11 count12)

nsplit id, d(2 3) gen(state_code mun_code)

*Reshaping to long format*
reshape long count, i(year id category type subtype state_code mun_code) j(month)
destring count, replace

*Creating crime variable wich groups all categories of "ROBO" in just one*
gen crime = category
replace crime = "ROBOS" if word(category,1) == "ROBO"

*Merging with population data base*
merge m:1 year state_code mun_code id using "$files/munpop.dta", keepusing(population) ///
			nogenerate

*Ordering variables and sorting cases*
order state_code mun_code year month crime category type subtype count population id
sort state_code mun_code year crime category type subtype month

*Exporting to .cvs and saveing in .dta*
export delimited using "$files/fuero-comun-municipios.csv", nolabel replace
save fuero-comun-municipios.dta, replace

*Stop here if you want to keep all files
*deleting files*
rm incidencia_m.zip
rm "$municip"
rm munpop.dta
rm fuero-comun-municipios.dta
rm IncidenciaDelictivaMunicipal.xlsx

************************End of do-file*****************************************

*******************************************************************************
* AUTHORS: Leonel Fernandez
* DATE:January, 2015
* PURPOSE: Create clean database and indicators of official crime data in Mexico
* DATA IN:
*			-IncidenciaDelictiva_FueroComun_Estatal_1997-Diciembre2014.xls
*			(Raw monthly data of 66 crime types in the 32 federative entities
*			of Mxico, 1997-actual [Updates in the 20th of each month])
*
*			-state-population.csv (Cleaned estimated mid-year population data at
*			the state and county level from the CONAPO (2010) by Diego Valle
*			https://github.com/diegovalle/conapo-2010
*
* DATA OUT:
*			fuero-comun-estados.csv
*
*******************************************************************************
clear
set more off

* Defining globals. Change working directory HERE*
global files "/Users/leoxnv/desktop/Micrositio/incidenciaDelictiva/data"

*Downloading datasets*
cd "$files"

**IMPORTANT: Change updated URL after the 20th of each month*
copy "http://www.secretariadoejecutivo.gob.mx/docs/pdfs/incidencia%20delictiva%20del%20fuero%20comun/IncidenciaDelictiva_FueroComun_Estatal_1997-Diciembre2014.zip" incidencia.zip, replace
unzipfile "incidencia.zip", replace
copy "https://raw.githubusercontent.com/diegovalle/conapo-2010/master/clean-data/state-population.csv" statepop.csv
copy "$files/IncidenciaDelictiva_FueroComun_Estatal_1997-Diciembre2014.xlsx" IncidenciaDelictivaEstatal.xlsx, replace

*Cleaning population data set*
clear
import delimited "$files/statepop.csv"
drop if year > 2014 | year <1997
rename statecode state_code
save statepop.dta, replace

*Cleaning SESNSP's crime data set*

clear
import excel using "IncidenciaDelictivaEstatal.xlsx", firstrow
drop if AO == .

**Erase leading, trailing and intermediate blank spaces in all string variables
replace  ENTIDAD = trim(itrim(ENTIDAD))
replace  MODALIDAD = trim(itrim(MODALIDAD))
replace  TIPO = trim(itrim(TIPO))
replace  SUBTIPO = trim(itrim(SUBTIPO))

*Encoding state codes*
encode ENTIDAD, gen(state_code)
drop ENTIDAD TOTALAO

recode state_code (5=45) (6=46)
recode state_code (7=5) (8=6)
recode state_code (45=7) (46=8)

label define state_code 5 "COAHUILA", modify
label define state_code 6 "COLIMA", modify
label define state_code 7 "CHIAPAS", modify
label define state_code 8 "CHIHUAHUA", modify

*renaming variables*
rename (AO MODALIDAD TIPO SUBTIPO ENERO FEBRERO MARZO ABRIL MAYO JUNIO JULIO ///
		AGOSTO SEPTIEMBRE OCTUBRE NOVIEMBRE DICIEMBRE) 						 ///
       (year category type subtype count1 count2 count3 count4 count5 count6 ///
		count7 count8 count9 count10 count11 count12)

*Reshaping to long format*
reshape long count, i(year category type subtype state_code) j(month)
destring count, replace

*Creating crime variable wich groups all categories of "ROBO" in just one*
gen crime = category
replace crime = "ROBO" if word(category,1) == "ROBO"

*Merging with population data base*
merge m:1 year state_code using "$files/statepop.dta", keepusing(total) ///
			nogenerate
rename total population

*Ordering variables and sorting cases*
order state_code year month crime category type subtype count population
sort state_code year crime category type subtype month

*Exporting to .cvs and saveing in .dta*
export delimited using "$files/fuero-comun-estados.csv", nolabel replace
save delitos-fuero-comun.dta, replace

*Stop here if you want to keep all files
*deleting files*
rm incidencia.zip
rm IncidenciaDelictiva_FueroComun_Estatal_1997-Diciembre2014.xlsx
rm statepop.dta
rm delitos-fuero-comun.dta

************************End of do-file*****************************************

/*ACTUALZIACIN MUNICIPIOS*/

clear
set more off

* Defining globals. Change working directory HERE*
global files "/Users/leoxnv/Crime rates in Mexico/Municipalities/Files"

*Downloading datasets*
cd "$files"

**IMPORTANT: Change updated URL after the 20th of each month*
copy "http://secretariadoejecutivo.gob.mx/Incidencia%20Municipal%20Diciembre/IncidenciaDelictiva-Municipal2011-2014.zip" incidencia.zip, replace
unzipfile "incidencia.zip", replace
copy "https://raw.githubusercontent.com/diegovalle/conapo-2010/master/clean-data/municipio-population2010-2030.csv" munpop.csv, replace
copy "$files/Incidencia Delictiva FC Municipal 2011 - 2014.xlsx" IncidenciaDelictivaMunicipal.xlsx, replace

*Cleaning population data set*
clear
import delimited "$files/munpop.csv"
drop if sex == "Males" | sex == "Females"
drop sex
drop if year == 2010 | year >2014
nsplit code, d(2 3) gen(state_code mun_code)
format code %05.0f
rename code id

save munpop.dta, replace

*Cleaning SESNSP's crime data set*

clear
set excelxlsxlargefile on
import excel using "IncidenciaDelictivaMunicipal.xlsx", firstrow
drop if AO == .
save delitos-fuero-comun.dta, replace

**Erase leading, trailing and intermediate blank spaces in all string variables
replace  ENTIDAD = trim(itrim(ENTIDAD))
replace  MUNICIPIO = trim(itrim(MUNICIPIO))
replace  MODALIDAD = trim(itrim(MODALIDAD))
replace  TIPO = trim(itrim(TIPO))
replace  SUBTIPO = trim(itrim(SUBTIPO))

replace SUBTIPO ="CON ARMA DE FUEGO" if SUBTIPO == "POR ARMA DE FUEGO"
replace SUBTIPO ="CON ARMA BLANCA" if SUBTIPO == "POR ARMA BLANCA"

*Encoding state codes*

drop ENTIDAD MUNICIPIO

*renaming variables*
rename (AO INEGI MODALIDAD TIPO SUBTIPO ENERO FEBRERO MARZO ABRIL MAYO JUNIO JULIO ///
		AGOSTO SEPTIEMBRE OCTUBRE NOVIEMBRE DICIEMBRE) 						 ///
       (year id category type subtype count1 count2 count3 count4 count5 count6 ///
		count7 count8 count9 count10 count11 count12)

nsplit id, d(2 3) gen(state_code mun_code)

*Reshaping to long format*
reshape long count, i(year id category type subtype state_code mun_code) j(month)
destring count, replace

*Creating crime variable wich groups all categories of "ROBO" in just one*
gen crime = category
replace crime = "ROBOS" if word(category,1) == "ROBO"

*Merging with population data base*
merge m:1 year state_code mun_code id using "$files/munpop.dta", keepusing(population) ///
			nogenerate

*Ordering variables and sorting cases*
order state_code mun_code year month crime category type subtype count population id
sort state_code mun_code year crime category type subtype month

*Exporting to .cvs and saveing in .dta*
export delimited using "$files/fuero-comun-municipios.csv", nolabel replace
save delitos-fuero-comun.dta, replace

*Stop here if you want to keep all files
*deleting files*
rm incidencia.zip
rm "Incidencia Delictiva FC Municipal 2011 - 2014.xlsx"
rm munpop.dta
rm delitos-fuero-comun.dta

************************End of do-file*****************************************


add games played

add games played

program make_baseball

make_hof

make_allstar

**make_match

read_mt

make_master

end

// This makes master baseball dataset
program make_master

insheet using ${lahman}Master.csv, clear

merge 1:1 playerid using ${lahman}hof, keep(match) nogen

// there are players who were hall of fame, but not all star
merge 1:1 playerid using ${lahman}allstar, keep(match master) nogen

// appearances
merge 1:1 playerid using ${lahman}appearances, keep(match master)

gen debutyear = substr(debut, 1,4)
gen finalyear = substr(finalgame, 1,4)

keep playerid birthyear debutyear finalyear deathyear name* everinducted *hof* numallstar firstallstar gp

destring, replace

gen playername = namefirst + " " + namelast

// this deletes and old player that we dont care about anyway
bysort playername: drop if _n > 1

// generate file for amazon to get me data on
drop if debutyear < 1940
drop if debutyear > 2000

merge 1:1 playerid using ${lahman}mt_bb_output, nogen

save ${lahman}bb_master, replace

end

program make_match

insheet using ${rawdata}match_wiki.csv, clear names
keep if sport == "Baseball"
drop  if wikiname == "None"

save ${rawdata}match_wiki, replace

end

// This Makes Hall of Fame
program make_hof

insheet using ${lahman}HallOfFame.csv, clear
bysort playerid: egen everinducted = max(inducted=="Y")
bysort playerid: gen num_hof_nominations = _N
sort playerid yearid
bysort playerid: gen hof_year_elected = yearid[_N]*everinducted
replace hof_year_elected = -1 if hof_year_elected == 0

bysort playerid: gen hof_year_first = yearid[1]

// TODO : calculate margin
bysort playerid: drop if _n > 1
keep playerid everinducted num_hof hof_year*

save ${lahman}hof, replace

end

// This Makes Allstar
program make_allstar

insheet using ${lahman}AllstarFull.csv, clear
bysort playerid: gen numallstar = _N
sort playerid yearid
bysort playerid: gen firstallstar = yearid[1]
bysort playerid: drop if _n > 1
keep playerid numallstar firstallstar
save ${lahman}allstar, replace

end

program make_appearance

insheet using ${lahman}Appearances.csv, clear

bysort playerid: egen gp = total(g_all)
bysort playerid: drop if _n > 1
keep playerid gp

save ${lahman}appearances, replace

end

////////////// MT
///////////////////////
program read_mt

insheet using ${lahman}mt_bb_output.csv, clear

gen wikihandle = subinstr(answer, "http://en.wikipedia.org/wiki/","",.)
keep inputp inputdisp wikihandle

gen tmp = word(inputd,1) + "_" + word(inputd,2)
list tmp wiki if tmp != wikihandle

fix_errors

rename inputplayerid playerid
keep wikihandle playerid

save ${lahman}mt_bb_output, replace

end

program fix_errors

replace wikihandle = tmp if tmp == "Sonny_Jackson"
replace wikihandle = tmp if tmp == "Bill_Bruton"
replace wikihandle = tmp if tmp == "Terry_Kennedy"
replace wikihandle = "Bill_Lee_(left-handed_pitcher)" if inputp == "leebi03"

replace wikihandle = tmp if tmp == "Todd_Jones"
replace wikihandle = tmp if tmp == "Mel_Stottlemyre"
replace wikihandle = "Mike_Jackson_(right-handed_pitcher)" if tmp == "Michael_Jackson"
replace wikihandle = tmp if tmp == "Gil_Hodges"
replace wikihandle = "Tony_Pe%C3%B1a" if tmp == "Tony_Pena"
replace wikihandle = tmp if tmp == "Bob_Porterfield"
replace wikihandle = tmp if tmp == "Milt_Pappas"
replace wikihandle = "Willie_Horton_(baseball)" if tmp == "Willie_Horton"
replace wikihandle = tmp if tmp == "Hank_Edwards"
replace wikihandle = tmp if tmp == "Cecil_Cooper"
replace wikihandle = tmp if tmp == "Chris_Short"
replace wikihandle = tmp if tmp == "Clint_Courtney"
replace wikihandle = tmp if tmp == "Danny_Jackson"
replace wikihandle = tmp if tmp == "Davey_Lopes"
replace wikihandle = tmp if tmp == "Del_Rice"
replace wikihandle = tmp if tmp == "Juan_Pizarro"

replace wikihandle = "Andy_Van_Slyke" if tmp == "Andy_Van"
replace wikihandle = tmp if tmp == "Fred_Gladding"
replace wikihandle = tmp if tmp == "Todd_Zeile"
replace wikihandle = tmp if tmp == "Walker_Cooper"
replace wikihandle = tmp if tmp == "Carlos_May"
replace wikihandle = tmp if tmp == "Roberto_Alomar"

end

// This makes baseball data for MT
program make_mt

use ${lahman}bb_master, clear

gen displayname = namefirst + " " + namelast + " " + namegiven
gen searchname = namefirst + "+" + namelast + "+" + namegiven + " baseball"

gen link = "http://en.wikipedia.org/w/index.php?title=Special%3ASearch&profile=default&search=" + searchname + "&fulltext=Search"

outsheet playerid displayname link using ${stash}mt_input_bb.csv, replace comma

end

program make_baseball

make_hof

make_allstar

**make_match

read_mt

make_master

end

// This makes master baseball dataset
program make_master

insheet using ${lahman}Master.csv, clear

merge 1:1 playerid using ${lahman}hof, keep(match) nogen

// there are players who were hall of fame, but not all star
merge 1:1 playerid using ${lahman}allstar, keep(match master)

gen debutyear = substr(debut, 1,4)
gen finalyear = substr(finalgame, 1,4)

keep playerid birthyear debutyear finalyear deathyear name* everinducted *hof* numallstar firstallstar

destring, replace

gen playername = namefirst + " " + namelast

// this deletes and old player that we dont care about anyway
bysort playername: drop if _n > 1

// generate file for amazon to get me data on
drop if debutyear < 1940
drop if debutyear > 2000

merge 1:1 playerid using ${lahman}mt_bb_output, nogen

save ${lahman}bb_master, replace

end

program make_match

insheet using ${rawdata}match_wiki.csv, clear names
keep if sport == "Baseball"
drop  if wikiname == "None"

save ${rawdata}match_wiki, replace

end

// This Makes Hall of Fame
program make_hof

insheet using ${lahman}HallOfFame.csv, clear
bysort playerid: egen everinducted = max(inducted=="Y")
bysort playerid: gen num_hof_nominations = _N
sort playerid yearid
bysort playerid: gen hof_year_elected = yearid[_N]*everinducted
replace hof_year_elected = -1 if hof_year_elected == 0

bysort playerid: gen hof_year_first = yearid[1]

// TODO : calculate margin
bysort playerid: drop if _n > 1
keep playerid everinducted num_hof hof_year*

save ${lahman}hof, replace

end

// This Makes Allstar
program make_allstar

insheet using ${lahman}AllstarFull.csv, clear
bysort playerid: gen numallstar = _N
sort playerid yearid
bysort playerid: gen firstallstar = yearid[1]
bysort playerid: drop if _n > 1
keep playerid numallstar firstallstar
save ${lahman}allstar, replace

end

////////////// MT
///////////////////////
program read_mt

insheet using ${lahman}mt_bb_output.csv, clear

gen wikihandle = subinstr(answer, "http://en.wikipedia.org/wiki/","",.)
keep inputp inputdisp wikihandle

gen tmp = word(inputd,1) + "_" + word(inputd,2)
list tmp wiki if tmp != wikihandle

fix_errors

rename inputplayerid playerid
keep wikihandle playerid

save ${lahman}mt_bb_output, replace

end

program fix_errors

replace wikihandle = tmp if tmp == "Sonny_Jackson"
replace wikihandle = tmp if tmp == "Bill_Bruton"
replace wikihandle = tmp if tmp == "Terry_Kennedy"
replace wikihandle = "Bill_Lee_(left-handed_pitcher)" if inputp == "leebi03"

replace wikihandle = tmp if tmp == "Todd_Jones"
replace wikihandle = tmp if tmp == "Mel_Stottlemyre"
replace wikihandle = "Mike_Jackson_(right-handed_pitcher)" if tmp == "Michael_Jackson"
replace wikihandle = tmp if tmp == "Gil_Hodges"
replace wikihandle = "Tony_Pe%C3%B1a" if tmp == "Tony_Pena"
replace wikihandle = tmp if tmp == "Bob_Porterfield"
replace wikihandle = tmp if tmp == "Milt_Pappas"
replace wikihandle = "Willie_Horton_(baseball)" if tmp == "Willie_Horton"
replace wikihandle = tmp if tmp == "Hank_Edwards"
replace wikihandle = tmp if tmp == "Cecil_Cooper"
replace wikihandle = tmp if tmp == "Chris_Short"
replace wikihandle = tmp if tmp == "Clint_Courtney"
replace wikihandle = tmp if tmp == "Danny_Jackson"
replace wikihandle = tmp if tmp == "Davey_Lopes"
replace wikihandle = tmp if tmp == "Del_Rice"
replace wikihandle = tmp if tmp == "Juan_Pizarro"

replace wikihandle = "Andy_Van_Slyke" if tmp == "Andy_Van"
replace wikihandle = tmp if tmp == "Fred_Gladding"
replace wikihandle = tmp if tmp == "Todd_Zeile"
replace wikihandle = tmp if tmp == "Walker_Cooper"
replace wikihandle = tmp if tmp == "Carlos_May"
replace wikihandle = tmp if tmp == "Roberto_Alomar"

end

// This makes baseball data for MT
program make_mt

use ${lahman}bb_master, clear

gen displayname = namefirst + " " + namelast + " " + namegiven
gen searchname = namefirst + "+" + namelast + "+" + namegiven + " baseball"

gen link = "http://en.wikipedia.org/w/index.php?title=Special%3ASearch&profile=default&search=" + searchname + "&fulltext=Search"

outsheet playerid displayname link using ${stash}mt_input_bb.csv, replace comma

end


* quotes around variables seem sane

* quotes around variables seem sane

redo-ifchange $1.linklist
INLIST=false

exec >&2  # redirect stdout to stderr

cat $1.linklist | while read LINE; do
    LINEMARKER=`echo $LINE | grep --only-matching --perl-regexp '^....(?= )' || echo`
    test "$LINEMARKER" = "HTML" && test "$INLIST" = "true" && echo '
' >> $3
    test "$LINEMARKER" = "HTML" && HTML=`echo $LINE | grep --only-matching --perl-regexp '(?<=^.... ).*$'`
    test "$LINEMARKER" = "HTML" && echo $HTML >> $3
    test "$LINEMARKER" = "HTML" && INLIST=false && continue

    TIMECODE=`echo $LINE | cut -d " " -f1`
    URL=`echo $LINE | cut -d " " -f2 | tr -d "<>"`

    # check if http/https URLs are accessible
    echo -n .
    echo "$URL" | head -c4 | grep -q "http" && (
        curl -A "redokast/2012-05-25" -m 20 -isk "$URL" | head -n1 | \
            egrep -q "200|301|302|401|405" || (
                echo; echo; echo "<$URL> is not accessible."
                echo; curl -A "redokast/2012-05-25" -m 20 -Isk "$URL"
                false
            )
    )

    TEXT=`echo $LINE | cut -d " " -f3-`
    test "$INLIST" = "false" && echo '
' >> $3
    cat << EOF >> $3

$TIMECODE
  $TEXT
EOF
    INLIST=true
done

echo  # newline, necessary because of the dots

cat << EOF >> $3

EOF

redo-ifchange $1.linklist
INLIST=false

exec >&2  # redirect stdout to stderr

cat $1.linklist | while read LINE; do
    LINEMARKER=`echo $LINE | grep --only-matching --perl-regexp '^....(?= )' || echo`
    test "$LINEMARKER" = "HTML" && test "$INLIST" = "true" && echo '
' >> $3
    test "$LINEMARKER" = "HTML" && HTML=`echo $LINE | grep --only-matching --perl-regexp '(?<=^.... ).*$'`
    test "$LINEMARKER" = "HTML" && echo $HTML >> $3
    test "$LINEMARKER" = "HTML" && INLIST=false && continue

    TIMECODE=`echo $LINE | cut -d " " -f1`
    URL=`echo $LINE | cut -d " " -f2 | tr -d "<>"`

    # check if http/https URLs are accessible
    echo -n .
    echo $URL | head -c4 | grep -q "http" && (
        curl -A "redokast/2011-10-22" -m 20 -isk $URL | head -n1 | \
            egrep -q "200|301|302|401|405" || (
                echo; echo; echo "<$URL> is not accessible."
                echo; curl -A "redokast/2011-10-22" -m 20 -Isk $URL
                false
            )
    )

    TEXT=`echo $LINE | cut -d " " -f3-`
    test "$INLIST" = "false" && echo '
' >> $3
    cat << EOF >> $3

$TIMECODE
  $TEXT
EOF
    INLIST=true
done

echo  # newline, necessary because of the dots

cat << EOF >> $3

EOF


Update author section

Update author section

{smcl}
{* *! version 1.0.0 Matthew White 30dec2014}{...}
{title:Title}

{phang}
stata-java {hline 2} Java library of helper classes for Stata plug-ins

{marker description}{...}
{title:Description}

{pstd}
stata-java is a Java library of helper classes for Stata plug-ins.
The SSC package contains no ado-files, and the library is meant for
indirect use in other packages.

{marker remarks}{...}
{title:Remarks}

{pstd}
The GitHub repository for stata-java is
{browse "https://github.com/matthew-white/stata-java":here}.

{marker author}{...}
{title:Author}

{pstd}Matthew White{p_end}

{pstd}For questions or suggestions, submit a
{browse "https://github.com/matthew-white/stata-java/issues":GitHub issue}
or e-mail researchsupport@poverty-action.org.{p_end}

{smcl}
{* *! version 1.0.0 Matthew White 30dec2014}{...}
{title:Title}

{phang}
stata-java {hline 2} Java library of helper classes for Stata plug-ins

{marker description}{...}
{title:Description}

{pstd}
stata-java is a Java library of helper classes for Stata plug-ins.
The SSC package contains no ado-files, and the library is meant for
indirect use in other packages.

{marker remarks}{...}
{title:Remarks}

{pstd}
The GitHub repository for stata-java is
{browse "https://github.com/matthew-white/stata-java":here}.

{marker author}{...}
{title:Author}

{pstd}Matthew White, Innovations for Poverty Action{p_end}
{pstd}mwhite@poverty-action.org{p_end}