[iegitaddmd] improved language

[iegitaddmd] improved language

Co-authored-by: Luiza Andrade <3e25dce60d6313c358f2ba3c7da4f9a53677fa7f@worldbank.org>

{smcl}
{* 5 Nov 2019}{...}
{hline}
help for {hi:iegitaddmd}
{hline}

{title:Title}

{cmd:iegitaddmd} {hline 2} Creates a placeholder file in subfolders of a GitHub repository folder, which allows committing folder structures with empty folders.

{phang2}For a more descriptive discussion on the intended usage and work flow of this
command please see the {browse "https://dimewiki.worldbank.org/wiki/Iegitaddmd":DIME Wiki}.

{title:Syntax}

{phang} {cmdab:iegitaddmd} , {opt folder(full_file_path)} [{opt comparefolder(full_file_path)} {cmd:customfile(}{it:{help filename}}{cmd:)} {opt all} {opt skip} {opt replace} {opt skipfolders(folder_name)}  {opt auto:matic} {opt dry:run}]

{marker opts}{...}
{synoptset 28}{...}
{synopthdr:options}
{synoptline}
{synopt :{opt folder(full_file_path)}}Specifies the folder path to the project folder clone where placeholder files will be created{p_end}
{synopt :{opt comparefolder(full_file_path)}}Specifies the top folder of a folder tree to be compared with {opt folder()}{p_end}
{synopt :{cmd:customfile(}{it:{help filename}}{cmd:)}}Specifies a file saved on disk that is used instead of the default file as placeholder{p_end}
{synopt :{cmd:all}}Creates the placeholder file in every subfolder of {opt folder()}, whether empty or not{p_end}
{synopt :{opt skip}}If option {opt all} is used and a folder has a file with same name as placeholder file, then nothing is done{p_end}
{synopt :{opt replace}}If option {opt all} is used and a folder has a file with same name as placeholder file, then the file is overwritten{p_end}
{synopt :{opt skipfolders(folder_name)}}List of folders to be skipped. The folder {it:.git} is always added to this list, even when option is not used{p_end}
{synopt :{opt auto:matic}}Makes the command create placeholder files without prompting the user for each file{p_end}
{synopt :{opt dry:run}}Makes the command list all the files that would have been created without this option.{p_end}
{synoptline}

{marker desc}
{title:Description}

{pstd}GitHub does not sync empty folders, or folder that only contain ignored files. However, it is common in research projects that these types of folders needs to be shared anyways. {cmd:iegitaddmd} provides a solution to this by creating placeholder files in in these folders. This placeholder file can then be shared to GitHub and that way the folder is also shared. {cmd:iegitaddmd} have two ways to identify which those folders are that are developed with the two use cases described below in mind.{p_end}

{pstd}{bf:Use case 1.} It is common in the beginning of a research project that a standardized folder structure is set up (for example with {help iefolder}) where some folders are created that are not yet needed. Examples of folders like that could be the output folders. If these folders are not shared at the time they are created, there is a risk that different team members name them differently which will be confusing and could lead to errors. {cmd:iegitaddmd} can solve this by scanning a folder and all its sub-folders for completely empty folders, and create a placeholder file in that folder.{p_end}

{pstd}The solution to use case 1 in {cmd:iegitaddmd} is a Stata adaptation of {it:Solution B} in {browse "http://bytefreaks.net/gnulinux/bash/how-to-add-automatically-all-empty-folders-in-git-repository" :this post}.{p_end}

{pstd}{bf:Use case 2.} A common method when transferring an already ongoing research project to GitHub is to copy all of the data work folder from wherever the files and folders where shared before (for example DropBox) to a clone of the repository. Then only the files that are meant to be shared on GitHub are uploaded to the cloud using a {it:.gitignore file}. Empty folders or folders where all files are ignored is then not shared through GitHub. {cmd:iegitaddmd} can solve this by scanning the DropBox folder (or wherever the project folder was shared before) for sub-folders that exist in that folder but not in the clone and crate that folder in the clone and put a placeholder in it.{p_end}

{pstd}The way you differentiate between these two use cases when using {cmd:iegitaddmd} is whether the option {opt comparefolder()} is used or not. If that option is used, then {cmd:iegitaddmd} will identify all folders that are in {opt comparefolder()} that is not in {opt folder()}, and create a placeholder file there. If {opt comparefolder()} is not used, then {cmd:iegitaddmd} will identify all completely empty folders (no files or subfolders) and create a placeholder file there.{p_end}

{pstd}The default placeholder file used if no file is specified using option {opt customfile()} is called README.md, which is a name and format recognized by GitHub.com so that the content of the file is displayed on GitHub.com when navigating to that folder using a browser. The placeholder file may be removed as	soon as files that are not ignored have been added to the folder, or even better is to keep the file but edit the content of it so it documents the content of that folder for other and future team members.{p_end}

{marker optslong}
{title:Options}

{phang}{opt folder(file_path)} is the folder path to the project folder clone where the placeholder files will be created. Unless option {opt comparefolder()} or {opt all} is used, then placeholder files will be created in all empty folders. See option {opt manual} for a way to manually confirm each folder before a placeholder file is created in it.{p_end}

{phang}{opt comparefolder(file_path)} is used to compare a file tree in a different location and create the folders with placeholder files in all folders that exist in {opt comparefolder()} but not in {opt folder()}. If a folder that does no exist in {opt folder()} has a subfolder, then both the folder and the subfolder will be created and the placeholder file is only crated in the sub-folder. This is enough to share both folder and the subfolder in GitHub. If the {opt comparefolder()} and {opt folder()} ends with folders with different names, for example {opt folder(${project}/datawork/baseline)} and {opt comparefolder(${project}/datawork)}, then the command will throw an error as it is likely that the paths do not point to corresponding folders in the two folder trees. See option {opt manual} for a way to manually confirm each folder before a placeholder file is created in it.{p_end}

{phang}{cmd:file(}{it:{help filename}}{cmd:)} allows the user to specify a file saved on
	the computer to be used as the placeholder file	instead of the default README.md file. This
	allows people and organizations to write their own template placeholder files according to their own
	preferences. We recommend that a file of type .md (markdown) and name README.md is used as
	GitHub.com recognizes this name and format and will display the content of the file when someone
	browse to that folder on GitHub.com. But this is not a technical requirement, any file type
	and name can be used as placeholder file.{p_end}

{phang}{opt all} creates the placeholder file in {opt folder()} and in every subfolder,
	regardless if they are empty or not. This allows the user to create a placeholder file in every
	folder that can be edited with documentation and instructions to the purpose and usage of
	each subfolder. This option can also be important when .{it:gitignore} is used, as {cmd:iegitaddmd} will not
	create files to subfolders that only has ignored file -- in which case the folder will not
	be synced by GitHub.{p_end}

{phang}{opt skip} and {opt replace} tells {cmd:iegitaddmd} what to do if the option {opt all} is used
	and if any of the folders contain a file with the same name as the placeholder file the command
	is trying to create. If a file with the same name as the placeholder file exist and neither of these options
	are used, then the command will throw an error. If {opt skip} is used, then nothing is done in the
	folder where the file with the same name exists and the command proceeds to the next folder.
	If {opt replace} is used then the file with the same name is overwritten with the new placeholder
	file before the command proceeds to the next folder.{p_end}

{phang}{opt skipfolders(folder_name)} can be used to tell {cmd:iegitaddmd} which folders
in which a placeholder file should never be created. The best example of this is the {it:.git}
folder in which placeholders never should be created. That name is always skipped regardless of
this option being used or not. Use this option to list additional folders to be skipped. You
should not list the full folder path in this option, just the folder name. All folders with
that name will be skipped regardless of location in the project folder. The folder names may
be listed with or without quotation marks as long as there are no spaces in the names. If any of
the folder names has spaces, then quotation marks must be used for all folder names.

{phang}{opt auto:matic} can be used to speed up the creation of placeholders by telling the
command to not prompt the users for confirmation for each file before it is created. The
default is that the command is asking the user before creating each place holder file.
This option should only be used when you are confident you have specified the correct folder
paths. We recommend that you use the {opt dryrun} with this option to make sure that the
folder paths are correct.{p_end}

{phang}{opt dry:run} can be used to safely test that the folder paths are specified correctly
before any placeholder files are created. When this option it used the command simply lists
the file that would have been created if this option were not used. Once you are confident
that list is correct, you can remove this option and re-run the command and the files will
be created.{p_end}

{title:Example}

{pstd}{inp:global github_folder "C:\Users\JohnSmith\Documents\GitHub\ProjectA"}{break}{inp:iegitaddmd , folder({it:"$github_folder"})}{p_end}

{pstd}In the example above, there is a GitHub repository in the folder ProjectA. This
	repository has a folder structure where some folders are still empty but will later
	be populated with files. In order to have all folders, even the empty ones, synced on all
	collaborators' cloned local copies of the repository, the folders need to contain at least
	one file, which is being created by the command.{p_end}

{title:Acknowledgements}

{pstd}I would like to acknowledge the help in testing and proofreading I received in relation to this command and help file from (in alphabetic order):{p_end}
{pmore}Guadalupe Bedoya{break}Luiza Cardoso de Andrade{break}Mrijan Rimal{break}Benjamin Daniels{break}

{title:Author}

{phang}All commands in ietoolkit is developed by DIME Analytics at DECIE, The World Bank's unit for Development Impact Evaluations.

{phang}Main author: Kristoffer Bjarkefur, DIME Analytics, The World Bank Group

{phang}Please send bug-reports, suggestions and requests for clarifications
		 writing "ietoolkit iegitaddmd" in the subject line to:{break}
		 dimeanalytics@worldbank.org

{phang}You can also see the code, make comments to the code, see the version
		 history of the code, and submit additions or edits to the code through {browse "https://github.com/worldbank/ietoolkit":the GitHub repository of ietoolkit}.{p_end}

{smcl}
{* 5 Nov 2019}{...}
{hline}
help for {hi:iegitaddmd}
{hline}

{title:Title}

{cmd:iegitaddmd} {hline 2} Creates a placeholder file in subfolders of a GitHub repository folder, which allows committing folder structures with empty folders.

{phang2}For a more descriptive discussion on the intended usage and work flow of this
command please see the {browse "https://dimewiki.worldbank.org/wiki/Iegitaddmd":DIME Wiki}.

{title:Syntax}

{phang} {cmdab:iegitaddmd} , {opt folder(full_file_path)} [{opt comparefolder(full_file_path)} {cmd:customfile(}{it:{help filename}}{cmd:)} {opt all} {opt skip} {opt replace} {opt skipfolders(folder_name)}  {opt auto:matic} {opt dry:run}]

{marker opts}{...}
{synoptset 28}{...}
{synopthdr:options}
{synoptline}
{synopt :{opt folder(full_file_path)}}Specifies the folder path to the project folder clone where placeholder files will be created{p_end}
{synopt :{opt comparefolder(full_file_path)}}Specifies the top folder of a folder tree to be compared with {opt folder()}{p_end}
{synopt :{cmd:customfile(}{it:{help filename}}{cmd:)}}Specifies a file saved on disk that is used instead of the default file as placeholder{p_end}
{synopt :{cmd:all}}Creates the placeholder file in every subfolder of {opt folder()}, whether empty or not{p_end}
{synopt :{opt skip}}If option {opt all} is used and a folder has a file with same name as placeholder file, then nothing is done{p_end}
{synopt :{opt replace}}If option {opt all} is used and a folder has a file with same name as placeholder file, then the file is overwritten{p_end}
{synopt :{opt skipfolders(folder_name)}}List of folders to be skipped. The folder {it:.git} is always added to this list, even when option is not used{p_end}
{synopt :{opt auto:matic}}Makes the command create placeholder files without prompting the user for each file{p_end}
{synopt :{opt dry:run}}Makes the command list all the files that would have been created without this option.{p_end}
{synoptline}

{marker desc}
{title:Description}

{pstd}GitHub does not sync empty folders, or folder that only contain ignored files. However, it is common in research projects that these types of folders needs to be shared anyways. {cmd:iegitaddmd} provides a solution to this by creating placeholder files in in these folders. This placeholder file can then be shared to GitHub and that way the folder is also shared. {cmd:iegitaddmd} have two ways to identify which those folders are that are developed with the two use cases described below in mind.{p_end}

{pstd}{bf:Use case 1.} It is common in the beginning of a research project that a standardized folder structure is set up (for example with {help iefolder}) where some folders are created that are not yet needed. Examples of folders like that could be the output folders. If these folders are not shared at the time they are created, there is a risk that different team members name them differently which will be confusing and could lead to errors. {cmd:iegitaddmd} can solve this by scanning a folder and all its sub-folders for completely empty folders, and create a placeholder file in that folder.{p_end}

{pstd}The solution to use case 1 in {cmd:iegitaddmd} is a Stata adaptation of {it:Solution B} in {browse "http://bytefreaks.net/gnulinux/bash/how-to-add-automatically-all-empty-folders-in-git-repository" :this post}.{p_end}

{pstd}{bf:Use case 2.} A common method when transferring an already ongoing research project to GitHub is to copy all of the data work folder from wherever the files and folders where shared before (for example DropBox) to a clone of the repository. Then only the files that are meant to be shared on GitHub are uploaded to the cloud using a {it:.gitignore file}. Empty folders or folders where all files are ignored is then not shared through GitHub. {cmd:iegitaddmd} can solve this by scanning the DropBox folder (or wherever the project folder was shared before) for sub-folders that exist in that folder but not in the clone and crate that folder in the clone and put a placeholder in it.{p_end}

{pstd}The way you differentiate between these two use cases when using {cmd:iegitaddmd} is whether the option {opt comparefolder()} is used or not. If that option is used, then {cmd:iegitaddmd} will identify all folders that are in {opt comparefolder()} that is not in {opt folder()}, and create a placeholder file there. If {opt comparefolder()} is not used, then {cmd:iegitaddmd} will identify all completely empty folders (no files or subfolders) and create a placeholder file there.{p_end}

{pstd}The default placeholder file used if no file is specified using option {opt customfile()} is called README.md, which is a name and format recognized by GitHub.com so that the content of the file is displayed on GitHub.com when navigating to that folder using a browser. The placeholder file may be removed as	soon as files that are not ignored have been added to the folder, or even better is to keep the file but edit the content of it so it documents the content of that folder for other and future team members.{p_end}

{marker optslong}
{title:Options}

{phang}{opt folder(file_path)} is the folder path to the project folder clone where the placeholder files will be created. Unless option {opt comparefolder()} or {opt all} is used, then placeholder files will be created in all empty folders. See option {opt manual} for a way to manually confirm each folder before a placeholder file is created in it.{p_end}

{phang}{opt comparefolder(file_path)} is used to compare a file tree in a different location and create the folders with placeholder files in all folders that exist in {opt comparefolder()} but not in {opt folder()}. If a folder that does no exist in {opt folder()} has a subfolder, then both the folder and the subfolder will be created and the placeholder file is only crated in the sub-folder. This is enough to share both folder and the subfolder in GitHub. If the {opt comparefolder()} and {opt folder()} ends with folders with different names, for example {opt folder(${project}/datawork/baseline)} and {opt comparefolder(${project}/datawork)}, then the command will throw an error as it is likely that the paths do not point to corresponding folders in the two folder trees. See option {opt manual} for a way to manually confirm each folder before a placeholder file is created in it.{p_end}

{phang}{cmd:file(}{it:{help filename}}{cmd:)} allows the user to specify a file saved on
	the computer to be used as the placeholder file	instead of the default README.md file. This
	allows people and organizations to write their own template placeholder files according to their own
	preferences. We recommend that a file of type .md (markdown) and name README.md is used as
	GitHub.com recognizes this name and format and will display the content of the file when someone
	browse to that folder on GitHub.com. But this is not a technical requirement, any file type
	and name can be used as placeholder file.{p_end}

{phang}{opt all} creates the placeholder file in {opt folder()} and in every subfolder,
	regardless if they are empty or not. This allows the user to create a placeholder file in every
	folder that can be edited with documentation and instructions to the purpose and usage of
	each subfolder. This option can also be important when .{it:gitignore} is used, as {cmd:iegitaddmd} will not
	create files to subfolders that only has ignored file -- in which case the folder will not
	be synced by GitHub.{p_end}

{phang}{opt skip} and {opt replace} tells {cmd:iegitaddmd} what to do if the option {opt all} is used
	and if any of the folders contain a file with the same name as the placeholder file the command
	is trying to create. If a file with the same name as the placeholder file exist and neither of these options
	are used, then the command will throw an error. If {opt skip} is used, then nothing is done in the
	folder where the file with the same name exists and the command proceeds to the next folder.
	If {opt replace} is used then the file with the same name is overwritten with the new placeholder
	file before the command proceeds to the next folder.{p_end}

{phang}{opt skipfolders(folder_name)} can be used to tell {cmd:iegitaddmd} which folders
in which a placeholder file should never be created. The best example of this is the {it:.git}
folder in which placeholders never should be created. That name is always skipped regardless if
this option is used or not. In this option additional folders to be skipped can be listed. You
should not list the full folder path in this option, just the folder name. All folders with
that name will be skipped regardless of location in the project folder. The folder names may
be listed with or without quotation marks as long as there are no spaces in the names. If any of
the folder names has spaces, then quotation marks must be used for all folder names.

{phang}{opt auto:matic} can be used to speed up the creation of placeholders by telling the
command to not prompt the users for confirmation for each file before it is created. The
default is that the command is asking the user before creating each place holder file.
This option should only be used when you are confident you have specified the correct folder
paths. We recommend that you use the {opt dryrun} with this option to make sure that the
folder paths are correct.{p_end}

{phang}{opt dry:run} can be used to safely test that the folder paths are specified correctly
before any placeholder files are created. When this option it used the command simply lists
the file that would have been created if this option were not used. Once you are confident
that list is correct, you can remove this option and re-run the command and the files will
be created.{p_end}

{title:Example}

{pstd}{inp:global github_folder "C:\Users\JohnSmith\Documents\GitHub\ProjectA"}{break}{inp:iegitaddmd , folder({it:"$github_folder"})}{p_end}

{pstd}In the example above, there is a GitHub repository in the folder ProjectA. This
	repository has a folder structure where some folders are still empty but will later
	be populated with files. In order to have all folders, even the empty ones, synced on all
	collaborators' cloned local copies of the repository, the folders need to contain at least
	one file, which is being created by the command.{p_end}

{title:Acknowledgements}

{pstd}I would like to acknowledge the help in testing and proofreading I received in relation to this command and help file from (in alphabetic order):{p_end}
{pmore}Guadalupe Bedoya{break}Luiza Cardoso de Andrade{break}Mrijan Rimal{break}Benjamin Daniels{break}

{title:Author}

{phang}All commands in ietoolkit is developed by DIME Analytics at DECIE, The World Bank's unit for Development Impact Evaluations.

{phang}Main author: Kristoffer Bjarkefur, DIME Analytics, The World Bank Group

{phang}Please send bug-reports, suggestions and requests for clarifications
		 writing "ietoolkit iegitaddmd" in the subject line to:{break}
		 dimeanalytics@worldbank.org

{phang}You can also see the code, make comments to the code, see the version
		 history of the code, and submit additions or edits to the code through {browse "https://github.com/worldbank/ietoolkit":the GitHub repository of ietoolkit}.{p_end}


iedup - comments and sec numbering - #146

*! version 5.5 26APR2018 DIME Analytics lcardosodeandrad@worldbank.org

	capture program drop ieduplicates
	program ieduplicates , rclass

	qui {

		syntax varname ,  FOLder(string) UNIQUEvars(varlist) [KEEPvars(varlist) tostringok droprest nodaily SUFfix(string)]

		version 11.0

		*Add version of Stata fix
		//Make sure that keepvars are still saved if saved if the duplicates file
		*	is generated on a subset of the data. For example, duplicates from
		*	version 1, 2 . If the command is run on only version 1. Then values
		*	for keeepvars in version 2 and 3 are dropped and not reloaded as those
		*	obs are not in current memory

		**Test that observations have not been deleted from the report before readind
		* it. Deleted in a way that the report does not make sense. Provide an error
		* message to this that is more informative.

		preserve

			/***********************************************************************
			************************************************************************

				Section 1 - Set up locals needed in data

				Saving a version of the data to be used before merging corrections
				back to the original data set and before correcting duplicates

			************************************************************************
			***********************************************************************/

			*Tempfiles to be uses
			tempfile originalData preppedReport datawithreportmerged dataToReturn

			** Save a version of original data that can be brought
			*  back withing the preserve/restore section
			save 	`originalData'

			* Create a local with todays date to date the reports
			local  date = subinstr(c(current_date)," ","",.)

			*Put idvar in a local with a more descriptive name
			local idvar `varlist'

			** Making one macro with all variables that will be
			*  imported and exported from the Excel file
			local argumentVars `idvar' `uniquevars' `keepvars'

			* Create a list of the varaibles created by this command to put in the report
			local excelvars dupListID dateListed dateFixed correct drop newID initials notes

			/***********************************************************************
			************************************************************************

				Section 2 - Test unique vars

				Test the unique vars so that they are identifying the data set and
				are not in a time format that might get corrupted by exporting to
				and importing from Excel. This is needed as the uniquevars are needed
				to merge the correct correction to the correct duplicat

			************************************************************************
			***********************************************************************/

			*Test that the unique vars fully and uniquely identifies the data set
			cap isid `uniquevars'
			if _rc {

				noi display as error "{phang}The variable(s) listed in uniquevars() does not uniquely and fully identifies all observations in the data set.{p_end}"
				isid `uniquevars'
				error 198
				exit
			}

			** Test that no unique vars are time format. Time values might be corrupted
			*  and changed a tiny bit when importing and exporting to Excel, which make merge not possible
			foreach uniquevar of local uniquevars {

				*Test if format is time format
				local format : format `uniquevar'
				if substr("`format'",1,2) == "%t" {

					noi display as error `"{phang}The variable {inp:`uniquevar'} listed in {inp:uniquevars()} is using time format which is not allowed. Stata and Excel stores and displays time slightly differently which can lead to small changes in the value when the value is imported and exported between Stata and Excel, and then the variable can no longer be used to merge the report back to the original data. Use another variable or create a string variable out of your time variable using this code: {inp: generate `uniquevar'_str = string(`uniquevar',"%tc")}.{p_end}"'
					noi di ""
					error 198
					exit
				}
			}

			/***********************************************************************
			************************************************************************

				Section 3 - Test input from Excel file

				If Excel repirt exist, import it and test for invalid corrections
				made in the excel report.

			************************************************************************
			***********************************************************************/

			/******************
				Section 3.1
				Check if report exists from before.
			******************/

			*Check if file exist. Suffix option can be used to create multiple reports in the same folder
			cap confirm file "`folder'/iedupreport`suffix'.xlsx"

			if !_rc {
				local fileExists 1
			}
			else {
				local fileExists 0
			}

			/******************
				Section 3.2
				If report exist, load file and check input, otherwise skip to section 4
			******************/

			if `fileExists' {

				*Load excel file. Load all vars as string and use meta data from Section 1
				import excel "`folder'/iedupreport`suffix'.xlsx"	, clear firstrow

				** All excelvars but dupListID and newID should be string. dupListID
				*  should be numeric and the type of newID should be based on the user input
				foreach excelvar of local excelvars {
					if !inlist("`excelvar'", "dupListID", "newID") {

						* Make original ID var string
						tostring `excelvar' , replace
						replace  `excelvar' = "" if `excelvar' == "."
					}
				}

				/******************
					Section 3.2.1
					Make sure that the ID variable and the uniquevars are
					not changed since last report.
				******************/

				*Copy a list of all variables in the imported report to the local existingexcelvars
				ds
				local existingexcelvars  `r(varlist)'

				*Test that the ID variable is in the imported report
				if `:list idvar in existingexcelvars' == 0 {

					noi display as error "{phang}The ID variable `idvar' does not exist in the previously exported Excle file. If you renamed the ID variable you need to rename it manually in the Excel report or start a new Excel report by renaming or moving the original report, then run the command again and create a new file and manually copy any corrections from the old file to the new. If you changed the ID varaible you need to start with a new report.{p_end}"
					noi di ""
					error 111
					exit

				}

				*Test that the unique variables are in the imported report
				if `:list uniquevars in existingexcelvars' == 0 {

					noi display as error "{phang}One or more unique variables in [`uniquevars'] do not exist in the previously exported Excel file. If you renamed or changed any variable used in uniquevars(), you need to start over with a new file. Rename or move the already existing file. Create a new file and carefully copy any corrections from the old file to the new.{p_end}"
					noi di ""
					error 111
					exit
				}

				/******************
					Section 3.3
					Make sure input is correct
				******************/

				*Temporary variables needed for checking input
				tempvar multiInp inputNotYes maxMultiInp notDrop yesCorrect groupNumCorrect toDrop anyCorrection groupAnyCorrection

				*Locals indicating in which ways input is incorrect (if any)
				local local_multiInp 		0
				local local_multiCorr		0
				local local_inputNotYes		0
				local local_notDrop			0

				/******************
					Section 3.3.1
					Make sure input is yes or y for the correct and drop columns
				******************/

				* Make string input lower case and change "y" to "yes"
				replace correct = lower(correct)
				replace drop 	= lower(drop)
				replace correct = "yes" if correct 	== "y"
				replace drop 	= "yes" if drop 	== "y"

				*Check that varaibles are either empty or "yes"
				gen `inputNotYes' = !((correct  == "yes" | correct == "") & (drop  == "yes" | drop == ""))

				*Set local to 1 if error should be outputted
				cap assert `inputNotYes' == 0
				if _rc local local_inputNotYes 	1

				/******************
					Section 3.3.2
					Make sure there are not too many corrections for a single observation
				******************/

				* Count the number of corrections (correct drop newID) per
				* observation. Only one correction per observation is allowed.
				egen `multiInp' = rownonmiss(correct drop newID), strok

				*Check that all rows have utmost one correction
				cap assert `multiInp' == 0 | `multiInp' == 1

				*Error will be outputted below
				if _rc local local_multiInp 	1

				/******************
					Section 3.3.3
					Test that maximum one duplicate per duplicate group is indicated as correct

				******************/

				*Generate dummy if correct column is set to yes
				gen `yesCorrect' = (correct == "yes")

				*Count number of duplicates within duplicates where that dummy is 1
				bys `idvar' : egen `groupNumCorrect' =  total(`yesCorrect')

				*Test if more than 1 duplicate in that duplicate group is correct
				count if `groupNumCorrect' > 1

				*Output error is more than one duplicate in a duplicate group is yes
				if `r(N)' != 0 local local_multiCorr 1

				/******************
					Section 3.3.4
					Make sure that either option droprest is specified, or that
					drop was correctly indicated for all observations. i.e.; if
					correct or newID was indicated for at least one duplicate in
					a duplicate group, then all other observations should be
					indicated as drop (unless droprest is specified)

				******************/

				*Generate dummy if there is any correction for this observation
				gen `anyCorrection' = !missing(correct) | !missing(newID)

				*Count number of observations with any correction in suplicates group
				bys `idvar' : egen `groupAnyCorrection' =  total(`anyCorrection')

				*Create dummy that indicates each place this error happens
				gen `notDrop' = (missing(drop) & `groupAnyCorrection' > 0 & `anyCorrection' == 0)

				* Check if option droprest is specified
				if "`droprest'" == "" {

					** If option droprest is not used, then all observations in a duplicate
					*  group where at least one observation has a correction must have a
					*  correction or have drop set to yes.
					cap assert `notDrop' == 0

					*Error will be outputted below
					if _rc local local_notDrop 	1

				}
				else {

					** Option -droprest- specified. Drop will be changed to yes
					*  for any observations without drop or any other correction
					*  explicitly specified if the observation is in a duplicate
					*  group with at least one observation has a correction
					replace drop 	= "yes" if `notDrop' == 1

				}

				/******************
					Section 3.4
					Throw errors if any of the tests were not passed
				******************/

				*Output errors if errors in the report were detected
				if `local_multiInp' == 1 | `local_inputNotYes' == 1 | `local_notDrop' == 1 | `local_multiCorr' == 1  {
					noi {
						di ""
						di ""
						di as error "{phang}{ul:The corrections made in the Excel report has the following errors:}{p_end}"
						di ""

						*Error multiple input
						if `local_multiInp' == 1 {
							display as error "{phang}The following observations have more than one correction. Only one correction (correct, drop or newID) per row is allowed{p_end}"
							list `idvar' dupListID correct drop newID `uniquevars' if `multiInp' > 1
							di ""
						}

						*Error multiple correct
						if `local_multiCorr' == 1 {
							display as error "{phang}The following observations are in a duplicate group where more than one observation is listed as correct. Only one observation per duplicate group can be correct{p_end}"
							list `idvar' dupListID correct drop newID `uniquevars' if `groupNumCorrect' > 1
							di ""
						}

						*Error in incorrect string
						if `local_inputNotYes' == 1 {
							display as error "{phang}The following observations have an answer in either correct or drop that is neither yes nor y{p_end}"
							list `idvar' dupListID correct drop `uniquevars' if `inputNotYes' == 1
							di ""
						}

						*Error is not specfied as drop
						if `local_notDrop' == 1 {
							display as error "{phang}The following observations are not explicitly indicated as drop while other duplicates in the same duplicate group are corrected. Either manually indicate as drop or see option droprest{p_end}"
							list `idvar' dupListID correct drop newID `uniquevars' if `notDrop' == 1
							di ""
						}

						*Same error for any incorrect input
						error 198
						exit
					}
				}

				/******************
					Section 3.5
					Save the prepared report to be used later
				******************/

				*Keep only the variables needed for matching and variables used for input in the Excel file
				keep 	`idvar' `uniquevars' `excelvars' `groupAnyCorrection'

				*Save imported data set with all corrections
				save	`preppedReport'
			}

			/***********************************************************************
			************************************************************************

				Section 4 - Merge report to original data

				Merge corrections with original data, test that there are no
				obs in report that are not in main data, and save this data
				in temp file

			************************************************************************
			***********************************************************************/

			*Re-load original data
			use `originalData', clear

			* Merge original data with imported Excel file (if Excel file exists)
			if `fileExists'  {

				*Create a tempvar for merging results
				tempvar iedup_merge

				*Merge the corrections with the data set
				merge 1:1 `uniquevars' using `preppedReport', generate(`iedup_merge')

				*Make sure that obsrevations listed in the duplicate report is still in the data set
				cap assert `iedup_merge' != 2

				*Display error message if assertion is not true and some duplicates in the excle file are no longer in the data set
				if _rc {

					display as error "{phang}One or several observations in the Excel report are no longer found in the data set. Always run ieduplicates on the raw data set that include all the duplicates, both new duplicates and those you have already identified. After removing duplicates, save the data set using a different name. You might also recieve this error if you are using an old ieduplicates Excel report on a new data set.{p_end}"
					error 9
					exit
				}

				*Explicitly drop temporary variable. Temporary variables might
				*be exported to excel so delete explicitly before that. Only
				*using tempvar here to create a name with no conflicts
				drop `iedup_merge'

			}

			*save the data set to be used when correcting the data
			save `datawithreportmerged'

			/***********************************************************************
			************************************************************************

				Section 5 - Generate the Excel report

				Test if there are duplicates in ID var. If any duplicates exist,
				then update the Excel file with new and unaddressed cases

			************************************************************************
			***********************************************************************/

			/******************
				Section 5.1
				Test if there are any duplicates in ID var
			******************/

			* Generate variable that is not 0
			* if observation is a duplicate
			tempvar dup
			duplicates tag `idvar', gen(`dup')

			*Test if there are any duplicates
			cap assert `dup'==0
			if _rc {

				/******************
					Section 5.2
					Keep only duplicates for the report
				******************/

				*Keep if observation is part of duplicate group
				keep if `dup' != 0

				if `fileExists'  {
					* If Excel file exists keep excel vars and
					* variables passed as arguments in the
					* command
					keep 	`argumentVars' `excelvars'
				}
				else {
					* Keep only variables passed as arguments in
					* the command and the string ID var as no Excel file exists
					keep 	`argumentVars'

					*Generate the excel variables used for indicating correction
					foreach excelvar of local excelvars {

						*Create all variables apart from dupListID as string vars
						if inlist("`excelvar'", "dupListID", "newID") {
							gen `excelvar' = .
						}
						else {
							gen `excelvar' = ""
						}
					}
				}

				/******************
					Section 5.3
					Update the excel vars that are not updated manually
				******************/

				* Generate a local that is 1 if there are new duplicates
				local unaddressedNewExcel 0
				count if missing(dateFixed)
				if `r(N)' > 0 local unaddressedNewExcel 1

				/******************
					Section 5.3.1 Date variables
				******************/

				* Add date first time duplicvate was identified
				replace dateListed 	= "`date'" if missing(dateListed)

				** Add today's date to variable dateFixed if dateFixed
				*  is empty and at least one correction is added
				replace dateFixed 	= "`date'" if missing(dateFixed) & (!missing(correct) | !missing(drop) | !missing(newID))

				/******************
					Section 5.3.2 Duplicate report list ID
				******************/

				** Sort after dupListID and after ID var for
				*  duplicates currently without dupListID
				sort dupListID `idvar'

				** Assign dupListID 1 to the top row if no duplicate
				*  list IDs have been generated so far.
				replace dupListID = 1 if _n == 1 & missing(dupListID)

				** Generate new IDs based on the row above instead of directly
				*  from the row number. That prevents duplicates in the list in
				*  case an observation is deleted. The first observation with
				*  missing value will have an ID that is one digit higher than
				*  the highest ID already in the list
				replace dupListID = dupListID[_n - 1] + 1 if missing(dupListID)

				/******************
					Section 5.4
					Keep and order the variables and output the Excel files
				******************/

				* If cases unaddressed then update the Excel file
				if `unaddressedNewExcel'  {

					keep 	`argumentVars' `excelvars'
					order	`idvar' `excelvars' `uniquevars' `keepvars'

					if "`daily'" == "" {

						*Returns 0 if folder does not exist, 1 if it does
						mata : st_numscalar("r(dirExist)", direxists("`folder'/Daily"))

						** If the daily folder is not created, just create it
						if `r(dirExist)' == 0  {

							*Create the folder since it does not exist
							mkdir "`folder'/Daily"
						}

						*Export the daily file
						cap export excel using "`folder'/Daily/iedupreport`suffix'_`date'.xlsx"	, firstrow(variables) replace nolabel

						*Print error if daily report cannot be saved
						if _rc {

							display as error "{phang}The Daily copy could not be saved to the `folder'/Daily folder. Make sure to close any old daily copy or see the option nodaily.{p_end}"
							error 603
							exit
						}

						*Prepare local for output
						local daily_output " and a daily copy have been saved to the Daily folder"
					}

					*Export main report
					export excel using "`folder'/iedupreport`suffix'.xlsx"	, firstrow(variables) replace  nolabel

					*Produce output
					noi di `"{phang}Excel file created at: {browse "`folder'/iedupreport`suffix'.xlsx":`folder'/iedupreport`suffix'.xlsx}`daily_output'.{p_end}"'
					noi di ""
				}
			}

		/***********************************************************************
		************************************************************************

			Section 6

			Update the data set and with the new corrections.

		************************************************************************
		***********************************************************************/

		* Load the original data set merged with correction. Duplicates
		* in all variables are already dropped in this data set
		use 	`datawithreportmerged', clear

		* If excel file exists, apply any corrections indicated (if any)
		if `fileExists' {

			/******************
				Section 6.1
				Drop duplicates listed for drop
			******************/

			drop if drop == "yes"

			/******************
				Section 6.2
				Update new ID. ID var can be either numeric or
				string. All numbers can be made strings but not
				all strings can be numeric. Therefore this
				section is complicated.
			******************/

			/******************
				Section 6.2.1
				ID var in original file is string. Either
				newID was imported as string or the variable
				is made string. Easy.
			******************/

			*Test if there are any corrections by new ID
			cap assert missing(newID)
			if _rc {

				local idtype 	: type `idvar'
				local idtypeNew : type newID

				*If ID var is string but newID is not, then just make it string
				if substr("`idtype'",1,3) == "str" & substr("`idtypeNew'",1,3) != "str" {

					tostring newID , replace
					replace  newID = "" if newID == "."
				}

				*If ID var is numeric but the newID is loaded as string
				else if substr("`idtype'",1,3) != "str" & substr("`idtypeNew'",1,3) == "str" {

					* Check if -tostringok- is specificed:
					if "`tostringok'" != "" {

						* Make original ID var string
						tostring `idvar' , replace
						replace  `idvar' = "" if `idvar' == "."

					}

					* Error, IDvar can not be updated
					else {

						* Create a local with all non-numeric values
						levelsof newID if missing(real(newID)), local(NaN_values) clean

						* Output error message
						di as error "{phang}The ID varaible `idvar' is numeric but newID has thes non-numeric values: `NaN_values'. Update newID to only contain numeric values or see option tostringok.{p_end}"
						error 109
						exit
					}
				}

				*After making sure that type is ok, update the IDs
				replace `idvar' = newID if !missing(newID)

				/******************
					Section 6.3
					Test that values in newID
					were neither used twice
					nor already existed
				******************/

				*Test if there are any duplicates after corrections (excluding observations in duplicate groups not yet corrected)
				tempvar newDup
				duplicates tag `idvar' if `groupAnyCorrection' != 0, generate(`newDup')

				*Consider missing values as no new duplicates, make conditions below easier
				replace `newDup' = 0 if missing(`newDup')

				cap assert `newDup' == 0
				if _rc {

					levelsof `idvar' if `newDup' != 0  , local(newDuplist)
					di as error "{phang}No corrections from the report are applied as it would lead to new duplicates in the following value(s): `newDuplist'.{p_end}"
					error 198
					exit
				}
			}

			/******************
				Section 6.4
				Drop Excel vars
			******************/

			drop `excelvars'
		}

		/***********************************************************************
		************************************************************************

			Section 7

			Return the data set without duplicates and
			output information regarding unresolved duplicates.

		************************************************************************
		***********************************************************************/

		* Generate a variable that is 1 if the observation is a duplicate in varlist
		tempvar dropDup
		duplicates tag `idvar',  gen(`dropDup')
		* Generate a list of the IDs that are still duplicates
		levelsof `idvar' 			if `dropDup' != 0 , local(dup_ids) clean
		* Drop the duplicates (they are exported in Excel)
		drop 						if `dropDup' != 0

		* Test if varlist is now uniquely and fully identifying the data set
		cap isid `idvar'
		if _rc {

			di as error "{phang}The data set is not returned with `idvar' uniquely and fully identifying the data set. Please report this bug to kbjarkefur@worldbank.org{p_end}"
			error 119
			exit

		}

		*Count number of duplicate groups still in the data
		local numDup	= `:list sizeof dup_ids'

		if `numDup' == 0 {
			noi di	"{phang}There are no unresolved duplicates in this data set. The data set is returned with `idvar' uniquely and fully identifying the data set.{p_end}"
		}
		else {
			noi di	"{phang}There are `numDup' duplicated IDs still unresolved. IDs still containing duplicates: `dup_ids'. The unresolved duplicate observations were exported in the Excel file. The data set is returned without those duplicates and with `idvar' uniquely and fully identifying the data set.{p_end}"
		}

		return scalar numDup	= `numDup'

		/***********************************************************************
		************************************************************************

			Section 8

			Save data set to be returned outside preserve/restore.
			Preserve/restore is used so that original data is returned
			in case an error is thrown.

		************************************************************************
		***********************************************************************/

		save 	`dataToReturn'

		restore

		** Using restore above to return the data to
		*  the orignal data set in case of error.
		use `dataToReturn', clear

	}
end

*! version 5.5 26APR2018 DIME Analytics lcardosodeandrad@worldbank.org

	capture program drop ieduplicates
	program ieduplicates , rclass

	qui {

		syntax varname ,  FOLder(string) UNIQUEvars(varlist) [KEEPvars(varlist) tostringok droprest nodaily SUFfix(string)]

		version 11.0

		*Add version of Stata fix
		//Make sure that keepvars are still saved if saved if the duplicates file
		*	is generated on a subset of the data. For example, duplicates from
		*	version 1, 2 . If the command is run on only version 1. Then values
		*	for keeepvars in version 2 and 3 are dropped and not reloaded as those
		*	obs are not in current memory

		**Test that observations have not been deleted from the report before readind
		* it. Deleted in a way that the report does not make sense. Provide an error
		* message to this that is more informative.

		preserve

			/***********************************************************************
			************************************************************************

				Section 1 - Set up locals needed in data

				Saving a version of the data to be used before merging corrections
				back to the original data set and before correcting duplicates

			************************************************************************
			***********************************************************************/

			*Tempfiles to be uses
			tempfile originalData preppedReport datawithreportmerged dataToReturn

			** Save a version of original data that can be brought
			*  back withing the preserve/restore section
			save 	`originalData'

			* Create a local with todays date to date the reports
			local  date = subinstr(c(current_date)," ","",.)

			*Put idvar in a local with a more descriptive name
			local idvar `varlist'

			** Making one macro with all variables that will be
			*  imported and exported from the Excel file
			local argumentVars `idvar' `uniquevars' `keepvars'

			* Create a list of the varaibles created by this command to put in the report
			local excelvars dupListID dateListed dateFixed correct drop newID initials notes

			/***********************************************************************
			************************************************************************

				Section 2 - Test unique vars

				Test the unique vars so that they are identifying the data set and
				are not in a time format that might get corrupted by exporting to
				and importing from Excel. This is needed as the uniquevars are needed
				to merge the correct correction to the correct duplicat

			************************************************************************
			***********************************************************************/

			*Test that the unique vars fully and uniquely identifies the data set
			cap isid `uniquevars'
			if _rc {

				noi display as error "{phang}The variable(s) listed in uniquevars() does not uniquely and fully identifies all observations in the data set.{p_end}"
				isid `uniquevars'
				error 198
				exit
			}

			** Test that no unique vars are time format. Time values might be corrupted
			*  and changed a tiny bit when importing and exporting to Excel, which make merge not possible
			foreach uniquevar of local uniquevars {

				*Test if format is time format
				local format : format `uniquevar'
				if substr("`format'",1,2) == "%t" {

					noi display as error `"{phang}The variable {inp:`uniquevar'} listed in {inp:uniquevars()} is using time format which is not allowed. Stata and Excel stores and displays time slightly differently which can lead to small changes in the value when the value is imported and exported between Stata and Excel, and then the variable can no longer be used to merge the report back to the original data. Use another variable or create a string variable out of your time variable using this code: {inp: generate `uniquevar'_str = string(`uniquevar',"%tc")}.{p_end}"'
					noi di ""
					error 198
					exit
				}
			}

			/***********************************************************************
			************************************************************************

				Section 3

				Import and prepare Corrections (if file exists)

			************************************************************************
			***********************************************************************/

			/******************
				Section 3.1
				Check if earlier report exists.
			******************/

			cap confirm file "`folder'/iedupreport`suffix'.xlsx"

			if !_rc {
				local fileExists 1
			}
			else {
				local fileExists 0
			}

			/******************
				Section 3.2
				If report exist, load file and check input
			******************/

			if `fileExists' {

				*Load excel file. Load all vars as string and use meta data from Section 1
				import excel "`folder'/iedupreport`suffix'.xlsx"	, clear firstrow

				** All excelvars but dupListID and newID should be string. dupListID
				*  should be numeric and the type of newID should be based on the user input
				foreach excelvar of local excelvars {
					if !inlist("`excelvar'", "dupListID", "newID") {

						* Make original ID var string
						tostring `excelvar' , replace
						replace  `excelvar' = "" if `excelvar' == "."
					}
				}

				/******************
					Section 3.2.1
					Make sure that the
					ID variable and
					the uniquevars are
					not changed since
					last report.
				******************/

				ds
				local existingexcelvars  `r(varlist)'

				if `:list varlist in existingexcelvars' == 0 {

					noi display as error "{phang}ID variable [`varlist'] does not exist in the previously exported Excle file. If you renamed or changed the ID variable, you need to start over with a new file. Rename or move the already existing file. Create a new file and carefully copy any corrections from the old file to the new.{p_end}"
					noi di ""
					error 111
					exit

				}
				if `:list uniquevars in existingexcelvars' == 0 {

					noi display as error "{phang}One or more unique variables in [`uniquevars'] do not exist in the previously exported Excel file. If you renamed or changed any variable used in uniquevars(), you need to start over with a new file. Rename or move the already existing file. Create a new file and carefully copy any corrections from the old file to the new.{p_end}"
					noi di ""
					error 111
					exit
				}

				/******************
					Section 3.3
					Make sure input is correct
				******************/

				*Temporary variables needed for checking input
				tempvar multiInp inputNotYes maxMultiInp notDrop yesCorrect groupNumCorrect toDrop anyCorrection groupAnyCorrection

				*Locals indicating in which ways input is incorrect (if any)
				local local_multiInp 		0
				local local_multiCorr		0
				local local_inputNotYes		0
				local local_notDrop			0

				/******************
					Section 3.3.1
					Make sure input is yes or y for the correct and drop columns
				******************/

				* Make string input lower case and change "y" to "yes"
				replace correct = lower(correct)
				replace drop 	= lower(drop)
				replace correct = "yes" if correct 	== "y"
				replace drop 	= "yes" if drop 	== "y"

				*Check that varaibles are either empty or "yes"
				gen `inputNotYes' = !((correct  == "yes" | correct == "") & (drop  == "yes" | drop == ""))

				*Set local to 1 if error should be outputted
				cap assert `inputNotYes' == 0
				if _rc local local_inputNotYes 	1

				/******************
					Section 3.3.2
					Make sure there are not too many corrections for a single observation
				******************/

				* Count the number of corrections (correct drop newID) per
				* observation. Only one correction per observation is allowed.
				egen `multiInp' = rownonmiss(correct drop newID), strok

				*Check that all rows have utmost one correction
				cap assert `multiInp' == 0 | `multiInp' == 1

				*Error will be outputted below
				if _rc local local_multiInp 	1

				/******************
					Section 3.3.3
					Test that maximum one duplicate per duplicate group is indicated as correct

				******************/

				*Generate dummy if correct column is set to yes
				gen `yesCorrect' = (correct == "yes")

				*Count number of duplicates within duplicates where that dummy is 1
				bys `idvar' : egen `groupNumCorrect' =  total(`yesCorrect')

				*Test if more than 1 duplicate in that duplicate group is correct
				count if `groupNumCorrect' > 1

				*Output error is more than one duplicate in a duplicate group is yes
				if `r(N)' != 0 local local_multiCorr 1

				/******************
					Section 3.3.4
					Make sure that either option droprest is specified, or that
					drop was correctly indicated for all observations. i.e.; if
					correct or newID was indicated for at least one duplicate in
					a duplicate group, then all other observations should be
					indicated as drop (unless droprest is specified)

				******************/

				*Generate dummy if there is any correction for this observation
				gen `anyCorrection' = !missing(correct) | !missing(newID)

				*Count number of observations with any correction in suplicates group
				bys `idvar' : egen `groupAnyCorrection' =  total(`anyCorrection')

				*Create dummy that indicates each place this error happens
				gen `notDrop' = (missing(drop) & `groupAnyCorrection' > 0 & `anyCorrection' == 0)

				* Check if option droprest is specified
				if "`droprest'" == "" {

					** If option droprest is not used, then all observations in a duplicate
					*  group where at least one observation has a correction must have a
					*  correction or have drop set to yes.
					cap assert `notDrop' == 0

					*Error will be outputted below
					if _rc local local_notDrop 	1

				}
				else {

					** Option -droprest- specified. Drop will be changed to yes
					*  for any observations without drop or any other correction
					*  explicitly specified if the observation is in a duplicate
					*  group with at least one observation has a correction
					replace drop 	= "yes" if `notDrop' == 1

				}

				/******************
					Section 3.4
					Throw errors if any of the tests were not passed
				******************/

				*Output errors if errors in the report were detected
				if `local_multiInp' == 1 | `local_inputNotYes' == 1 | `local_notDrop' == 1 | `local_multiCorr' == 1  {
					noi {
						di ""
						di ""
						di as error "{phang}{ul:The corrections made in the Excel report has the following errors:}{p_end}"
						di ""

						*Error multiple input
						if `local_multiInp' == 1 {
							display as error "{phang}The following observations have more than one correction. Only one correction (correct, drop or newID) per row is allowed{p_end}"
							list `idvar' dupListID correct drop newID `uniquevars' if `multiInp' > 1
							di ""
						}

						*Error multiple correct
						if `local_multiCorr' == 1 {
							display as error "{phang}The following observations are in a duplicate group where more than one observation is listed as correct. Only one observation per duplicate group can be correct{p_end}"
							list `idvar' dupListID correct drop newID `uniquevars' if `groupNumCorrect' > 1
							di ""
						}

						*Error in incorrect string
						if `local_inputNotYes' == 1 {
							display as error "{phang}The following observations have an answer in either correct or drop that is neither yes nor y{p_end}"
							list `idvar' dupListID correct drop `uniquevars' if `inputNotYes' == 1
							di ""
						}

						*Error is not specfied as drop
						if `local_notDrop' == 1 {
							display as error "{phang}The following observations are not explicitly indicated as drop while other duplicates in the same duplicate group are corrected. Either manually indicate as drop or see option droprest{p_end}"
							list `idvar' dupListID correct drop newID `uniquevars' if `notDrop' == 1
							di ""
						}

						*Same error for any incorrect input
						error 198
						exit
					}
				}

				/******************
					Section 3.5
					Save the prepared report to be used later
				******************/

				*Keep only the variables needed for matching and variables used for input in the Excel file
				keep 	`idvar' `uniquevars' `excelvars' `groupAnyCorrection'

				*Save imported data set with all corrections
				save	`preppedReport'
			}

			/***********************************************************************
			************************************************************************

				Section 4

				Merge corrections with original data

			************************************************************************
			***********************************************************************/

			*Re-load original
			use `restart', clear

			* Merge original data with imported Excel file (if Excel file exists)
			if `fileExists'  {

				*Create a tempvar for merging results
				tempvar iedup_merge

				*Merge the corrections with the data set
				merge 1:1 `uniquevars' using `preppedReport', generate(`iedup_merge')

				*Make sure that obsrevations listed in the duplicate report is still in the data set
				cap assert `iedup_merge' != 2

				*Display error message if assertion is not true and some duplicates in the excle file are no longer in the data set
				if _rc {

					display as error "{phang}One or several observations in the Excel report are no longer found in the data set. Always run ieduplicates on the raw data set that include all the duplicates, both new duplicates and those you have already identified. After removing duplicates, save the data set using a different name. You might also recieve this error if you are using an old ieduplicates Excel report on a new data set.{p_end}"
					error 9
					exit
				}

				*Explicitly drop temporary variable. Temporary variables might
				*be exported to excel so delete explicitly before that. Only
				*using tempvar here to create a name with no conflicts
				drop `iedup_merge'

			}

			*save the data set to be used when correcting the data
			save `datawithreportmerged'

			/***********************************************************************
			************************************************************************

			/***********************************************************************
			************************************************************************

				Section 6

				Test if there are duplicates in ID var. If any duplicates exist,
				tehn update the Excel file with new and unaddressed cases

			************************************************************************
			***********************************************************************/

			/******************
				Section 6.1
				Test if there are any duplicates in ID var
			******************/

			* Generate variable that is not 0
			* if observation is a duplicate
			tempvar dup
			duplicates tag `idvar', gen(`dup')

			*Test if there are any duplicates
			cap assert `dup'==0
			if _rc {

				/******************
				/******************
					Section 6.3
					Keep only duplicates for the report
				******************/

				*Keep if observation is part of duplicate group
				keep if `dup' != 0

				if `fileExists'  {
					* If Excel file exists keep excel vars and
					* variables passed as arguments in the
					* command
					keep 	`argumentVars' `excelvars'
				}
				else {
					* Keep only variables passed as arguments in
					* the command and the string ID var as no Excel file exists
					keep 	`argumentVars'

					*Generate the excel variables used for indicating correction
					foreach excelvar of local excelvars {

						*Create all variables apart from dupListID as string vars
						if inlist("`excelvar'", "dupListID", "newID") {
							gen `excelvar' = .
						}
						else {
							gen `excelvar' = ""
						}
					}

				}

				/******************
					Section 6.4
					Update the excel vars that are not updated manually
				******************/

				* Generate a local that is 1 if there are new duplicates
				local unaddressedNewExcel 0
				count if missing(dateFixed)
				if `r(N)' > 0 local unaddressedNewExcel 1

				/******************
					Section 6.4.1 Date variables
				******************/

				* Add date first time duplicvate was identified
				replace dateListed 	= "`date'" if missing(dateListed)

				** Add today's date to variable dateFixed if dateFixed
				*  is empty and at least one correction is added
				replace dateFixed 	= "`date'" if missing(dateFixed) & (!missing(correct) | !missing(drop) | !missing(newID))

				/******************
					Section 6.4.2 Duplicate report list ID
				******************/

				** Sort after dupListID and after ID var for
				*  duplicates currently without dupListID
				sort dupListID `idvar'

				** Assign dupListID 1 to the top row if no duplicate
				*  list IDs have been generated so far.
				replace dupListID = 1 if _n == 1 & missing(dupListID)

				** Generate new IDs based on the row above instead of directly
				*  from the row number. That prevents duplicates in the list in
				*  case an observation is deleted. The first observation with
				*  missing value will have an ID that is one digit higher than
				*  the highest ID already in the list
				replace dupListID = dupListID[_n - 1] + 1 if missing(dupListID)

				/******************
					Section 6.5
					Keep and order the variables and output the Excel files
				******************/

				* If cases unaddressed then update the Excel file
				if `unaddressedNewExcel'  {

					keep 	`argumentVars' `excelvars'
					order	`idvar' `excelvars' `uniquevars' `keepvars'

					if "`daily'" == "" {

						*Returns 0 if folder does not exist, 1 if it does
						mata : st_numscalar("r(dirExist)", direxists("`folder'/Daily"))

						** If the daily folder is not created, just create it
						if `r(dirExist)' == 0  {

							*Create the folder since it does not exist
							mkdir "`folder'/Daily"
						}

						*Export the daily file
						cap export excel using "`folder'/Daily/iedupreport`suffix'_`date'.xlsx"	, firstrow(variables) replace nolabel

						*Print error if daily report cannot be saved
						if _rc {

							display as error "{phang}There the Daily copy could not be saved to the `folder'/Daily folder. Make sure to close any old daily copy or see the option nodaily{p_end}"
							error 603
							exit

						}

						*Prepare local for output
						local daily_output "and a daily copy have been saved to the Daily folder"
					}

					*Export main report
					export excel using "`folder'/iedupreport`suffix'.xlsx"	, firstrow(variables) replace  nolabel

					*Produce output
					noi di `"{phang}Excel file created at: {browse "`folder'/iedupreport`suffix'.xlsx":`folder'/iedupreport`suffix'.xlsx} `daily_output'{p_end}"'
					noi di ""
				}
			}

		/***********************************************************************
		************************************************************************

			Section 7

			Update the data set and with the new corrections.

		************************************************************************
		***********************************************************************/

		* Load the original data set merged with correction. Duplicates
		* in all variables are already dropped in this data set
		use 	`datawithreportmerged', clear

		* If excel file exists, apply any corrections indicated (if any)
		if `fileExists' {

			/******************
				Section 7.1
				Drop duplicates listed for drop
			******************/

			drop if drop == "yes"

			/******************
				Section 7.2
				Update new ID. ID var can be either numeric or
				string. All numbers can be made strings but not
				all strings can be numeric. Therefore this
				section is complicated.
			******************/

			/******************
				Section 7.2.1
				ID var in original file is string. Either
				newID was imported as string or the variable
				is made string. Easy.
			******************/

			*Test if there are any corrections by new ID
			cap assert missing(newID)
			if _rc {

				local idtype 	: type `idvar'
				local idtypeNew : type newID

				*If ID var is string but newID is not, then just make it string
				if substr("`idtype'",1,3) == "str" & substr("`idtypeNew'",1,3) != "str" {

					tostring newID , replace
					replace  newID = "" if newID == "."
				}

				*If ID var is numeric but the newID is loaded as string
				else if substr("`idtype'",1,3) != "str" & substr("`idtypeNew'",1,3) == "str" {

					* Check if -tostringok- is specificed:
					if "`tostringok'" != "" {

						* Make original ID var string
						tostring `idvar' , replace
						replace  `idvar' = "" if `idvar' == "."

					}

					* Error, IDvar can not be updated
					else {

						* Create a local with all non-numeric values
						levelsof newID if missing(real(newID)), local(NaN_values) clean

						* Output error message
						di as error "{phang}The ID varaible `idvar' is numeric but newID has thes non-numeric values: `NaN_values'. Update newID to only contain numeric values or see option tostringok.{p_end}"
						error 109
						exit
					}
				}

				*After making sure that type is ok, update the IDs
				replace `idvar' = newID if !missing(newID)

				/******************
					Section 6.3
					Test that values in newID
					were neither used twice
					nor already existed
				******************/

				*Test if there are any duplicates after corrections (excluding observations in duplicate groups not yet corrected)
				tempvar newDup
				duplicates tag `idvar' if `groupAnyCorrection' != 0, generate(`newDup')

				*Consider missing values as no new duplicates, make conditions below easier
				replace `newDup' = 0 if missing(`newDup')

				cap assert `newDup' == 0
				if _rc {

					levelsof `idvar' if `newDup' != 0  , local(newDuplist)
					di as error "{phang}No corrections from the report are applied as it would lead to new duplicates in the following value(s): `newDuplist'.{p_end}"
					error 198
					exit
				}
			}

			/******************
				Section 7.4
				Drop Excel vars
			******************/

			drop `excelvars'
		}

		/***********************************************************************
		************************************************************************

			Section 8

			Return the data set without duplicates and
			output information regarding unresolved duplicates.

		************************************************************************
		***********************************************************************/

		* Generate a variable that is 1 if the observation is a duplicate in varlist
		tempvar dropDup
		duplicates tag `idvar',  gen(`dropDup')
		* Generate a list of the IDs that are still duplicates
		levelsof `idvar' 			if `dropDup' != 0 , local(dup_ids) clean
		* Drop the duplicates (they are exported in Excel)
		drop 						if `dropDup' != 0

		* Test if varlist is now uniquely and fully identifying the data set
		cap isid `idvar'
		if _rc {

			di as error "{phang}The data set is not returned with `varlist' uniquely and fully identifying the data set. Please report this bug to kbjarkefur@worldbank.org{p_end}"
			error 119
			exit

		}

		*Count number of duplicate groups still in the data
		local numDup	= `:list sizeof dup_ids'

		if `numDup' == 0 {
			noi di	"{phang}There are no unresolved duplicates in this data set. The data set is returned with `idvar' uniquely and fully identifying the data set.{p_end}"
		}
		else {
			noi di	"{phang}There are `numDup' duplicated IDs still unresolved. IDs still containing duplicates: `dup_ids'. The unresolved duplicate observations were exported in the Excel file. The data set is returned without those duplicates and with `idvar' uniquely and fully identifying the data set.{p_end}"
		}

		return scalar numDup	= `numDup'

		/***********************************************************************
		************************************************************************

			Section 9

			Save data set to be returned outside preserve/restore.
			Preserve/restore is used so that original data is returned
			in case an error is thrown.

		************************************************************************
		***********************************************************************/

		save 	`dataToReturn'

		restore

		** Using restore above to return the data to
		*  the orignal data set in case of error.
		use `dataToReturn', clear

	}
	end


Update 02_hhchar.do

Update 02_hhchar.do

/*------------------------------------------------------------------------------
# Name:		02_hhchar
# Purpose:	Process household characteristics and education characteristics
# Author:	Tim Essam
# Created:	2015/2/26
# License:	MIT License
# Ado(s):	labutil, labutil2 (ssc install labutil, labutil2)
# Dependencies: copylables, attachlabels, 00_SetupFoldersGlobals.do
#-------------------------------------------------------------------------------
*/

clear
capture log close
log using "$pathlog/02_hhchar", replace
di in yellow "`c(current_date)' `c(current_time)'"

use "$pathraw/GSEC2.dta", replace

* Merge education data to household roster using the force command
merge 1:1 HHID PID using "$pathraw/GSEC4.dta", force
ren _merge gsecMerge
merge 1:1 HHID PID using "$pathraw/GSEC3.dta", force

/* Demographic list to calculate
1. Head of Household Sex
2. Relationship Status
*/

* Create head of household variable based on primary respondent and sex
g byte hoh = h2q4 == 1
la var hoh "Head of household"

g byte femhead = h2q3 == 2 & h2q4 == 1
la var femhead "Female head of household"

g agehead = h2q8 if hoh == 1
la var agehead "Age of head of household"

g ageSpouse = h2q8 if h2q4 == 2
la var ageSpouse "Age of spouse"

g ageheadsq = agehead^2
la var ageheadsq "Squared age of the head (for non-linear effects)"

* Relationship status variables
g byte marriedHohm = h2q10 == 1 & hoh==1
la var marriedHohm "married HOH monogamous"

g byte marriedHoh = hoh ==1 & inlist(h2q10, 1, 2)
la var marriedHoh "married head of household (any type)"

g byte marriedHohp = h2q10 == 2 & hoh==1
la var marriedHohp "married HOH polygamous"

g byte divorcedHead = (h2q10 == 3 & hoh==1)
la var divorcedHead "divorced HoH"

g byte divorcedFemhead = (h2q10 == 3 & femhead == 1)
la var divorcedFemhead "divorced Female head of household"

g byte widowHead = (h2q10 == 4 & hoh==1)
la var widowHead "widowed HoH"

g byte widowFemhead = (h2q10 == 4 & femhead == 1)
la var widowFemhead "Widowed Female head of household"

g byte singleHead = (h2q10==5 & hoh==1)
la var singleHead "single HoH"

g byte singleFemhead = (h2q10==5 & femhead)
la var singleFemhead "single HoH"

* Calculate household demographics (size, adult equivalent units, dep ratio, etc).

/* Household size - Household size refers to the number of usual members in a
household. Usual members are defined as those who have lived in the household
for at least 6 months in the past 12 months. However, it includes persons who
may have spent less than 6 months during the last 12 months in the household
but have joined the household with intention to live permanently or for an
extended period of time.
* http://www.ubos.org/UNHS0910/chapter2_householdcharacteristics.html */

*Create a flag variable for determining who is a usual member from above.
g byte hhmemb = (inlist(h2q7, 1, 2) & h2q5 >= 6)
egen hhsize = total(hhmemb), by(HHID)
la var hhsize "household size"

* Create sex ratio for households
g byte male = h2q3 == 1 & hhmemb == 1
g byte female = h2q3 == 2 & hhmemb == 1
la var male "male hh members"
la var female "female hh members"

egen msize = total(male), by(HHID)
la var msize "number of males in hh"

egen fsize = total(female), by(HHID)
la var fsize "number of females in hh"

* Create a gender ratio variable
g sexRatio = msize/fsize
recode sexRatio (. = 0) if fsize==0
la var sexRatio "Ratio of males to females in HH"

* Calculate age demographics (Youth)
*** REDO USING EGEN + CUT ***

g byte under15tmp = h2q8 <= 15 & hhmemb ==1
g byte under24tmp = h2q8 <= 24 &  hhmemb==1
g byte youth15to35tmp = h2q8 >=15 & h2q8 <=35
g byte youth15to24tmp = h2q8 >=15 & h2q8 <=24
g byte child0to5tmp = h2q8 <= 5
g byte child6to14tmp = h2q8 >= 6 & h2q8 <= 14

egen under15 = total(under15tmp), by(HHID)
la var under15 "number of hh members under 15 years old"

egen under24 = total(under24tmp), by(HHID)
la var under24 "number of hh members under 24 years old"

egen under15male = total(under15tmp) if male==1, by(HHID)
la var under15male "number of male hh members under 15"

egen under15female = total(under15tmp) if female==1, by(HHID)
la var under15female "number of female hh members under 15"

egen under24male = total(under24tmp) if male==1, by(HHID)
la var under24male "number of male hh members under 24"

egen under24female = total(under24tmp) if female==1, by(HHID)
la var under24female "number of female hh members under 24"

* Continue to make youth variables (by gender)
egen youth15to35 = total(youth15to35tmp), by(HHID)
la var youth15to35 "number of youth 15 to 35"

egen youth15to24 = total(youth15to24tmp), by(HHID)
la var youth15to24 "number of youth 15 to 24"

egen youth15to35male = total(youth15to35tmp) if male==1, by(HHID)
la var youth15to35male "number of male youth 15 to 35"

egen youth15to35female = total(youth15to35tmp) if female==1, by(HHID)
la var youth15to35female "number of female youth 15 to 35"

egen youth15to24male = total(youth15to24tmp) if male==1, by(HHID)
la var youth15to24male "number of male youth 15 to 24"

egen youth15to24female = total(youth15to24tmp) if female==1, by(HHID)
la var youth15to24female "number of female hh members under 24"

egen child0to5 = total(child0to5tmp), by(HHID)
la var child0to5 "number of children under 5"

egen child6to14 = total(child6to14tmp), by(HHID)
la var child6to14 "number of children 6 to 14"

/* Create intl. HH dependency ratio (age ranges appropriate for Bangladesh)
# HH Dependecy Ratio = [(# people 0-14 + those 65+) / # people aged 15-64 ] * 100 #
The dependency ratio is defined as the ratio of the number of members in the age groups
of 0â€“14 years and above 60 years to the number of members of working age (15â€“60 years).
The ratio is normally expressed as a percentage (data below are multiplied by 100 for pcts.*/
g byte numDepRatio = (h2q8 < 15 | h2q8 > 64) & hhmemb == 1
g byte demonDepRatio = numDepRatio != 1 & hhmemb == 1
egen totNumDepRatio = total(numDepRatio), by(HHID)
egen totDenomDepRatio = total(demonDepRatio), by(HHID)

* Check that numbers add to hhsize
assert hhsize == totNumDepRatio+totDenomDepRatio if hhmemb==1
g depRatio = (totNumDepRatio/totDenomDepRatio)*100 if totDenomDepRatio!=.
recode depRatio (. = 0) if totDenomDepRatio==0
la var depRatio "Dependency Ratio"

* Calculate household labor shares (ages 12 - 60)
/* Household Labor Shares */
g byte hhLabort = (h2q8>= 12 & h2q8<60) & hhmemb == 1
egen hhlabor = total(hhLabort), by(HHID)
la var hhlabor "hh labor age>11 & < 60"

g byte mlabort = (h2q8>= 12 & h2q8<60 & male == 1)
egen mlabor = total(mlabort), by(HHID)
la var mlabor "hh male labor age>11 & <60"

g byte flabort = (h2q8>= 12 & h2q8<60 & female == 1)
egen flabor = total(flabort), by(HHID)
la var flabor "hh female labor age>11 & <60"
drop hhLabort mlabort flabort

* Male/Female labor share in hh
g mlaborShare = mlabor/hhlabor
recode mlaborShare (. = 0) if hhlabor == 0
la var mlaborShare "share of working age males in hh"

g flaborShare = flabor/hhlabor
recode flaborShare (. = 0) if hhlabor == 0
la var flaborShare "share of working age females in hh"

* Generate adult equivalents in household
g male10 	= 1
g fem10_19 	= 0.84
g fem20		= 0.72
g child10	= 0.60

g ae = .
replace ae = male10 if (h2q8 >=10 ) & male == 1
replace ae = fem10_19 if (h2q8 >= 10 & h2q8 < 20) & female == 1
replace ae = fem20 if (h2q8 >= 20) & female == 1
replace ae = child10 if (h2q8) < 10 & hhmemb == 1
la var ae "Adult equivalents in household"

egen adultEquiv = total(ae), by(HHID)
la var adultEquiv "Total adult equivalent units"

* Ethnicity codes of household members - create mixed hh codes or homogenous hh
* Codes found in UNPS2010.Household.Woman.Qx.Manual.pdf (pp. 132)
la def eth 11 "Acholi" 12 "Alur" 13 "Baamba" 14 "Babukusu" 15 "Babwisi" /*
*/ 16 "Bafumbira" 17 "Baganda" 18 "Bagisu" 19 "Bagungu" 20 "Bagwe" 21 "Bagwere"  /*
*/ 22 "Bahehe" 23 "Bahororo" 24 "Bakenyi" 25 "Bakiga" 26 "Bakhonzo" /*
*/ 27 "Banyabindi" 28 "Banyakole" 29 "Banyara" 30 "Banyarwanda" 31 "Banyole" /*
*/ 32 "Banyoro" 33 "Baruli" 34 "Basamia" 35 "Basoga" 36 "Basongora" /*
*/ 37 "Batagwenda" 38 "Batoro" 39 "Batuku" 40 "Batwa" 41 "Chope" 42 "Dodoth" /*
*/ 43 "Ethur" 44 "Ik (Teuso)" 45 "Iteso" 46 "Indian" 47 "Japadhola" 48 "Jie" /*
*/ 49 "Jonam" 50 "Kakwa" 51 "Karimojong" 52 "Kebu" 53 "Kuku" 54 "Kumam" /*
*/ 55 "Langi" 56 "Lendu" 57 "Lugbara" 58 "Madi" 59 "Mening" 60 "Mvuba" 61 "Napore"/*
*/ 62 "Nubi" 63 "Nyangia" 64 "Pokot" 65 "Sabiny" 66 "So (Tepeth)" 67 "Vonoma"
la val h3q9 eth

* Create variable reflectin whether or not husband/wife are same ethnic mix
g ethHeadtmp = h3q9 if h2q4 == 1
egen ethHead = max(ethHeadtmp), by(HHID)

g ethSpousetmp = h3q9 if h2q4 == 2
egen ethSpouse = max(ethSpousetmp), by(HHID)

g byte mixedEth = (ethHead != ethSpouse) if marriedHoh==1 & ethHead!=. & ethSpouse !=.
replace mixedEth = 0 if mixedEth ==.

* Create a varible for homogenous hh
* Take summ of hh eth and divide by hhsize

**********************
* Education outcomes *
**********************
/* Literacy is defined as oneâ€™s ability to read with understanding and to
 write meaningfully in any language. */
g byte literateHoh = h4q4 == 4 & hoh == 1
g byte literateSpouse = h4q4 == 4 & h2q4 == 2 & hhmemb == 1

la var literateHoh "Hoh is literate"
la var literateSpouse "Spouse is literate"

/* Education level values found in h4q7 defined using the following:
http://microdata.worldbank.org/index.php/catalog/565/datafile/F2/V110
http://www.classbase.com/countries/Uganda/Education-System
	No Education (0)
	Pre-Primary (Less than Primary Year 1)
	Primary Level (Years 1 - 7)
	Post-Primary Specialized Training or Certificate
	Junior Vocational/Technical (Years 8 - 10)
	Lower Secondary (Years 8 - 11)
	Upper Secondary (Years 11 - 13)
	Post-Secondary Specialized Training or Certificate
	Tertiary (Above Secondary other than Post-Secondary Specialized Training or Cer
tificate)
*/
g educ = .
la var educ "Education levels"
* No education (This includes:"Don't Know" and "2" Responses))
replace educ = 0 if inlist(h4q7, 2, 99)
* Pre-primary
replace educ = 1 if inlist(h4q7, 10)
* Primary
replace educ = 2 if inlist(h4q7, 11, 12, 13, 14, 15, 16, 17)
* Post-Primary Specialized Training or Certificate
replace educ = 3 if inlist(h4q7, 41)
* Junior Techincal/Vocational
replace educ = 4 if inlist(h4q7, 21, 22, 23)
* Lower Secondary
replace educ = 5 if inlist(h4q7, 31, 32, 33, 34)
* Upper Secondary
replace educ = 6 if inlist(h4q7, 35, 36)
* Post-Secondary Specialized Training or Certificate
replace educ = 7 if inlist(h4q7, 51)
* Tertiary
replace educ = 8 if inlist(h4q7, 61)

g educHoh = educ if hoh == 1
g educSpouse = educ if h2q4 == 2 & hhmemb == 1

lab def ed 0 "No education" 1 "Pre-primary" 2 "Primary" 3 "Post-Primary" /*
*/ 4 "Junior Techincal/Vocational " 5 "Lower Secondary" 6 "Upper Secondary" /*
*/ 7 "Post-Secondary Specialized" 8 "Tertiary"
la values educ ed
la values educHoh ed
la values educSpouse ed

* Create variable to reflect the maximum level of education in the household for those 25+
egen educAdult = max(educ) if h2q8>24 & hhmemb ==1, by(HHID)
egen educAdultM = max(educ) if h2q8>24 & hhmemb ==1 & male == 1, by(HHID)
egen educAdultF = max(educ) if h2q8>24 & hhmemb ==1 & female == 1, by(HHID)

la var educAdult "Highest adult education in household"
la var educAdultM "Highest male adult education in household"
la var educAdultF "Highest female adult education in household"
la var educHoh "Education of Hoh"
la var educSpouse "Education of spouse"

* Calculate school expenses for all regular household members
egen totSchoolExptmp = rsum2(h4q15a h4q15b h4q15c h4q15d h4q15e h4q15f)
replace totSchoolExptmp = h4q15g if totSchoolExp == 0
egen totSchoolExp = total(totSchoolExptmp), by(HHID)
la var totSchoolExp "Total school expenses for household"

g byte aidSchool = (h4q16 == 1)
la var aidSchool "HH member received school scholarship (all sources)"

* Household member slept under mosquito net
g byte mosqTreatnet = h3q10 == 2 & hhmemb == 1
g byte mosqNet = (inlist(h3q10, 1, 2) & hhmemb == 1)

* Occupation of mother and father (do not know codes yet, need to get them).
g byte occupFath = h3q4 if hhmemb == 1
g byte oocupMoth = h3q7 if hhmemb == 1

drop totNumDepRatio totDenomDepRatio demonDepRatio numDepRatio male10 fem10_19 fem20 child10 /*
*/ under15tmp under24tmp hhmemb totSchoolExptmp

* Retain only derived data for collapsing
ds(h2q* h4* T6* T2* LocID PID gsecMerge h3q* _merge educ), not
keep `r(varlist)'

* Collapse everything down to HH-level using max values for all vars
* Copy variable labels to reapply after collapse
include "$pathdo/copylabels.do"

ds(HHID), not
collapse (max) `r(varlist)', by(HHID)

* Reapply variable lables & value labels
include "$pathdo/attachlabels.do"

* Summarize collapsed data and review for potential coding errors
sum

foreach x of varlist  educHoh educSpouse educAdult educAdultM educAdultF educHoh educSpouse {
	la values `x' ed
	tab `x'
	}
*end

merge 1:1 HHID using "$pathout/Geovars.dta", gen(geo_merge)

* Save
save "$pathout/hhchar.dta", replace
bob
* Keep a master file of only household id's for missing var checks
use "$pathraw/GSEC2", replace
keep HHID PID
save "$pathout/hhid.dta", replace

* Create an html file of the log for internet sharability
log2html "$pathlog/02_hhchar", replace
log close

/*------------------------------------------------------------------------------
# Name:		02_hhchar
# Purpose:	Process household characteristics and education characteristics
# Author:	Tim Essam
# Created:	2015/2/26
# License:	MIT License
# Ado(s):	labutil, labutil2 (ssc install labutil, labutil2)
# Dependencies: copylables, attachlabels, 00_SetupFoldersGlobals.do
#-------------------------------------------------------------------------------
*/

clear
capture log close
log using "$pathlog/02_hhchar", replace
di in yellow "`c(current_date)' `c(current_time)'"

use "$pathraw/GSEC2.dta", replace

* Merge education data to household roster using the force command
merge 1:1 HHID PID using "$pathraw/GSEC4.dta", force
ren _merge gsecMerge
merge 1:1 HHID PID using "$pathraw/GSEC3.dta", force

/* Demographic list to calculate
1. Head of Household Sex
2. Relationship Status
*/

* Create head of household variable based on primary respondent and sex
g byte hoh = h2q4 == 1
la var hoh "Head of household"

g byte femhead = h2q3 == 2 & h2q4 == 1
la var femhead "Female head of household"

g agehead = h2q8 if hoh == 1
la var agehead "Age of head of household"

g ageheadsq = agehead^2
la var ageheadsq "Squared age of the head (for non-linear effects)"

* Relationship status variables
g byte marriedHohm = h2q10 == 1 & hoh==1
la var marriedHohm "married HOH monogamous"

g byte marriedHoh = hoh ==1 & inlist(h2q10, 1, 2)
la var marriedHoh "married head of household (any type)"

g byte marriedHohp = h2q10 == 2 & hoh==1
la var marriedHohp "married HOH polygamous"

g byte divorcedHead = (h2q10 == 3 & hoh==1)
la var divorcedHead "divorced HoH"

g byte divorcedFemhead = (h2q10 == 3 & femhead == 1)
la var divorcedFemhead "divorced Female head of household"

g byte widowHead = (h2q10 == 4 & hoh==1)
la var widowHead "widowed HoH"

g byte widowFemhead = (h2q10 == 4 & femhead == 1)
la var widowFemhead "Widowed Female head of household"

g byte singleHead = (h2q10==5 & hoh==1)
la var singleHead "single HoH"

g byte singleFemhead = (h2q10==5 & femhead)
la var singleFemhead "single HoH"

* Calculate household demographics (size, adult equivalent units, dep ratio, etc).

/* Household size - Household size refers to the number of usual members in a
household. Usual members are defined as those who have lived in the household
for at least 6 months in the past 12 months. However, it includes persons who
may have spent less than 6 months during the last 12 months in the household
but have joined the household with intention to live permanently or for an
extended period of time.
* http://www.ubos.org/UNHS0910/chapter2_householdcharacteristics.html */

*Create a flag variable for determining who is a usual member from above.
g byte hhmemb = (inlist(h2q7, 1, 2) & h2q5 >= 6)
egen hhsize = total(hhmemb), by(HHID)
la var hhsize "household size"

* Create sex ratio for households
g byte male = h2q3 == 1 & hhmemb == 1
g byte female = h2q3 == 2 & hhmemb == 1
la var male "male hh members"
la var female "female hh members"

egen msize = total(male), by(HHID)
la var msize "number of males in hh"

egen fsize = total(female), by(HHID)
la var fsize "number of females in hh"

* Create a gender ratio variable
g sexRatio = msize/fsize
recode sexRatio (. = 0) if fsize==0
la var sexRatio "Ratio of males to females in HH"

* Calculate age demographics
g byte under15tmp = h2q8 <= 15 & hhmemb ==1
g byte under24tmp = h2q8 <= 24 &  hhmemb==1

egen under15 = total(under15tmp), by(HHID)
la var under15 "number of hh members under 15 years old"

egen under24 = total(under24tmp), by(HHID)
la var under24 "number of hh members under 24 years old"

egen under15male = total(under15tmp) if male==1, by(HHID)
la var under15male "number of male hh members under 15"

egen under15female = total(under15tmp) if female==1, by(HHID)
la var under15female "number of female hh members under 15"

egen under24male = total(under24tmp) if male==1, by(HHID)
la var under24male "number of male hh members under 24"

egen under24female = total(under24tmp) if female==1, by(HHID)
la var under24female "number of female hh members under 24"

/* Create intl. HH dependency ratio (age ranges appropriate for Bangladesh)
# HH Dependecy Ratio = [(# people 0-14 + those 65+) / # people aged 15-64 ] * 100 #
The dependency ratio is defined as the ratio of the number of members in the age groups
of 0–14 years and above 60 years to the number of members of working age (15–60 years).
The ratio is normally expressed as a percentage (data below are multiplied by 100 for pcts.*/
g byte numDepRatio = (h2q8 < 15 | h2q8 > 64) & hhmemb == 1
g byte demonDepRatio = numDepRatio != 1 & hhmemb == 1
egen totNumDepRatio = total(numDepRatio), by(HHID)
egen totDenomDepRatio = total(demonDepRatio), by(HHID)

* Check that numbers add to hhsize
assert hhsize == totNumDepRatio+totDenomDepRatio if hhmemb==1
g depRatio = (totNumDepRatio/totDenomDepRatio)*100 if totDenomDepRatio!=.
recode depRatio (. = 0) if totDenomDepRatio==0
la var depRatio "Dependency Ratio"

* Calculate household labor shares (ages 12 - 60)
/* Household Labor Shares */
g byte hhLabort = (h2q8>= 12 & h2q8<60) & hhmemb == 1
egen hhlabor = total(hhLabort), by(HHID)
la var hhlabor "hh labor age>11 & < 60"

g byte mlabort = (h2q8>= 12 & h2q8<60 & male == 1)
egen mlabor = total(mlabort), by(HHID)
la var mlabor "hh male labor age>11 & <60"

g byte flabort = (h2q8>= 12 & h2q8<60 & female == 1)
egen flabor = total(flabort), by(HHID)
la var flabor "hh female labor age>11 & <60"
drop hhLabort mlabort flabort

* Male/Female labor share in hh
g mlaborShare = mlabor/hhlabor
recode mlaborShare (. = 0) if hhlabor == 0
la var mlaborShare "share of working age males in hh"

g flaborShare = flabor/hhlabor
recode flaborShare (. = 0) if hhlabor == 0
la var flaborShare "share of working age females in hh"

* Generate adult equivalents in household
g male10 	= 1
g fem10_19 	= 0.84
g fem20		= 0.72
g child10	= 0.60

g ae = .
replace ae = male10 if (h2q8 >=10 ) & male == 1
replace ae = fem10_19 if (h2q8 >= 10 & h2q8 < 20) & female == 1
replace ae = fem20 if (h2q8 >= 20) & female == 1
replace ae = child10 if (h2q8) < 10 & hhmemb == 1
la var ae "Adult equivalents in household"

egen adultEquiv = total(ae), by(HHID)
la var adultEquiv "Total adult equivalent units"

* Ethnicity codes of household members - create mixed hh codes or homogenous hh
* Codes found in UNPS2010.Household.Woman.Qx.Manual.pdf (pp. 132)
la def eth 11 "Acholi" 12 "Alur" 13 "Baamba" 14 "Babukusu" 15 "Babwisi" /*
*/ 16 "Bafumbira" 17 "Baganda" 18 "Bagisu" 19 "Bagungu" 20 "Bagwe" 21 "Bagwere"  /*
*/ 22 "Bahehe" 23 "Bahororo" 24 "Bakenyi" 25 "Bakiga" 26 "Bakhonzo" /*
*/ 27 "Banyabindi" 28 "Banyakole" 29 "Banyara" 30 "Banyarwanda" 31 "Banyole" /*
*/ 32 "Banyoro" 33 "Baruli" 34 "Basamia" 35 "Basoga" 36 "Basongora" /*
*/ 37 "Batagwenda" 38 "Batoro" 39 "Batuku" 40 "Batwa" 41 "Chope" 42 "Dodoth" /*
*/ 43 "Ethur" 44 "Ik (Teuso)" 45 "Iteso" 46 "Indian" 47 "Japadhola" 48 "Jie" /*
*/ 49 "Jonam" 50 "Kakwa" 51 "Karimojong" 52 "Kebu" 53 "Kuku" 54 "Kumam" /*
*/ 55 "Langi" 56 "Lendu" 57 "Lugbara" 58 "Madi" 59 "Mening" 60 "Mvuba" 61 "Napore"/*
*/ 62 "Nubi" 63 "Nyangia" 64 "Pokot" 65 "Sabiny" 66 "So (Tepeth)" 67 "Vonoma"
la val h3q9 eth

* Create variable reflectin whether or not husband/wife are same ethnic mix
g ethHeadtmp = h3q9 if h2q4 == 1
egen ethHead = max(ethHeadtmp), by(HHID)

g ethSpousetmp = h3q9 if h2q4 == 2
egen ethSpouse = max(ethSpousetmp), by(HHID)

g byte mixedEth = (ethHead != ethSpouse) if marriedHoh==1
replace mixedEth = 0 if mixedEth ==.

* Create a varible for homogenous hh
* Take summ of hh eth and divide by hhsize

**********************
* Education outcomes *
**********************
/* Literacy is defined as one’s ability to read with understanding and to
 write meaningfully in any language. */
g byte literateHoh = h4q4 == 4 & hoh == 1
g byte literateSpouse = h4q4 == 4 & h2q4 == 2 & hhmemb == 1

la var literateHoh "Hoh is literate"
la var literateSpouse "Spouse is literate"

/* Education level values found in h4q7 defined using the following:
http://microdata.worldbank.org/index.php/catalog/565/datafile/F2/V110
http://www.classbase.com/countries/Uganda/Education-System
	No Education (0)
	Pre-Primary (Less than Primary Year 1)
	Primary Level (Years 1 - 7)
	Post-Primary Specialized Training or Certificate
	Junior Vocational/Technical (Years 8 - 10)
	Lower Secondary (Years 8 - 11)
	Upper Secondary (Years 11 - 13)
	Post-Secondary Specialized Training or Certificate
	Tertiary (Above Secondary other than Post-Secondary Specialized Training or Cer
tificate)
*/
g educ = .
la var educ "Education levels"
* No education (This includes:"Don't Know" and "2" Responses))
replace educ = 0 if inlist(h4q7, 2, 99)
* Pre-primary
replace educ = 1 if inlist(h4q7, 10)
* Primary
replace educ = 2 if inlist(h4q7, 11, 12, 13, 14, 15, 16, 17)
* Post-Primary Specialized Training or Certificate
replace educ = 3 if inlist(h4q7, 41)
* Junior Techincal/Vocational
replace educ = 4 if inlist(h4q7, 21, 22, 23)
* Lower Secondary
replace educ = 5 if inlist(h4q7, 31, 32, 33, 34)
* Upper Secondary
replace educ = 6 if inlist(h4q7, 35, 36)
* Post-Secondary Specialized Training or Certificate
replace educ = 7 if inlist(h4q7, 51)
* Tertiary
replace educ = 8 if inlist(h4q7, 61)

g educHoh = educ if hoh == 1
g educSpouse = educ if h2q4 == 2 & hhmemb == 1

lab def ed 0 "No education" 1 "Pre-primary" 2 "Primary" 3 "Post-Primary" /*
*/ 4 "Junior Techincal/Vocational " 5 "Lower Secondary" 6 "Upper Secondary" /*
*/ 7 "Post-Secondary Specialized" 8 "Tertiary"
la values educ ed
la values educHoh ed
la values educSpouse ed

* Create variable to reflect the maximum level of education in the household for those 25+
egen educAdult = max(educ) if h2q8>24 & hhmemb ==1, by(HHID)
egen educAdultM = max(educ) if h2q8>24 & hhmemb ==1 & male == 1, by(HHID)
egen educAdultF = max(educ) if h2q8>24 & hhmemb ==1 & female == 1, by(HHID)

la var educAdult "Highest adult education in household"
la var educAdultM "Highest male adult education in household"
la var educAdultF "Highest female adult education in household"
la var educHoh "Education of Hoh"
la var educSpouse "Education of spouse"

* Calculate school expenses for all regular household members
egen totSchoolExptmp = rsum2(h4q15a h4q15b h4q15c h4q15d h4q15e h4q15f)
replace totSchoolExptmp = h4q15g if totSchoolExp == 0
egen totSchoolExp = total(totSchoolExptmp), by(HHID)
la var totSchoolExp "Total school expenses for household"

g byte aidSchool = (h4q16 == 1)
la var aidSchool "HH member received school scholarship (all sources)"

* Household member slept under mosquito net
g byte mosqTreatnet = h3q10 == 2 & hhmemb == 1
g byte mosqNet = (inlist(h3q10, 1, 2) & hhmemb == 1)

* Occupation of mother and father (do not know codes yet, need to get them).
g byte occupFath = h3q4 if hhmemb == 1
g byte oocupMoth = h3q7 if hhmemb == 1

drop totNumDepRatio totDenomDepRatio demonDepRatio numDepRatio male10 fem10_19 fem20 child10 /*
*/ under15tmp under24tmp hhmemb totSchoolExptmp

* Retain only derived data for collapsing
ds(h2q* h4* T6* T2* LocID PID gsecMerge h3q* _merge educ), not
keep `r(varlist)'

* Collapse everything down to HH-level using max values for all vars
* Copy variable labels to reapply after collapse
include "$pathdo/copylabels.do"

ds(HHID), not
collapse (max) `r(varlist)', by(HHID)

* Reapply variable lables & value labels
include "$pathdo/attachlabels.do"

* Summarize collapsed data and review for potential coding errors
sum

foreach x of varlist  educHoh educSpouse educAdult educAdultM educAdultF educHoh educSpouse {
	la values `x' ed
	tab `x'
	}
*end

* Save
save "$pathout/hhchar.dta", replace

* Keep a master file of only household id's for missing var checks
use "$pathraw/GSEC2", replace
keep HHID PID
save "$pathout/hhid.dta", replace

* Create an html file of the log for internet sharability
log2html "$pathlog/02_hhchar", replace
log close