Long-missed helpfile maintenance

Long-missed helpfile maintenance

{smcl}
{* *! version 0.0.1  28may2015}{...}
{vieweralsosee "[R] regress" "mansection R regress"}{...}
{viewerjumpto "Syntax" "svm##syntax"}{...}
{viewerjumpto "Description" "svm##description"}{...}
{viewerjumpto "Options" "svm##options"}{...}
{viewerjumpto "Stored results" "svm##results"}{...}
{viewerjumpto "Remarks" "svm##remarks"}{...}
{viewerjumpto "Examples" "svm##examples"}{...}
{viewerjumpto "Copyright" "svm##copyright"}{...}
{viewerjumpto "Authors" "svm##authors"}{...}
{viewerjumpto "References" "svm##references"}{...}
{...}{* NB: these hide the newlines }
{...}
{...}
{title:Title}

{p2colset 5 18 20 2}{...}
{p2col :{cmd:svm} {hline 2}}Support Vector Machines{p_end}
{p2colreset}{...}

{marker syntax}{...}
{title:Syntax}

{p 8 16 2}
{help svm##svm:svm} {depvar} {indepvars} {ifin} [{cmd:,} {it:options}]

{p 8 16 2}
{help svm##svm:svm} {indepvars} {ifin}, type({help svm##one_class:one_class}) [{it:options}]

{synoptset 20 tabbed}{...}
{synopthdr}
{synoptline}
{syntab:Model}
{synopt :{opth t:ype(svm##type:type)}}Type of model to fit: {opt c_svc}, {opt nu_svc}, {opt epsilon_svr}, or {opt nu_svr}, or {opt one_class}. Default: {cmd:type(c_svc)}{p_end}
{synopt :{opth k:ernel(svm##kernel:kernel)}}SVM kernel function to use: {opt linear}, {opt poly}, {opt rbf}, {opt sigmoid}, or {opt precomputed}. Default: {cmd:kernel(rbf)}{p_end}

{* XXX the division between 'tuning' and 'model' parameters is hazy; e.g. you could in theory cross-validate to choose degree (and people do this with neural networks), or even to choose the kernel . hmmmmm}{...}
{syntab:Tuning}
{synopt :{opth c:(svm##c:#)}}For {opt c_svc}, {opt epsilon_svr} and {opt nu_svr} SVMs, the weight on the margin of error. Should be > 0. Default: {cmd:c(1)}{p_end}
{synopt :{opth eps:ilon(svm##epsilon:#)}}For {opt epsilon_svr} SVMs, the margin of error allowed within which observations will be support vectors. Default: {cmd:eps(0.1)}{p_end}
{synopt :{opth nu:(svm##nu:#)}}For {opt nu_svc}, {opt one_class}, and {opt nu_svr} SVMs, tunes the proportion of expected support vectors. Should be in (0, 1]. Default: {cmd:nu(0.5)}{p_end}

{synopt :{opth g:amma(svm##gamma:#)}}For {opt poly}, {opt rbf} and {opt sigmoid} kernels, a scaling factor for the linear part of the kernel. Default: {cmd:gamma(1/[# {indepvars}])}{p_end}
{synopt :{opth coef0:(svm##coef0:#)}}For {opt poly} and {opt sigmoid} kernels, a bias ("intercept") term for the linear part of the kernel. Default: {cmd:coef0(0)}{p_end}
{synopt :{opth deg:ree(svm##degree:#)}}For {opt poly} kernels, the degree of the polynomial to use. Default: cubic ({cmd:degree(3)}){p_end}

{synopt :{opt shrink:ing}}Whether to use {help svm##shrinking:shrinkage} heuristics to improve the fit. Default: disabled{p_end}

{syntab:Features}
{* {synopt :{opt norm:alize}}Whether to {help svm##normalize:center and scale} the data. NOT IMPLEMENTED. Default: disabled{p_end} }
{synopt :{opt prob:ability}}Whether to {help svm##probability:precompute} for "predict, prob" during estimation. Only applicable to classification problems. Default: disabled{p_end}
{synopt :{opth sv:(svm##sv:newvarname)}}If given, an indicator variable to generate to mark each row as a support vector or not. Default: disabled{p_end}

{syntab:Performance}
{synopt :{opth tol:erance(svm##tolerance:#)}}The stopping tolerance used to decide convergence. Default: {cmd:epsilon(0.001)}{p_end}
{synopt :{opt v:erbose}}Turns on {help svm##verbose:verbose mode}. Default: disabled{p_end}
{synopt :{opth cache:_size(svm##cache_size:#)}}The amount of RAM used to cache kernel values during fitting, in megabytes. Default: 100MB ({cmd:cache_size(100)}){p_end}

{synoptline}
{pstd}All variables must be numeric, including categorical variables.
If you have categories stored in strings use {help encode} before {cmd:svm}.
{p_end}
INCLUDE help fvvarlist

{p 8 16 2}
{help svm##predict:predict} {newvar} {ifin}, [{cmdab:prob:ability}]

{synoptset 20 tabbed}{...}
{synopthdr}
{synoptline}
{synopt :{opt prob:ability}}If specified, estimate class probabilities for each observation. The fit must have been previously made with {opt probability}.{p_end}
{synopt :{opt v:erbose}}Turns on {help svm##verbose:verbose mode}. Default: disabled{p_end}

{p 8 16 2}
{help svm##export:svm_export} using {filename}

{p 8 16 2}
{help svm##export:svm_import} using {filename}

{synoptset 20 tabbed}{...}
{synopthdr}
{synoptline}
{synopt :{opt using}}The filename to export to or import from a fitted libsvm model.{p_end}
{synopt :{opt v:erbose}}Turns on {help svm##verbose:verbose mode}. Default: disabled{p_end}

{marker description}{...}
{title:Description}

{pstd}
{cmd:svm} fits a support vector machine (SVM) model.
SVM is not one, but several, variant models each based upon the principles of
splitting hyperplanes and the culling of unimportant observations.

{pstd}
The basic SVM idea is to find a linear boundary---a hyperplane---in high-dimensional space:
for classification, this is a boundary between two classes;
for regression it is a line {help svm##epsilon:near} which points should be--much like in {help regess:OLS},
while simultaneously minimizing the number of observations required to distinguish
this hyperplane.
The unimportant observations are ignored after fitting is done, which makes SVM very memory efficient.

{pstd}
Each observation can be thought of as a vector,
so the {it:support vectors} are those observations which the algorithm deems critical to the fit.

{pstd}

{pstd}
This package is a thin wrapper for the widely deployed {help svm##libsvm:libsvm},
so in addition to the Stata package, {bf:libsvm must be installed}.
On Windows, libsvm.dll is bundled with the package,
and you can find it in your {help adopath} (try {cmd:findfile libsvm.dll} to verify this).
On OS X, libsvm is available in both {browse "https://brew.sh":brew} and {browse "https://www.macports.org":macports}.
On Linux, search for libsvm in your distribution's package manager.
You can also compile and install libsvm from source,
if you cannot find it in your package manager or if you want the latest libsvm.
If you are having plugin load errors, please {help svm##authors:contact the authors},
as we want to make the experience as smooth as possible for our users across as many platforms as possible.

{pstd}
The thinness of this wrapper is an intentional feature. It means work done under
Stata-SVM should be replicable with other libsvm wrappers such as
{browse "http://weka.wikispaces.com/LibSVM":Weka} or
{browse "http://scikit-learn.org/stable/modules/svm.html":sklearn}.
We will happily {help svm##authors:field} feature requests,
since they mean that people are using this software,
but unless the feature really is Stata-specific
we will try to get features rolled directly into libsvm
so that everyone can benefit before trying to squeeze it on top.

{pstd}
See {help svm##svmtutorial:this SVM tutorial} for a gentle introduction to the method.
IF this manual is confusing, refer to the authoritative
the libsvm {browse "http://www.csie.ntu.edu.tw/~cjlin/libsvm/faq.html":FAQ},
{browse "https://github.com/cjlin1/libsvm/blob/master/README":README},
and {help svm##libsvm:implementation paper}.
Then please write us with suggestions for clarification.

{marker options}{...}
{title:Options}

{marker svm}{...}
{dlgtab:svm}{* this is a misuse of dlgtab because I have no corresponding dialog, but it drastically helps readability }

{pstd}
{cmd:svm} fits an SVM model, fitting {depvar} to {indepvars} except under {opt type(one_class)} which only uses {indepvars}.

{pstd}
libsvm has several algorithms with a single entry point. Since this is a thin wrapper, so do we,
which means {it:not all combinations of options are valid}.
Usually libsvm will give an error if you specify an invalid combination,
but sometimes it just ignores parameters {it:without telling you}.
Further, amongst valid combinations, not all options and datasets give good results.
Our goal is to have sane defaults,
so that the only choice you usually need to make is what {help svm##type:type} and {help svm##kernel:kernel} to use,
but there is no way to give universal default parameters.

{pstd}
Rather than guessing at the {help svm##tuning_params:tuning parameters},
you should almost always use cross-validated grid-search to find them.
Which parameters you need to tune depend on which model you pick; for example,
for ({opt type(c_svc)}, opt{kernel(rbf)}) you only need to find ({opt c()}, {opt gamma()}).
You can grid-search on a subset of your full data, so long as it is a
representative sample, to quickly find approximations for the optimal parameters.
The {help svm##libsvmguide:libsvm guide} explains this in-depth.

{* MODEL PARAMS: }
{phang}
{marker type}{...}
{opt t:ype(type)} specifies which SVM model to run.{p_end}
{pmore}{opt c_svc} and {opt nu_svc} perform classification.{p_end}
{pmore2}{depvar} should be a variable containing categories.
Multiclass classification is automatically handled if necessary using
the {browse "http://en.wikipedia.org/wiki/Multiclass_classification":class-against-class} method.

{pmore2}
If you try to use floating point values with classification you
will find that they are truncated mercilessly to their integer parts,
so you may need to recode your categories before giving them to {cmd: svm}.
If you end up with almost as many classes as observations,
you have probably used a continuous {depvar} and
should use regression instead.{p_end}

{pmore}{opt epsilon_svr} and {opt nu_svr} perform regression.{p_end}
{pmore2}{depvar} should be a variable containing continuous values.{p_end}
{pmore2}Rather than try to find a hyperplane which separates data as far as possible,
this tries to find a hyperplane which most data is near [CITATION NEEDED].
See {help svm##svr_tutorial:the SVR tutorial} for more details.{p_end}

{marker one_class}{...}
{pmore}{opt one_class} separates outliers from the bulk of the data.{p_end}
{pmore2}{opt one_class} is a form of unsupervised learning.
It estimates the support of a distribution by distinguishing "class" from "outlier",
based only on the features given to it. Therefore, it does not take a {depvar}.
Its predictions give 1 for "class" and -1 for "outlier".{p_end}
{pmore3}{bf:Tip:} You may use the same {varlist} as with the other types.
 {opt one_class} just then interprets your {depvar} as one of its {indepvars},
 giving it more information to work with.{p_end}

{pmore}To learn about the {help svm##nu:nu} variants, see {help svm##nusvm:Chen and Lin's ν-SVM tutorial}.{p_end}

{pmore}{it:type} is case insensitive.{p_end}

{phang}
{marker kernel}{...}
{opt k:ernel(kernel)} gives a kernel function to use.

{pmore}
Much like {help glm:GLMs}, the {browse "https://en.wikipedia.org/wiki/Kernel_Method":kernel trick}
extends the linear SVM algorithm to be capable of fitting nonlinear data.
Kernels bend a non-linear space into a linear one by applying a high-dimensional mapping.
Under high enough dimensions, any set of data looks close to linear.
See {browse "https://www.youtube.com/watch?v=3liCbRZPrZA"} for a visualization of this process.
{p_end}
{pmore}
The trick part---and the reason why the set of kernels is hardcoded---is that
for certain kernels [CITATION NEEDED] the fit can be done efficiently without
actually constructing the high-dimensional points, as the estimation only cares
scoring coefficients u using the output of the kernel, not the output of the values the kernel
is, in theory, operating upon.{p_end}

{pmore}Kernels available in this implementation are:{p_end}
{pmore2}{opt linear} is the dot-product you are probably familiar with from {help regress:OLS}: u'*v{p_end}
{pmore2}{opt poly} is (gamma*u'*v + coef0)^degree. This extends the linear kernel with wiggliness.{p_end}
{pmore2}{opt rbf} stands for Radial Basis Functions, and treats the coefficients
      as a mean to smoothly approach in a ball, with the form exp(-gamma*|u-v|^2);
	  this kernel tends to be a good generalist option for non-linear data.{p_end}
{pmore2}{opt sigmoid} is a kernel which bends the linear kernel to fit in -1 to 1
   with tanh(gamma*u'*v + coef0), similar to the {help logistic} non-linearity.{p_end}
{pmore2}{opt precomputed} assumes that {depvar} is actually a list of precomputed kernel values.
 With effort, you can use this to use custom kernels with your data [TODO].{p_end}

{pmore}{it:kernel} is case insensitive.{p_end}

{* TUNING PARAMS: }
{phang}
{marker tuning_params}{...}
{marker c}{...}
{opt c(#)} weights (regularizes) the error term used in {opt c_svc}, {opt epsilon_svr} and {opt nu_svr}.
Larger allows less error, but too large will lead to underfitting.

{phang}
{marker epsilon}{...}
{opt eps:ilon(#)} is the margin of error allowed by {opt epsilon_svr}.
Larger makes your fit more able to incorporate more observations, but can lead to underfitting.
Smaller can lead to overfitting.

{phang}
{marker nu}{...}
{opt nu(#)} is used in the nu variants.
The nu variants are a reparamaterization of regular SVM which lets you directly tune,
using {opt nu}, the tradeoff between error and computation.
{opt nu} is an upper bound on the fraction of training errors and a lower bound of the
fraction of support vectors, so that [TODO smaller means ... and larger means .....]
See {help svm##nusvm:the ν-SVM tutorial} for details.
{* ..wait... this doesn't make any sense. nu == 0.1 means there are at most 10% (training) errors and at least 10% are support vectors.}{...}
{*      nu = 0.9 means there are at most 90% errors and at least 90% SVs.   you should always choose 0, then, to get perfect prediction and zero memory usage}{...}

{phang}
{marker gamma}{...}
{opt g:amma(#)} is used in the non-linear {opt poly}, {opt rbf} and {opt sigmoid}
kernels as a scaling factor for the linear part. Larger weights the data more.

{phang}
{marker coef0}{...}
{opt coef0(#)} similarly is used in the non-linear {opt poly} and {opt sigmoid}
kernels as a pseudo-intercept term.

{phang}
{marker degree}{...}
{opt deg:ree(#)} selects the degree of the polynomial used by the {opt poly} kernel.
This literally controls the degree of freedom in the {opt poly} fit:
setting this too low results in underfitting and sometimes even non-convergence (notice that at {opt degree(1)}, this is just the {opt linear} kernel);
setting this too high will result in overfitting.

{phang}
{marker shrinking}{...}
{opt shrink:ing} invokes the shrinkage heuristics,
which can improve the fit by trading bias for variance. [CITATION NEEDED]

{* FEATURE PARAMS: }{...}
{* {marker normalize} }{...}
{* {phang} }{...}
{* {opt normalize} instructs the estimation to first center and scale the data }{...}
{* as SVM tends to be very sensitive to scaling issues. }{...}
{* This normalizes all data to [0,1] using min-max normalization, as suggested in the {help svm##libsvmguide:libsvm guide}. }{...}
{* Normalization creates temporary variables, so you may prefer to preprocess the data yourself---destructively and in-place---to save time on re-estimations and memory for variables, }{...}
{* especially if you are bumping up against your Stata system limits. You may find {cmd:ssc install center} helpful }{...}

{phang}
{marker probability}{...}
{opt prob:ability} enables the use of "{help svm##predict_prob:predict, prob}".
That does {browse "https://en.wikipedia.org/wiki/Platt_scaling":Platt scaling},
so for each class-against-class this precomputes a logistic regression
which is tuned with 5-fold cross-validation.
Internally, libsvm shuffles the data before cross-validation using the OS random number generator,
which is unrelated to {help set seed:Stata's RNG}, so {it:different runs will give different results}.
Enabling this demands a great deal of additional CPU and RAM.

{phang}
{marker sv}{...}
{opt sv(newvarname)} records in the given variable a boolean indicating whether each observation was determined to be a support vector.

{* PERFORMANCE PARAMS: }
{phang}
{marker tolerance}{...}
{opt tol:erance(#)} is the stopping tolerance used by the numerical optimizer. You could widen this if you are finding convergence is slow,
 but be aware that this usually non-convergence is a deeper problem.
 You could also tighten this if you have a powerful enough machine and want to get slightly more accurate estimates.

{phang}
{marker verbose}{...}
{opt v:erbose} enables output from the low level libsvm code for the duration of the operation.

{phang}
{marker cache_size}{...}
{opt cache:_size(#)} controls a time-memory tradeoff during estimation.
Value is how many megabytes (MB) of RAM to set aside for caching kernel values
Generally, more is faster, at least until you run out of RAM or cause your machine to start swapping.
On modern machines, a reasonable choice is {opt cache_size(1024)}.

{...}
{...}

{marker predict}{...}
{dlgtab:predict}

{pstd}After training you can ask svm to {cmd:predict} what the category (classification) or outcome value (regression)
      should be for each given observation. Results are placed into {newvar}.{p_end}
{pstd}{newvar} must not exist, so if you want to repredict your choices are {cmd:drop {newvar}} or to pick a new name, e.g. {cmd:predict {newvar}2}.{p_end}

{marker predict_prob}{...}
{phang}For classification ({opt c_svc}, {opt nu_svc}) problems, {opt probability} requests, for each observation, the probability of it being each class.
{newvar} is used as a stem for the new columns.
Both probabilities and predictions are decided with Platt Scaling, rather than the normal method, and is not guaranteed to give the same results
as non-probability predictions. If you are getting inconssistent results, investigate the {help svm##tuning_params:tuning} parameters.
This option is not valid for other SVM types.{p_end}

{phang}
{opt verbose}: see {help svm##verbose:svm, verbose}.

{pstd}
Prediction implicitly uses the same {indepvars} as during estimation, so be careful about renaming or dropping variables.
{...}
{...}

{marker export}{...}
{dlgtab:import/export}

{pstd}
libsvm has an ad-hoc format it uses to save trained models.
The command line programs {cmd:svm-train} and {cmd:svm-predict} which come with libsvm communicate via it.
We support this format for interoperability.

{pstd}
{cmd:svm_export} will write a fitted model to disk.
This can only be run if there is one in memory, either from {help svm##svm:svm} or {help svm##export:svm_import}.
Conventionally you should name your file with '.model' for a file extension.

{pstd}
{cmd:svm_import} will load a model from disk, replacing any previous in-memory fit.
When you import, [TODO: some properties] will be missing because the import was done without reference to any dataset.

{phang}
{opt verbose}: see {help svm##verbose:svm, verbose}.

{pstd}
Do not confuse these commands with {help svmlight:import_svmlight and export_svmlight}.

{marker results}{...}
{title:Stored results}

{pstd}
{cmd:svm} and {cmd:svm_import} stores the following in {cmd:e()}:

{synoptset 20 tabbed}{...}
{* Note: svm_model components left unexposed: }{...}
{*      - SV, the actual SVs found. If you want this in a new fit, you can use the sv() option; the use case of inspecting old SVs on an imported fit is rare, we assume, and we do not support it; (they will still be loaded to memory }{...}
{*      - sv_indices, the indexes of which support vectors were found; exposed indirectly with the sv() option }{...}
{*      - probA, probB, the coefficients used for predict, prob; these are not, by themselves, interesting }{...}
{*      - label, the "labels" of the classes (which are the integers libsvm casts out of the initial dataset; exported with strLabels to be used for labelling rho and sv_coef, but otherwise not directly interesting and  }{...}
{*      - nSV, the number of SVs per class; this is only interesting for classifications, and it duplicates what you can get out of "tab `e(depvar)' SV" }{...}
{*      - free_sv, internal libsvm flag which is a hack to stretch svm_model to handle creation from both svm_train() and svm_import() }{...}
{p2col 5 20 24 2: Scalars}{p_end}
{synopt:{cmd:e(N)}}number of observations{p_end}
{synopt:{cmd:e(N_class)}}number of classes, in a classification problem. {opt 2} in a regression problem.{p_end}
{synopt:{cmd:e(N_SV)}}number of support vectors.
If {opt e(N_SV)}/{opt e(N)} is close to 100% your fit is inefficient; perhaps you need to adjust your {help svm##kernel:kernel}.
{p_end}

{synoptset 20 tabbed}{...}
{p2col 5 20 24 2: Macros}{p_end}
{synopt:{cmd:e(cmd)}}"{cmd:svm}"{p_end}
{synopt:{cmd:e(cmdline)}}command as typed{p_end}
{synopt:{cmd:e(depvar)}}name of dependent variable{p_end}
{synopt:{cmd:e(title)}}title in estimation output{p_end}
{synopt:{cmd:e(model)}}"{cmd:svm}"{p_end}
{synopt:{cmd:e(svm_type)}}SVM type string, as above{p_end}
{synopt:{cmd:e(svm_kernel)}}kernel string, as above{p_end}
{synopt:{cmd:e(predict)}}program used to implement {cmd:predict}{p_end}
{synopt:{cmd:e(levels)}}list of the classes detected, in the order they were detected, if applicable{p_end}
{* {synopt:{cmd:e(estat_cmd)}}program used to implement {cmd:estat}{p_end} }{...}

{synoptset 20 tabbed}{...}
{p2col 5 20 24 2: Matrices}({help svm##remarks:may be missing}){p_end}
{synopt:{cmd:e(sv_coef)}}The coefficients of the support vectors for each fitted hyperplane. [TODO]{p_end}
{synopt:{cmd:e(rho)}}The intercept term for each fitted hyperplane. It is lower-triangular and {cmd:e(N_class)}^2 large, with each entry [i,j] representing the hyperplane between class i and class j.{p_end}

{marker remarks}{...}
{title:Remarks}

{pstd}
{bf:Memory Limits}: The cheaper versions of Stata allow only allow less variables and smaller matrices to be used.
As machine learning problems typically are on very large datasets,
it is easy to inadvertently instruct this package to construct more columns or larger matrices than you can afford.
If you overflow {help maxvar}, you will receive an error, the operation will fail, and the dataset will be left untouched.
If you overflow {help matsize}, the matrix that overflowed will be missing, but operation will otherwise succeed.

{pmore}
If Stata's memory limits are an impossible hurdle,
your best option is to give up on Stata and switching to libsvm's companion {cmd:svm-train} program.
This will have been installed with the libsvm package if you used a package manager, or
you can get it {browse "http://www.csie.ntu.edu.tw/~cjlin/cgi-bin/libsvm.cgi?+http://www.csie.ntu.edu.tw/~cjlin/libsvm+zip":from its authors};
You can use {help svmlight:export svmlight} to extract your dataset for use with {cmd:svm-train}.

{marker examples}{...}
{title:Examples}

INCLUDE help svm_examples

{marker copyright}{...}
{title:Copyright}

{pstd}
The wrapper is licensed:

{pmore}
The MIT License (MIT)

{pmore}
Except where otherwise noted in the code, Copyright (c) 2015
Nick Guenther <nguenthe@uwaterloo.ca>, Matthias Schonlau <schonlau@uwaterloo.ca>

{pmore}
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

{pmore}
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

{pmore}
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

{pstd}
libsvm is licensed:

{pmore}
Copyright (c) 2000-2014 Chih-Chung Chang and Chih-Jen Lin
All rights reserved.

{pmore}
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

{pmore}
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.

{pmore}
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

{pmore}
3. Neither name of copyright holders nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.

{pmore}
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

{marker authors}{...}
{title:Authors}

{pstd}
Though the license does not obligate you in any way to do so, if you find
this software useful we would be curious and appreciative to hear about your
adventures in machine learning with Stata.{p_end}
{pmore}Thank you.

{pstd}You can contact us at{p_end}
{pmore}* Nick Guenther <nguenthe@uwaterloo.ca>{p_end}
{pmore}* Matthias Schonlau <schonlau@uwaterloo.ca>{p_end}

{marker references}{...}
{title:References}

{marker sourcecode}{...}
{phang}
Guenther, Nick and Schonlau, Matthias. 2015.
{it:Stata-SVM}.
{browse "https://github.com/kousu/statasvm/"}.
{p_end}

{marker svmtutorial}{...}
{phang}
Bennett, Kristin P., and Colin Campbell. 2000.
{it:Support Vector Machines: Hype or Hallelujah?}
SIGKDD Explor. Newsl. 2.2: 1–13.
{browse "http://www.svms.org/tutorials/BennettCampbell2000.pdf"}.
{p_end}

{marker libsvm}{...}
{phang}
Chang, Chih-Chung and Lin, Chih-Jen. 2011.
{it:LIBSVM: a library for support vector machines.}
ACM Transactions on Intelligent Systems and Technology, 2:27:1--27:27.
{browse "http://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf"}.
Software available at {browse "http://www.csie.ntu.edu.tw/~cjlin/libsvm"}
{p_end}

{marker libsvmguide}{...}
{phang}
Hsu, Chih-Wei, Chang, Chih-Chung, and Lin, Chih-Jen. April 15, 2010.
{it:A Practical Guide to Support Vector Classification}.
{browse "http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf"}.
{p_end}

{marker svr_tutorial}{...}
{phang}
Smola, Alex J., and Schölkopf, Bernhard. 2004.
{it:A tutorial on support vector regression}.
Statistics and Computing 14.3: 199–222.
{* This one is behind a paywall, so the best we can do is a give a DOI link }{...}
{browse "http://dx.doi.org/10.1023/b:stco.0000035301.49549.88"}.
{p_end}

{marker nusvm}{...}
{phang}
Chen, Pai-Hsuen, Lin Chih-Jen, and Schölkopf, Bernhard. 2005.
{it:A Tutorial on ν-Support Vector Machines}.
Applied Stochastic Models in Business and Industry 21.2: 111–136.
{browse "http://www.csie.ntu.edu.tw/~cjlin/papers/nusvmtutorial.pdf"}.
{p_end}

{smcl}
{* *! version 0.0.1  28may2015}{...}
{vieweralsosee "[R] regress" "mansection R regress"}{...}
{viewerjumpto "Syntax" "svm##syntax"}{...}
{viewerjumpto "Description" "svm##description"}{...}
{viewerjumpto "Options" "svm##options"}{...}
{viewerjumpto "Stored results" "svm##results"}{...}
{viewerjumpto "Remarks" "svm##remarks"}{...}
{viewerjumpto "Examples" "svm##examples"}{...}
{viewerjumpto "Copyright" "svm##copyright"}{...}
{viewerjumpto "Authors" "svm##authors"}{...}
{viewerjumpto "References" "svm##references"}{...}
{...}{* NB: these hide the newlines }
{...}
{...}
{title:Title}

{p2colset 5 18 20 2}{...}
{p2col :{cmd:svm} {hline 2}}Support Vector Machines{p_end}
{p2colreset}{...}

{marker syntax}{...}
{title:Syntax}

{p 8 16 2}
{help svm##svm:svm} {depvar} {indepvars} {ifin} [{cmd:,} {it:options}]

{p 8 16 2}
{help svm##svm:svm} {indepvars} {ifin}, type({help svm##one_class:one_class}) [{it:options}]

{synoptset 20 tabbed}{...}
{synopthdr}
{synoptline}
{syntab:Model}
{synopt :{opth t:ype(svm##type:type)}}Type of model to fit: {opt c_svc}, {opt nu_svc}, {opt epsilon_svr}, or {opt nu_svr}, or {opt one_class}. Default: {cmd:type(c_svc)}{p_end}
{synopt :{opth k:ernel(svm##kernel:kernel)}}SVM kernel function to use: {opt linear}, {opt poly}, {opt rbf}, {opt sigmoid}, or {opt precomputed}. Default: {cmd:kernel(rbf)}{p_end}

{* XXX the division between 'tuning' and 'model' parameters is hazy; e.g. you could in theory cross-validate to choose degree (and people do this with neural networks), or even to choose the kernel . hmmmmm}{...}
{syntab:Tuning}
{synopt :{opth c:(svm##c:#)}}For {opt c_svc}, {opt epsilon_svr} and {opt nu_svr} SVMs, the weight on the margin of error. Should be > 0. Default: {cmd:c(1)}{p_end}
{synopt :{opth eps:ilon(svm##epsilon:#)}}For {opt epsilon_svr} SVMs, the margin of error allowed within which observations will be support vectors. Default: {cmd:p(0.1)}{p_end}
{synopt :{opth nu:(svm##nu:#)}}For {opt nu_svc}, {opt one_class}, and {opt nu_svr} SVMs, tunes the proportion of expected support vectors. Should be in (0, 1]. Default: {cmd:nu(0.5)}{p_end}

{synopt :{opth g:amma(svm##gamma:#)}}For {opt poly}, {opt rbf} and {opt sigmoid} kernels, a scaling factor for the linear part of the kernel. Default: {cmd:gamma(1/[# {indepvars}])}{p_end}
{synopt :{opth coef0:(svm##coef0:#)}}For {opt poly} and {opt sigmoid} kernels, a bias ("intercept") term for the linear part of the kernel. Default: {cmd:coef0(0)}{p_end}
{synopt :{opth deg:ree(svm##degree:#)}}For {opt poly} kernels, the degree of the polynomial to use. Default: cubic ({cmd:degree(3)}){p_end}

{synopt :{opt shrink:ing}}Whether to use {help svm##shrinking:shrinkage} heuristics to improve the fit. Default: disabled{p_end}

{syntab:Features}
{* {synopt :{opt norm:alize}}Whether to {help svm##normalize:center and scale} the data. NOT IMPLEMENTED. Default: disabled{p_end} }
{synopt :{opt prob:ability}}Whether to {help svm##probability:precompute} for "predict, prob" during estimation. Only applicable to classification problems. Default: disabled{p_end}
{synopt :{opth sv:(svm##sv:newvarname)}}If given, an indicator variable to generate to mark each row as a support vector or not. Default: disabled{p_end}

{syntab:Performance}
{synopt :{opth tol:erance(svm##tolerance:#)}}The stopping tolerance used to decide convergence. Default: {cmd:epsilon(0.001)}{p_end}
{synopt :{opt v:erbose}}Turns on {help svm##verbose:verbose mode}. Default: disabled{p_end}
{synopt :{opth cache:_size(svm##cache_size:#)}}The amount of RAM used to cache kernel values during fitting, in megabytes. Default: 100MB ({cmd:cache_size(100)}){p_end}

{synoptline}
{pstd}All variables must be numeric, including categorical variables.
If you have categories stored in strings use {help encode} before {cmd:svm}.
{p_end}
INCLUDE help fvvarlist

{p 8 16 2}
{help svm##predict:predict} {newvar} {ifin}, [{cmdab:prob:ability}]

{synoptset 20 tabbed}{...}
{synopthdr}
{synoptline}
{synopt :{opt prob:ability}}If specified, estimate class probabilities for each observation. The fit must have been previously made with {opt probability}.{p_end}
{synopt :{opt v:erbose}}Turns on {help svm##verbose:verbose mode}. Default: disabled{p_end}

{p 8 16 2}
{help svm##export:svm_export} using {filename}

{p 8 16 2}
{help svm##export:svm_import} using {filename}

{synoptset 20 tabbed}{...}
{synopthdr}
{synoptline}
{synopt :{opt using}}The filename to export to or import from a fitted libsvm model.{p_end}
{synopt :{opt v:erbose}}Turns on {help svm##verbose:verbose mode}. Default: disabled{p_end}

{marker description}{...}
{title:Description}

{pstd}
{cmd:svm} fits a support vector machine (SVM) model.
SVM is not one, but several, variant models each based upon the principles of
splitting hyperplanes and the culling of unimportant observations.

{pstd}
The basic SVM idea is to find a linear boundary---a hyperplane---in high-dimensional space:
for classification, this is a boundary between two classes;
for regression it is a line {help svm##epsilon:near} which points should be--much like in {help regess:OLS},
while simultaneously minimizing the number of observations required to distinguish
this hyperplane.
The unimportant observations are ignored after fitting is done, which makes SVM very memory efficient.

{pstd}
Each observation can be thought of as a vector,
so the {it:support vectors} are those observations which the algorithm deems critical to the fit.

{pstd}

{pstd}
This package is a thin wrapper for the widely deployed {help svm##libsvm:libsvm},
so in addition to the Stata package, {bf:libsvm must be installed}.
On Windows, libsvm.dll is bundled with the package,
and you can find it in your {help adopath} (try {cmd:findfile libsvm.dll} to verify this).
On OS X, libsvm is available in both {browse "https://brew.sh":brew} and {browse "https://www.macports.org":macports}.
On Linux, search for libsvm in your distribution's package manager.
You can also compile and install libsvm from source,
if you cannot find it in your package manager or if you want the latest libsvm.
If you are having plugin load errors, please {help svm##authors:contact the authors},
as we want to make the experience as smooth as possible for our users across as many platforms as possible.

{pstd}
The thinness of this wrapper is an intentional feature. It means work done under
Stata-SVM should be replicable with other libsvm wrappers such as
{browse "http://weka.wikispaces.com/LibSVM":Weka} or
{browse "http://scikit-learn.org/stable/modules/svm.html":sklearn}.
We will happily {help svm##authors:field} feature requests,
since they mean that people are using this software,
but unless the feature really is Stata-specific
we will try to get features rolled directly into libsvm
so that everyone can benefit before trying to squeeze it on top.

{pstd}
See {help svm##svmtutorial:this SVM tutorial} for a gentle introduction to the method.
IF this manual is confusing, refer to the authoritative
the libsvm {browse "http://www.csie.ntu.edu.tw/~cjlin/libsvm/faq.html":FAQ},
{browse "https://github.com/cjlin1/libsvm/blob/master/README":README},
and {help svm##libsvm:implementation paper}.
Then please write us with suggestions for clarification.

{marker options}{...}
{title:Options}

{marker svm}{...}
{dlgtab:svm}{* this is a misuse of dlgtab because I have no corresponding dialog, but it drastically helps readability }

{pstd}
{cmd:svm} fits an SVM model, fitting {depvar} to {indepvars} except under {opt type(one_class)} which only uses {indepvars}.

{pstd}
libsvm has several algorithms with a single entry point. Since this is a thin wrapper, so do we,
which means {it:not all combinations of options are valid}.
Usually libsvm will give an error if you specify an invalid combination,
but sometimes it just ignores parameters {it:without telling you}.
Further, amongst valid combinations, not all options and datasets give good results.
Our goal is to have sane defaults,
so that the only choice you usually need to make is what {help svm##type:type} and {help svm##kernel:kernel} to use,
but there is no way to give universal default parameters.

{pstd}
Rather than guessing at the {help svm##tuning_params:tuning parameters},
you should almost always use cross-validated grid-search to find them.
Which parameters you need to tune depend on which model you pick; for example,
for ({opt type(c_svc)}, opt{kernel(rbf)}) you only need to find ({opt c()}, {opt gamma()}).
You can grid-search on a subset of your full data, so long as it is a
representative sample, to quickly find approximations for the optimal parameters.
The {help svm##libsvmguide:libsvm guide} explains this in-depth.

{* MODEL PARAMS: }
{phang}
{marker type}{...}
{opt t:ype(type)} specifies which SVM model to run.{p_end}
{pmore}{opt c_svc} and {opt nu_svc} perform classification.{p_end}
{pmore2}{depvar} should be a variable containing categories.
Multiclass classification is automatically handled if necessary using
the {browse "http://en.wikipedia.org/wiki/Multiclass_classification":class-against-class} method.

{pmore2}
If you try to use floating point values with classification you
will find that they are truncated mercilessly to their integer parts,
so you may need to recode your categories before giving them to {cmd: svm}.
If you end up with almost as many classes as observations,
you have probably used a continuous {depvar} and
should use regression instead.{p_end}

{pmore}{opt epsilon_svr} and {opt nu_svr} perform regression.{p_end}
{pmore2}{depvar} should be a variable containing continuous values.{p_end}
{pmore2}Rather than try to find a hyperplane which separates data as far as possible,
this tries to find a hyperplane which most data is near [CITATION NEEDED].
See {help svm##svr_tutorial:the SVR tutorial} for more details.{p_end}

{marker one_class}{...}
{pmore}{opt one_class} separates outliers from the bulk of the data.{p_end}
{pmore2}{opt one_class} is a form of unsupervised learning.
It estimates the support of a distribution by distinguishing "class" from "outlier",
based only on the features given to it. Therefore, it does not take a {depvar}.
Its predictions give 1 for "class" and -1 for "outlier".{p_end}
{pmore3}{bf:Tip:} You may use the same {varlist} as with the other types.
 {opt one_class} just then interprets your {depvar} as one of its {indepvars},
 giving it more information to work with.{p_end}

{pmore}To learn about the {help svm##nu:nu} variants, see {help svm##nusvm:Chen and Lin's ν-SVM tutorial}.{p_end}

{pmore}{it:type} is case insensitive.{p_end}

{phang}
{marker kernel}{...}
{opt k:ernel(kernel)} gives a kernel function to use.

{pmore}
Much like {help glm:GLMs}, the {browse "https://en.wikipedia.org/wiki/Kernel_Method":kernel trick}
extends the linear SVM algorithm to be capable of fitting nonlinear data.
Kernels bend a non-linear space into a linear one by applying a high-dimensional mapping.
Under high enough dimensions, any set of data looks close to linear.
See {browse "https://www.youtube.com/watch?v=3liCbRZPrZA"} for a visualization of this process.
{p_end}
{pmore}
The trick part---and the reason why the set of kernels is hardcoded---is that
for certain kernels [CITATION NEEDED] the fit can be done efficiently without
actually constructing the high-dimensional points, as the estimation only cares
scoring coefficients u using the output of the kernel, not the output of the values the kernel
is, in theory, operating upon.{p_end}

{pmore}Kernels available in this implementation are:{p_end}
{pmore2}{opt linear} is the dot-product you are probably familiar with from {help regress:OLS}: u'*v{p_end}
{pmore2}{opt poly} is (gamma*u'*v + coef0)^degree. This extends the linear kernel with wiggliness.{p_end}
{pmore2}{opt rbf} stands for Radial Basis Functions, and treats the coefficients
      as a mean to smoothly approach in a ball, with the form exp(-gamma*|u-v|^2);
	  this kernel tends to be a good generalist option for non-linear data.{p_end}
{pmore2}{opt sigmoid} is a kernel which bends the linear kernel to fit in -1 to 1
   with tanh(gamma*u'*v + coef0), similar to the {help logistic} non-linearity.{p_end}
{pmore2}{opt precomputed} assumes that {depvar} is actually a list of precomputed kernel values.
 With effort, you can use this to use custom kernels with your data [TODO].{p_end}

{pmore}{it:kernel} is case insensitive.{p_end}

{* TUNING PARAMS: }
{phang}
{marker tuning_params}{...}
{marker c}{...}
{opt c(#)} weights (regularizes) the error term used in {opt c_svc}, {opt epsilon_svr} and {opt nu_svr}.
Larger allows less error, but too large will lead to underfitting.

{phang}
{marker epsilon}{...}
{opt eps:ilon(#)} is the margin of error allowed by {opt epsilon_svr}.
Larger makes your fit more able to incorporate more observations, but can lead to underfitting.
Smaller can lead to overfitting.

{phang}
{marker nu}{...}
{opt nu(#)} is used in the nu variants.
The nu variants are a reparamaterization of regular SVM which lets you directly tune,
using {opt nu}, the tradeoff between error and computation.
{opt nu} is an upper bound on the fraction of training errors and a lower bound of the
fraction of support vectors, so that [TODO smaller means ... and larger means .....]
See {help svm##nusvm:the ν-SVM tutorial} for details.
{* ..wait... this doesn't make any sense. nu == 0.1 means there are at most 10% (training) errors and at least 10% are support vectors.}{...}
{*      nu = 0.9 means there are at most 90% errors and at least 90% SVs.   you should always choose 0, then, to get perfect prediction and zero memory usage}{...}

{phang}
{marker gamma}{...}
{opt g:amma(#)} is used in the non-linear {opt poly}, {opt rbf} and {opt sigmoid}
kernels as a scaling factor for the linear part. Larger weights the data more.

{phang}
{marker coef0}{...}
{opt coef0(#)} similarly is used in the non-linear {opt poly} and {opt sigmoid}
kernels as a pseudo-intercept term.

{phang}
{marker degree}{...}
{opt deg:ree(#)} selects the degree of the polynomial used by the {opt poly} kernel.
This literally controls the degree of freedom in the {opt poly} fit:
setting this too low results in underfitting and sometimes even non-convergence (notice that at {opt degree(1)}, this is just the {opt linear} kernel);
setting this too high will result in overfitting.

{phang}
{marker shrinking}{...}
{opt shrink:ing} invokes the shrinkage heuristics,
which can improve the fit by trading bias for variance. [CITATION NEEDED]

{* FEATURE PARAMS: }{...}
{* {marker normalize} }{...}
{* {phang} }{...}
{* {opt normalize} instructs the estimation to first center and scale the data }{...}
{* as SVM tends to be very sensitive to scaling issues. }{...}
{* This normalizes all data to [0,1] using min-max normalization, as suggested in the {help svm##libsvmguide:libsvm guide}. }{...}
{* Normalization creates temporary variables, so you may prefer to preprocess the data yourself---destructively and in-place---to save time on re-estimations and memory for variables, }{...}
{* especially if you are bumping up against your Stata system limits. You may find {cmd:ssc install center} helpful }{...}

{phang}
{marker probability}{...}
{opt prob:ability} enables the use of "{help svm##predict_prob:predict, prob}".
That does {browse "https://en.wikipedia.org/wiki/Platt_scaling":Platt scaling},
so for each class-against-class this precomputes a logistic regression
which is tuned with 5-fold cross-validation.
Internally, libsvm shuffles the data before cross-validation using the OS random number generator,
which is unrelated to {help set seed:Stata's RNG}, so {it:different runs will give different results}.
Enabling this demands a great deal of additional CPU and RAM.

{phang}
{marker sv}{...}
{opt sv(newvarname)} records in the given variable a boolean indicating whether each observation was determined to be a support vector.

{* PERFORMANCE PARAMS: }
{phang}
{marker tolerance}{...}
{opt tol:erance(#)} is the stopping tolerance used by the numerical optimizer. You could widen this if you are finding convergence is slow,
 but be aware that this usually non-convergence is a deeper problem.
 You could also tighten this if you have a powerful enough machine and want to get slightly more accurate estimates.

{phang}
{marker verbose}{...}
{opt v:erbose} enables output from the low level libsvm code for the duration of the operation.

{phang}
{marker cache_size}{...}
{opt cache:_size(#)} controls a time-memory tradeoff during estimation.
Value is how many megabytes (MB) of RAM to set aside for caching kernel values
Generally, more is faster, at least until you run out of RAM or cause your machine to start swapping.
On modern machines, a reasonable choice is {opt cache_size(1024)}.

{...}
{...}

{marker predict}{...}
{dlgtab:predict}

{pstd}After training you can ask svm to {cmd:predict} what the category (classification) or outcome value (regression)
      should be for each given observation. Results are placed into {newvar}.{p_end}
{pstd}{newvar} must not exist, so if you want to repredict your choices are {cmd:drop {newvar}} or to pick a new name, e.g. {cmd:predict {newvar}2}.{p_end}

{marker predict_prob}{...}
{phang}For classification ({opt c_svc}, {opt nu_svc}) problems, {opt probability} requests, for each observation, the probability of it being each class.
{newvar} is used as a stem for the new columns.
Both probabilities and predictions are decided with Platt Scaling, rather than the normal method, and is not guaranteed to give the same results
as non-probability predictions. If you are getting inconssistent results, investigate the {help svm##tuning_params:tuning} parameters.
This option is not valid for other SVM types.{p_end}

{phang}
{opt verbose}: see {help svm##verbose:svm, verbose}.

{pstd}
Prediction implicitly uses the same {indepvars} as during estimation, so be careful about renaming or dropping variables.
{...}
{...}

{marker export}{...}
{dlgtab:import/export}

{pstd}
libsvm has an ad-hoc format it uses to save trained models.
The command line programs {cmd:svm-train} and {cmd:svm-predict} which come with libsvm communicate via it.
We support this format for interoperability.

{pstd}
{cmd:svm_export} will write a fitted model to disk.
This can only be run if there is one in memory, either from {help svm##svm:svm} or {help svm##export:svm_import}.
Conventionally you should name your file with '.model' for a file extension.

{pstd}
{cmd:svm_import} will load a model from disk, replacing any previous in-memory fit.
When you import, [TODO: some properties] will be missing because the import was done without reference to any dataset.

{phang}
{opt verbose}: see {help svm##verbose:svm, verbose}.

{pstd}
Do not confuse these commands with {help svmlight:import_svmlight and export_svmlight}.

{marker results}{...}
{title:Stored results}

{pstd}
{cmd:svm} and {cmd:svm_import} stores the following in {cmd:e()}:

{synoptset 20 tabbed}{...}
{* Note: svm_model components left unexposed: }{...}
{*      - SV, the actual SVs found. If you want this in a new fit, you can use the sv() option; the use case of inspecting old SVs on an imported fit is rare, we assume, and we do not support it; (they will still be loaded to memory }{...}
{*      - sv_indices, the indexes of which support vectors were found; exposed indirectly with the sv() option }{...}
{*      - probA, probB, the coefficients used for predict, prob; these are not, by themselves, interesting }{...}
{*      - label, the "labels" of the classes (which are the integers libsvm casts out of the initial dataset; exported with strLabels to be used for labelling rho and sv_coef, but otherwise not directly interesting and  }{...}
{*      - nSV, the number of SVs per class; this is only interesting for classifications, and it duplicates what you can get out of "tab `e(depvar)' SV" }{...}
{*      - free_sv, internal libsvm flag which is a hack to stretch svm_model to handle creation from both svm_train() and svm_import() }{...}
{p2col 5 20 24 2: Scalars}{p_end}
{synopt:{cmd:e(N)}}number of observations{p_end}
{synopt:{cmd:e(N_class)}}number of classes, in a classification problem. {opt 2} in a regression problem.{p_end}
{synopt:{cmd:e(N_SV)}}number of support vectors.
If {opt e(N_SV)}/{opt e(N)} is close to 100% your fit is inefficient; perhaps you need to adjust your {help svm##kernel:kernel}.
{p_end}

{synoptset 20 tabbed}{...}
{p2col 5 20 24 2: Macros}{p_end}
{synopt:{cmd:e(cmd)}}"{cmd:svm}"{p_end}
{synopt:{cmd:e(cmdline)}}command as typed{p_end}
{synopt:{cmd:e(depvar)}}name of dependent variable{p_end}
{synopt:{cmd:e(title)}}title in estimation output{p_end}
{synopt:{cmd:e(model)}}"{cmd:svm}"{p_end}
{synopt:{cmd:e(svm_type)}}SVM type string, as above{p_end}
{synopt:{cmd:e(svm_kernel)}}kernel string, as above{p_end}
{synopt:{cmd:e(predict)}}program used to implement {cmd:predict}{p_end}
{synopt:{cmd:e(levels)}}list of the classes detected, in the order they were detected, if applicable{p_end}
{* {synopt:{cmd:e(estat_cmd)}}program used to implement {cmd:estat}{p_end} }{...}

{synoptset 20 tabbed}{...}
{p2col 5 20 24 2: Matrices}({help svm##remarks:may be missing}){p_end}
{synopt:{cmd:e(sv_coef)}}The coefficients of the support vectors for each fitted hyperplane. [TODO]{p_end}
{synopt:{cmd:e(rho)}}The intercept term for each fitted hyperplane. It is lower-triangular and {cmd:e(N_class)}^2 large, with each entry [i,j] representing the hyperplane between class i and class j.{p_end}

{marker remarks}{...}
{title:Remarks}

{pstd}
{bf:Memory Limits}: The cheaper versions of Stata allow only allow less variables and smaller matrices to be used.
As machine learning problems typically are on very large datasets,
it is easy to inadvertently instruct this package to construct more columns or larger matrices than you can afford.
If you overflow {help maxvar}, you will receive an error, the operation will fail, and the dataset will be left untouched.
If you overflow {help matsize}, the matrix that overflowed will be missing, but operation will otherwise succeed.

{pmore}
If Stata's memory limits are an impossible hurdle,
your best option is to give up on Stata and switching to libsvm's companion {cmd:svm-train} program.
This will have been installed with the libsvm package if you used a package manager, or
you can get it {browse "http://www.csie.ntu.edu.tw/~cjlin/cgi-bin/libsvm.cgi?+http://www.csie.ntu.edu.tw/~cjlin/libsvm+zip":from its authors};
You can use {help svmlight:export svmlight} to extract your dataset for use with {cmd:svm-train}.

{marker examples}{...}
{title:Examples}

INCLUDE help svm_examples

{marker copyright}{...}
{title:Copyright}

{pstd}
The wrapper is licensed:

{pmore}
The MIT License (MIT)

{pmore}
Except where otherwise noted in the code, Copyright (c) 2015
Nick Guenther <nguenthe@uwaterloo.ca>, Matthias Schonlau <schonlau@uwaterloo.ca>

{pmore}
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

{pmore}
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

{pmore}
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

{pstd}
libsvm is licensed:

{pmore}
Copyright (c) 2000-2014 Chih-Chung Chang and Chih-Jen Lin
All rights reserved.

{pmore}
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

{pmore}
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.

{pmore}
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

{pmore}
3. Neither name of copyright holders nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.

{pmore}
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

{marker authors}{...}
{title:Authors}

{pstd}
Though the license does not obligate you in any way to do so, if you find
this software useful we would be curious and appreciative to hear about your
adventures in machine learning with Stata.{p_end}
{pmore}Thank you.

{pstd}You can contact us at{p_end}
{pmore}* Nick Guenther <nguenthe@uwaterloo.ca>{p_end}
{pmore}* Matthias Schonlau <schonlau@uwaterloo.ca>{p_end}

{marker references}{...}
{title:References}

{marker sourcecode}{...}
{phang}
Guenther, Nick and Schonlau, Matthias. 2015.
{it:Stata-SVM}.
{browse "https://github.com/kousu/statasvm/"}.
{p_end}

{marker svmtutorial}{...}
{phang}
Bennett, Kristin P., and Colin Campbell. 2000.
{it:Support Vector Machines: Hype or Hallelujah?}
SIGKDD Explor. Newsl. 2.2: 1–13.
{browse "http://www.svms.org/tutorials/BennettCampbell2000.pdf"}.
{p_end}

{marker libsvm}{...}
{phang}
Chang, Chih-Chung and Lin, Chih-Jen. 2011.
{it:LIBSVM: a library for support vector machines.}
ACM Transactions on Intelligent Systems and Technology, 2:27:1--27:27.
{browse "http://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf"}.
Software available at {browse "http://www.csie.ntu.edu.tw/~cjlin/libsvm"}
{p_end}

{marker libsvmguide}{...}
{phang}
Hsu, Chih-Wei, Chang, Chih-Chung, and Lin, Chih-Jen. April 15, 2010.
{it:A Practical Guide to Support Vector Classification}.
{browse "http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf"}.
{p_end}

{marker svr_tutorial}{...}
{phang}
Smola, Alex J., and Schölkopf, Bernhard. 2004.
{it:A tutorial on support vector regression}.
Statistics and Computing 14.3: 199–222.
{* This one is behind a paywall, so the best we can do is a give a DOI link }{...}
{browse "http://dx.doi.org/10.1023/b:stco.0000035301.49549.88"}.
{p_end}

{marker nusvm}{...}
{phang}
Chen, Pai-Hsuen, Lin Chih-Jen, and Schölkopf, Bernhard. 2005.
{it:A Tutorial on ν-Support Vector Machines}.
Applied Stochastic Models in Business and Industry 21.2: 111–136.
{browse "http://www.csie.ntu.edu.tw/~cjlin/papers/nusvmtutorial.pdf"}.
{p_end}


added label headers

added label headers

program summary

est clear
use ${stash}master, clear

labelvar

estpost tabstat img text bd traf debut treat, s(mean sd median min max) columns(statistics) nototal by(isbasket)

esttab using "${tables}summary.tex", cells ("mean(fmt(2) label(Mean)) sd(label(SD)) p50(label(Median)) min(label(Min)) max(label(Max))" ) coeflabels("Mean" "SD" "Median" "Min" "Max") replace nonum noobs booktabs width(\hsize) alignment(rrrrr) varwidth(30) label

end

program labelvar

label variable img "Images"
label variable text "Text"
label variable bd "Citations"
label variable traf "Traffic"

label variable lnimg "Log(Images)"
label variable lntext "Log(Text)"
label variable lnbd "Log(Citations)"
label variable lntraf "Log(Traffic)"

label variable debutyear "Debut Year"
label variable treat "Out-Of-Copy"

label define isbaseball_l 0 "Basketball" 1 "Baseball"
label values isbaseball isbaseball_l

gen isbasket = !isbaseball
label define isbasket_l 1 "\textbf{Panel B -- Basketball}" 0 "\textbf{Panel A -- Baseball}"
label values isbasket isbasket_l

end

program summary

est clear
use ${stash}master, clear

labelvar

estpost tabstat img text bd traf debut treat, s(mean sd median min max) columns(statistics) nototal by(isbasket)

esttab using "${tables}summary.tex", cells ("mean(fmt(2) label(Mean)) sd(label(SD)) p50(label(Median)) min(label(Min)) max(label(Max))" ) coeflabels("Mean" "SD" "Median" "Min" "Max") replace nonum noobs label booktabs width(\hsize) alignment(rrrrr) varwidth(30)

end

program labelvar

label variable img "Images"
label variable text "Text"
label variable bd "Citations"
label variable traf "Traffic"

label variable lnimg "Log(Images)"
label variable lntext "Log(Text)"
label variable lnbd "Log(Citations)"
label variable lntraf "Log(Traffic)"

label variable debutyear "Debut Year"
label variable treat "1(Debut Before 1964)"

label define isbaseball_l 0 "Basketball" 1 "Baseball"
label values isbaseball isbaseball_l

gen isbasket = !isbaseball
label define isbasket_l 1 "\textbf{Basketball}" 0 "\textbf{Baseball}"
label values isbasket isbasket_l

end


graphs added

graphs added

Added graphs but the number of variables is still hardcoded

cap	program drop	ieimpgraph
	program define 	ieimpgraph

	syntax varlist

	mat beta_ = e(b)

	local counter = 0
	local textfile = "Mrijan"
	local date = "$S_DATE"

	qui{

	foreach var of local varlist{
		cap assert inlist(`var',0,1) | missing(`var')
			if _rc {
				noi display as error "{phang} The variable `var' is not a dummy. Treatment variable needs to be a dummy(0 or 1) variable. {p_end}"
				noi display ""
				error 149
				}
			else {
				}

		local counter = `counter' + 1
		di `counter'

		scalar coeff_`var' = _b[`var']
		noi display coeff_`var'

		scalar coeff_se_`var' = _se[`var']
		noi display coeff_se_`var'

		count if e(sample) == 1 & `var' == 1 //Count one tmt group at the time
		scalar obs_`var' = r(N)
		noi display obs_`var'

	}

	//local count: word count `varlist'
	//tokenize "`varlist'"
	//noi di `count'

	//Make all vars tempvars (maybe do later)
	//Make sure that missing is properly handled
	tempvar anyTMT
	egen `anyTMT' = rowmax(`varlist') if e(sample) == 1
	tempvar control
	gen `control' = (`anyTMT' == 0) if !missing(`anyTMT')

	noi sum `e(depvar)' if e(sample) == 1 & `control' == 1
	scalar ctl_N		= r(N)
	scalar ctl_mean	  	= r(mean)
	scalar ctl_mean_sd 	= r(sd)

		foreach var of local varlist {
			scalar  conf_int_min_`var'   =	coeff_`var'-(1.96*coeff_se_`var') + ctl_mean
			scalar  conf_int_max_`var'   =	coeff_`var'+(1.96*coeff_se_`var') + ctl_mean

			noi test `var'
			local star_`var'

			scalar  pvalue =r(p)
			if pvalue < 0.10 {
				local star_`var' "*"
			}
			if pvalue < 0.05 {
				local star_`var' "**"
			}
			if pvalue < 0.01 {
				local star_`var' "***"
			}
			noi display "`star_`var''"
			scalar tmt_mean_`var' = ctl_mean + coeff_`var'
		}

	tempname 	newHandle
	tempfile	newTextFile
	cap file close 	`newHandle'
	file open  	`newHandle' using "`newTextFile'", text write append
	file write `newHandle' ///
		"order" 	_tab "xLabeled"	_tab "mean" _tab "coeff" _tab "conf_int_min" _tab "conf_int_max" _tab "obs" _tab "star" _n 	///
		%9.3f (1) _tab "Control"  _tab %9.3f (ctl_mean)      _tab  		 			_tab 		 			  	  _tab 			 				_tab %9.3f (ctl_N) 	   _tab			  	  _n
	file close `newHandle'

	tempvar newCounter
	gen `newCounter' = 2
	foreach var in `varlist' {
		file open  	`newHandle' using "`newTextFile'", text write append
		file write `newHandle' ///
		%9.3f (`newCounter') _tab "`var'" _tab %9.3f (tmt_mean_`var') _tab  %9.3f (coeff_`var')  _tab %9.3f (conf_int_min_`var') _tab %9.3f (conf_int_max_`var') _tab %9.3f (obs_`var')   _tab "`star_`var''"  _n
		file close `newHandle'
		replace `newCounter' = `newCounter' + 1
		}

	copy "`newTextFile'"  "mainMasterDofile.txt" , replace

	insheet using mainMasterDofile.txt, clear

	local graphname	gph_new
		graph twoway (bar mean order if order == 1, color("215 25 28")	 ) ///
			(bar mean order if order == 2 , color("253 174 97") 		) ///
			(bar mean order if order == 3 , color("255 255 191" ) 		 ) ///
			(rcap conf_int_max conf_int_min order, lc(gs) 						) ///
			(scatter  mean order, msym(none) mlab() mlabs(medium) mlabpos(10) mlabcolor(black))	, ///
			legend(order(1 2 3 4) lab(1 ) lab(2 xLabeled) lab(3 "Shared Demo Treatment")) ///
			 plotregion(margin(b+3 t+3)) ///
			xlabel(	1 `" (N = `obs_ctrl') "' ///
					2 `" "(N = `obs_reg')" " `star_reg' " "' ///
					3 `" "(N = `obs_sdp')" " `star_sdp' " "' , noticks labsize(medsmall)) ///
			graphregion( lcolor("182 222 255") fcolor("182 222 255")) ///
			plotregion(fcolor("247 247 247") ) ///
		   	xtitle("")   ///
			title("`title'") saving(`graphname'.gph, replace)
			graph export `graphname'.png, replace

}
end

cap	program drop	ieimpgraph
	program define 	ieimpgraph

	syntax varlist

	mat beta_ = e(b)

	local counter = 0
	local textfile = "Mrijan"
	local date = "$S_DATE"

	qui{

	foreach var of local varlist{
		cap assert inlist(`var',0,1) | missing(`var')
			if _rc {
				noi display as error "{phang} The variable `var' is not a dummy. Treatment variable needs to be a dummy(0 or 1) variable. {p_end}"
				noi display ""
				error 149
				}
			else {
				}

		local counter = `counter' + 1
		di `counter'

		scalar coeff_`var' = _b[`var']
		noi display coeff_`var'

		scalar coeff_se_`var' = _se[`var']
		noi display coeff_se_`var'

		count if e(sample) == 1 & `var' == 1 //Count one tmt group at the time
		scalar obs_`var' = r(N)
		noi display obs_`var'

	}

	local count: word count `varlist'
	tokenize "`varlist'"
	noi di `count'

	//Make all vars tempvars (maybe do later)
	//Make sure that missing is properly handled
	tempvar anyTMT
	egen `anyTMT' = rowmax(`varlist') if e(sample) == 1
	tempvar control
	gen `control' = (`anyTMT' == 0) if !missing(`anyTMT')

	noi sum `e(depvar)' if e(sample) == 1 & `control' == 1
	scalar ctl_N		= r(N)
	scalar ctl_mean	  	= r(mean)
	scalar ctl_mean_sd 	= r(sd)

		foreach var of local varlist {
			scalar  conf_int_min_`var'   =	coeff_`var'-(1.96*coeff_se_`var') + ctl_mean
			scalar  conf_int_max_`var'   =	coeff_`var'+(1.96*coeff_se_`var') + ctl_mean

			noi test `var'
			local star_`var'

			scalar  pvalue =r(p)
			if pvalue < 0.10 {
				local star_`var' "*"
			}
			if pvalue < 0.05 {
				local star_`var' "**"
			}
			if pvalue < 0.01 {
				local star_`var' "***"
			}
			noi display "`star_`var''"
			scalar tmt_mean_`var' = ctl_mean + coeff_`var'
		}

	tempname 	newHandle
	tempfile	newTextFile
	cap file close 	`newHandle'
	file open  	`newHandle' using "`newTextFile'", text write append
	file write `newHandle' ///
		"Variable" 	_tab "Tmt-Status"	_tab "mean" _tab "coeff" _tab "conf_int_min" _tab "conf_int_max" _tab "obs" _tab "star" _n
	file close `newHandle'
	copy "`newTextFile'"  "mainMasterDofile.txt" , replace
}
end


Do not forget to set the path

Do not forget to set the path

// DO NOT FORGET TO SET PATH!
use "101innov.dta", clear

local phase_row analysis
local phase_col write

qui ds tool_`phase_row'_*, has(type byte)
local vlist_row `r(varlist)'
qui ds tool_`phase_col'_*, has(type byte)
local vlist_col `r(varlist)'

local llr=2+wordcount("`vlist_row'")
local llc=2+wordcount("`vlist_col'")

set trace on
set tracedepth 1
mata: T=J(`llr',`llc',.);
local Lr=length("tool_`phase_row'_")+1
local Lc=length("tool_`phase_col'_")+1
local rownames ""
local colnames ""
local ii 0
forval row=1/`llr'	{
	local n=substr("`: word `row' of `vlist_row''",`Lr',.)
	local rownames `rownames' `n'
//	macro list _rownames
	local `++ii'
	forval col=1/`llc'	{
		if `ii'==`col'	{
			local n=substr("`: word `col' of `vlist_col''",`Lc',.)
			local colnames `colnames' `n'
//			macro list _colnames
		}
		// if one-but-last column
		if `col'==`=`llc'-1'	{
			if `row'>=`=`llr'-1'	continue
			summ `: word `row' of `vlist_row'', mean
			scalar p=r(mean)
			mata: T[`row',`col']=st_numscalar("p")
			continue
		}
		// if last column
		if `col'==`llc'	{
			if `row'>=`=`llr'-1'	continue
			summ `: word `row' of `vlist_row'', mean
			scalar s=r(sum)
			mata: T[`row',`col']=st_numscalar("s")
			continue
		}
		// if one-but-last row
		if `row'==`=`llr'-1'	{
			if `col'>=`=`llc'-1'	continue
			summ `: word `col' of `vlist_col'', mean
			scalar p=r(mean)
			mata: T[`row',`col']=st_numscalar("p")
			continue
		}
		// if last row
		if `row'==`llr'	{
			if `col'>=`=`llc'-1'	continue
			summ `: word `col' of `vlist_col'', mean
			scalar s=r(sum)
			mata: T[`row',`col']=st_numscalar("s")
			continue
		}
		summ `: word `col' of `vlist_col'' if `: word `row' of `vlist_row''==1, mean
		scalar p=r(mean)
		mata: T[`row',`col']=st_numscalar("p")

	}
}
set trace off
mata:st_matrix("T",T)
matrix rownames T=`rownames' Average Users
matrix colnames T=`colnames' Average Users
matrix list T, format(%3.2f)

use 101innov.dta", clear

local phase_row analysis
local phase_col write

qui ds tool_`phase_row'_*, has(type byte)
local vlist_row `r(varlist)'
qui ds tool_`phase_col'_*, has(type byte)
local vlist_col `r(varlist)'

local llr=wordcount("`vlist_row'")
local llc=wordcount("`vlist_col'")

//set trace on
set tracedepth 1
mata: T=J(`llr',`llc',.);
local Lr=length("tool_`phase_row'_")+1
local Lc=length("tool_`phase_col'_")+1
local rownames ""
local colnames ""
local ii 0
forval row=1/`llr'	{
	local n=substr("`: word `row' of `vlist_row''",`Lr',.)
	local rownames `rownames' `n'
//	macro list _rownames
	local `++ii'
	forval col=1/`llc'	{
		if `ii'==`col'	{
			local n=substr("`: word `col' of `vlist_col''",`Lc',.)
			local colnames `colnames' `n'
//			macro list _colnames
		}
		summ `: word `col' of `vlist_col'' if `: word `row' of `vlist_row''==1, mean
		scalar p=r(mean)
		mata: T[`row',`col']=st_numscalar("p")
	}
}
//set trace off
mata:st_matrix("T",T)
matrix rownames T=`rownames'
matrix colnames T=`colnames'
matrix list T, format(%3.2f)


change setup

change setup

/*-------------------------------------------------------------------------------
# Name:		00_SetupFolderGlobals
# Purpose:	Create series of folders Food for Uganda Vulnerability Analysis
# Author:	Tim Essam, Ph.D.
# Created:	10/31/2014; 02/19/2015.
# Owner:	USAID GeoCenter | OakStream Systems, LLC
# License:	MIT License
# Ado(s):	see below
#-------------------------------------------------------------------------------
*/

/* RUN/READ ME FIRST -- Make directories for the study.
 1. Requires root name for file folder structure
 2. Requires branch names for sub-folders
 3. Sets global macros for the study; These are used through the do files.
 4. TODO -- Script DOES NOT copy in raw data at this point.
 5. TODO -- Create program folder for calling custom programs.
 6. TODO -- Convert setup .do file into a program (inputs are project path & folder name)
*/

* install the confirm directory ado if not already installed
* list all known user-written .ados needed for project
local required_ados confirmdir mvfiles fs spatgsa  adolist labellist winsor2
foreach x of local required_ados {
	capture findfile `x'.ado
		if _rc==601 {
			cap ssc install `x'
		}
		else disp in yellow "`x' currently installed."
	}
*end

* Determine path for the study
* global projectpath "U:\"
* global projectpath "C:\Users\Tim\Documents\"
global projectpath "C:\Users\t\Documents\"
cd "$projectpath"

* Run a macro to set up study folder (needs to be modified)
* Name the file path below -- replace "nigerlsms" with your folder name
local pFolder Uganda
foreach dir in `pFolder' {
	confirmdir "`dir'"
	if `r(confirmdir)'==170 {
		mkdir "`dir'"
		display in yellow "Project directory named: `dir' created"
		}
	else disp as error "`dir' already exists, not created."
	cd "$projectpath\`dir'"
	}
* end

* Run initially to set up folder structure
* Choose your folders to set up as the local macro `folders'
local folders Rawdata Stata Datain Log Output Dataout Excel PDF Word Graph GIS Export R Python Programs Sensitive_Data FinalProducts
foreach dir in `folders' {
	confirmdir "`dir'"
	if `r(confirmdir)'==170 {
			mkdir "`dir'"
			disp in yellow "`dir' successfully created."
		}
	else disp as error "`dir' already exists. Skipped to next folder."
}
*end

/*---------------------------------
# Set Globals based on path above #
-----------------------------------*/
global date $S_DATE
local dir `c(pwd)'
global path "`dir'"
global pathdo "`dir'\Stata"
global pathdo2 "`dir'\Github\Uganda\Stata"
global pathlog  "`dir'\Log"
global pathin "`dir'\Datain"
global pathout "`dir'\Dataout"
global pathgraph "`dir'\Graph"
global pathxls "`dir'\Excel"
global pathreg "`dir'\Output"
global pathgis "`dir'\GIS"
global pathraw "`dir'\Rawdata"
global pathexport "`dir'\Export"
global pathR "`dir'\R"
global pathPython "`dir'\Python"
global pathProgram "`dir'\Program"
global pathPdrive "P:\GeoCenter\GIS\Projects\LVAM\Uganda"
global pathSensitive "`dir'\Sensitive_Data"
global pathProducts "`dir'\FinalProducts"
* Project macros are defined as:
macro list

* Add in subfolders for GIS and Final Products Folders
cd "$pathgis"
local gisfolders FeatureClass Shapefiles csv Layers mxd Rasters scratch
foreach dir in `gisfolders' {
	confirmdir "`dir'"
	if `r(confirmdir)'==170 {
			mkdir "`dir'"
			disp in yellow "`dir' successfully created."
		}
	else disp as error "`dir' already exists. Skipped to next folder."
}
*end

cd "$pathProducts"
local subfolders AI PDFs PosterJPG PPTJPG
foreach dir in `subfolders' {
	confirmdir "`dir'"
	if `r(confirmdir)'==170 {
			mkdir "`dir'"
			disp in yellow "`dir' successfully created."
		}
	else disp as error "`dir' already exists. Skipped to next folder."
}
*end

/*------------------------------------------------------------
# Manually copy raw data  into Datain Folder #

/*-------------------------------------------------------------------------------
# Name:		00_SetupFolderGlobals
# Purpose:	Create series of folders Food for Uganda Vulnerability Analysis
# Author:	Tim Essam, Ph.D.
# Created:	10/31/2014; 02/19/2015.
# Owner:	USAID GeoCenter | OakStream Systems, LLC
# License:	MIT License
# Ado(s):	see below
#-------------------------------------------------------------------------------
*/

/* RUN/READ ME FIRST -- Make directories for the study.
 1. Requires root name for file folder structure
 2. Requires branch names for sub-folders
 3. Sets global macros for the study; These are used through the do files.
 4. TODO -- Script DOES NOT copy in raw data at this point.
 5. TODO -- Create program folder for calling custom programs.
 6. TODO -- Convert setup .do file into a program (inputs are project path & folder name)
*/

* install the confirm directory ado if not already installed
* list all known user-written .ados needed for project
local required_ados confirmdir mvfiles fs spatgsa  adolist labellist winsor2
foreach x of local required_ados {
	capture findfile `x'.ado
		if _rc==601 {
			cap ssc install `x'
		}
		else disp in yellow "`x' currently installed."
	}
*end

* Determine path for the study
*global projectpath "U:\"
global projectpath "C:\Users\Tim\Documents\"
cd "$projectpath"

* Run a macro to set up study folder (needs to be modified)
* Name the file path below -- replace "nigerlsms" with your folder name
local pFolder Uganda
foreach dir in `pFolder' {
	confirmdir "`dir'"
	if `r(confirmdir)'==170 {
		mkdir "`dir'"
		display in yellow "Project directory named: `dir' created"
		}
	else disp as error "`dir' already exists, not created."
	cd "$projectpath\`dir'"
	}
* end

* Run initially to set up folder structure
* Choose your folders to set up as the local macro `folders'
local folders Rawdata Stata Datain Log Output Dataout Excel PDF Word Graph GIS Export R Python Programs Sensitive_Data FinalProducts
foreach dir in `folders' {
	confirmdir "`dir'"
	if `r(confirmdir)'==170 {
			mkdir "`dir'"
			disp in yellow "`dir' successfully created."
		}
	else disp as error "`dir' already exists. Skipped to next folder."
}
*end

/*---------------------------------
# Set Globals based on path above #
-----------------------------------*/
global date $S_DATE
local dir `c(pwd)'
global path "`dir'"
global pathdo "`dir'\Stata"
global pathdo2 "`dir'\Github\Uganda\Stata"
global pathlog  "`dir'\Log"
global pathin "`dir'\Datain"
global pathout "`dir'\Dataout"
global pathgraph "`dir'\Graph"
global pathxls "`dir'\Excel"
global pathreg "`dir'\Output"
global pathgis "`dir'\GIS"
global pathraw "`dir'\Rawdata"
global pathexport "`dir'\Export"
global pathR "`dir'\R"
global pathPython "`dir'\Python"
global pathProgram "`dir'\Program"
global pathPdrive "P:\GeoCenter\GIS\Projects\LVAM\Uganda"
global pathSensitive "`dir'\Sensitive_Data"
global pathProducts "`dir'\FinalProducts"
* Project macros are defined as:
macro list

* Add in subfolders for GIS and Final Products Folders
cd "$pathgis"
local gisfolders FeatureClass Shapefiles csv Layers mxd Rasters scratch
foreach dir in `gisfolders' {
	confirmdir "`dir'"
	if `r(confirmdir)'==170 {
			mkdir "`dir'"
			disp in yellow "`dir' successfully created."
		}
	else disp as error "`dir' already exists. Skipped to next folder."
}
*end

cd "$pathProducts"
local subfolders AI PDFs PosterJPG PPTJPG
foreach dir in `subfolders' {
	confirmdir "`dir'"
	if `r(confirmdir)'==170 {
			mkdir "`dir'"
			disp in yellow "`dir' successfully created."
		}
	else disp as error "`dir' already exists. Skipped to next folder."
}
*end

/*------------------------------------------------------------
# Manually copy raw data  into Datain Folder #


Pushed from snowflake Tue Jul 26 07:35:02 PDT 2016

= to.do: =

  o bloggish directories should have a "make wc" target that subtracts the word count of
    the template file.  Or, better, use sed to filter out tags.

========================================================================================>
Work Log:
========

2015:
=====

1111We
  @ OpenSSH/Cookbook/Multiplexing - Wikibooks
    Makes ssh much faster, and circumvents dreamhost's connection limits.
    > mkdir ~/.ssh/controlmasters; chmod 700 ~/.ssh/controlmasters
  * Add the following to .ssh/config, in the global section at the top:
      ControlPath ~/.ssh/controlmasters/%r@%h:%p
      ControlPersist 10m
      ControlMaster auto
  @ https://github.com/blipvert/ljupdate/commits/master marginally more up to date
    * update emacs/depends.make

2016:
=====

0214Su
  * much hacking in gnome-setup and font setup over the last week or so.

0215Mo
  * added adobe-source-code-pro font, tell emacs to use system font.

... Really ought to copy the config-related stuff out of starport/Admin/admin.to.do

0626
  ~ alltray looked useful - lets you put an icon for any program into the tray.
    It starts the program if it's not already running, but fails in xmonad because it's
    waiting for the program to be iconified.
  * gsimplecal, fbpanel, and probably others have subdirectories in .config
    probably best is to make another directory called dotconfig, parallel to dotfiles

0627Mo
  * made the Makefiles independendent of Tools/Makefile.

0704Mo
  * git subtree add --prefix bin ../bin master
  * The idea is that if there's a pre-existing ~/bin, we can symlink into it.

0705Tu
  * regular dotfiles should start with "_" rather than ".", to make them visible.
    -> move dotfiles/.* -> dotfiles/_*; update dotfiles/Makefile

0716Sa
  . Some dotfiles may need to be modified by a script rather than simply symlinked.
    That's because things like username and email address change from one system to
    another.  In most cases, though, one can use include files generated by make install.

0719Tu
  * When site-dependent parameters can be in separate files (include files/modules).
    the various install targets can initialize them from templates.  A good place to start
    would be .xmonad/Local.hs.  Could also do this for .bashrc; a conditional include
    would make it unnecessary to generate the file if it's not needed. (0716)
    : for gitconfig - note that the file doesn't need to exist
      [include]
	  path = /path/to/file

0725Mo
  * color-coded workspace labels.  ws N makes an xclock with WS=N and the appropriate
    resistor color code.
  * the ability to make ad-hoc workspace labels might help.  Put the list in a file that
    can be read in either xmonad or the bottom-bars script.  Possibly a top dbar with
    vertical menu. (0719) -> add a second argument to ws.
    Re-running ws N should grep ps for the appropriate color code and kill it.
    An alternative to xclock would be dzen -p.

========================================================================================>|
Local Variables:
    fill-column:90
End:

= to.do: =

  o bloggish directories should have a "make wc" target that subtracts the word count of
    the template file.  Or, better, use sed to filter out tags.

  o the ability to make ad-hoc workspace labels might help.  Put the list in a file that
    can be read in either xmonad or the bottom-bars script.  Possibly a top dbar with
    vertical menu. (0719)

========================================================================================>
Work Log:
========

2015:
=====

1111We
  @ OpenSSH/Cookbook/Multiplexing - Wikibooks
    Makes ssh much faster, and circumvents dreamhost's connection limits.
    > mkdir ~/.ssh/controlmasters; chmod 700 ~/.ssh/controlmasters
  * Add the following to .ssh/config, in the global section at the top:
      ControlPath ~/.ssh/controlmasters/%r@%h:%p
      ControlPersist 10m
      ControlMaster auto
  @ https://github.com/blipvert/ljupdate/commits/master marginally more up to date
    * update emacs/depends.make

2016:
=====

0214Su
  * much hacking in gnome-setup and font setup over the last week or so.

0215Mo
  * added adobe-source-code-pro font, tell emacs to use system font.

... Really ought to copy the config-related stuff out of starport/Admin/admin.to.do

0626
  ~ alltray looked useful - lets you put an icon for any program into the tray.
    It starts the program if it's not already running, but fails in xmonad because it's
    waiting for the program to be iconified.
  * gsimplecal, fbpanel, and probably others have subdirectories in .config
    probably best is to make another directory called dotconfig, parallel to dotfiles

0627Mo
  * made the Makefiles independendent of Tools/Makefile.

0704Mo
  * git subtree add --prefix bin ../bin master
  * The idea is that if there's a pre-existing ~/bin, we can symlink into it.

0705Tu
  * regular dotfiles should start with "_" rather than ".", to make them visible.
    -> move dotfiles/.* -> dotfiles/_*; update dotfiles/Makefile

0716Sa
  . Some dotfiles may need to be modified by a script rather than simply symlinked.
    That's because things like username and email address change from one system to
    another.  In most cases, though, one can use include files generated by make install.

0719Tu

  * When site-dependent parameters can be in separate files (include files/modules).
    the various install targets can initialize them from templates.  A good place to start
    would be .xmonad/Local.hs.  Could also do this for .bashrc; a conditional include
    would make it unnecessary to generate the file if it's not needed. (0716)
    : for gitconfig - note that the file doesn't need to exist
      [include]
	  path = /path/to/file

==============================================================================
Local Variables:
    fill-column:90
End:


Typos correction

Typos correction

	** The globals in this file are made up globals for this example. See the
	*  DIME Wiki article on file paths for explanation.

	* Load monitor raw data

	import delimited using "$monitorRaw/farmer_monitoring_data.csv"

	*Common different spelling
	replace first_name = "Christopher" 	if first_name == "Christoffer" 	& last_name == "Smith" 		 & village == "River Creek"

	*Translitteration discrepancies from languages with different script
	replace first_name = "Mahmood" 		if first_name == "Mahmod" 		& last_name == "Al-Meghribi" & village == "Mountain Bend"

	*Applying the case convention used in the master data set
	replace first_name = "Rose" 		if first_name == "ROSE" 		& last_name == "Jones" 		 & village == "Meadow Spring"

	/*Merge the monitoring data set with the master data set. This step will
		require a lot of preparation and it is common that one could have to go
		back and make numerous additions above before the merge works without
		errors, but after this work is done here, it is done once and for all and
		your data sets will merge and append effortlessly throughout your project.
		Here some important details on the command below:
		-Only use 1:1 matching in this case. Stata will then give you an
		 error if there is any ambiguity in which farmer is which. You have
		 to add more identifying varaibles or solve ambiguous cases manually
		 before you can proceed.
		-assert(using match). This option tests that all observation after the
		 merge is either in both data sets (match) or only in the master data
		 set (using). We should not proceede if an observation is only found in
		 the monitor data set. For all those cases we want to follow up manually.
		 We want to see if there is a typo that causing that the observation is
		 not matched. If we are 100% convinced that this is an observation we
		 have not encountered before, then we should go back and add that
		 obvservation to the data set before proceeding with this script. In
		 order to know what observations cause an error in assert(match using)
		 you can remove that option, run the code and inspect the variable _merge.
		 However, your task is not complete before you can run the command below
		 when you include assert(match using).
		-keep(match). In the end we do not want to add any observation from the
		 master data set to the monitoring data set, so we only keep the
		 observations that are in both.
		-keepusing(farmer_ID). We do not need to copy all variables from the
		 master data set. This option allows us to list the varaibles we want to
		 include. The examples given here is the ID variable and a dummy for being
		 in the baseline sample. We always want to include the ID variable, but
		 what other variibles you include varies between every project.

	*/
	merge 1:1 first_name last_name village using "$masterFinal/farmer_master_data_set.dta",  assert(match using) keep(match) keepusing(farmer_ID baseline_sample)

	**Save data with an ID that we can use when merging this data set to any
	* other data set in our project.
	save "$monitorInt/farmer_monitoring_data.dta"

	** The globals in this file are made up globals for this example. See the
	*  DIME Wiki article on file paths for explination.

	* Load monitor raw data

	import delimited using "$monitorRaw/farmer_monitoring_data.csv"

	*Common different spelling
	replace first_name = "Christopher" 	if first_name == "Christoffer" 	& last_name == "Smith" 		 & village == "River Creek"

	*Translitteration decreptancies from languages with different script
	replace first_name = "Mahmood" 		if first_name == "Mahmod" 		& last_name == "Al-Meghribi" & village == "Mountain Bend"

	*Applying the case convention used in the maste data set
	replace first_name = "Rose" 		if first_name == "ROSE" 		& last_name == "Jones" 		 & village == "Meadow Spring"

	/*Merge the monitoring data set with the master data set. This step will
		require a lot of preperation and it is common that one have to go back
		and make numerous additions above before the merge works without errors,
		but after this work is done here, it is done once and for all and your
		data sets will merge and append effortlessly throughout your project.
		Here some important detail on the command below:
		-Only use 1:1 matching in this case. Stata will then give you an
		 error if there is any ambiguity in which farmer is which. You have
		 to add more identifying varaibles or solve ambiguis cases manually
		 before you can proceede
		-assert(using match). This options test that all observation after the
		 merge is either in both data sets (match) or only in the master data
		 set (using). We should not proceede if an observation is only found in
		 the monitor data set. For all those cases we want to follow up manually.
		 We want to see if there is a typo that causing that the observation is
		 not matched. If we are 100% convinced that this is an observation we
		 have not encountered before, then we should go back and add that
		 obvservation to the data set before proceeding with this script. In
		 order to know what observations casue an error ins assert(match using)
		 you can remove that option, run the code and inspect the variable _merge.
		 However, your task is not complete before you can run the command below
		 when you include assert(match using).
		-keep(match). In the end we do not want to add any observation from the
		 master data set to the monitoring data set, so we only keep the
		 observations that are in both.
		-keepusing(farmer_ID). We do not need to copy all variables from the
		 master data set. This option allows us to list the varaibles we want to
		 include. The examples given here is the ID variable and a dummy for being
		 in the baseline sample. We always want to include the ID variable, but
		 what other variibles you include varies between every project.

	*/
	merge 1:1 first_name last_name village using "$masterFinal/farmer_master_data_set.dta",  assert(match using) keep(match) keepusing(farmer_ID baseline_sample)

	**Save data with an ID that we can use when merging this data set to any
	* other data set in our project.
	save "$monitorInt/farmer_monitoring_data.dta"


updated stata file

updated stata file

* script to read in results of R spatial microsim using ipf and run weighted stats
* in theory this should also be done in R when I work out how!

* change this to your path!
local where "/Users/ben/Documents/Work/Papers and Conferences/spatial-microsim-r-course"
local path "`where'/spatial-microsim-book-git/fractional_weights"

insheet using "`path'/final_micro_fractional_cakes_geo.csv", clear

* create 'dummy' variables ready to collapse to weighted frequencies
gen NCakes_rare = 0
replace NCakes_rare = 1 if ncakes == "rarely"

gen NCakes_l1 = 0
replace NCakes_l1 = 1 if ncakes == "<1"

gen NCakes_1_2 = 0
replace NCakes_1_2 = 1 if ncakes == "1-2"

gen NCakes_3_5 = 0
replace NCakes_3_5 = 1 if ncakes == "3-5"

gen NCakes_6m = 0
replace NCakes_6m = 1 if ncakes == "6+"

* keep the data in memory
* immediately obvious = stata's inability to hold 2 or more datasets at a time!
preserve
* collapse cakes by zone
* this is what the weighted agregate in R should do
collapse (sum) NC* [iw=weight], by(zone_name)

* list first 5 lines as a check
li in 1/5

* save the results so we can add to a map
outsheet using "`path'/cakes_geo.csv", comma replace

* put the data back so we can do other stuff
restore

* script to read in results of R spatial microsim using ipf and run weighted stats
* in theory this should also be done in R when I work out how!

* change this to your path!
local where "/Users/ben/Documents/Work/Papers and Conferences/spatial-microsim-r-course"
local path "`where'/spatial-microsim-book-fork-git"

insheet using "`path'/final_micro_fractional_cakes_geo.csv", clear

* create 'dummy' variables ready to collapse to weighted frequencies
gen NCakes_rare = 0
replace NCakes_rare = 1 if ncakes == "rarely"

gen NCakes_l1 = 0
replace NCakes_l1 = 1 if ncakes == "<1"

gen NCakes_1_2 = 0
replace NCakes_1_2 = 1 if ncakes == "1-2"

gen NCakes_3_5 = 0
replace NCakes_3_5 = 1 if ncakes == "3-5"

gen NCakes_6m = 0
replace NCakes_6m = 1 if ncakes == "6+"

* keep the data in memory
* immediately obvious = stata's inability to hold 2 or more datasets at a time!
preserve
* collapse cakes by zone
* this is what the weighted agregate in R should do
collapse (sum) NC* [iw=weight], by(zone_name)

* list first 5 lines as a check
li in 1/5

* save the results so we can add to a map
outsheet using "`path'/cakes_geo.csv", comma replace

* put the data back so we can do other stuff
restore


minor bugfix

minor bugfix


Update Country_SpecificQs.do

Update Country_SpecificQs.do

** this program reads in the grand merge file and prints each country specific question code to the console
** read the grand_merge file
use "C:\Users\zoorobmj\Desktop\grand_merge.dta"

* this loop will get every country specific question
** to instead find any particular country, delete outer loop and replace pais with number equal to pais of interest (line 13)
*** plausibly one could toggle with r(c) == [num] if you wanted to, say, find questions asked in <5 countries, etc
**** note that if you do not neeed to know which qs go with which country, delete outer loop
***** and 2nd if statement for drastically improved performance.
foreach i of num 1/17 21/29 40 41 {
	display "These questions are for Country ", "`i'"
	foreach var of varlist q1-q11nr {
		capture quietly tab `var' pais
		* I know it's country specific if r(c) == 1
		if r(c) == 1 {
		quietly sum `var' if pais == `i'
		* if it's ctry specific AND has results for Nicaragua, it is a Nicaragua-only question
			if r(N) > 0 {
				display "`var'"
				}
		}
	}
}

*** simple version (does not identify countries)
foreach var of varlist q1-q11nr {
	capture quietly tab `var' pais
	* I know it's country specific if r(c) == 1
	if r(c) == 1 {
		display "`var'"
		}
}