# SimBA Tutorial:

To faciliate the initial use of SimBA, we provide several use scenarios. We have created these scenarios around a hypothetical experiment that take a user from initial use (completely new start) all the way through analyzing a complete experiment and then adding additional experimental datasets to an initial project.

All scenarios assume that the videos have been [pre-processed](https://github.com/sgoldenlab/simba/blob/master/docs/tutorial_process_videos.md) and that [DLC behavioral tracking .CSV dataframes](https://github.com/sgoldenlab/simba/blob/master/docs/Tutorial_DLC.md) have been created.

# **Hypothetical Experiment**:
Three days of resident-intruder testing between aggressive CD-1 mice and subordinante C57 intruders. Each day of testing has 10 pairs of mice, for a total of 30 videos recorded across 3 days. Recordings are 3 minutes in duration, in color, at 30fps.

# **Scenario 3**: Updating a classifier with further annotated data.
In this Scenario you have generated a classifier using SimBA, but its possible (or, likely) that the classifier could be improved by including more training data. We suggest that you iterate between Scenario 1 and Scenerio 3 a few times before using a classifier on Experimental data, as done in Scenario 2. The goal is that you will only need to create a classifier once, so the better it is before experimental use, the less work you will have to do in the future!

>**Note 1**: If you have received a classifier from another lab, or have notably changed the experimental recording configuration in your own lab, you will likely need create a new version of your classifier that now includes training examples from your own lab's behavioral setup or from the new recording conditions. While classifiers can genreralize well, they will always do better if the training dataset includes examples from your own behavioral setup! If your predictions are not going well with someone else's classifier, or after you have changed your behavioral equipment, this is a very likely reason and and easy fix.

>**Note 2**: For the above reason, it is very important that you create clear operational definitions of annotated behaviors. Please see these examples for [mouse](https://github.com/sgoldenlab/simba/blob/master/misc/Operational_definitions_mouse_resident_intruder.pdf) and [rat](https://github.com/sgoldenlab/simba/blob/master/misc/Operational_definitions_rat_resident_intruder.pdf) predictive classifiers for aggressive behaviors in resident-intruder protocols.  

>**Note 3**: If you plan to share a predictive classifier with other users, you will need to share the operationalized definitions, the `project_folder/csv/targets_inserted` CSV files for **all** the videos used in the initial training set, and the *video_info.csv* file generated by SimBA which is located in the `project_folder\logs` directory (further detailed below in *Part 1*).

*HINT*: SimBA inludes several model evaluation tools that cant be selected by the user when generating the predictive classfiers. Among other things, this includes learning curves, which indicates how the predictive classifier performs with increasing amounts of data. For more information on how to use SimBa to generate learning curves for your predictive classifier, click [here](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md#step-7-train-machine-model). For more general information on learning curves in machine learning and their potential for informing on the benefit of additional data, click [here](https://scikit-learn.org/stable/auto_examples/model_selection/plot_learning_curve.html). 

In this tutorial, we will use SimBA to add further annotations to the predictive classifier generated in [Scenario 1](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md), and create an new classifier that now includes the training data from [Scenario 1](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md) as well as the new additional data. 

>*Note:* In practice, what we are doing in this Scenario is to generate a brand new classifier that contains previously annotated data as well as new annotated data, rather than building on an already generated classifier.    

## Part 1: Previously annotated CSV files. 

**1. Locate the previously annotated CSV files**. We will need to start with a project directory tree that does not contain any other data than the data we want to use to create the new predictive classifier. If you are coming along from [Scenario 1](https://github.com/sgoldenlab/simba/edit/master/docs/Scenario1.md) or [Scenario 2](https://github.com/sgoldenlab/simba/edit/master/docs/Scenario2.md) you will have one or more project directory trees already. The project tree from [Scenario 1](https://github.com/sgoldenlab/simba/edit/master/docs/Scenario1.md) will contain critical files that we want to use when creating the new predictive classifier in this current Scenario 3. The critical files are located in the `project_folder/csv/targets_inserted` directory of the [Scenario 1](https://github.com/sgoldenlab/simba/edit/master/docs/Scenario1.md) project. In this folder, there is one CSV file for each video, and each CSV file represents a video. Each CSV file contains the (i) pose-estimation data, (ii) the extracted features, and (iii) annotated behaviours that were created using the [SimBA behavioral annotation GUI](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md#step-6-label-behavior-ie-create-annotations-for-predictive-classifiers). Copy these CSV files to a safe memorable place, or alternatively, just be aware of their location. We will use them in later in this Scenario 3.

>**Note**: If a predictive classifier has been generated by someone else, and you want to add further data to this classifer in order to generate a new classifier with more training data, you will need three things:

- You will need the operational definitions of the behaviors, so that you can score the behaviors identically.

- You will need the files from the *targets_inserted* folder from the project where the classifier was generated. 

- You will also need information on the resolution, pixels per mm, and fps from the videos used to create the original predictive classifier. This infomation is stored in the *video_info.csv* file generated by SimBA which is located in the `project_folder\logs` directory.

- You **do not** have to have the frames, or the video files, for the videos that *were used to generate the original classifier*.

## Part 2: Create a project. 

To create a new project, follow the instructions for creating a new project in either of these tutorials: [1](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md#part-1-create-a-new-project-1), [2](https://github.com/sgoldenlab/simba/blob/master/docs/tutorial.md#step-1-generate-project-config). Instead of importing the 'pilot' data as indicated in the tutorial from [Scenario 1](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md#part-1-create-a-new-project-1), import the pose estimation data and videos you want to 'add' to the predictive classifier. 

**Important I**: In the final portion of the tutorials for creating a new project (*Step 4*), we extract the frames from the videos we have imported into the project. Having the frames is necessery in this current Scenario 3, as we will render them for annotating behaviors in the [SimBA behavioral annotation GUI](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md#step-6-label-behavior-ie-create-annotations-for-predictive-classifiers). However, you **do not** have to have the frames for the videos that *were used to generate the original classifier*. 

**Important II**: Make sure the file names of the new videos, and new DeepLabCut pose estimation data, are not identical to the file names of the previously annotated videos.  

**Important III**: Make sure the the classifier name(s), which is defined when [creating the project](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md#step-1-generate-project-config) in the SML Settings menu, is **the same** as the classifier name used to create the previous classifier. 

## Part 3: Load project and process your new tracking data

In Part 2 above, we created a project that contains the data you wish to add to the predictive classifier. To continue working with this project, we **must** load it. To load the project and process your new data, follow the instructions for **Step 1-6** in either of these tutorials: [1](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md#part-1-create-a-new-project-1), [2](https://github.com/sgoldenlab/simba/blob/master/docs/tutorial.md#step-1-generate-project-config). 

When you have completed **Step 1-6**, you will have (i) corrected the outliers present in your new data, (ii) extracted features from your new data, (iii) and annotated behaviours in your new data. For example, if you annotated an additional, new, dataset contain 5 videos, you should now see 5 CSV files in the `project_folder\csv\targets_inserted` directory.

## Part 4: Add the previously annotated CSV files to your `project_folder\csv\targets_inserted` directory.

We have now created a new project and have annotated data in this project, and we now want to import the annotations used to create the previous classifier into this project. **We will do this manually**. 

**1. Navigate** to the folder that stores the previously annotated CSV files (see **Part 1**). Copy and paste them into your current current project into the `project_folder\csv\targets_inserted` directory. 

You now have all the annotated files that will be used to create your predictive classifier in one place. 

**2. Create a new, updated, video_info.csv file**. In the the current project directory, the *video_info.csv* file generated by SimBA,  which is located in the `project_folder\logs` directory, will so far only include the information from the new videos that you wish to *add* to the classifier. It does not contain the information from the videos that were used to create the previous predictive classifier. To update the *video_info.csv* file so that it contains both the new and the older video information, first open the *video_info.csv* file from the **earlier project** (see **Step 1**) and copy its content. Next, open the *video_info.csv* file for the **current project**  and paste the content into the file. You should now have a *video_info.csv* that contains a row of information for each of the CSV files represented in the `project_folder\csv\targets_inserted` directory. The content of the file may look similar to this:

![](/images/video_info.JPG "video_info")

>*Note I*: In the example screenshoot above the videos have the same resolution, pixels per millimeter, and fps. However, they can be in any different combination of values across different videos. 

>*Note II*: Make sure that the column *Video* (the first column on the left), has the names of the video files in the `project_folder\csv\targets_inserted` directory (excluding the file format, .csv).    

## Part 5: Generate a new predictive classifier with the new dataset. 

You now have all the necessery data organised and you are ready to generate a new predictive classifier that includes the data used to generate the original classfier, plus the data from new annotated videos. To generate a new predictive classifier with these data, see [**Step 7**](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md#step-7-train-machine-model) and onwards in the [Scenario 1](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md) tutorial.

Once you have completed **Part 5** of the current Scenario 3, you may want to proceed to either:

- Visualize and validate the new classifier on a video not used during training of the classifier. For validating a classifier on a new video, see Scenario 1 - [Critical validation step before running machine model on new data](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario1.md#critical-validation-step-before-running-machine-model-on-new-data).

- Run the new classifier on new data. To run the classifier on new data, see Scenario 2 - [Using a classifier on new experimental data](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario2.md#scenario-2-using-a-classifier-on-new-experimental-data).

- Visualize machine predictions on one or several videos. To create visualizations, see Scenario 2 - [Visualizing machine predictions](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario2.md#part-5--visualizing-machine-predictions). *Note*: You can only visualize the predictions on videos for which you have extracted the frames and have feature data for. For more information on extracting frames - click [here](https://github.com/sgoldenlab/simba/blob/master/docs/tutorial.md#step-4-extract-frames-into-project-folder).

**Congratulations! You now have a new classifier of your behavior that is better than the original, or that has been updated from another lab's behavioral setup to work with your own! If you are following the Scenarios, you also now are ready to add the next two days of Experimental data to your project. Please see [Scenario 4](https://github.com/sgoldenlab/simba/blob/master/docs/Scenario4.md) for instructions.**


##
Author [Simon N](https://github.com/sronilsson)
