Report: Dynamic Eye Movement Matching and Visualization Tool in Neuro Gesture
Qiangeng Xu John Kender
cs.NE cs.AI cs.CV




headings


 

Dynamic Eye Movement Matching and Visualization Tool in Neuro Gesture




Computer Science Department,

	Columbia University

	qx2128@columbia.edu, jrk@cs.columbia.edu


 
Report: Dynamic Eye Movement Matching and Visualization Tool in Neuro Gesture
    Qiangeng Xu , John R. Kender
    December 30, 2023
=============================================================================


In the research of the impact of gestures using by a lecturer, one challenging task is to infer the attention of a group of audiences. Two important measurements that can help infer the level of attention are eye movement data and Electroencephalography (EEG) data. Under the fundamental assumption that a group of people would look at the same place if they all pay attention at the same time, we apply a method, "Time Warp Edit Distance", to calculate the similarity of their eye movement trajectories. Moreover, we also cluster eye movement pattern of audiences based on these pair-wised similarity metrics. Besides, since we don't have a direct metric for the "attention" ground truth, a visual assessment would be beneficial to evaluate the gesture-attention relationship. Thus we also implement a visualization tool.







§ INTRODUCTION


It is well studied that people's gazing position would reveal the visual stimuli that people receive. The saliency of an item, defined by the state or quality by which it stands out relative to its neighbors is considered to be a key attention mechanism that facilitates in many learning tasks <cit.>. Several recent neural science studies have discovered specific tissues and neurons that would support such mechanism. For example, Snow et al.<cit.> studied pulvinar nuclei which modulates physical/perceptual salience in attentional selection. Besides, Baliki et al.<cit.> discovered D1-type medium spiny neurons and D2-type medium spiny neurons within the NAcc shell assigns aversive motivational salience to aversive stimuli. 

Several previous studies concentrated on the relation between viewer's attention and visual stimuli such as Loschky et al.<cit.>, Dewhurst et al.<cit.> and Valuch et al.<cit.>. They used eye tracking data in order to discover the similar eye trajectories across viewers. More interestingly, the study introduced by Burleson-Lesser et al.<cit.> has revealed that video material also synchronises judgments and the similarity of eye movement could help predict viewer's preference. 

Thus, under the assumption that higher similarity of eye movement indicates higher attention of a group of audience, we try to use eye tracking data to find the similarity of eye movement and cluster the audiences into subgroups based on their eye movement similarities. We believe such effort would help us to find the more "attractive moment" in a lecture video that would get higher attention from audiences, indicated by more "homogeneous" eye movement clustering result. On the other hand, we would also expect such clustering could help us to find some outlier among audiences(e.g. some people don't pay attention to anything). 

Another work we have done is to re-design the visualization tool previously implemented in Visual Studio. To cope with the diverse working environment that our team members have, we conduct frame-work selection between a few popular candidates. We select Qt in the end and added EEG graph into the visualization tool. The contributions of our work to the Neuro Gesture project are two folds:

	

  
  ∙ We apply Time Warp Edit Distance methods to calculate the pair-wise similarity between each eye movement trajectories pair. Then we use the similarity matrix to cluster the audience eye movement pattern by using relevant communities clustering method introduced by Le Martelot et al.<cit.>
  
  ∙ We implement a visualization tool. The main functionalities include: 1. eye fixation and trails superimposing on the video and 2. 64 channels of EEG graph on sync with the video lapse. After considered several framework, we select Qt, the most compatible cross-platform SDK.
 
 
The rest of the report is organized as follows. We introduce the similarity matrix and clustering method in Section 2 and visualization tool in section 3. In section 2, the related work and model selection would be discussed in 2.1, the detailed algorithm of similarity matrix would be shown in 2.2 and the parameter tuning would be shown in 2.3. The clustering algorithm would be shown in 2.4. The study of correlation between similarity matrix and question answer correctness would be shown in 2.5. In section 3, we introduce the visualization tool's design in 3.1 and illustrate the display in 3.2. In the end, we summarize our work and future challenges in section 4.
	  


§ EYE MOVEMENT SIMILARITY MATCHING AND CLUSTERING


The form of eye-tracking data is essentially multi-dimensional time-series data of multiple objects. Thus here we discuss several time-series matching methods and their strength and weakness if applied our task.



 §.§ Related Works and Model Selection



  §.§.§ Inter-subject correlation of eye movement
 is an emerging topic which would lead toward a better understanding of correlated activity between individuals under stimuli. Some previous studies such as Burleson-Lesser et al.<cit.> employed tools from statistical mechanics which is very powerful to explain emergent properties of the local interactions between subjects. They have demonstrated that the audiences' eye movement would be distributed under a balance between randomness and alignment. They modeled the distribution of direction as it is entirely random yet the observed mean μ_i and pair-wise correlations C_ij could be derived. The model is also studied by Mora et al.<cit.>:

    p(σ) ∝ exp(-∑_ih⃗_⃗i⃗·σ_i - ∑_i,j > iJ_ijσ_i·σ_j)

Adopting same measurement in the statistic mechanics studies such as Cavagna et al.<cit.>, Bialek et al.<cit.> and Tkačik et al.<cit.>, they computed the correlation among eye movement using direction of every pair of two viewers:

    C_ij = ⟨σ_i(t)·σ_j(t)⟩_t - μ_i·μ_j
    
    	where: μ_i = ⟨σ_i(t)⟩_t

However, considering our setting of the experiment(we let each viewer look at the lecture video in a individual room), the homophily of eye movement could only be explained by the video content and the attention of an individual at a certain moment. The "interaction" between audiences doesn't have a physical meaning but only serves as an analogy. More importantly, on purpose of inferring the attention of audience to visual stimuli, we choose to rely on fixation instead of eye movement direction. Since We have discovered many cases that would cause viewer having different eye movement direction while actually getting attracted by the same subject. For example, when the camera is shifting and the scene is switching, people's eye gazing position would be random at first, then, concentrated on the same object that camera is switching to. The movement direction during this period would be very diverse, which hinders the fact that viewers actually have similar "attention". Besides, the method used in <cit.> would provide little tolerance to mismatch on temporal dimension. From our observation, individual viewers always have different response time to visual stimuli. Many medical studies such as Jain et al.<cit.> has discovered such variation between individuals. Thus we explore the methods in time series matching by using fixation data to serve our research.



  §.§.§ Time series similarity measures
 is a central task in the domain of analysis, prediction or classification of information unfolding over time. Given time series T_1 and T_2, a similarity function calculate one point of T_1 and one corresponding point of T_2 having the same timestamp belongs to the family of Lock-Step Measure. The lock-step measures including Euclidean Distance matching introduced by Faloutsos et al.<cit.> along with its variants using L_p-norms(including Manhattan norm, Maximum norms, etc.) introduced by Yi et al.<cit.>. The advantage of these methods is their simplicity. The matching map is shown in [Figure 1 is adapted from <cit.>] Figure <ref>(a). However, in the case of eye-tracking data, because of the time  inaccuracy in eye fixation measurement and different response time among individuals, these methods would give us large distance between an audience having faster response and an audience who is relatively slower, even both their trajectories are following the same pattern. 

Another rather novel family of approaches is Threshold-Based Measure. For example, the TQuEST distance introduced by Aßfalg et al.<cit.> is using a threshold parameter τ and transform a time series to threshold-crossing time intervals. Each time interval is treated as a point in two dimensional space. the Minkowski sum of the two sequences served as the similarity. The matching map is shown in Figure <ref>(b). In general this approach would be more suitable to data that has similar amplitude value(fixation of eye gazing point) but different temporal value. However, the eye-tracking data we collect shows diverse pattern both in temporal value and amplitude value.

Other families of approaches also have been well studied. The Feature-Based Measures such as Fourier coefficients matching introduced by Oppenheim et al.<cit.> used discrete Fourier transforms of the raw time series to filter out high-frequency coefficients, making it a efficient methods to remove rapidly fluctuating signal components. Yet, in our case, it also suffered from not being able to adapt to the differences among individual eye responses. The family of Pattern-Based Measure including SpADe introduced by Chen et al.<cit.> matches segments within entire time series by adjusting(shifting or scaling) value in both temporal and amplitude dimensions. 
Although this method concentrates on matching patterns between time series, in our case, shifting the amplitude value(e.g. fixation) would be problematic. Two audiences would generally pay attention to same object(e.g. the lecturer or picture in slides), thus similar fixation, if they are both attracted by the lecture. 

The most effective family of matching methods for our data might be Elastic Measure. Methods in this family generally would allowed matching between one-to-many and one-to-none points. In our case, some of the fixation data points are missing and some people's attention would stay relatively longer in an object or move slower than others. The most popular method in this family would be dynamic time warping(DTW) introduced by Berndt et al.<cit.> The method aligns the time series in the temporal domain that minimize the cost of the matching distance after the alignment. Dynamic programming could be applied recursively to solve this problem. 

    Minimize: d_DTW(X,Y) = D_M,N
     
    	D_i,j = f(x_i,y_j)  + Min{[   D_i,j-1;   D_i-1,j; D_i-1,j-1;           ]. 
    
    	while: i= 1,...,M, j = 1,...,N

In the study of Wang et al.<cit.>,Very few measures have been proposed that systematically outperform DTW for a number of different data sources. The matching map is shown in Figure <ref>(c). However, the DTW method would entirely ignore the time lag between eye trails, which might also be an unwanted behavior. Instead, we study the edit distance time warping methods such as Edit distance on real sequence introduced in <cit.>, Edit
Distance with Real Penalty (ERP) introduced in <cit.> and Time Warp Edit Distance(TWED) introduced by Marteau et al.<cit.>. Among those extensions, TWED is the most suitable one to the eye-tracking data since it incorporates both time stamp differences and editable series matching. TWED includes a deletion penalty λ and stiffness parameter γ to cope with three different conditions each step. The whole objective could be summarized as follows:

    Minimize    : d_TWED(x,y) = δ_M,N
    
        having    :


    δ_λ,γ(A_1^p, B_1^q)    = Min{[ δ_λ,γ(A_1^p-1, B_1^q) + d(a_p, a_p-1) + γ d(t_a_p, t_a_p-1) + λ deleteA;               δ_λ,γ(A_1^p-1, B_1^q-1) + d(a_p, b_q) + d(a_p-1, a_p-1) +;                          γ d(t_a_p,t_b_q) + γ d(t_a_p-1,t_b_q-1)  match; δ_λ,γ(A_1^p, B_1^q-1) + d(b_q-1, b_q) + γ d(t_b_q, t_b_q-1) + λ deleteB;                                                                         ].

The matching map of TWED is shown in Figure <ref>(d).
By looking at the form of TWED, we can easily find the DTW, ERP, LCSS methods are all TWED's special case. We will illustrate the detail of TWED algorithm applied to eye-tracking data in the next section.





  §.§.§ Clustering based on pair-wise relationship between data points
 is a well studied problem with various solutions. However, since the number of group or the granularity of the cluster is unknown, adopting traditional clustering method would force us to determine this number. However, to seek the optimal number of clusters might be an unwanted operation since we want to make the clustering criteria consistent through whole lecture(we separate video to several sub sections to do the clustering independently). Some part of the lecture might simply be more attractive and people would pay more attention to similar object, other parts may not be delivering interesting information thus people starting looking around. Under certain scenario, the cluster number should not be the same but vary accordingly. Thus we adopt community detection to solve this problem. 

Community detection, compared to traditional clustering method, the number of cluster(community) is always unknown. Most of the methods rank communities by using a criterion. Also studied by Simon et al.<cit.>, a dataset often have several levels of granularity resulting various clustering at several resolutions. Small clusters could be generated by fine scale analysis but a large scale analysis could result in larger clusters. Many studies including Le Martelot et al.<cit.> and Huang et al.<cit.> have proposed criteria designated to suffice multi-scale analysis.  However, they are limited in their efficiency and accuracy. In stead, we find the method proposed by Le Martelot et al.<cit.> could enable fast multi-scale community detection on large networks with global and local criteria. Their method have also been effectively adopted in eye-tracking clustering by Burleson-Lesser et al.<cit.>. The detail implementation and result would be introduced in following sections.



 §.§ Method and implementation of eye movement similarity matrix

Our project code Eye-movement-similarity-clustering are released at

<https://github.com/Xharlie/Eye-movement-similarity-clustering>


  §.§.§ Data Preprocessing

:

Our data come from the proprietary software which runs the eye-tracking machine while recording the audience eye fixation. The output file information and format are illustrated in Appendix A. We adopt the preprocessing implementation in Burleson-Lesser et al.<cit.>. We filter the fixation data with 80 millisecond triangular window. Participants would be excluded from the video if they had over 20% missing data in a video. All missing data would be set to 0. A sparse principal component analysis was run on each dimension. The operation inserted a linear interpolation over viewers to missing samples. We also divide the data for a video into 30 second data files. The sample file is shown in the project's data folder. Each file represent x or y position for an object in a 30 seconds session.


  §.§.§ Similarity Matrix
 
:

The frame rate of the eye tracking data could be very high(capable to be more than 1000 frames per second). Even after preprocessing, we still have eye-tracking frame rate around 120 per second. However, since the video's(we use TED talk as an example) frame rate are only 32 frames per second. Thus first of all, we down-sample the eye-tracking data to the same frame rate as the video.
After that, we will calculate the similarity matrix by using TWED. TWED has two hyper parameters: deletion penalty λ and stiffness parameter γ in order to cope with three different conditions each step. The "deletion" penalty or according to Serra et al.<cit.> the "mismatch" penalty is a constant cost setting to compensate the situation whenever we decide one of the series would stay at the same point while the other one would step forward. The stiffness parameter would add a cost proportional to the distance between two series temporal distance at a certain step. In the original literature of the algorithm in <cit.>, the "elastic cost" has the form γ d(t_a_p-1,t_b_q-1). Here, since the distance between t_a_p and t_b_q or t_a_p-1 and t_a_p are proportional to the difference of index no matter what time unit we choose, we can just write the distance of time as the distance of frame indices so that the time unit could be absorbed into γ. Now, the simpler equation for each step is:

    δ_λ,γ(A_1^p, B_1^q)    = Min{[  δ_λ,γ(A_1^p-1, B_1^q) + d(a_p, a_p-1) + γ  + λ deleteA; δ_λ,γ(A_1^p-1, B_1^q-1) + d(a_p, b_q) + d(a_p-1, a_p-1);                                        + 2γ|p-q|  match;   δ_λ,γ(A_1^p, B_1^q-1) + d(b_q-1, b_q) + γ + λ deleteB;                                                         ].

Here we just fill in all the possible steps from bottom left corner to upper right corner(see Fig.<ref>). Each element contains the minimum possible cost to reach that element from the origin. 

The algorithm of TWED in our application is listed in Appendix B The implementation is inside the TWED.py file in our project repository. 

After we enumerate through all combination of eye trail pairs, we would get a N by N distance matrix in which each cell C_i,j contains the TWED result for eye trail i and j.
To transform the distance matrix to similarity matrix, we simply normalize each element by dividing the largest value among all the elements in the matrix and subtracting it by 1. Thus, the similarity matrix we get for a video in a time window would contains all pair-wise similarity values between 0 to 1. The matrix is a symmetric matrix with all value 1 on its diagonal(since the ith eye trail would have 100% similarity with itself). 



  §.§.§ Parameter Selection of TWED
 
:

The hyper-parameter in TWED algorithm λ,γ would affect the importance of temporal step matching. With λ,γ both set to zero, the TWED algorithm would automatically select the matching between two time series with minimum distance. However, in our case, such matching strategy would be problematic if we match two viewer's gazing fixations at an object's position before and after a shifting of that object in the video scene. Here we will exam the impact of the parameter selection by visual evaluation. We pick 6 × 6 combination of λ,γ (λ and γ iterate with values [0, 1000, 5000, 10000, 20000]). Then we pick several window of 5 second for the TED talk Carol and visualize the movement of different pairs and their values under different hyper-parameter selections. We implement both 3d and 2d visualization of 2 eye movement trails, a sample visualization is shown in Fig. <ref>. We also select parula color map to get a better illustration of the similarity matrix. We would like to get some quantitative ground truth to facilitate the parameter selection, but seems the visual assessment is the only feasible method for now.

So far, depends on visual assessment, we would select λ as 5000 and γ as 5000. More comparisons between different parameter settings could be found in Appendix C. After selecting the optimal parameters, we set the optimal parameters and run through all the video segments with time window of 5 seconds as well as 30 seconds. We then, save the result under the "result" folder in our project repository.



  §.§.§ Eye Movement Clustering

We adopt most of the clustering code in <cit.>. Besides replacing the C_ij vector direction adjacenct matrix by our similarity matrix, we also directly use the C_ij as the J_ij in <cit.>. To better visualize the clustering, we also map the object's index in TWED result file to the object's name. The algorithm is shown in Appendix D and the matlab code is located in the "clustering" folder in our project repository.
After saving the clustering result into the location:
\, we also use the Gephi to visualize the clustering result. We select a 30 second window staring the 290 second and a 30s window starting the 290s in TED Talk Carol video. The clustering result is shown at Fig.<ref>



  §.§.§ Correlation Between Similarity Matrix and Question Answer Correctness

We also examine the relationship between the TWED similarity matrix and question answer correctness. 

Because of lacking ground truth of "attention", we explore the possibility to use question  answer correctness as the evaluation standard. We select a 30 seconds window(the 405s to the 435s) in TED talk Simon video in which the content is related to 5 questions(s3C, s1B, s1E, s3F, s3G). The timing of these questions could be found at <https://docs.google.com/spreadsheets/d/CCMoVJfCyiCMZxf0D1mexE4uELeodQPhRrfBnikXDtU/edit> and the correctness of these questions could be found at <https://docs.google.com/spreadsheets/d/1bDehGoP68VTa7ycggp7G9yOi2FHYNyl6S51Ne4Rolzc/edit> 

Under the assumption that more similar the viewers answering the questions, more likely they pay the same level of attention to the lecture. We calculate a N by N answer distance penalty matrix holding the answer difference penalty between each pair of viewers. If both viewers have correct answer to one question, we would add 1 to their distance penalty value. If both or one of the viewers have incorrect answer, we don't add value to their distance penalty. The higher the distance penalty, the more similar the two viewers' attention is in this time window. We then, along with the similarity matrix of this time window, plot the value of answer distance penalty as x value and similarity as y value. The result is showing in Fig. <ref>


As we can see, since the number of questions is too small, it is relatively hard to apply a regression to reveal the correlation. However, by only looking at this plot, we can see in this period of time, the similarities of eye movement between viewers are relatively low and the similarities of their answers are low at the same time.
  


§ EYE-TRACKING OVERLAY AND EEG VISUALIZATION TOOL

The code of the visualization tool could be found at <https://github.com/hectorcho/visproj> The newest developed version is inside its  folder.


 §.§ Design and Framework

Due to compatibility issue across different development environment, we decide to implement a visualization tool by using PyQt5. QT is basically the most flexible framework for cross-platform GUI development. In addition to PyQt5, we also rely on pyqtgraph, a library help plot dynamic data. The detail of dependencies is shown in Appendix E. 

The requirement of our visualization tool includes a video player, a eye fixation canvas super-imposed above the video player and a EEG data graph that can dynamically update EEG data for all channels. 

An important concept of Qt GUI is widget, which is a complex development unit that contains various basic functionalities. After a few structure changes, we customize our own widget by inheriting the most basic widget and adding corresponding functions. The tool includes three components: 1.a eye movement and video playing widget; 2.an EEG widget; 3.a main window widget that in charge of the GUI layout and the inter-widget communication. We apply observer-listener pattern to register several events to make the eye movement widget and EEG widget on sync. A widget level design is shown in Fig. <ref>



To include with the functionality of eye movement super-imposing, we choose not to create a separate widget but adding a graph video element as the video player's mirror inside a graph view and dynamically add and delete graphic elements such as dots and lines to illustrate the eye movement. The "open Eye" button will read in several eye tracking files, preprocess the fixation data and generate a dictionary. The dictionary holds every object's eye gazing dot, the eye trail lines of previous steps for each eye movement record and a designated color to draw these elements. Whenever the slider of the video moved, the drawing will be triggered to add, delete, move or transform those graphic elements(gazing dots, trail lines). Moreover, the gazing dots would change their size depends on the duration of gazing at each record. 

By selecting audiences in the comboBox, the corresponding eye movement graphic elements would be added to or removed from the graphic view. The position of the fixation has been normalized according to the original screen's resolution and the video player's resolution.

We use the pyqtgraph library to develop the EEG graph widget. The EEG data contains 64 channels, but we allow user to select any numbers of channels to be shown in the canvas. Their plots would be added or removed accordingly. If the video lapse slider change its position, the EEG graphic would also be triggered to update the corresponding EEG graph. We choose to display all EEG values between 5 seconds before and after the current video time. The detail of tool's functions is shown in Appendix F 




 §.§ The Display of the Visualization Tool


In this section, we show several screen shots for all functionalities and views. To better present the eye trail lines, we select a moment when viewers are reading the slides. Similarly, we choose a moment the viewers are gazing at the speaker to show the fixation. Two screen shots are illustrated in Fig.<ref>

We also show the combobox when user selecting the audience's eye movement and the channels of EEG in Fig.<ref>




§ CONCLUSIONS AND FUTURE WORKS


We have compared different methods to calculate the similarity of eye movement between pairs of viewers. After studied several approaches including mechanical statistics, etc., we decide to use time series matching methods. Adopting TWED method, we are able to get similarity matrix for any specified time window. However, due to the fact we don't have reliable attention ground-truth, the hyper-parameter selection and the assessment of the similarity matrix remains a challenge. We also try to use question answer correctness as attention ground-truth but also find it's unreliable due to lack of enough questions. However, the visual assessment by showing eye movement path indicates promising trend. By using the similarity matrix, we are able to cluster the audience by using community detection algorithm. Finding the attention ground-truth seems to be a critical task for future study. By tackle this problem, we would be able to better select the model, hyper-parameters and assess the similarity matrix and clustering result. Another work we have done is developing the visualization tool. Although the functionality is relatively basic, but after exploring the suitable architecture and pattern, we would spent much less effort when adding clustering visualization or other functionalities in the future.


splncs








§ OUTPUT FORMAT OF EYE-TRACKING DATA

The files are located in 
 \ 
 
There two kinds of files came out of the eye-tracking machine: 


  
  ∙ 1.The asc files, which contains the raw data out of the machine.
 Each audience for a lecture would have his or her own file. 

The file's name usually is arranged as , for example, . 

The file could be opened as simple txt file. Although this kind of file contains all the raw information, most of the useful information inside would also be included in the xls file with better aggregation. The most useful information to us in the file is MSG 0 . For example, 0  0 0 1023 767. 
  
  ∙ 2. The xls files, which contains the aggregated data from the raw data. 
 Each audience for a lecture would have his or her own file. 

  The file's name usually is arranged as , for example, . The file could be opened as a simple excel file.

The file contains 222 columns. Many of them containing event data such as , which in most case is empty. Among with columns, , ,  are most important to us. The unit of the columns  and 

 are all millisecond. The unit of the position columns 
  and  are the pixel location relative to the left up corner of the screen.




§ TWED ALGORITHM IN OUR APPLICATION


Modified iterative implementation of the TWED distance


code modified based on "Iterative implementation of the TWED distance" in Marteau et al.<cit.>


§ TWED HYPERPARAMETER SETTING COMPARISON

  




§ FAST MULTI-SCALE COMMUNITY DETECTION ALGORITHM







§ DEPENDENCIES OF DEVELOPMENT







§ GUI OPERATION







