Understanding and Improving CNNs with Complex Structure Tensor: A Biometrics Study
Kevin Hernandez-Diaz Josef Bigun Fernando Alonso-Fernandez

















































ISDD Lab, Halmstad University, Kristian IV:s väg 3, 
 Halmstad, 30250, Halland, Sweden









[cor1]Corresponding author email: kevin.hernandez-diaz@hh.se




Our study provides evidence that CNNs struggle to effectively extract orientation features. We show that the use of Complex Structure Tensor, which contains compact orientation features with certainties, as input to CNNs consistently improves identification accuracy compared to using grayscale inputs alone. Experiments also demonstrated that our inputs, which were provided by mini complex conv-nets, combined with reduced CNN sizes, outperformed full-fledged, prevailing CNN architectures. This suggests that the upfront use of orientation features in CNNs, a strategy seen in mammalian vision, not only mitigates their limitations but also enhances their explainability and relevance to thin-clients. Experiments were done on publicly available data sets comprising periocular images for biometric identification and verification (Close and Open World) using 6 State of the Art CNN architectures. We reduced SOA Equal Error Rate (EER) on the PolyU dataset by 5-26% depending on data and scenario.
















Periocular Biometrics Texture Complex Structure Tensor









§ INTRODUCTION



Computer Vision (CV) defines texture as a repetitive pattern <cit.>. Extracting texture features amounts to finding a vector of parameters from subsets of the pattern to uniquely characterize it. Ideally, it is then necessary for the 
vector to be invariant to translation inside the same texture and different in another <cit.>. 
The squared magnitude of Fourier Transform, FT, a.k.a. the power spectrum, of an image is invariant to image translations.  Consequently, the power spectrum of local images has been a main source of texture perception studies <cit.>, and texture feature vectors <cit.><cit.><cit.>. Even Convolutional Neural Networks (CNN) are biased towards texture information <cit.>, even showing that some first-layer ImageNet trained CNNs' filters have similar appearance as members of a Gabor filter-bank <cit.>. Since Hubel and Wiesel’s Nobel Prize-winning work, <cit.>, we know that mammalian vision is enabled by cells, each tuned to a unique orientation of translating bars in its visual field, and are upfront in the brain’s data flow, at the visual cortex. There exist cell layers that are invariant to translation, with tune-in angles uniformly and densely sampling [0-π], like Gabor functions <cit.> magnitude responses <cit.>. 



Texture plays a crucial role in many image-based biometric recognition systems such as the case of fingerprints <cit.> or iris <cit.>. In 2009, the authors of <cit.> showed the potential of the periocular region for biometric recognition. This region encompasses the iris, eyebrows, eyelids, commissures, skin texture, and the general eye's shape that can be used for recognition. This information-rich area has shown not only to be quite stable but also to provide high performance for soft-biometrics recognition <cit.>. The periocular area provides flexibility regarding acquisition and occlusion as a middle ground between face and iris recognition. Like its biometric siblings, it has been the target of several texture extraction algorithms for recognition purposes with good performance <cit.><cit.>.

In this paper, we propose a set of complex mini conv-nets specialized in extracting texture presence and orientation with texture pattern complexity perfectly representable up to the n-th order moments of the power spectrum <cit.> in a region to enhance the performance of CNNs. The main contributions of this paper are as follows:



     
  * We show that CNN networks can benefit from compact orientation features as texture features at the input by improving their a) explainability and focus, b) convergence, c) network size, and d) performance.
     
  * We suggest complex Structure Tensors (CST), which is a further generalization of the ordinary Structure Tensor (ST), as a mini conv-net, relying on complex scalar products and complex non-linearities to provide compact inputs to CNNs. The features provided are dense equals of the local power spectrum, including one provided by a full Gabor filter bank, which approximates cell layers of mammalian vision well.
     
  * Our evidence from biometric recognition confirms more explicitly previous indications that SOA CNN networks alone cannot produce orientation features significant for decisions on training data with reasonable size  (≈ ten thousand images). 
    
  * Results indicate that the findings above generalize across different CNN architectures and spectra. 
 
 
 



§ BACKGROUND AND RELATED WORK




This paper uses a method based on the Structure Tensor Theory (ST) <cit.> as a compact texture orientation descriptor combined with CNNs. The descriptor, explained in detail in Section <ref>, employs Complex Derivative of Gaussians to extract gradient information and detect the presence and orientation of line patterns. Besides biometrics, STs have been studied and used in numerous applications, including image enhancement, <cit.>, and medical image analysis <cit.>, but also as a fundamental tool in computer vision, to detect and track key-point with <cit.> or without scale-space <cit.>, to generate dense optical flow, <cit.>, as well as orientation maps <cit.>.
Symmetry descriptors have been used previously in biometric recognition with success, given their resilience to viewpoint or scale changes <cit.>, but their possible contribution has not yet been studied in the context of Deep Learning. Regarding periocular recognition, the works <cit.> used the presence and orientation of several symmetry patterns at different scales around selected image keypoints for both periocular and forensic fingerprint recognition, achieving performance comparable to SOA. 


Traditional methods like ST <cit.>, LBP <cit.>, HOG <cit.>, and Gabor <cit.> try to quantify the presence of line patterns in images, specifically their orientations. However, lately, their use has been superseded by CNNs. Even other studies on CNNs show the importance of quantifying orientation. While filtering profiles similar to Gabor filters are observed in the initial layers of CNNs <cit.><cit.>, such filtering i) lacks systematic uniform orientation tuning and ii) the (non-linear) magnitude processing for translation invariance. This gap may arise from the assumption that all layers in CNNs must be determined by training data, whereas mammalian vision quantifies orientation uniformly up-front and maintains it throughout adult life.


To exploit potential synergies, researchers have tried to create robust models by combining traditional methods and CNNs in different ways. The authors in <cit.> combined LBP and RGB features with CNNs for face anti-spoofing, increasing performance while reducing the network size. In <cit.>, the authors proposed a double stream CNN that takes RGB ocular images and Orthogonal Combination-Local Binary Coded Pattern (OCLBCP) texture descriptor as inputs, improving recognition performance. In <cit.>, the authors created a new CNN layer called Local Binary Convolution (LCB) as an alternative to convolutional layers. The network, called Local Binary Convolutional Neural Network (LBCNN), performed the same as regular CNNs while reducing network size and learnable parameters between 9 and 169 times.


In <cit.>, the authors used HOG features and CNN to create an efficient pedestrian tracking and reidentification model across multiple camera views. The authors of <cit.> used handcrafted HOG features, pre-trained CNNs features, and gender information extracted from a shallow network for periocular recognition. The work <cit.> used a HOG descriptor over spatial and temporal CNNs shallow features for face expression recognition in video. 


The work <cit.> introduced GaborNet, a modified CNN in which the first layer filters are set to fit the Gabor Function by learning Gabor filter parameters instead of filter values directly. They showed that this modified CNN can outperform classic CNN while reducing the number of learnable parameters. 
The authors of <cit.> apply a bank of Gabor filters to face images to obtain responses. Each response image was used to train its own CNN network to achieve face classification. In <cit.>, the authors used the output of two consecutive Gabor filters to train a CNN to recognize between seven possible facial emotions. They showed that Gabor outputs can help CNNs achieve better emotion classification and faster convergence. The work <cit.> showed that using the response of Gabor filters as input of CNNs can improve recognition of soft-biometrics recognition on face images in  CNNs and capsule networks.













§ STRUCTURE TENSOR AND POWER SPECTRUM




The Structure Tensor, ST <cit.>, was originally introduced to provide a compact parametrization of the local power spectrum, which provides rich translation invariant features in texture. ST fits the total least square error line to the local spectrum and calculates the best and worst possible errors. Linearly symmetric patterns family in a local patch, those having iso-curves that are parallel, excite ST the most. In Figures <ref> a), b) two (local) patches exemplify the family, here defined as real parts of two distinct planar waves, Aexp (iω_x x
+iω_yy).



𝐒𝐓 is the 2×2 symmetric real semipositive definite matrix obtained by averaging the tensor products of image gradients 𝐒𝐓 =⟨∇ f ∇^T f⟩ with a filter representing the neighborhood. Its three real elements encode the presence of an oriented texture in the neighborhood.


Satisfying 𝐒𝐓  u=λ u, eigenvalues 0≤λ_2≤λ_1 with corresponding (orthogonal) eigenvectors u_2 and u_1 constitute another representation of ST, useful to explain the neighborhood in terms of measurable human observation, texture orientation.

The matrix 𝐒𝐓 can be decomposed as 

    𝐒𝐓 =(λ_1-λ_2)u_1u_1^T+ λ_2
    (u_1u_1^T+u_2u_2^T)

where u_1u_1^T+u_2u_2^T=E is the unity matrix. Accordingly, 𝐒𝐓 can be uniquely decomposed into an oriented “line” component (encoded by u_1 u_1^T) with weight-power of λ_1-λ_2 and a non-oriented, “balanced”, component (encoded by identity) with the weight-power of λ_2. 







If and only if the neighborhood is linearly symmetric, i.e. it possesses texture orientation, its entire spectral power collapses to Dirac distributions located on a line through the (spectral) origin. In the solution, the error of the best fit is λ_2 (ideally zero for direction u_1), and the worst fit is λ_1 (for direction u_2). Accordingly, the difference λ_1-λ_2

represents the confidence in the fit, whereas θ=∠ u_1 is the direction of the pattern. However, the confidence is energy dependent, and can vary with the neighborhood contrast.



To exploit the ST theory more effectively, complex numbers can be used to represent the unique decomposition. 

The main advantage of this “overhaul” is the mathematical completability of STs by power series expansion enabled via the complex representation, allowing to express more elaborate oriented textures, those possessing n-folded symmetries.

Complex numbers reduce the unique decomposition  (<ref>) to second order complex moments[
  I_pq≜∫
  (ω_x+iω_y)^p
  (ω_x-iω_y)^q |F(ω_x,ω_y) |^2
  dω
  where F is FT of f.
  ]  of the neighborhood power spectrum, the complex I_20, and real I_11,

    𝐂𝐒𝐓= [ I_20,
      I_11 ]=[ (λ_1  - λ_2)exp(2i∠ u_1 ) , λ_1 + λ_2 ]

Here i=√(-1),  and the 𝐂𝐒𝐓 vector has 3 degrees of freedom. Components are readily connected to the unique decomposition of 𝐒𝐓. The CST vector is thus the complex equivalent of ST.  Replacing the tensor mapping u_1u_1^T in 2D, the complex mapping 2∠ u_1 is similarly invariant to pattern rotation with π i.e. 2(∠ u_1+π)=2∠ u_1.

The sum λ_1+λ_2=I_11 is a tight upper bound for the confidence 0≤λ_1-λ_2 ≤λ_1+λ_2 as λ_2 approaches to zero. An exploitable fact is then, that confidence | I_20 | equals the upper bound I_11 if and only if the pattern possesses texture orientation (=linearly symmetric). 













The first benefit is that we can estimate these moments with just 3 separable filters, i.e., without a massive filtering process (with a  Gabor filter bank), producing the local power spectrum after taking magnitude responses. This is because the axis fitting process is directly available in 𝐂𝐒𝐓.
As illustrated by Figure <ref>, the moments can be obtained directly via a mini-complex convolutional network, comprising convolution of the original image with complex filters given by

    Γ^{ n, σ^2}(x,y)= (D_x+iD_y)^n G_σ^2 (x,y)}

where D_x, D_y are partial derivatives, and

    G_σ^2 (x,y)=1/2πσ^2exp(- x^2+y^2/2σ^2 ),

is the familiar Gaussian function with variance σ^2.  In the case of texture orientation of linear symmetry type  (n=1), the first (linear) box becomes the gradient computation.














The linear symmetry algorithm generates elements of CST, via three
steps each, as follows.

  
  * Convolve gray image f  with the filter Γ^{n,σ^2}, by setting n=1, and σ=σ_1.
The result is a complex valued image, r.
  
  * Square each pixel in r while emphasizing (gradient)
magnitudes with γ to yield two  outputs, complex r_a=|r|^γexp(2∠ r) having double gradient
angles, and real r_b=|r|^γ







  
  3a Convolve the r_a strand of Step 2 with the Gaussian
filter Γ^{n,σ^2}  where  n=0, and σ=σ_2 to provide the complex I_20.
  
  3b Convolve the r_b strand of Step 2 with the same Gaussian
filter to output the real  I_11.






Textures with unique orientation have their 2D power spectra concentrated into a line, as illustrated in Figure <ref> a), b).
However, multiple orientations can occur in textures such as repeated squarish patterns, as exemplified by Figure <ref> c), which will then produce a cross-like pattern in the power spectrum. This makes  I_20 with n=1 insufficient, requiring more sophisticated models of orientation presence.

The texture model represented in the above algorithm can then be
completed in the mathematical sense by increasing n to  n=2 in
the first box, <cit.>. This changes only
the used filter in the box to  Γ^{2,σ^2},
(<ref>), and the same algorithm will instead compute
elements of the vector (I_2n,0, I_n,n), completing the
descriptive ability of lower "n"s. Such increases yield even orders of complex
moments of the (local or Gabor) power spectrum, which is even, to
quantify its n-folded symmetry concentration. Instead of 1
axis as in n=1, each (I_2n,0,I_n,n ) pair fits n evenly
distributed axes through the origin. The orientation delivered in I_2n,0 is texture group orientation, e.g., how many radians the lines of a squarish
pattern deviate from a reference direction.





Using 𝐂𝐒𝐓,  we present evidence that CNNs are not able to reach the valuable information embedded in texture orientations. Likewise, the evidence supports that if the orientation of power spectrum concentrations is extracted and presented to CNNs via complex moments, then they can make decisions with improved accuracy.







§ METHODOLOGY








We follow the pipeline shown in Figure <ref> to introduce the CST into CNN architectures.
The  σ_1 of the first step determines the 
frequency band in which CST acts, whereas σ_2 in the third step determines neighborhood size. Figure <ref> a) shows the effect of using different σ values.
Preliminary results showed us that the highest frequencies in our
datasets contained the most discriminative information. The neighborhood size only
influenced methods where alignment was an issue. We settled therefore
for a value for σ_1 of 0.6 to extract high-frequency
gradients and σ_2 of 4.0 to
define minimum neighborhood size. Furthermore, we set γ to 0.1 to alleviate the difference in magnitudes between the regions with subtle and fast changes. Figure <ref> b) shows the type of texture extracted according to different values of n and γ for the selected σ_1 and σ_2.
















To corroborate conclusions, we carried out biometric recognition experiments by six widely used CNN architectures:  ResNet50 <cit.>, DenseNet121 <cit.>, Xception <cit.>, InceptionV3 <cit.>, MobileNetV2 <cit.>.  







We used Tensorflow-Keras as our Deep Learning framework to download, initialize, train, and test the models. The CST process was transparent to the network and not affected by the training process. We use the standard models provided by the Keras library, only changing the last fully connected layer to fit the number of users in the database and the input size to fit the type of data used. We trained the networks using Stochastic Gradient Descent with a learning rate of 0.001 and a momentum of 0.9. The networks were trained for 100 epochs with a batch size of 16. No data augmentation was used during training. Model checkpoint was used to monitor the validation loss and recover the best-performing version after training. All training was conducted on a Windows 10 machine with an i7-8700, 32GB of RAM, and an Nvidia Titan V. 
































§ DATABASES, METRICS, AND PROTOCOL





 §.§ Databases






We employed two commonly used periocular datasets in the experimentation: Cross-Eyed <cit.>, and PolyU <cit.>. Figure <ref> shows some examples of each database.

The Cross-Eyed dataset is a cross-spectral periocular database that offers Near-Infrared (NIR) and Visible (VIS) images from subjects of different nationalities, ethnicities, and eye colors using a custom dual-spectrum image sensor under normal indoor illumination. The database contains 8 images of each eye from 120 subjects per spectrum, for a total of 3,840 images. Images were normalized to have the same sclera radius, center, orientation, size, and dimensions.



The PolyU database is a bi-spectral iris database captured using simultaneous bi-spectral imaging by the Hong Kong Polytechnic University. It provides images in NIR and VIS, captured simultaneously, offering exact pixel correspondence between both spectral image versions. The database consists of 15 instances for each spectrum of both eyes of 209 different subjects for a total of 12,540 iris images. Since the periocular region in this dataset is smaller, images are zero-padded to be square while maintaining the aspect ratio.





 §.§ Metrics and Protocol




To evaluate the performance of the proposed system, we carried out both biometric identification and verification experiments. For the identification cases, we did 5-fold cross-validation and tested across all user samples using softmax and cross-entropy loss. We performed biometric verification to frame our methods across the SOA results for the used databases. We followed the same protocols as in <cit.>: the Close-World (CW) protocol, in which the training and test partition contains images of all users without overlap, and the Open-World (OW) protocol, in which the users are split into training and testing along with all their images. The networks used for biometric verification were previously trained for identification using the same training and validation partitions. For testing, we extracted the features from the second to last layer and compared them using the cosine similarity.













§ RESULTS AND DISCUSSION




 §.§ Effect of CST input variables


























The first step in our experimentation was to investigate how the representation and presence of I_20 and I_11 affected the training and performance of the network. Since I_20 is complex, we can use either the real (I_20) and imaginary (I_20 representation or the polar coordinates | I_20 |  and ∠(I_20). The network input was modified by concatenating the selected representation through the channel axis. Table <ref> presents the identification accuracy results for these experiments. Additionally, Figure <ref> shows the validation loss during the training process. 




From Table <ref>, we can see that the incremental improvement observed with the addition of each component in the polar coordinates indicates that magnitude, angle, and upper fitting boundary all contain beneficial information offering comparable or better performance than the original grayscale image. Furthermore, the angle information (fourth row) improves the accuracy further for all cases, outperforming the grayscale version for both datasets. This shows that the network can benefit from the orientation information upfront, even if the angle is discontinuous at 0/360 degrees in the polar representation. 
Providing the real and imaginary parts of I_20 alone, which also implicitly encodes magnitude and angle, gives similar or better results than the polar counterparts, plus I_11. The addition of I_11 to this new case (sixth row) gives extra improvement, highlighting the valuable contribution of all elements in either representation. 
We also have studied the combination of the original grayscale image with the best variants of the CST representation (8th and 9th row). While also providing the grayscale image can help improve the accuracy, this may not hold true for all the options (sixth vs. ninth row). 
The best overall results are obtained with the real and imaginary parts, | I_20 |, and I_11 (seventh row). This case provides not only direct information on the presence of the linear pattern in the area, but the orientation information is also provided by the real and imaginary values. Another consideration of these results is that our observations hold for the two databases and the two spectra, indicating the generalization capabilities of the proposed method.






We then analyze the benefits of the proposed methods during network training (Figure <ref>). We do so for selected cases in Table <ref>. We can observe that just providing the real and imaginary parts already allows for faster convergence and lower loss than when using the grayscale version. However, as with the results in Table <ref>, providing just the magnitude | I_20 | is insufficient to observe tangible benefits. Again, the best case overall corresponds with the addition of I_11 and | I_20 | to I_20, which can be observed regardless of the database and spectrum employed.





 §.§ Effect of CST derivative order




Another relevant parameter related to the network’s input is how the symmetry order (n in Equation <ref>) affects the network. The higher the order, the more intricate texture orientation the model will extract (Figure <ref>). However, the presence of highly complex patterns is dataset and application dependent. Nonetheless, higher-order symmetries still have some response, making an interesting case for the selection of the input data. In this paper, we experiment with values of n from 1 to 3. For simplicity, for each value of n, we use the input combination consisting of (I_20), (I_20), and I_11. Table <ref> shows the results of these experiments. We can see that when only one is used, the first-order derivative of Gaussian (n=1) performs the best, although the second-order still matches or improves the grayscale version. In general, the performance decreases as we increase n. This is an indication that classes could be separated already by low-order orientation quantification. Even though combining different orders improved the accuracy in some database/method combinations, the effect was not consistently observable. 




 §.§ Effect of network compression












Since ready-to-use features are provided up-front, there is potential for network compression and speed improvement. Thus, we also experimented with different network depths, widths, and input resolutions. A coefficient α was used to reduce the number of filters per layer (width) of ResNet50 with values (0.2, 0.4, 0.6, 0.8, and 1.0), being 1.0 the standard network size. In addition, the input image size was reduced by one-fourth each time while simultaneously changing the network depth to compensate, going from an input size of 256x256 pixels to 32x32 pixels in its shallowest version. We maintained the network's abstraction level and output resolution by decreasing the depth of the network accordingly.



Figure <ref> shows the effect that the amount of parameters has on performance. The amount of parameters depends on the network's width and depth of the network. The first thing to notice is that all curves exhibit very similar behavior regardless of the database, spectrum, and input type. The slight offset in terms of the number of parameters in comparison to grayscale images is due to the increase in the number of channels at the input. This offset (6,272 parameters) is minor compared to the number of parameters of the original ResNet50 (25.5 million). 
It can also be seen that CST outperforms the grayscale images in all cases, except for one point in the Cross-Eyed dataset and VIS spectrum at around 1 million parameters. At this point, the network depth is at its maximum, but the width is at its minimum (α=0.2). Such extreme imbalance is the explanation for the sudden drop in performance observed in the graphs for both datasets. Another observation from the graphs is that the network can go through compression to a large extent without loss in performance, with our solution systematically providing better accuracy for any compression level. This suggests that our method could support compression, producing smaller networks while still benefiting from an increase in performance.




 §.§ Consistency across other CNN architectures
























































































We have also applied the proposed method to other popular CNN architectures to corroborate its benefits. Table <ref> shows the results of such experimentation. It can be seen that for all networks and databases, our complex structure tensor version outperforms the baseline grayscale version. For the Cross-Eyed database. For PolyU, our ResNet50 network improves the baseline results by 3.3% for the NIR case and 1.9% for VIS. It is also worth noting the improvements of InceptionV3 and MobileNetV2 with PolyU-NIR (2.7% and 2.8%, respectively). In absolute terms, the best-performing network for both the baseline and our method is Xception, reaching up to 99,6% accuracy for the VIS Cross-Eyed data and 98%-99% accuracy in all other cases. In summary, the observed accuracy improvements with our solution demonstrate that it also generalizes to other CNN architectures.




 §.§ Performance comparison with previous studies



Table <ref> summarizes and compares our results with previous works using the same databases <cit.><cit.>. The whole performance for each system can be seen in the ROC curves presented in Figures <ref> and <ref> for the Close-World and Open-World cases, respectively. This sub-section follows the same protocol as those previous works to enable the comparison as explained in Section <ref>. 

The CST method in these experiments used the input combination of BW, (I_20),(I_20), I_11. Including the grayscale image in these experiments proposes a more general approach of using CST with Deep Learning. 










    




















































































































In this case, results are more mixed than in the previous subsections. In particular, the CST approach does not outperform the use of original grayscale images in all cases. This contrast can be partially attributed to the use of 5-fold cross-validation in the previous sections, which shows a more reliable indication of the performance of the system by testing over the whole dataset. 

This effect mostly happens with the Cross-Eyed database. We attribute this to the more limited training data available in Cross-Eyed since the networks are trained from scratch, and our CST approach involves a higher amount of input channels. 
Conversely, with PolyU, our approach works better or the same as the original grayscale images in most cases.



From the ROC curves in Figures <ref> and <ref>, one can see that in some cases, despite a higher EER in Table <ref>, the performance favors ConvNet models using CST information when ROC curves enter the low FAR area. Such cases are the Xception network on the PolyU dataset for both the NIR case and Close-World Protocol, as well as for the VIS spectrum in the Open-World Protocol. 

For the Open-World case, our CST model performs better at a lower FAR for every case, with the exception of PolyU in the VIS spectrum, where the difference between models is negligible. For DenseNet121, the performance difference in Cross-Eyed and NIR for the Close-World setting becomes smaller at lower FAR. Despite the variability in the results due to single-partition testing, ROC curve results confirm predictions of the identification sections. Texture orientation information brings measurable performance benefits in recognition compared to using only grayscale information when networks were not pre-trained at all.



The results of Table <ref> indicate the EER of CST and the SOA results from previous studies <cit.><cit.> in the same protocol. Despite CST having no access to pre-training, it is able to deliver SOA performance with Xception and DenseNet121 networks in PolyU in CW (both spectra) and in the difficult OW with PolyU in NIR. In the case of ResNet50, which delivers the best performance of state-of-the-art in the protocol, CST was able to come closer than 1 percentage point to the state-of-the-art in CW, PolyU NIR CW, and in the difficult OW PolyU NIR cases. In other cases, CST was lagging behind by not more than 2-3 percentage points.

The method of <cit.> used a ResNet50 network pre-trained for face recognition on the very large VGGFace2 and MS1M databases. 

This leads us to hypothesize that our strategy of using mini complex ConvNets to complement grayscale with texture orientation at the input of CNNs could also benefit from pre-training on other modalities, such as face, where much larger databases exist. 

On the other hand, the method of <cit.> underwent an exhaustive search to find layers of the networks that could provide enhanced verification performance. 











§ CONCLUSIONS


This study examined the benefits of the Complex Structure Tensor theory to improve CNNs by providing measurements representing the presence of unique texture orientation and its angle at the input. We investigated the effect of changing the input image of the network from the original grayscale to a preprocessed image where the joint presence of up to n orientations in the pixels' neighborhood is highlighted. This is achieved via second-order complex moments of local power spectra, which produce a complex-value representation of the image that encodes both the magnitude and angle of the dominant texture (group) orientations. We used periocular recognition as a case study and six well-known network architectures. We employed two common datasets used in periocular research, Cross-Eyed and PolyU, which contain images both in near-infrared (NIR) and visible (VIS) spectra. This allowed us to explore the applicability of our findings to different spectra.

Typically a Gabor filter bank representing the local spectrum is obtained by using 30-100 (=dimension) different filters. In our method, we extract single frequency band texture components in the image, which are then averaged over a neighborhood (configurable via two parameters, σ_1 and σ_2, respectively). This approach compresses the feature space dimension of the image that is provided to the network while maintaining translation invariant information of the local power-spectra, improving accuracy and convergence.

Improvement is observed regardless of the database, spectrum, or network architecture. This suggests that standard CNNs are not able to effectively reach these features from the gray-scale images, especially when training data is limited.


We show that, with our method, thanks to the reduction of the feature space, network compression is possible while maintaining superior performance. In terms of comparison with prior studies, our method is shown to improve the verification performance when sufficient training data is available. 


With orientation features at the input, we also gained explainability as to CNNs' decision basis. Networks can then focus on what has proven helpful for mammalian vision upfront. Even if CNNs have been shown to extract select texture orientations in first layers, <cit.><cit.>, our results indicate that providing them upfront reduced the burden of learning and extracting them with higher mathematical systemacy, at no cost (downsizable network). The systemacy included <cit.> mathematical completeness, interpolative direction, independence of filter bank size for direction accuracy, certainty measure, and translation invariance within textures.






To fully leverage the presented method's benefits, we aim to go further by making the hyperparameters of the employed filters learnable. Our use of local texture features and the increase in convergence, network compression, and the separability of the filters used, make our model an interesting case for newer architectures like hybrid vision transformers, where recent works focus on models' accuracy-latency trade-off <cit.>. Future work also includes applications in CV where texture information and/or certainty play important roles, such as image segmentation <cit.>, or image generation <cit.>.




§ ACKNOWLEDGEMENTS


This work was supported by Vetenskapsrådet [grant numbers 2016-03497 and 2021-05110] and VINNOVA [grant number 2022-00919].













 elsarticle-num 
 












