Drawing the Line: Deep Segmentation for Extracting Art from Ancient Etruscan Mirrors
Rafael Sterzinger Simon Brenner Robert Sablatnig




Extracting Art from Ancient Etruscan Mirrors





R. Sterzinger et al.



Computer Vision Lab, TU Wien, Vienna, AUT

{firstname.lastname}@tuwien.ac.at

Drawing the Line:
Deep Segmentation for Extracting Art from Ancient Etruscan MirrorsThis project has received funding from the Austrian Science Fund/Österreichischer Wissenschaftsfonds (FWF) under grant agreement No. P 33721.
    Rafael Sterzinger0009-0001-0029-8463 Simon Brenner0000-0001-6909-7099 Robert Sablatnig0000-0003-4195-1593
    April 29, 2024
=================================================================================================================================================================================================================================




Etruscan mirrors constitute a significant category within Etruscan art and, therefore, undergo systematic examinations to obtain insights into ancient times.
A crucial aspect of their analysis involves the labor-intensive task of manually tracing engravings from the backside.
Additionally, this task is inherently challenging due to the damage these mirrors have sustained, introducing subjectivity into the process.
We address these challenges by automating the process through photometric-stereo scanning in conjunction with deep segmentation networks which, however, requires effective usage of the limited data at hand.
We accomplish this by incorporating predictions on a per-patch level, and various data augmentations, as well as exploring self-supervised learning.
Compared to our baseline, we improve predictive performance w.r.t. the pseudo-F-Measure by around 16%.
When assessing performance on complete mirrors against a human baseline, our approach yields quantitative similar performance to a human annotator and significantly outperforms existing binarization methods.
With our proposed methodology, we streamline the annotation process, enhance its objectivity, and reduce overall workload, offering a valuable contribution to the examination of these historical artifacts and other non-traditional documents.





§ INTRODUCTION


Comprising more than 3,000 identified specimens, Etruscan hand mirrors represent one of the most extensive categories in Etruscan art.
These ancient artworks, which date from the second half of the sixth century to the first century B.C.E., are predominantly made from bronze.
On the front, they feature a highly polished surface acting as a mirror, whereas the back side typically depicts engraved and/or chased linear figurative illustrations of Greek mythology which may be supplemented by inscriptions in Etruscan <cit.>.
An exemplary mirror showcasing such drawings is depicted in <Ref>.

These mirrors have been systematically examined since the 1980s to better understand the Etruscan culture, with findings published in the scholarly series Corpus Speculorum Etruscorum (e.g., <cit.>).
One main component of their examination is the labor- and cost-intensive task of tracing the linear figurative artworks typically featured on their backside.
Although performed by experts, i.e., archaeologists specialized in Etruscan culture, this task is prone to subjectivity: over time, these mirrors experienced erosion/corrosion, which increases the difficulty of distinguishing between artwork, damage, and background.[1]©KHM-Museumsverband (Kunsthistorisches Museum Vienna)




Based on these challenges, this work presents the first approach for automated segmentation of Etruscan hand mirrors to enhance annotation quality and objectivity while reducing the workload of manual tracing.
For this, we collect high-resolution 3D surface models via photometric stereo and train a deep segmentation network to recognize intentional lines over scratches.
Our approach is grounded in a thorough ablation study where we evaluate various surface modalities and explore different techniques to address the scarcity of data and annotations within this domain, including performing predictions on a per-patch level, incorporating various data augmentations, and self-supervised learning, as well as examine different architectural components, such as encoders, decoders, and loss functions.
Compared to an initial baseline, our modifications improve performance w.r.t. the pseudo-F-Measure by around 16%. 
Moreover, we juxtapose our method with a human baseline to underscore its efficacy as it achieves results on par with a human annotator.

Summarizing our contributions: our methodology streamlines the overall annotation process of Etruscan hand mirrors; It not only expedites the process of manual tracing but also offers a supplementary perspective, contributing to increased quality and objectivity.
Finally, we provide public access to both the code and data utilized in this work (see https://github.com/RafaelSterzinger/etmira-segmentationgithub.com/RafaelSterzinger/etmira-segmentation) to promote transparency and reproducibility.




§ RELATED WORK

Photometric Stereo (PS) is a powerful tool to capture unique insights into an object's surface geometry.
McGunnigle et al. <cit.> demonstrate its usefulness in segmentation by extracting handwriting on paper based on depth profiles.
Similarly, Landström et al. <cit.> have applied this technique to detect cracks in steel surfaces, showing its versatile application.
Moreover, the work by Zhang et al. <cit.>, where they compute shape index and curvedness from PS for leaf venation extraction, or by Tao et al. <cit.>, in which they detect air voids in concrete surfaces, further exemplify its broad applicability.
Its ability to distinguish subtle surface variations makes it particularly suitable for analyzing complex patterns such as those found on Etruscan hand mirrors.
However, traditional segmentation methods might fall short due to damage these mirrors experienced, increasing the difficulty of distinguishing between fore- and background.

In the field of image segmentation, the state of the art is dominated by deep learning architectures such as the UNet <cit.>, DeepLabV3++ <cit.>, Pyramid Scene Parsing Network <cit.>, Pyramid Attention Network <cit.>, or the Feature Pyramid Network <cit.>.
A driving force for such innovations is, for instance, the medical sector, which demands high-detailed segmentations as well.
One example is the segmentation of vascular structures found in the retina (e.g., see <cit.>), which is key in determining retinal diseases.
In this context, Kamran et al. <cit.> proposed a multiscale, GAN-based segmentation system, designed to reduce loss in fidelity.
More recent works propose transformer-based <cit.> as well as diffusion-based architectures <cit.> for brain tumor, gland, or nuclei segmentation.

Saiz et al. <cit.> showcased the effectiveness of combining PS scanning with deep segmentation networks.
With this paradigm, they perform defect detection in steel component manufacturing, paving the way for future applications where intricate and detailed segmentation is vital. 
Lastly, to give an example in the context of cultural heritage, Redon et al. <cit.> present a deep learning-based reconstruction pipeline that transforms RGB images into high-resolution normal maps to digitize Bayeux tapestry.
Additionally, they couple the reconstruction with segmentation, enabling cost-effective and quick production of 3D-printed bas-reliefs to enhance tactile exploration for visually impaired individuals.
Such examples showcase the potential efficacy of PS when combined with deep learning to segment figurative drawings such as the ones found on Etruscan mirrors.



§ METHODOLOGY

In the following, we describe the setup for data acquisition, the captured dataset, and outline our pipeline in detail, focusing on preprocessing and the subsequent segmentation.
For the latter, we summarize our explored augmentation/self-supervised learning techniques as well as point out training and inference specifics.





 §.§ Data Acquisition


We employ PS, a 3D reconstruction technique introduced by Woodham <cit.> in 1980. 
Compared to other 3D acquisition methods, e.g., structured-light scanning or photogrammetry, it is especially adept at capturing local surface details <cit.>.
PS accomplishes this by capturing a series of images with consistent camera settings but variable lighting conditions to then, for each pixel, determine surface orientation, distance, and illumination-independent brightness.
These properties can be encoded across the entire image to normal, depth (computed from the normal map via integration <cit.>), and albedo maps, illustrated in <Ref>. 
 
For recording the images, we employ a PhaseOne IQ260 Achromatic camera with an 8,964×6,716 pixel medium-format sensor and a Schneider-Kreuznach 120 mm LS Macro lens.
Using this setup, we obtained a surface resolution of 38.5 ppm (∼978 dpi) for most of our mirrors.
However, as imaging distance changed between locations, this value is not consistent throughout the images, ranging from 29.8 ppm to 58.3 ppm.
For more information, we refer the reader to Brenner et al. <cit.> from where we derived our setup.



 §.§ Dataset

Our dataset includes a wide variety of Etruscan mirrors preserved in public collections in Austria.
Its core comprises 53 specimens located at the Kunsthistorischen Museum (KHM) Vienna and 6 additional ones scattered throughout Austria.
We acquired annotations for 19 out of these 59 mirrors, whereby 19 backsides and 10 fronts were annotated, leading to 29 examples in total.
Note that engravings mainly decorate the backside where they do not interfere with reflectance.
Sometimes, however, they can also be found on the front near the handle or around the border, though with less dense drawings.

We split these labels into training, validation, and test sets, however, given their limited amount, their varying density of engravings per mirror, and their condition, this step is not straightforward: first, mirrors with dense engravings offer a stronger learning signal and are hence more preferable for the training, second, using mirrors of different condition for validation and testing, will cause discrepancies in performance, rendering model selection difficult.
Therefore, we select the backsides of three mirrors of different conditions and densities in engravings for evaluation, one mirror from Wels and two from the KHM Vienna.
We create non-overlapping patches of size 512×512 pixels, shuffle them, and split them in half to yield our validation and test set, to create splits with similar underlying distributions.
Out of the remaining 26 examples, we remove one, representing an outlier, due to its different art style (points instead of lines), leaving us with 25 annotated examples for training. 





 §.§ Preprocessing


With three modalities at hand, albedo, depth, and normal, it is unclear which ones obtain the best predictive performance.
In order to allow for a fair performance evaluation, preprocessing is crucial as it directly influences the result of subsequent tasks.
For instance, employing the depth modality and not performing any preprocessing renders the segmentation of fine artistic lines impossible; its effect is illustrated in <Ref>.
Here, we filter low frequencies by subtracting a Gaussian-filtered version of the depth map.
Afterward, we cap the depth values to μ - 3σ as the minimum and μ + 3σ as the maximum intensity, where μ is the mean and σ is the standard deviation.
Regarding the other two modalities, i.e., normal and albedo, we standardize the normal vectors to be of unit length, whereas we keep the latter modality unmodified.

In addition to that, we generate segmentation masks to capture the location of a mirror within a shot using the Segment Anything Model (SAM) <cit.>.
Although these masks are not necessarily required, we employ them for Cross-Pseudo Supervision (CPS) <cit.> as well as during inference to mask out regions that impossibly contain engravings.
Lastly, using the acquired masks, we ignore non-mirror parts for the calculation of per-channel means and standard deviations which are later on used to normalize the input.

As mentioned, to alleviate our lack of annotations, we perform inference on a per-patch level:
For validation and testing, we extract non-overlapping, quadratic patches of size 512×512 pixels.
Regarding our training data, we pad four pixels, since 6720 ≡ 0 2240, to the original resolution (8,964×6,716 pixels) to extract 25 overlapping tiles of size 2,988×2,240 pixels 
using a stride of half the size; tiles, containing no annotation, are discarded.
Later, during training, to create varied training data for one epoch, we extract ten patches per tile, all of the same size as our validation/test patches.
Finally, all patches are resized to a height and width of 256×256 pixels to reduce model complexity.





 §.§ Segmentation

Our proposed architecture consists of an EfficientNet-B6 <cit.> encoder combined with a UNet <cit.> decoder, employing augmentations such as rotations, flips, and shifts.
A high-level overview of our methodology is depicted in <Ref>, which illustrates the per-patch level training (512×512, stride 256; resized to 256×256) that we employ to alleviate our lack of annotations.
At inference, predictions are performed for all patches and are recombined to form the complete segmentation mask of a mirror.



  §.§.§ Training

All our models are trained on an NVIDIA RTX A5000 until convergence, i.e., no improvement ≥ 1e-3 w.r.t. the pseudo-F-Measure (pFM; see Equation <ref>) for ten consecutive epochs, using a batch size of 32 and a learning rate of 3e-4.
As a loss function, we employ a generalized Dice overlap (Dice Loss; see Sudre et al. <cit.>), which is well suited for highly unbalanced data, and optimize it using Adam <cit.>.
Additionally, we incorporate a learning rate scheduler that also monitors the pFM on our validation set.
If there is no improvement for five consecutive training epochs, the learning rate is halved. 



  §.§.§ Augmentation

In order to enhance the robustness and generalization capability of our deep learning model, we explore augmentation techniques during the training phase.
In detail, we experimented with two techniques:
First, the CutMix augmentation strategy, proposed by Yun et al. <cit.>.
This technique involves cutting patches from one training image and pasting them onto another.
Second, MixUp, as proposed by Zhang et al. <cit.>, involves training our model on convex combinations of pairs of examples.
For this, we randomly sampled interpolation levels between 0.4 and 0.6.
Besides these two techniques, we also use standard augmentations, including flipping, shifting, and rotating.



  §.§.§ Self-Supervised Learning

Given the limited amount of annotations and the large amount of unlabelled data, we opt to explore a semi-supervised learning technique.
In the works by Chen et al. <cit.>, the authors introduce CPS, a new consistency regularization.
CPS enforces consistency by perturbing two networks with different initializations for the same input image, i.e., the pseudo-one-hot segmentation generated by one network is employed to supervise the other network and vice versa.
For this, we employ the masks generated by SAM to select only image patches that include mirror parts, and hence potentially include engravings. 
Lastly, an important hyperparameter is the weighing λ of this regularization term which needs to be tuned.



  §.§.§ Inference


After training our model, we need to adjust the methodology for inference to obtain a segmentation mask not only on a patch level but for whole mirrors.
After having acquired masks for individual patches, we recombine them using a custom weight map to avoid visual artifacts such as border edges.
Finally, we use the masks from SAM to mask False Positives (FP) from non-mirror parts.



§ EVALUATION

In this section, we evaluate our design choices.
In detail, we consider the effectiveness of employing different modalities, augmenting the input via CutMix, MixUp, and applying standard augmentations, training the model in a semi-supervised manner, and making global decisions such as exchanging the encoder/loss function.
Our initial Baseline, comprising an EfficientNet-B4 <cit.> as encoder and a UNet <cit.> as decoder, trains on all modalities initially optimizing the Binary Cross-Entropy (BCE) loss and does not incorporate augmentations nor semi-supervised learning. 

During this evaluation process, we report standard segmentation metrics such as the Intersection-over-Union (IoU), the Sørensen-Dice coefficient (Dice), the F-Measure (FM), as well as the pseudo-F-Measure (pFM).
The pFM is a metric commonly used for evaluating the binarization quality of handwritten documents and, hence, well-suited for our binarization task, i.e., a task where shifting the mask by a single pixel will have a major impact on per-pixel metrics such as the IoU.
Compared to the traditional FM the pFM relies on an alternative metric for recall, known as the pseudo-Recall (p-Recall), which is calculated based on the skeletonized version of the ground truth mask <cit.>:

    pFM = 2 ×p-Recall×Precision/p-Recall + Precision






 §.§ Ablation Study




Generally, for this section, tables are sorted in ascending order according to the pFM, excluding the first entry which denotes the previous best model, i.e., the model where the currently evaluated modification is not applied.
An overview of kept design choices and their impact are denoted quantitatively in <Ref> and qualitatively in <Ref>.
First, we remove the albedo and normal modality to yield a pFM of 43.8. Next, we incorporate augmentations such as flipping, rotating, and shifting, improving pFM performance to 46.2. The most significant change visually is achieved by incorporating the Dice Loss, pushing performance to a pFM of 47.7.
Finally, by employing a larger encoder (EfficientNet-B6), compared to our baseline, we improved predictive performance w.r.t. the pFM by around +16%, from 41.7 to 48.3.



  §.§.§ Modalities
 
Starting from our baseline, we first considered the usefulness of employing all modalities at hand versus different subsets.
For this evaluation, we stack subsets of modalities and evaluate their corresponding predictive performance.
The results of this can be viewed in <Ref>.
Inspecting the model's performance on these different subsets, we found that training solely on the depth modality performed significantly better (pFM of 43.8) than when trained on other combinations.
For instance, training solely on surface normals achieves a lower pFM of 42.3.
However, compared to other variations (all modalities and depth/normal plus albedo), the difference in performance is more noticeable. 
Hence, we opt for training solely on the depth modality due to its significantly better performance (two-tailed P-value of 0.032 over three runs) and simplicity.
An additional benefit is that depth maps are readily suited for standard augmentations as opposed to normal maps.





  §.§.§ Augmentation

Next, we consider the effectiveness of CutMix, MixUp, and standard augmentations applied combined or individually.
We observe that augmentation is important in our setting as applying flips, rotations, and/or shifts is beneficial, resulting in an improved pFM of 46.2.
This is in contrast to CutMix and MixUp, as their application did not achieve better results, at least w.r.t. the pFM.
Considering the Dice score, however, we find that applying MixUp alone yields an improvement of ca. +5%.
Hence, we evaluate employing MixUp in combination with standard augmentations which results in similar performance as applying only standard augmentations (two-tailed P-value of 0.6415 over three runs).
For more detailed results, we refer to <Ref>.





  §.§.§ Architecture

We continue with revising our choice in the loss function, testing replacing the BCE loss with a Focal loss <cit.>, Dice loss <cit.>, as well as a the multi-loss, consisting of all three (see Wazir and Fraz <cit.>).
Here, we found the Dice loss to improve performance by around +3%, achieving a pFM of 47.7, qualitatively significantly improving appearance.
Results of this experiment are denoted in <Ref>.
Furthermore, we consider different encoders/decoders for our architecture:
For the encoder, we upgraded to an EfficientNet-B6 (pFM of 48.3; considered variants from B5 up to B8) but kept the decoder the same, as no performance improvements have been observed.
Information on decoders tested and their corresponding performance are detailed in <Ref>.







  §.§.§ Cross-Pseudo Supervision

Finally, to potentially make use of the large portion of unlabelled data, we explore semi-supervised learning via CPS.
For this, we evaluate different weightings λ∈{0, 0.3, 0.5, 1} for the CPS loss term:
Employing a weight of λ=0.5 achieves a pFM of 48.1.
Compared to this, using a weight of λ=0.3 is slightly worse, achieving a pFM of 47.7, and λ=1 yields a pFM of 44.8.
However, the best performance is achieved by not employing CPS, as this obtains a pFM of 48.3.
For this reason, we discard this option and mention these results only because we consider them valuable to others faced with similar tasks, e.g., binarization of intricate handwriting or calligraphy.

On a final note, we want to point out that, although our explored augmentation techniques, i.e., CutMix and MixUp, as well as CPS, did not improve performance, we still recommend them in settings where the normal modality, a modality where standard augmentations are not readily available, outperforms the depth modality. 
In experiments different to these, when we trained solely on surface normals, employing CutMix in conjunction with MixUp improved the IoU by ca. +4% and adding CPS on top gained ca. +2%.







§ RESULTS


After verifying our design choices, we move away from evaluating the performance of our model on a patch level, towards testing it on whole mirrors: we select two mirrors of different conditions from our validation/test set and evaluate our model's performance on them as well as consider the influence of the thresholding value.
On top of this, we compare the impact of having a segmentation mask of the mirror object, in other words, a mask to remove FP from the background (non-mirror parts), onto the pFM. 

Inspecting <Ref>, we observe that the pFM is heavily dependent on a mirror's condition: considering mirror “Wels-11944”, the pFM is significantly higher than for the more damaged mirror “ANSA-VI-1701”.
As discussed in the beginning, this is expected as extracting drawings becomes ever more susceptible to subjectivity, depending on the level of damage a mirror has experienced.
We also note the significant differences in optimal threshold values, showcasing the difficulty of finding one to fit them all; however, the difference in performance is negligible, ranging only between 1 and 1.5.
Moreover, the use of a constant, naive threshold of 0.5 further narrows this range, suggesting skipping tuning altogether.

Next, under inspection of mirror “ANSA-VI-1700”, we assess the predictive performance qualitatively.
<Ref> illustrates the mirror with the prediction overlaid. 
Here, we note three areas to discuss in more detail:
First, at the left bottom lies an area where the interpretation of lines is subject to debate.
Despite this ambiguity, our model maintains objectivity, extracting only the lines that are clearly discernible.
Moving to the top, we encounter an area where the model struggles to distinguish between damage and deliberate lines, mistaking surface cracks for intentional engravings.
Lastly, in the bottom right corner, the model overlooks unmistakable engravings of the border decoration, failing to identify them as engravings, presumably attributable to its sloping surface rather than a flat one.
Note that additional examples of mirrors and respective predicted segmentation masks are provided in the Appendix.





 §.§ Human Baseline


Finally, we evaluate our model against a human baseline.
For this, we selected a mirror to be drawn twice to better understand the extent of subjectivity in annotations when an artwork is traced. 
Given that some of our mirrors are in pristine condition, we select a heavier damaged one, i.e., mirror “ANSA-VI-1700” from the KHM, such that the segmentation task is more prone to misinterpretation.
Although the same annotator traced this mirror twice, we kept four months in between to reduce the influence of the first tracing on the second.
With an IoU of around 37 and a pFM of 56, <Ref> illustrates the large room for interpretation when annotating engravings from Etruscan mirrors.



For comparison to these results, we evaluate the predictive performance of our model and contrast it with one local (Sauvola <cit.>) and one global (Otsu <cit.>) binarization method; results are denoted in <Ref>.
With a pFM of around 57.5/53.6, depending on the considered annotation, our proposed methodology scores better/slightly worse, achieving performance comparable to a human annotator.
Furthermore, it clearly sets itself apart from existing binarization methods, performing ca. 76%/74% better.



§ CONCLUSION

In conclusion, our work presents a pioneering effort in automating the segmentation of Etruscan hand mirrors, aiming to enhance objectivity, improve annotation quality, and reduce the associated workload.
By leveraging photometric stereo alongside deep segmentation networks, we have tackled the labor-intensive task of manually tracing engravings from the backside of these historical artifacts.
After conducting thorough experimentation and refining our methodology, including data augmentation, semi-supervised learning, as well as general architectural changes such as different loss strategies, we have surpassed our initial baseline by approximately 16% in terms of the pseudo-F-Measure.
Moreover, compared to a human baseline, our approach demonstrates similar performance to human annotators in quantitative terms.
By mitigating the inherent subjectivity introduced by damage on these mirrors, our methodology not only streamlines the annotation process by drastically reducing workload but also improves the quality and objectivity of annotations by offering a supplementary perspective, overall contributing to the examination process of Etruscan hand mirrors.

In summary, our work offers a practical solution to the challenges associated with analyzing these historic artifacts, representing advancements in the application of advanced imaging techniques and deep segmentation in the context of non-traditional documents and cultural heritage in general.
For future work, we aim to further improve objectivity by addressing the subjectivity of underlying annotations by creating a golden standard, i.e., a set of annotations that experts generally agree on.
Another avenue we want to explore is the idea of collaborative annotation, where our model works in tandem with human annotators to iteratively refine annotations, i.e., suggestions provided by humans serve as valuable priors to update the posterior of our model, to further accelerate and qualitatively enhance the annotation process.rafael was here







splncs04







§ APPENDIX

In the following, we append additional visual results for the reader to enjoy.
On top of this, we add a visualization of the weight map used to combine predicted patches to form a complete mirror.
It is designed such that the center is most influential and less influenced by the border of adjacent patches.














