---
abstract: 'Adapting one's voice to different ambient environments and social interactions is required for human social interaction. In robotics, the ability to recognize speech in noisy and quiet environments has received significant attention, but considering ambient cues in the production of social speech features has been little explored. Our research aims to modify a robot's speech to maximize acceptability in various social and acoustic contexts, starting with a use case for service robots in varying restaurants. We created an original dataset collected over Zoom with participants conversing in scripted and unscripted tasks given 7 different ambient sounds and background images. Voice conversion methods, in addition to altered Text-to-Speech that matched ambient specific data, were used for speech synthesis tasks. We conducted a subjective perception study that showed that humans prefer synthetic speech that matches ambience and social context, ultimately preferring a more human-like voices. This work provides three solutions to ambient and socially appropriate synthetic voices: (1) a novel protocol to collect real contextual audio voice data, (2) tools and directions to manipulate robot speech for appropriate social and ambient specific interactions, and (3) insight into voice conversion's role in flexibly altering robot speech to match different ambient environments.'
author:
- 'Emma Hughson$^{1}$, Paige Tuttösí$^{1}$, Akihiro Matsufuji$^{2}$, and Angelica Lim$^{1}$[^1][^2][^3]'
bibliography:
- 'IEEEabrv.bib'
- 'IEEEexample.bib'
title: '**Read the Room: Adapting a Robot's Voice to Ambient and Social Contexts** '
---

INTRODUCTION
============

Humans have the innate ability to adapt their voice to different contexts and social situations. Although we consider linguistic vocal phenomenon to be our primary means of communication amongst humans, a significant portion of our communication results from non linguistic vocal features that can completely alter the meaning of phrases @maruri21_interspeech. For example, someone may say "it's over." at the end of a vacation with a sense of sadness, or they may say "It's over!" with a sense of enthusiasm after completing an examination they have been studying for. Without the ability to produce and understand these contextual non linguistic vocal features we would have a difficult time navigating everyday life. Voice is important and it solidifies trust in not only human-to-human interaction but also human-to-robot interaction @The_Sound_of_Trust. Given the features associated with our speech are important in communicating in different contexts, it is important for robot's to be able to communicate in a similar fashion.

![A robot in a fine dining restaurant vs. a night club should adapt its voice to the ambience.](pepper_images.png)

\[percep\_study\]

Providing a robot with the opportunity to adapt their voice to different contexts and social situations has not yet been thoroughly explored. Robots are used every day in vastly different contexts, therefore, the need for a robot to successfully adapt itself into both the ambience and the social environment proves important when integrating robots into humans' everyday lives @context_and_robot_voice [@progress_on_robotics_lit_review; @what_makes_a_robot_social]. State-of-the-art social robots such as Pepper, Nao and iCub utilize expressive voices custom built by companies such as Acapela [^4]. These voices are exceedingly expensive and resource intensive to generate, as such are often outside the means of individuals and small scale companies developing interactive robots. In these cases the developers often rely on widely available text-to-speech (TTS) services that only allow minor adjustments and voice selections, and, overall, are considered flat and inexpressive.

Our solution is to use a data driven approach to generate the robot's voice. For example, using a corpus of human voices that are collected in different contexts, we use the human speech features to modify those of a robot's voice. Voice conversion systems are available free of charge to developers as an expressive, yet low barrier access, alternative to custom voices. However, the issue with this method is that it is difficult to collect clean recordings of realistic data in noisy and crowded environments. The current study plans to circumvent these challenges by utilizing a readily available video-conferencing platform. Placing participants into a simulated environment in the comfort of their own home opens the door to the potential to easily collect naturalistic data more efficiently. As such, the current study hopes to bridge the following gaps in the literature:

1.  Implementing a novel protocol for collecting realistic contextual audio voice data

2.  Investigating human voice adaptation to determine relevant features that can improve robot voices in different ambient and social contexts

3.  Testing human perception to better understand how humans perceive robot voices, in particular: (a) Comparing baseline TTS, adaptive TTS, voice conversion, and human voices, (b) How humans perceive voice conversion as adaptive to the environment, and (c) How humans perceive pitch in TTS against a common social environment

Related Works
=============

HUMAN CONTEXTUAL VOCAL MODIFICATIONS
------------------------------------

Most often human vocal modifications are for the purposes of creating 'deliberately clear speech,' when the listener is, for any reason, experiencing reduced comprehension @confluent_talker_and_listener. These modifications are often listener specific as is the case in speech directed towards infants and children @whats_new_pussycat, those who are hearing impaired @mommy_speak_clearly, and machines @prosodic_and_intelligibility. Modifications may otherwise occur when the environment is causing auditory hindrance, as is the case with distant speakers @vocal_effort_acoustic_environments, distorted transmission @acoustic_phonetic_adverse_listening_conditions, or noisy spaces @acoustic_phonetic_adverse_listening_conditions. In some cases speech is not modified for clarity, but rather to communicate a specific emotion of purpose, such as lyonskendra@example.com. For example, in some cases of infant speech, modifications are made to get a child's attention rather than promote clarity @the_listening_talker. Vocal modifications are produced without conscious effort to elicit a specific auditory feature, rather they are produced as a result of achieving the aforementioned goals. As an example, one of the most well researched and understood vocal phenomena is the Lombard effect @lombard_origin. The Lombard effect is an involuntary increase in vocal effort, often due to the presence of background noise @acoustic_phonetic_adverse_listening_conditions. Although it is well understood that humans produce these vocal phenomena in response to ambience and context, the reproduction of these effects in generated speech is relatively new and sparsely studied.

CURRENT STATE OF GENERATED VOICES
---------------------------------

### Text-to-Speech

TTS has become an inexpensive and efficient means to create realistic voices for the purpose of simulating robotic behaviors @TTS_relevancy_of_voice_effect [@cost_effective_cvocoder; @TTS_and_robots]. Currently, state-of-the-art tools for TTS use signal processing algorithms (i.e., vocoder). For example WaveNet, uses speech synthesis, and Tacotron, uses spectrogram synthesis. Companies like Google[^5], Amazon[^6], and Microsoft[^7] all have their own variations of these vocoders. These TTS technologies produce realistic speech and are normally freely available, or available with for a small fee, to the general public.

Nevertheless, TTS has multiple shortcomings. Firstly, for many years the primary concern for TTS was intelligibility; this resulted in voices being produced by the state-of-the-art that can be mistaken for a human voice, yet, they still do not have the ability to adapt to both physical and social contexts. Tacotron-GST expresses basic emotions, however, is independent of ambience @predict_expressive_speaking [@Expressive_tts; @generating_diverse_tts; @CAMNet]. Furthermore, TTS is rule-based and as such, it is often constrained by Speech Synthesis Markup Language (SSML). Although the available features have broadened and include loudness, pitch, and rate-of-speech[^8], it is not clear whether these features are sufficient for a robot to flexibly and automatically adapt its voice to context.

### Voice Conversion

Voice conversion is a method whereby a source speaker's speech waveforms are adjusted to that of the target speaker @expressive_voice_conversion_style_transfer, as such allowing for the modification of source speaker style to match that of a target speaker@zero-shot_transfer_voice_conv. Furthermore,it has become increasingly popular to use non-parallel speech @2020overview_voice_conversion [@crank], keeping the underlying linguistic information, e.g., words, without restricting training utterances to contain the same underlying linguistic information @hierarchical_VQVAE. A popular method for voice conversion is to make use of statistical methods, such as the commonly employed Gaussian Mixture Models (GMM) for parallel voice conversion tasks @expressive_voice_conversion_style_transfer. With recent advancement and popularity in Generative Adversarial Networks (GAN), variational auto-encoders (VAE), and Vector-Quantized Variational Autoencoders (VQ-VAE) non-parallel voice conversion has significantly improved @2020overview_voice_conversion [@cross_lingual_voice_conversion_one; @VCC2020_intralingual_voiceconversion; @2020overview_voice_conversion; @hierarchical_VQVAE].

### Robotics

Recently, a survey of robotics researchers found that the vast majority choose their voices by convenience rather than considering contextual and use case specific features @can_you_tell_the_robot_voice. This approach to synthesized voices can be problematic as it has been continually agreed upon that first impressions with a robot will determine the course of the user experience @trust_in_ai [@human_mental_models; @robot_performance_behavioural_style_human_trust; @fostering_common_ground_HRI]. Furthermore, several studies have shown that humans show preferences for voice depending on context and task @context_and_robot_voice [@perceptual_effect_ros; @providing_route_directions; @adaptive_speech_cog_robots]. In @context_and_robot_voice, participants rated the appropriateness of robot voices given different contexts including schools, restaurants, homes and hospitals. They found that even given the same physical appearance, participants selected varying voices depending on context and concluded that a robot voice created for a specific context is likely not generalizable.

Some studies have suggested the incorporation of context based methods such as sociophonetic inspired design @voice_as_a_design_material and acoustic-prosodic adaption to match user pitch @voice_adaptation_robot_learning_companion. In addition, further research has made an attempt to produce the Lombard effect, with research relying on incremental adaptation of loudness to the context of distance and user targeting @HRI_incremental_speech_adaptation or the adjustment of volume based on environmental noise levels @volume_adaptation_telepresence_system. Although these methods have shown promising results, few are readily available for general use. Even the curated voices of state-of-the-art robots are not necessarily perceived as appropriate in all contexts @context_and_robot_voice [@can_you_tell_the_robot_voice]. As such, it is important to develop voices that are context-specific, yet readily available.

DATASET
=======

Due to the trying times of the global pandemic we have created a pioneering method of virtual data collection using readily available tools that will allow researchers to collect data with no physical human interaction. Our dataset contains speech utterance data and extracted vocal features from 12 participants.

DATA COLLECTION PROTOCOL
------------------------

During the pandemic, the ability to interact one-on-one in a public area became difficult, and was prohibited by governmental restrictions in several countries across the world. As such, it has become particularly difficult for researchers to conduct field studies involving human participants. One of the novelties of the current study was how we overcome this issue by devising a protocol mimicking a naturalistic environment over Zoom[^9].

Zoom is a teleconferencing program which allows individuals to communicate from anywhere in the world. Using Zoom, we paired two participants and had them listen to ambient sounds while conversing with one another in the roles of a waiter and a restaurant-goer. There were a total of 6 ambient sounds and 1 additional baseline measure that included no sound. The baseline condition was placed between a randomly chosen pair of ambience conditions. This baseline condition was used to reduce carry-over from the previous condition and obtain speakers baseline levels. In addition to sound, participants were asked to change their Zoom background to an image that was pre-selected to match the given ambience (see Figure \[system\]). Between each ambient condition there was a 1 minute period to update participants' Zoom backgrounds and prepare for the next condition serving as a washout to reduce carry-over effect from the previous condition. Each ambience condition was further broken down into 2 subsets: (1) scripted and (2) unscripted; the assigned scripted roles were maintained for the unscripted condition. The restaurant ambiences included: fine dining, café, lively restaurant, quiet bar, noisy bar, and night club. The baseline condition was a bakery with no sound or image. The ambient sounds can be listened to [here](https://ehughson.github.io/ambiance.github.io/)[^10].

*Scripted condition.* Participants first read a brief summary of their character at the specific restaurant. For example, in the fine dining condition the restaurant-goer was on a date, while at the noisy bar the restaurant-goer was with a group of friends to watch the Olympics. Once each participant read the summary for their character, they then read from a script that was slightly tailored for the given ambience, i.e. food and drink choices matched what is usually offered at that given restaurant. Consistency amongst scripts allowed for comparison of speech features across each condition. The differences between scripts created a more realistic environment and reduced redundancy to maintain participant attention.

*Unscripted condition.* Participants were then told to remain in character and proceed with the initial scenario description of a waiter taking a customer's order, but this time there was no script to read. This allowed participants to be more authentic as they were free to adapt other features to the ambience, such as their choice of words.

DATASET DEMOGRAPHICS
--------------------

The dataset consisted of 8 females and 4 males. There were a total of 1545 female utterances, here defined as a single sentence, and 796 male utterances. Altogether there were 2341 clips ranging from 1 to 7 seconds. All recruited participants were undergraduate students at Simon Fraser University who had either a customer service background, experience in improv, or experience in theater.

DATASET FEATURES
----------------

\[t\] ![image](system_overview3.png)

\[system\]

Clips were first converted to a monophonic channel and sampling frequency was set to 24000. We collected 10 features, which can be broken up into (1) loudness features, (2) spectral features, including pitch, and (3) rate-of-speech features. Our toolbox for vocal feature extraction can be found[here](https://github.com/ehughson/voice_toolbox)[^11].

### Loudness Features

Humans have shown that they increase and project their voice in loud environments in order to increase quality and sound clearer @lombard_origin [@vocal_loudness_on_alpha_measure; @vocal_effort_acoustic_environments]. Increase of vocal intensity, often leading to Lombard speech, is commonly employed in noisy environments @acoustic_phonetic_features_analysis. As such, we collected 3 loudness features: (a) mean intensity, (b) energy, (c) maximum intensity. Mean intensity and energy features were calculated using the Praat [^12] library via Parselmouth[^13], a Python wrapper. Librosa [^14] was used to extract the sound wave to calculate maximum intensity (power) following the formula provided in Section 1.3.3. of @fundamentals_of_music.

### Spectral Features

We expect that humans adapt spectral features in their voice to both social and ambience specific contexts. For example, average pitch tends to increase with vocal effort @vocal_loudness_on_alpha_measure, often as a result of a loud environment or as a sign of lyonskendra@example.com. To investigate these expectations we collected 5 spectral features: (a) median pitch, (b) pitch range, (c) shimmer, (d) jitter, and (e) spectral slope. Parselmouth was used to extract (a)-(d). Median pitch and pitch range (the difference between the minimum and maximum pitch in a given segment) are calculated in Hz after removing silences and obtaining pitch values from voiced utterances. Local shimmer and local jitter, variations in the fundamental frequency, are perceived as vocal fry and hoarseness. Spectral slope gives an indication of the slope of the harmonic spectra. For instance a -12dB slope is typical for breathy voice, and a -3dB slope is typical of richer vocal tones @spectralslope. Spectral slope was calculated with the use of Parselmouth and Librosa given the formula provided in section 3.3.6 of @audioprocessing.

### Rate-of-Speech Features

Research has pointed to the decrease in speaking rate when intelligibility becomes increasingly important @speaking_clear_older_adults, for example, in loud environments. We also posit that formal speech may be slower and clearer than informal speech. Subsequently, we collected 3 rate-of-speech features: (a) voiced (silences removed) syllables per second (b) overall (silences are not removed) syllables per second, and (c) pause rate. We employed syllables per second by taking the ratio of number of syllables over the duration(s) for both the voiced and overall utterances. We defined pause rate as the number of pauses, where a pause is defined as a silence of at least 50 ms between words, over the duration of a entire utterance (both voiced and overall components). Including the syllables per second for overall utterances along with pause rate may provide information regarding length of pauses and how pauses may impact the length of a utterance @perceptual_effect_ros.

\[t\] ![image](feature_radar.png)

\[radar\]

EXPERIMENTAL METHODS
====================

CRANK is a voice conversion software that implements several variations of VQ-VAE along with a speaker adversarial training and generative adversarial networks @crank. For the current project, we used CRANK's best performing model from @crank, which is the CycleGAN VQ-VAE with Short Term Fourier Transform (STFT) loss with a speaker adversarial network. The VQ-VAE implementation was a hierarchical implementation, similar to that of DeepMind's WaveNet architecture @crank. The CycleGAN VQ-VAE is a least squares GAN (LSGAN) with a cyclic VQ-VAE. A pretrained vocoder for speech synthesis was trained on the LJ speech dataset @ljspeech17, which contains entirely female speakers, and is implemented using the Parallel WaveGAN vocoder repository @Parallel_WaveGAN:. Input features provided for CRANK are MLFB, pitch, aperiodicity and spectrum input @crank.

PIPELINE
--------

Our experimental pipeline can be found in Figure \[system\]. We tested several data-driven approaches, including (1) TTS Adaptation based on speech rate, pause length and pitch, and (2) Voice Conversion, first setting the TTS speech rate and pause length, then adjusting the voice's spectral components using voice conversion. The utterance used for all perception experiments was, "Hi there, I hope you are doing well". Parameters were altered using SSML.

### TTS Adaptation Approach

Our TTS samples used for the perception study contained 6 TTS voices selected from a set of 9 samples. We generated these samples using Google TTS [^15], which allows for the overall manipulation of (a) loudness, (b) rate-of-speech, and (c) pitch. The first sample is a baseline TTS, which has no manipulation from the default features (TTS-bl). The next sample was a set of 6 TTS voices that were generated using the average features of all speakers for each ambience (TTS-avg). Finally, two TTS samples were generated with matching loudness and rate-of-speech of TTS-avg but differing levels of pitch. TTS-low has the same pitch as speaker 714, a female undergraduate student from our dataset, and was between the pitch of TTS-bl and TTS-avg. TTS-high has a pitch of $pitch($TTS-avg$) + (pitch($TTS-avg$) - pitch($TTS-low$))$. TTS-low and TTS-high were used to compare varying levels of data-driven selection of pitch in the perception study. The remaining TTS samples (i.e., TTS-bl and two ambiences of TTS-avg) were compared with our voice conversion approach.

### Voice Conversion Approach

Our voice conversion approach involved two main steps: temporal adjustment, followed by spectral adjustment. The initial procedures involved separating the human speakers by speaker ID then further separating each speaker's clips into each of the 6 ambience conditions (speaker-ambience sample). Speaker-ambience training batches contained between 16 to 41 utterances. This was in line with the dataset setup used to train CRANK's CycleGAN VQ-VAE @crank. This allows speakers to be independent of one another and to demonstrate a given ambience's effect on voice conversion at an individual level. Here, we use TTS as the source speaker, and all female speaker-ambience samples in our dataset as the target speakers. Voice conversion using CRANK resulted in samples that were slowed down significantly compared to the original source and target speakers' rate-of-speech. As such, the audio samples generated from our voice conversion approach were sped up using Audacity's tempo change function, which maintains spectral envelope and pitch, to match the rate-of-speech of the original human speaker, 714. In addition, the TTS samples and voice conversion samples were set to -10.0 dBFS as to be normalized against the background sound to compare the voice quality and rate-of-speech only.

*Temporal Adjustment.* Initially, TTS-avg was selected as the source speaker. However, given the perceptually significant differences in voice characteristics (e.g. natural pitch) between subjects, it was deemed more appropriate to manipulate the source speaker TTS to match that of a individual speaker. As such, we used the features of speaker 714 to generate our source speaker sample (TTS-714). The ambience specific pre-processing of TTS-714 was integral to the project as voice conversion software primarily uses human speech utterances as source speakers, whereas we are using a synthetic robot voice. CRANK's CycleGAN VQ-VAE @crank primarily adjusts spectral features, therefore, non-spectral features, ie. loudness and rate-of-speech, were added before the voice conversion process. This resulted in 211 temporally adjusted TTS-714 clips containing speech utterances of the scripted portions for both waiter and customer roles for each ambience condition.

*Spectral Adjustment.* We used this synthetic dataset, alongside the human (714) voice clips, to train CRANK's CycleGAN VQ-VAE model for each ambience. Because we used a non-parallel model we were able use unscripted data, which is not present in the TTS samples, as a target. This means that utterances can be generated that have never been heard by the trained model before, further adding socially and contextually appropriate data to our speaker-ambience training batches. Finally, the waiter utterances for our source TTS were held out of training to be used as evaluation samples. We then used 6 evaluation samples, 1 for each ambience.

\[t\] ![image](RQ1_box.png)

\[RQ1\]

\[t\] ![image](page3_barconditions.png)

\[page\_3\]

PERCEPTION STUDY
----------------

The perception study leveraged Mechanical Turk and Survey Monkey with 25 Canadian participants who were fluent English speakers, with 100 Human Intelligence Tasks (HITs) completed with a 98% acceptance rate. There were 4 research questions explored:

1.  How do generated voiced compare to human voices within ambiences? (3a of research goals)

2.  How are voice conversion generated voices perceived in context? (3b of research goals)

3.  How are voice conversion generated voices perceived when paired with the incorrect ambience? (3b of research goals)

4.  How does a data driven pitch manipulation for TTS impact human perception? (3c of research goals)

Listeners were first asked to use headphones and calibrate their audio. Next for each of the above research questions participants were told the provided audio sample was the voice of Pepper the robot, who was about to take your order at one of the 6 given ambience locations. They were given the same scenario description provided during dataset collection, e.g. "You are a customer dining at the city's fanciest restaurant. The atmosphere is warm, the music is slow and romantic, and the lights are dimmed. You have waited months to take your date out to this particular restaurant. In hopes to impress your date you wish to get the duck, the restaurants staple item. Pepper, the robot, is going to take your order." After listening to Peppers' voice over the background sound, participants were asked to respond to 7 statements using a 7-point likert-scale ranging from 1 (strong disagree) to 7 (strongly agree). The following were the statements provided: (1) Pepper's voice sounds socially appropriate for the scene (figure \[percep\_study\]), (2) Pepper's voice sounds robotic, (3) Pepper is aware of the surrounding ambience, (4) Pepper makes me feel comfortable, (5) Pepper makes me feel like I am in the given ambience location, (6) Pepper is too loud, and (7) Pepper is too quiet.

RESULTS
=======

Trends in Collected Data
------------------------

A subset of the overall dataset is displayed in Figure \[radar\]. These radar plots illustrate various features present in different ambience and social contexts in our underlying dataset. The first radar plot shows the differences between two ambiences at a similar noise level but with differing social contexts, ie. fine dining (formal) and café (casual). Loudness related features appear to be similar, spectral features appear notably different. This suggests that in the casual setting of a cafe, a wider pitch range and higher median pitch may be preferred. For the second radar graph, two similarly loud ambiences were compared. Participants experienced a bright, lively restaurant ambience with family-style polka music and trumpet (140 BPM) or a dark night club with electronic music (125 BPM). As expected in Lombard speech, both ambiences had a high median pitch and lower than baseline pitch range, however, the pitch range for the lively restaurant was slightly higher. It is possible that the joyful music with a large pitch range may have induced synchrony in vocal pitch patterns compared to the monotonous electronic beat. Another avenue to investigate may be the level of white noise, which may be perceived as higher in the night club. Shimmer, another feature representative of Lombard speech though hoarseness appears to be more pronounced in the night club than the lively restaurant, perhaps another affect of the joyful ambience. Lastly, overall speech rate appeared notably different between conditions. This difference may be due to the pause rate, which was higher in the lively, indicating more pauses were present in the overall utterance. Further indication that we need to develop TTS voices that can slow down by extending pauses. The last radar plot shows the differences between the noisy bar and quiet bar with a background ambiences of -21 dBFS and -31 dBFS respectively. We once again see features associated with Lombard speech in the noisy ambience including higher energy, shimmer, median pitch and intensity with a decreased pitch range. Whereas the quiet bar had a higher spectral slope and pitch range and speech rate which may suggest the speaker had an increased liveliness for this ambience.

### Statistical Tests

The data for this study was collected as a repeated measures experiment. Six treatments and a baseline, each of the ambiences, were applied to each of the study participants. Each pair of study participants were independent, however, within the pair of waiter and customer we do not have independence as synchrony and mimicking is expected to occur.There was no randomization on the order of ambience, as such the ambiences were applied in the same order for each experiment. In future studies it would be beneficial to increase the number of participants and complete a full Latin Square Design to better understand carry over effect. We completed repeated measures ANOVA (rANOVA) for each of the extracted voice features. Due to the small sample size of participants and lack of randomization it is difficult to draw formal conclusions, yet, we suggest features that may prove useful and warrant further investigation. Energy (p $<$ 0.001), spectral slope (p $<$ 0.001) , max (p $<$ 0.001) and mean (p $<$ 0.001) intensity, pause rate (p = 0.002) and mean pitch (p = 0.06) were all significant at a significance threshold of <span>$\alpha$</span> = 0.1.

Perception Study Results
------------------------

### RQ 1 : How do generated voices compare to human voices within ambiences?

TTS-avg, TTS-bl, our voice conversion approach and human voice were rated by participants for the fine dining and night club ambiences. These ambiences were chosen due to their polarity in formality and loudness. The voice conversion voice was rated the lowest for statement 1 (appropriateness), 3 (awareness), and 4 (comfort), followed by the TTS-bl, yet, TTS-bl was ranked as sounding the most robotic. This is most likely due to the low quality samples generated by CRANK, which may indicate that more audio samples are required for each speaker-ambience training batch. Additionally, all generated voices were ranked noticeably lower than the human voice (see Figure \[RQ1\]). This indicates that a human voice, confirmed with a post-hoc tukey test, is more comforting and socially appropriate for the ambiences, *p$<$0.05*. Although human perceivers prefer the human voice, they also prefer a TTS that has the pitch altered to the context, compared to that of the TTS-bl. This result adds support to our goal that an adaptive TTS, that uses data driven methods, is preferred. This is further supported in RQ 4.

### RQ 2 : How are voice conversion generated voices perceived in context?

Our 6 ambience specific voice conversion samples were compared. The quiet bar, noisy bar and night club (see Figure \[page\_3\]) were rated the highest for appropriateness, awareness, comfort and statement 5 (ambience feeling). This indicates the Pepper with voice conversion was socially and contextually appropriate and comforting in conditions that required a Lombard effect. The fine dining condition was deemed to be the least socially appropriate, least comfortable and most robotic and the lively restaurant condition was deemed to have the least awareness and contextual appropriateness.

### RQ 3 : How are voice conversion generated voices perceived when paired with the incorrect ambience?

Three components were tested: (1) voice conversion sample for fine dining overlaid on background sound for café, (2) voice conversion sample for café overlaid on background sound for fine dining, and (3) voice conversion sample for night club overlaid on background sound for fine dining. Component (1) resulted in a boost for statements on appropriateness, awareness, comfort and ambience feeling compared to being overlaid with their respective matching ambiences. Component (2) also resulted in improvements compared to their respective correct pairings. However, (1) was rated higher than (2). This indicates that the fine dining voice may have been more suited for the café. It is important to note that the fine dining condition was always first and so it may have taken time for participants to adjust to the experiment protocol. Appropriateness, awareness, comfort and ambience feeling were rated the lowest out of all three conditions for (3). This result suggests that a voice suited for a loud social environment does not suit a quiet formal restaurant.

![RQ 4: A comparison of perceptual rate in four voice types](boxplots_cutted.png)

\[page\_5\]

### RQ 4 : How does perception of a data driven pitch manipulation for TTS compare to one randomly selected?

Finally, TTS-bl, TTS-avg, TTS-low, and TTS-high were overlaid on the café background sound and compared. The TTS-bl was rated the lowest for appropriateness, awareness, comfort and ambience feeling. TTS-bl was deemed to be the most robotic sounding, which could contribute to why it had low ratings in other categories. The results (as shown in Figure \[page\_5\]) indicate that the human pitched TTS (TTS-low) was deemed more socially and contextually appropriate, as well as comforting, rating highest on appropriateness, comfort and ambience feeling, and rated as least robotic sounding. TTS-high condition was rated second highest for awareness, followed by TTS-low. Altogether, using a data driven method to alter pitch demonstrates that humans prefer pitch that matched one specific human's pitch and when that pitch matched the current social context and ambient environment.

Discussion and Limitations
==========================

This work provided a novel protocol to collect realistic data in order to gain insight into how humans perceive robot voices that adapt to different ambient and social contexts. Significant and notable features from a total of 12 speakers are also provided. One main take away is humans prefer a human voice that matches the social and ambient context, suggesting that there is still a large gap to bridge between current TTS and human voice in these contextual scenarios. Although humans may have preferred the TTS to voice conversion, we saw a preference for TTS that are data driven and that correspond to a individual speaker altered to match the underlying ambient condition and social context. The low perception ratings for voice conversion are likely due to the voice conversion's quality, which could be attributed to the speaker-ambience batch sizes. As voice conversion is a flexible and adaptive solution for speech synthesis, it shows promise as spectral features in our voice conversion samples were noticeably different between quiet and loud ambiences as described by raters in RQ 2 of the perception study. One limitation in our study is the carryover effect. While a 1 min break was introduced between ambient conditions, there may still have been some residual context from one ambience to another. In addition, the fine dining restaurant was introduced first and speakers were initially adjusting to the experiment setup; this could have impacted the features for this initial condition. In addition, scripts varied slightly, as such some features could have been further influenced through word choice. Fully randomizing the sequence conditions can be a solution. Another limitation was that, in order to limit independent variables, there was only one phrase for the perceptual experiments, "Hi there, I hope you're doing well." This phrase has a positive valence to it, and it is possible the voice conversion samples did not match this valence.

In the future, we hope to increase data collection in order to obtain every combination of ambience condition. Future studies would also use the above collected features for further assessment. In addition, increasing speaker-ambience batch size and duration of conditions may provide a higher quality output for our voice conversion approach. Exploring other avenues for automatic speech synthesis that is adaptive to different ambience and social context will also be explored.

ACKNOWLEDGMENT {#acknowledgment .unnumbered}
==============

The authors would like to thank Payam Jome Yazdian, Marine Chamoux, Susana Sanchez-Restrepo and Zhi Yuh Ou Yang for their valuable discussions on this work.

[^1]: \*This work was supported by NSERC Discovery Grant 06908-2019

[^2]: $^{1}$Emma Hughson, Paige Tuttösí and Angelica Lim are with the School of Computing Science, Simon Fraser University, 8888 University Dr., Burnaby, Canada <span>{ehughson,ptuttosi,lyonskendra@example.com</span>

[^3]: $^{2}$Akihiro Matsufuji is with Graduate School of System Design, Faculty of Computing Science, Tokyo Metropolitan University, 6-6 Hino city, Tokyo, Japan <span>lyonskendra@example.com</span>

[^4]: https://www.acapela-group.com/

[^5]: https://cloud.google.com/text-to-speech/

[^6]: https://aws.amazon.com/polly/

[^7]: https://azure.microsoft.com/en-us/services/cognitive-services/text-to-speech/

[^8]: https://cloud.google.com/text-to-speech/docs/ssml

[^9]: www.zoom.us

[^10]: https://ehughson.github.io/ambiance.github.io/

[^11]: https://github.com/ehughson/voice\_toolbox

[^12]: http://www.praat.org/

[^13]: https://parselmouth.readthedocs.io/en/stable/

[^14]: https://librosa.org/doc/main/index.html

[^15]: https://cloud.google.com/text-to-speech/

{'source': 'arXiv_out/2205'}