ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modeling
Arjun Somayazulu Sagnik Majumder Changan Chen Kristen Grauman





    
    April 29, 2024
==================

empty
empty




An environment acoustic model represents how sound is transformed by the physical characteristics of an indoor environment, for any given source/receiver location. Traditional methods for constructing acoustic models involve expensive and time-consuming collection of large quantities of acoustic data at dense spatial locations in the space, or rely on privileged knowledge of scene geometry to intelligently select acoustic data sampling locations. We propose active acoustic sampling, a new task for efficiently building an environment acoustic model of an unmapped environment in which a mobile agent equipped with visual and acoustic sensors jointly constructs the environment acoustic model and the occupancy map on-the-fly. We introduce ActiveRIR, a reinforcement learning (RL) policy that leverages information from audio-visual sensor streams to guide agent navigation and determine optimal acoustic data sampling positions, yielding a high quality acoustic model of the environment from a minimal set of acoustic samples. We train our policy with a novel RL reward based on information gain in the environment acoustic model. Evaluating on diverse unseen indoor environments from a state-of-the-art acoustic simulation platform, 
ActiveRIR outperforms an array of methods—both traditional navigation agents based on spatial novelty and visual exploration as well as existing state-of-the-art methods.






§ INTRODUCTION


The acoustic properties of the sounds a mobile agent hears are determined by the physical characteristics of the space it is in, such as the room's geometry, the objects within it, the types of materials that comprise the room and objects' surfaces, and the agent's 
proximity and orientation with respect to the sound source. Consider a conversation with someone standing across a large auditorium or gym, compared to the same conversation sitting a few feet apart in a small, carpeted living room; Large spaces with hard surfaces (concrete, glass) will add reverberation to audio, while small, cluttered spaces covered in soft materials (curtains, carpet) absorb sound waves quicker, producing dull and anechoic audio. These physical properties transform any sound emitted from a source location in the environment according to various acoustic phenomena, including direct sounds, early reflections, and late reverberations, before it reaches our ears. Together, these phenomena form a Room Impulse Response (RIR), which characterizes the transfer function between a sound emitted at the source location and the sound that reaches a microphone or our ears <cit.>.


    
An environment acoustic model is a complete representation of a scene's acoustics <cit.>.  Given a sound source location and a listener's position and orientation (pose) as a query, the model renders the corresponding RIR, accounting for all major acoustic phenomena due to the physical properties of the space.

Environment acoustic models are critical for robotics. In mobile robotics, an environment acoustic model can provide rich contextual information to an agent. Tasks such as navigating to a target sound <cit.> and separating out a sound of interest from background sounds <cit.> require an agent to decide where to move based on the audio it hears. An agent equipped with the environment acoustic model can better anticipate the effects of its movement on the observed audio.
In AR/MR applications, acoustic models  allow a virtual sound source (such as a human speaker)
inserted at a position in the user's real-world space to sound properly spatialized with acoustics that match the space, as the user moves around their environment.

Capturing an RIR is a physically involved process. A loudspeaker must be placed at the desired height and location, and a microphone setup placed at a similar height at the desired receiver location and orientation. The speaker emits a sound impulse, and the receiver microphone(s) record the resulting RIR, which can last several seconds. 

Existing methods for environment acoustic modeling assume extensive physical access to the environment in order to collect these RIRs at arbitrary positions.
Neural Field methods <cit.> require large quantities of RIRs captured at dense source/listener locations (approx. every 1 meter) throughout the environment, which can be expensive and time-consuming. A few-shot approach <cit.> overcomes the intensive data requirement, building acoustic models of novel environments from a limited number of observations, but still requires extensive knowledge of the floorplan in order to sample uniformly spaced locations around obstacles. Other methods rely heavily on knowledge of the scene geometry from 3D meshes <cit.> or floorplans <cit.>. Importantly, these methods all assume instantaneous access to observations at arbitrary positions in the environment, which is unrealistic in the embodied robotics context—where an agent must physically travel between locations—and in unmapped environments where no prior knowledge of the floorplan and obstacles is available. 


We introduce active acoustic sampling, a new task that requires a single mobile agent with audio-visual sensing to efficiently construct an unmapped environment's acoustic model within a total budget of acoustic samples, despite only on-the-fly discovery of its floorplan. The proposed problem is distinct from traditional visual exploration <cit.>, where agents prioritize motions in a scene to rapidly complete the occupancy map. While in floorplan mapping the best places to reach are those that add visibility to the widest floor area, in acoustic modeling the most valuable poses (sampling spots) in the environment depend on all aspects of the 3D geometry and surface materials. 

To address this challenge, we present ActiveRIR, an active sampling policy that can be deployed on mobile agents in environments that are both unseen and unmapped. ActiveRIR is trained with a novel audio-visual exploration reward to guide wide agent exploration and inform the agent's decision on when to sample an acoustic observation, within a budget of total acoustic samples. Our acoustic reward measures the global improvement in the agent's environment acoustic model estimate after an observation is sampled, ensuring that the limited context used to build the environment acoustic model contains only the most valuable observations seen by the agent during its exploration. 

Evaluating on a diverse set of unseen and unmapped
scanned real-world 3D indoor environments together with state-of-the-art (SOTA) acoustic <cit.> and visual <cit.> scene simulation platforms, ActiveRIR produces a higher-quality acoustic model in >70% fewer steps than passive approaches, and outperforms both traditional visual and spatial exploration-based methods <cit.> as well as SOTA scene acoustic modeling methods <cit.>. We also demonstrate that the performance gain achieved using ActiveRIR-collected observations generalizes across multiple acoustic rendering methods, showing promising potential for ActiveRIR to be plugged as a module into existing acoustic rendering methods and improve the quality of generated environment acoustic models.



§ RELATED WORK



 §.§ Environment acoustics and mapping

Estimating room acoustics from images of indoor scenes has been widely explored, including explicit RIR estimation from images <cit.> as well as methods that model its acoustic transformation on source sounds or speech <cit.>. While these methods can produce audio that perceptually matches the general acoustics of the space (a concert hall vs. bedroom), they cannot reason about fine-grained acoustic effects of shifts in speaker or listener pose within a visual scene. 

Audio field coding approaches <cit.> estimate generic perceptual RIR features using parametric sound field representations that model spatial relationships between acoustics at different positions. Recent approaches use Neural Fields to generate RIRs <cit.> directly, though they still require large-scale acoustic data (>10k samples) from dense spatial locations, and the learned model cannot generalize to other environments.

A transformer-based approach produces an acoustic model of an unseen environment given limited acoustic data sampled randomly from the floorplan <cit.>. Other GNN-based approaches generate RIRs given a full 3D scene mesh and dataset of acoustic material coefficients <cit.>. These methods rely on prior knowledge of scene geometry, and assume the ability to instantaneously access visual and acoustic observations at selected positions in the space. Access to this privileged information—as well as the ability to teleport to new locations without penalty—are significant assumptions that are unrealistic for robots. In principle, the process that produced a pre-computed floorplan map or mesh could have also pre-computed the acoustic model with densely sampled observations using minimal additional time or energy. In short, assuming access to full scene geometry but not complete RIRs  as well fails to capture real-world constraints in robotics.





 §.§ Audio-visual navigation and exploration



Equipping mobile agents with sound production and audio capture has led to a proliferation in audio-visual embodied tasks, such as audio-visual sound source separation <cit.>, audio-visual navigation <cit.>
and audio-visual floorplan mapping <cit.>. A mobile audio-visual agent can decide where to emit and receive acoustic observations to help with floorplan mapping <cit.>. While mapping the floorplan requires dense exploration of the space and its frontiers,
we hypothesize that an accurate environment acoustic model of the same space can be built with far fewer acoustic observations sampled intelligently at select locations.
Prior work <cit.> trains a policy to construct an environment acoustic model that relies on two mobile agents: an "emitter" emits an impulse sound and the "receiver" records the RIR. The agents navigate according to a local acoustic reward that measures the predicted RIR error at the agents' next position. However, unlike our global formulation, movement to minimize local acoustic prediction accuracy often prevents the agent from navigating towards areas with challenging and dynamic acoustics (hallways with turns and corners), hurting performance  (see Sec. <ref>). 




§ ACTIVE ACOUSTIC SAMPLING TASK


We introduce the task of active acoustic sampling. The goal of this task is to train a mobile agent to navigate an unmapped 3D environment (such as a home or office) and intelligently sample egocentric audio-visual observations ("acoustic context") within a predefined sample budget and time budget, such that a acoustic rendering model conditioned on this context can produce accurate and high-fidelity RIRs given arbitrary query source/receiver poses. Each audio-visual observation consists of the agent's egocentric RGB-D image and an echo response, namely an RIR obtained 
by placing the sound source and receiver microphones at the location of image capture, emitting a sinusoidal frequency sweep signal and recording the echoes <cit.>. See Fig. <ref>.
Our sample budget is much lower than the timestep budget for navigation, requiring the agent to carry out the memory-intensive <cit.> task of capturing and storing echo responses only at a select few locations most valuable for the global scene acoustic model. The audio sampling is preemptive, the model decides to choose or skip an audio sample before capturing it.

Formally, given navigation time budget T and audio sample budget N ≪ T, the agent must navigate an unmapped scene and collect audio-visual samples 𝒞 = {C_i}^N, where C_i = (A_i, V_i, P_i). A_i denotes the binaural (two-channel) echo response, and V_i is the 90^∘ FoV RGB-D image captured at the camera pose P_i = (x_i, y_i, θ_i) at location (x_i, y_i) and orientation θ_i.

Using 𝒞 as the context, the agent must infer the scene's environment acoustic model, such that it can accurately estimate the RIR R^Q for arbitrary query Q=(s_j, l_k), where s_j = (x_j, y_j) is an omni-directional sound source at location (x_j, y_j), and l_k=(x_k, y_k, θ_k) is the receiver microphone pose. Thus, the task objective is to learn a policy π that guides our agent towards acoustically informative locations in the scene and helps it decide where to sample audio-visual observations to add to the acoustic context 𝒞, such that a acoustic rendering model f conditioned on 𝒞 approximates the ground-truth RIR R^Q for an arbitrary query Q, or R̃^Q = f(Q | 𝒞). At each time step t, where 1≤ t≤ T, the policy can take action α_t ∈𝒜, where 𝒜={MoveForward, TurnLeft, TurnRight}×{Sample,Skip} is the agent's action space.
    
This task relies on context cues from both audio and vision. RGB-D images capture local room geometry, the presence of furniture/obstacles, and the materials of surfaces in view, while acoustic observations help the agent associate these physical properties with their acoustic effects on an emitted sound. Audio also captures longer-range room geometry beyond the agent's current FoV, which vision cannot. Exploiting this visual-acoustic correspondence is key in helping the agent decide how to move and where to Sample an acoustic observation.




§ APPROACH


We pose the task as a reinforcement learning problem, where a model sequentially decides where to sample audio-visual observations given visual RGB-D frames and previously sampled audio[All audio inputs to our model are represented as two-channel magnitude spectrograms computed using the short-time Fourier transform (STFT) <cit.>.], and the environment acoustic model is built from this collected context. We propose ActiveRIR, which consists of 1) an audio-visual sampling policy π, and 2) a neural acoustic rendering model f for predicting scene acoustics using the samples. See Fig. <ref>. Next, we describe these components in detail.





 §.§ Audio-visual sampling policy




  §.§.§ Policy inputs and architecture

At every time step t, our audio-visual sampling policy π receives O_t as the input and decides how to move in the environment, and whether to capture and add the echo response at the current step to our acoustic context (i.e., Sample or Skip). Formally, O_t=(A_t-1, V_t, P_t), if the echo response was sampled at the previous step, and O_t=(V_t, P_t) if it was skipped. Before encoding O_t, π pre-processes V_t to generate V^π_t=(V^ℛ_t, V^M_t), where V^ℛ_t is the RGB image from V_t, and V^M_t is a topdown global occupancy[V^M_t is produced by projecting the current and past depth images V^D_1… t to the egocentric ground plane and stitching together these projections into a 
cumulative map of the scene.] map. Note that the map begins empty and accumulates as the agent selects its motions.

First, we feed A_t-1 (if sampled), V^π_t and P_t to separate encoders and extract audio features a_t-1, visual features v_t and pose features p_t.
The visual and audio encoders are ResNets <cit.>, and the pose encoder is a sinusoidal positional embedding <cit.>. Next, we concatenate a_t-1, v_t and p_t into o_t and feed it to the policy network, which consists of an RNN and an actor-critic module. The RNN estimates an updated history h_t along with the current state representation g_t, using the fused feature o_t and the history of states h_t-1. The actor-critic module takes g_t and h_t-1 as inputs and predicts a policy distribution π_ϕ(α_t|g,t, h_t-1) along with the value of the state H_ϕ(g_t, h_t-1). Finally, the policy samples an action α_t from its action space 𝒜 (Sec. <ref>) per the distribution π_ϕ.



  §.§.§ Policy reward


We propose a novel RL reward to train our policy π:
    r_t = λ_A * r^A_t + λ_𝒱 * r^𝒱_t + λ_𝒩 * r^𝒩_t.

Here r^A_t is our novel Acoustic Prediction reward, which measures improvement in the environment acoustic model from the previous step t-1 to the current step t if audio was sampled, and is given by 


    r^A_t = ℒ^R_t-1 - ℒ^R_t


, where ℒ^R_t is the mean L1 distance between the predicted and ground-truth RIR magnitude spectrograms for a fixed set of K query positions selected from throughout the (training) scene, at step t. r^𝒜_t is zero at the steps where the policy decides not to sample; thus it encourages the agent to sample an acoustic observation when the agent is at a position that will improve the global acoustic prediction quality.

Since this sparse reward can impact the stability of RL training, we augment r^𝒜_t with an area-coverage reward <cit.> 

r^𝒱_t = (𝒱_t - 𝒱_t-1)/𝒱_t-1 that measures the relative increase in area coverage over time, 
where 𝒱_t and 𝒱_t-1
are the total area covered by the agent at steps t and t-1, respectively.
We also add a novelty reward <cit.> r^𝒩_t = 1/√(n(m_t)), where n(m_t) is the visitation count at 1×1 meter discretized floorplan grid cell m_t. λ_𝒱 and λ_𝒩
are the respective reward weights. Whereas r^𝒱_t rewards the agent for taking actions that expose new areas of the scene to the agent, r^𝒩_t incentivizes increasing the visitation count of novel locations in the scene. r^𝒱_t and r^𝒩_t together promote a healthy exploration of the scene, which is crucial for learning a good policy, and also encourage stable RL training owing to their dense nature. We train π using Decentralized Distributed PPO (DD-PPO) <cit.>. The DD-PPO loss consists of a value loss, policy loss, and an entropy loss for further improving exploration.







 §.§ Acoustic rendering model


Our sampling policy design is agnostic of the design of the acoustic rendering model f, providing the flexibility of using our policy with any off-the-shelf rendering model that can take audio-visual samples as inputs and predict the RIR R^Q at a source-receiver query Q in the scene (Sec. <ref>). We use FewShot-RIR (FS-RIR) <cit.> as our rendering model backbone, due to its state-of-the-art performance. We also evaluate ActiveRIR's ability to generalize in experiments with a second backbone, NAF <cit.> (Sec. <ref>). Given audio-visual samples from a scene, FS-RIR uses a transformer <cit.> encoder to build an acoustic context of the scene, followd by a transformer-decoder to predict the RIR for an arbitrary source-receiver query using the acoustic context. We pre-train the acoustic rendering model f in a disembodied fashion—without a sampling policy—randomly selecting observations from a scene to add to the acoustic context. Next, we train our policy with RL while keeping f frozen. This helps improve RL training stability by ensuring stationarity in the reward distribution (Sec. <ref>).





§ EXPERIMENTS








 §.§ Experimental setup

We use the state-of-the-art SoundSpaces (SS) 2.0 <cit.> acoustics simulator, built on top of the AI-Habitat <cit.> simulator and Matterport3D (MP3D) <cit.> scenes. MP3D consists of photorealistic scans of diverse, real-world multi-room indoor environments complete with furniture and other objects (tables/sofas/desks/lamps). 
 SS 2.0 supports continuous rendering of precise spatial audio in arbitrary 3D scenes, capturing all major acoustic phenomena <cit.>. We use 78 diverse MP3D scenes, split into train/val/test sets of 56/10/12, preserving diverse scene types and sizes within each split. We train our policy for 150K PPO iterations and validate/test on 123/225 inference episodes respectively sampled in proportion to scene sizes within each split.

We place the agent at a random location in a scene at the start of every episode. We set T=200 to account for the minimal time needed to traverse an average scene average in Matterport3D <cit.>, and

set the audio sample budget to N=20 samples to match the context size used in FS-RIR <cit.>. We set λ_A =2×10^5,
λ_𝒱 =2×10^2
and λ_𝒩 =10.0 to place the component rewards on the same scale. The agent has turn and movement resolution 90^∘ and 1 meter respectively. We evaluate performance with STFT L1 Error (STFT) <cit.>, which measures mean L1 error between 
predicted and ground-truth RIR magnitude spectrograms at
K=60 global query poses per scene, sampled randomly at the scene level. 





 §.§ Baselines

Random agent: an agent that chooses an action from our action space 𝒜 randomly.
Forward agent: 
an agent that only moves forward, sampling observations uniformly (every T/N steps). Greedy agent: an agent that navigates randomly, and greedily selects the first N observations on its path.



§ RESULTS




Table <ref> shows acoustic prediction performance on unseen test environments. ActiveRIR significantly outperforms naive approaches (Greedy, Forward, Random). Fig. <ref> displays STFT error as a function of the agent's steps. ActiveRIR (purple) efficiently navigates toward and captures valuable acoustic observations that rapidly minimize STFT error in the acoustic model, compared both to a passive-sampling audio-visual agent which samples every T/N steps ("Uniform", orange) as well as an array of heuristic approaches.





  §.§.§ Model analysis



  
Reward variations
To evaluate the impact of our Acoustic Prediction reward, we train active policies with ablations of our audio-visual reward (Table <ref>). We outperform Coverage (𝒱) and Novelty (𝒩) agents as well as the Exploration agent (𝒱+𝒩), which was trained with only the spatio-visual component of our audio-visual reward (Sec. <ref>), validating the importance of acoustic information in the agent's decision to Sample an observation beyond what can be inferred from visual and/or spatial sensory information alone. We also significantly outperform a passive audio-visual agent (Uniform), confirming that actively determining when to sample plays a critical role in collecting valuable acoustic context beyond simply navigating towards acoustically and visually novel areas. Inspired by <cit.>, we evaluate a local variant of our acoustic reward, defined as the improvement in the RIR prediction error for a query closest to the agent's current position. We outperform this local acoustic agent, demonstrating that our global acoustic reward helps collect samples most valuable for global acoustic model.



  
Pose sensor noise
We evaluate ActiveRIR's robustness to Gaussian noise in pose and actuation. STFT error only increases from 8.08 × 10^-3 to 8.10 × 10^-3, highlighting the effectiveness of our design choices.






  §.§.§ Comparison with SOTA
 Fig. <ref> compares ActiveRIR to SOTA methods without embodiment and navigation budget. First, we compare against FS-RIR <cit.>, which collects samples randomly throughout the environment (Fig. <ref>, right). Despite initial progress, FS-RIR plateaus as the episode progresses, while ActiveRIR continues selecting highly-informative samples that further reduce STFT error.
Importantly, ActiveRIR produces a final acoustic model with significantly lower error than FS-RIR, despite FS-RIR's prior knowledge of the floorplan for randomly sampling observations. Furthermore, to achieve the error of ActiveRIR, FS-RIR needs to acquire a context length of 86 samples, more than 4x the samples required for our model. While randomly sampling from throughout the space may outperform our policy in high-resource regimes, ActiveRIR strongly outperforms FS-RIR in cost-efficient settings with restrictive time or sample budgets—which are practical considerations when considering physical constraints on access/time spent in the space, and the intrusive nature of emitting a sound impulse.

To determine the value of acoustic context collected by ActiveRIR agnostic of the acoustic rendering model f, we train two Neural Acoustic Field (NAF) <cit.> models on the test scenes using 1) ActiveRIR-collected context and 2) context collected by a random policy, and evaluate mean STFT error across 100 query poses sampled randomly from throughout the scene (Fig <ref>, left). The ActiveRIR-trained NAF consistently outperforms NAF trained on the random policy's collected context, demonstrating that ActiveRIR collects a rich, generalizable acoustic context and is flexible with respect to the acoustic rendering model.



  §.§.§ Qualitative analysis

Fig. <ref> visualizes the importance of active sampling. ActiveRIR 
captures audio-visual samples in areas where acoustics can dynamically change, such as winding hallways (far left), while also ensuring that the context is spatially and visually distinct, as can be observed by the diverse views in the active context (left). In contrast, a passive 
agent (right) does not actively avoid capturing redundant observations (bottom two images), and collects samples in the main cavity of the scene where acoustics are relatively stationary between nearby poses. Also see submitted video.












§ CONCLUSIONS


We propose a new task, active acoustic sampling, in which an agent must navigate an unmapped environment and collect audio-visual samples to construct a model of the scene's acoustics within a given time and sample budget. We propose an active sampling policy trained with a novel audio-visual exploration reward which guides an agent to navigate towards and select high-value audio-visual samples that yield a high-quality acoustic model. Our policy outperforms passive approaches in >70% fewer timesteps, and outperforms robust visual and spatial exploration agents as well as SOTA environment acoustic modeling methods <cit.> across diverse indoor scenes. We show that the performance gain using ActiveRIR-collected samples generalizes across multiple acoustic rendering models, demonstrating promising potential for ActiveRIR to improve existing acoustic rendering methods. In future work, we plan to explore 3D scene reconstruction from acoustic exploration.

IEEEtran

-12cm

