HoloLens 2 Research Mode as a Tool for Computer Vision Research

Mixed reality headsets, such as the Microsoft HoloLens 2, are powerful sensing devices with integrated compute capabilities, which makes it an ideal platform for computer vision research. In this technical report, we present HoloLens 2 Research Mode, an API and a set of tools enabling access to the raw sensor streams. We provide an overview of the API and explain how it can be used to build mixed reality applications based on processing sensor data. We also show how to combine the Research Mode sensor data with the built-in eye and hand tracking capabilities provided by HoloLens 2. By releasing the Research Mode API and a set of open-source tools, we aim to foster further research in the fields of computer vision as well as robotics and encourage contributions from the research community.


Introduction
Mixed reality technologies have tremendous potential to fundamentally transform the way we interact with our environment, with other people, and the physical world. Mixed reality headsets such as the Microsoft HoloLens 1 & 2 have already seen great adoption in a number of fields, particularly in first-line worker scenarios, ranging from assisted surgery to remote collaboration and from task guidance to overlaying digital twin on the real world. Despite existing adoption in these fields, the general space of mixed reality is still in its infancy. Oftentimes, the development of new mixed reality applications requires fundamental research and the novel combination of different sensors. The entry barrier to computer vision research in this area is significantly lowered by access to tools enabling us to effectively collect raw sensor data and develop new computer vision algorithms that run on-device.
The first-generation HoloLens Research Mode, released in 2018, enabled computer vision research on device by pro- viding access to all raw image sensor streams -including depth and IR. Released together with a public repository collecting auxiliary tools and sample applications [4], Research Mode promoted the use of HoloLens as a powerful tool for doing research in computer vision and robotics [1].
HoloLens 2 ( Fig. 1), announced in 2019, brings a number of improvements with respect to the first-generation device -like a dedicated DNN core, articulated hand tracking and eye gaze tracking [2]. However, being a novel platform built on new hardware, HoloLens 2 is not compatible with the previous version of Research Mode.
In this technical report, we introduce the secondgeneration HoloLens Research Mode, made available in 2020. Research Mode provides a set of C++ APIs and tools to access the HoloLens 2 sensor streams. We discuss the main novelties with respect to the previous version and present a set of applications built on top of it. For additional material and API documentation, we refer the reader to the GitHub repository [3].
Research Mode is designed for academic and industrial researchers exploring new ideas in the fields of computer vision and robotics. It is not intended for applications deployed to end-users. Additionally, Microsoft does not provide assurances that Research Mode will be supported in future hardware or OS updates.
The rest of this technical report is organized as follows. Section 2 introduces the HoloLens 2 device, detailing in particular its input streams. Section 3 provides an overview of the Research Mode API, while Section 4 showcases a few example applications. Finally, Section 5 sums up our contributions.

HoloLens 2
The HoloLens 2 mixed reality headset brings a set of improvements with respect to the first-generation deviceincluding a larger field of view, a custom DNN core, fully articulated hand tracking and eye gaze tracking.
The device features a second-generation custom-built Holographic Processing Unit (HPU 2.0), which enables low-power, real-time computer vision. The HPU runs all the computer vision algorithms on device (head tracking, hand tracking, eye gaze tracking, spatial mapping etc.) and hosts the DNN core. It is located on the front part of the device, near the sensors (Fig. 2, top). The CPU on the SoC (a Qualcomm SnapDragon 850) remains fully available for applications. The SoC is located on the rear (Fig. 2, bottom).
The device is equipped with a depth and an RGB camera, four grayscale cameras, and an Inertial Measurement Unit (IMU), as shown in Fig. 3. Audio is captured with a microphone array (5 channels).
Research Mode for HoloLens 2 enables access to the following input streams: • Four visible-light tracking cameras (VLC): Grayscale cameras (30 fps) used by the system for real-time visual-inertial SLAM.
• A depth camera, which operates in two modes: -AHAT (Articulated HAnd Tracking), highframerate (45 fps) near-depth sensing used for hand tracking. As hands are supported up to 1 meter from the device, the HoloLens 2 saves power by calculating only "aliased depth" from the phase-based time of flight camera. This means that the signal contains only the fractional part of the distance from the device when expressed in meters (see Fig. 4).
• Two depth modes of the IR stream (Active Brightness, AB in short), computed from the same modulated IR signal for depth computation. These images are illuminated by infrared and unaffected by ambient visible light (see Fig. 4, right).
• Inertial Measurement Unit (IMU): -Accelerometer, used by the system to determine the linear acceleration along the x, y and z axes as well as gravity.
-Gyroscope, used by the system to determine rotations.
-Magnetometer, used by the system for absolute orientation estimation.
For each stream, Research Mode provides interfaces to retrieve frames and associated information (e.g., resolution and timestamps), and to map the sensors with respect to the device and to the world. Table 1 summarizes the main characteristics of each (camera) input stream. In the following section, we provide an overview of the API.

Research Mode API
In this section, we describe the main objects exposed by Research Mode and how they are used to access the sensor  input streams. We refer the reader to [3] for a detailed API documentation.
The Research Mode main sensor loop (Sec. 3.1) starts by creating a ResearchModeDevice object, which is used to obtain a list of available Sensors. Sensor objects expose methods to retrieve and process frames (Sec. 3.2) and to locate the sensors with respect to the device and the world (Sec. 3.3).

Main Sensor Loop
The main sensor processing loop involves instantiating a ResearchModeDevice, getting sensor descriptors, opening sensor streams and fetching frames: pSensor->Release(); } pSensorDevice->EnableEyeSelection(); pSensorDevice->Release(); return hr; Note that OpenStream and GetNextBuffer need to be called from the same thread. Since GetNextBuffer calls are blocking, per-sensor frame loops should be run on their own thread. This allows sensors to be processed at their own framerate.

Sensors and Sensor Frames
The IResearchModeSensor interface introduced in the previous section abstracts the Research Mode sensors. It provides methods and properties common to all sensors: OpenStream, CloseStream, GetFriendlyName, GetSensorType, GetNextBuffer. Sensors can be of the following types: An analogous distinction exists between the IResearchModeSensorFrame interface and persensor specializations.
Once a sensor is in streaming mode, sensor frames are retrieved with IResearchModeSensor::GetNextBuffer. All sensor frames have a common IResearchModeSensorFrame interface that returns frame information common to all types of frames: timestamps and sample size in bytes. As above, camera and IMU frame specializations are obtained by calling QueryInterface: IResearchModeSensorFrame * pSensorFrame; // ... Obtain pSensorFrame via GetNextBuffer IResearchModeSensorVLCFrame * pVLCFrame = nullptr; HRESULT hr = pSensorFrame->QueryInterface( IID_PPV_ARGS(&pVLCFrame)); Camera and IMU frames can be used to access the following information: Camera frames provide getters for resolution, exposure, gain. In addition, • VLC frames return grayscale buffers; • Long Throw depth frames return a depth buffer, a sigma buffer and an active brightness buffer; • AHAT depth frames return a depth buffer and an active brightness buffer.
The active brightness buffer returns a so-called IR reading. The value of pixels in the clean IR reading is proportional to the amount of light returned from the scene. The image looks similar to a regular IR image. The sigma buffer for Long Throw is used to invalidate unreliable depth based on the invalidation mask computed by the depth algorithm. For AHAT, for efficiency purposes, the invalidation code is embedded in the depth channel itself.
The following code exemplifies how to access Long Throw depth: pDepthFrame->Release(); } IMU frames store batches of sensor samples. Accelerometer frames store accelerometer measurements and temperature; gyroscope frames store gyroscope measurements and temperature; magnetometer frames store magnetometer measurements.

Sensor Coordinate Frames
All sensors are positioned in a device-defined coordinate frame, which is defined as the rigNode. Each sensor returns its transform to the rigNode (device origin) expressed as an extrinsics rigid body transform (rotation and translation). On HoloLens 2, the device origin corresponds to the Left Front Visible Light Camera; therefore, the transformation returned by this sensor corresponds to the identity transformation. Figure 5 shows camera coordinate frames relative to the rigNode.
Instead, they expose MapImagePointToCameraUnitPlane and MapCameraSpaceToImagePoint methods to convert 3D coordinates in the camera reference frame into 2D image coordinates, and vice versa.

The HoloLens2ForCV Repository
The HoloLens2ForCV repository [3] provides a few sample UWP (Universal Windows Platform) apps showing how to access and process Research Mode input streams:

Sample Apps
In its initial version, the HoloLens2ForCV repository collects four UWP apps. We plan to expand this initial set over time, and welcome contributions from the community. In particular, we provide sample code to a) visualize input streams live on device and process them using OpenCV [5] and b) record streams to disk for offline postprocessing.
Sensor Visualization and Processing: The SensorVisualization app allows one to visualize depth, VLC (and optionally IMU) input streams live on device (see Fig. 7). This can be particularly useful to test, for example, sensor performance in different environments. With the CameraW-ithCVAndCalibration app, we show how to process these streams on device: we use OpenCV to detect arUco markers in the two frontal VLC cameras, and triangulate the detections (Fig. 6). The CalibrationVisualization app can be used to visualize VLC and depth coordinate frames.
Stream Recorder: The StreamRecorder app allows one to capture simultaneously Research Mode streams (depth and VLC), the RGB stream from the HoloLens PV (Pho-toVideo) camera, head, hand and eye tracking. The application provides a simple user interface to start and stop capture (Fig. 8, top). Streams are captured and stored on device, and can be downloaded and postprocessed by using a set of python scripts. Namely, we provide scripts to correlate Research Mode, PV, head, hand and eye tracking streams in  time and put them in a common coordinate frame. This allows us to reconstruct the scene in 3D and analyze how the user interacts with it by looking at their eye gaze, head and hand pose (Fig. 8, bottom). It is possible to add also audio capture, accessible via Windows Media APIs [6], though this functionality is not implemented at the time of writing.
It is worth noting that the StreamRecorder app relies on another library, Cannon (available on [3]). Cannon is a collection of wrappers and utility code for building native mixed reality apps using C++, Direct3D and Windows Per- Figure 9. TSDF Volume Integration [9] result obtained from HoloLens depth and RGB input streams.
ception APIs. It can be used as-is outside Research Mode for fast and easy native development.

Computer Vision with Research Mode
By leveraging Research Mode, one can use the HoloLens as a powerful tool for computer vision and robotics research. To showcase how easily this can be done, we provide two examples: TSDF Volume Integration and SLAM.
TSDF Volume Integration: We provide a script which runs TSDF Volume Integration using as input Long throw depth frames, RGB frames and head poses captured with the StreamRecorder app. We simply combine the app output with an off-the-shelf library [9]. Figure 9 shows the reconstruction obtained by running the algorithm on almost 700 frames (head trajectory shown in red).

SLAM:
We feed Long Throw depth frames and RGB frames as input to a recently proposed SLAM method (BAD SLAM [8]). The camera poses estimated by the SLAM algorithm can be compared and evaluated against the head poses returned by the HoloLens.

Conclusion
We presented Research Mode for HoloLens 2, an API and a set of functionalities enabling access to the raw sensor streams on device. Together with Research Mode, we release a public repository collecting sample apps and examples showcasing how to leverage Research Mode to build computer vision applications based on HoloLens. With these tools, we hope to facilitate further research in the fields of computer vision and robotics, and encourage contributions from the research community.