We present a challenging dataset, the TartanAir, for robot navigation task and more. The data is collected in photo-realistic simulation environments in the presence of various light conditions, weather and moving objects. By collecting data in simulation, we are able to obtain multi-modal sensor data and precise ground truth labels, including the stereo RGB image, depth image, segmentation, optical flow, camera poses, and LiDAR point cloud. We set up a large number of environments with various styles and scenes, covering challenging viewpoints and diverse motion patterns, which are difficult to achieve by using physical data collection platforms. In order to enable data collection in such large scale, we develop an automatic pipeline, including mapping, trajectory sampling, data processing, and data verification. We evaluate the impact of various factors on visual SLAM algorithms using our data. Results of state-of-the-art algorithms reveal that the visual SLAM problem is far from solved, methods that show good performance on established datasets such as KITTI don’t perform well in more difficult scenarios. Although we use the simulation, our goal is to push the limits of Visual SLAM algorithms in the real world by providing a challenging benchmark for testing new methods, as well as large diverse training data for learning-based methods.
The dataset is published using Azure Open Dataset platform. Please checkout here for the instruction of accessing the data.
Sample trajectories can be downloaded here.
Simultaneous Localization and Mapping (SLAM) is one of the most fundamental capabilities necessary for robots. Due to the ubiquitous availability of images, Visual SLAM (V-SLAM) has become an important component of many autonomous systems. Impressive progress has been made with both geometric-based methods and learning-based methods. However, developing robust and reliable SLAM methods for real-world applications is still a challenging problem. Real-life environments are full of difficult cases such as light changes or lack of illumination, dynamic objects, and texture-less scenes. Current popular benchmarks such as KITTI, TUM RGB-D SLAM datasets, and EuRoC MAV cover relatively limited scenarios and motion patterns compared to real-world cases.
We collect a large dataset using photo-realistic simulation environments. We minimize the sim2real gap by utilizing a large number of environments with various styles and diverse scenes. A special goal of our dataset is to focus on the challenging environments with changing light conditions, adverse weather, and dynamic objects. State-of-the-art SLAM algorithms are struggled in tracking the camera pose in our dataset and constantly getting lost on some challenging sequences. We propose a metric to evaluate the robustness of the algorithm. In addition, we develop an automatic data collection pipeline, which allows us to process more environments with minimum human intervention.
The four most important features of our dataset are:
- Large size diverse realistic data
- Multimodal ground truth labels
- Diversity of motion patterns
- Challenging Scenes
We have adopted 30 photo-realistic simulation environments in the Unreal Engine. The environments provide us a wide range of scenarios that cover many interesting yet challenging situations. The simulation scenes consist of
- Indoor and outdoor scenes with detailed 3D objects. We have multi-room, richly decorated indoor environments. For outdoor simulations, there are various kinds of buildings, trees, terrains, and landscapes.
- Special purpose facilities and ordinary household scenes.
- Rural and urban scenes.
- Real-life and sci-fi scenes.
Challenging visual effects
In some simulations, we simulated multiple types of challenging visual effects.
- Hard lighting conditions. Day-night alternating. Low-lighting. Rapidly changing illuminations.
- Weather effects. Clear, raining, snowing, windy, and fog.
- Seasonal change.
Diverse ego motions
In each simulated environment, we gather data by following multiple routes and making movements with different levels of aggressiveness. The virtual camera can move slowly and smoothly without sudden jittering actions. Or it can have intensive and violent actions mixed with significant rolling and yaw motions.
Multimodal ground truth labels
By unleashing the power of the Unreal Engine and AirSim, we can extract various types of ground truth labels including depth, semantic segmentation tag, and camera pose. From the extracted raw data, we further compute other ground truth labels such as optical flow, stereo disparity, simulated multi-line LiDAR points, and simulated IMU readings.
Data acquisition pipeline
We develop a highly automated pipe-line to facilitate data acquisition. For each environment, we build an occupancy map by incremental mapping. Base on the map, we then sample a bunch of trajectories for the virtual camera to follow. A set of virtual cameras follow the trajectories to capture raw data from Unreal Engine and AirSim. Raw data are processed to generate labels such as optical flow, stereo disparity, simulated LiDAR points, and simualated IMU readings. Data verification. Verify the data synchronization and the accuracy of the derived labels.
Tunatoa seti ya data ya changamoto, TartanAir, kwa ajili ya kazi ya urambazaji wa roboti na zaidi. Takwimu ni zilizokusanywa katika mazingira ya simulation photo-kweli katika uwepo wa hali mbalimbali za mwanga, hali ya hewa na vitu kusonga. Kwa kukusanya data katika simulation, tunaweza kupata data ya sensor ya multimodal na vitambulisho sahihi vya ukweli wa ardhi, ikiwa ni pamoja na picha ya RGB ya stereo, picha ya kina, mgawanyiko, mtiririko wa macho, nafasi za kamera, na wingu la pointi la LiDAR. Sisi kuweka idadi kubwa ya mazingira na mitindo mbalimbali na matukio, kufunika changamoto mtazamo na mifumo mbalimbali ya mwendo, ambayo ni vigumu kufikia kwa kutumia vifaa vya kukusanya data. Ili kuwezesha ukusanyaji wa data kwa kiwango kikubwa, tunaendeleza bomba la moja kwa moja, ikiwa ni pamoja na ramani, sampuli ya trajectory, usindikaji wa data, na uthibitisho wa data. Kwa hivyo, kwa sababu ya athari za sababu za kipekee za algorithms za SLAM, tunaweza kuamua jinsi ya kuamua matokeo ya algorithms za SLAM. Matokeo ya hali ya sanaa algorithms kufunua kwamba tatizo visual SLAM ni mbali na kutatuliwa, mbinu kwamba kuonyesha utendaji mzuri juu ya seti ya data imara kama vile KITTI si kufanya vizuri katika matukio ngumu zaidi. Ingawa sisi kutumia simulation, lengo letu ni kushinikiza mipaka ya Visual SLAM algorithms katika ulimwengu halisi kwa kutoa benchmark changamoto kwa ajili ya kupima mbinu mpya, kama vile kubwa mbalimbali ya mafunzo data kwa ajili ya kujifunza-msingi mbinu. dataset ni kuchapishwa kwa kutumia Azure Open Dataset jukwaa. Tafadhali angalia hapa kwa ajili ya maelekezo ya kupata data. Mifumo ya sampuli ya trajectories inaweza kupakuliwa hapa. Simultaneous Localization na Mapping (SLAM) ni moja ya uwezo wa msingi muhimu kwa ajili ya roboti. Kwa sababu ya upatikanaji wa picha kila mahali, Visual SLAM (V-SLAM) imekuwa sehemu muhimu ya mifumo mingi ya kujitegemea. Maendeleo ya kuvutia yamefanywa na mbinu zote mbili za msingi wa kijiometri na mbinu za msingi wa kujifunza. Hata hivyo, kuendeleza mbinu thabiti na ya kuaminika SLAM kwa ajili ya maombi ya ulimwengu halisi bado ni tatizo changamoto. Mazingira ya maisha halisi yana mambo magumu kama vile mabadiliko ya nuru au ukosefu wa taa, vitu vyenye nguvu, na mandhari zisizo na muundo. Kwa sasa, viwango maarufu kama vile KITTI, TUM RGB-D SLAM na EuROC MAV vinashughulikia hali ndogo na mifumo ya mwendo ikilinganishwa na kesi halisi. Sisi kukusanya seti kubwa ya data kutumia mazingira photo-kweli simulation. Sisi kupunguza sim2real pengo kwa kutumia idadi kubwa ya mazingira na mitindo mbalimbali na matukio mbalimbali. Lengo maalum la seti yetu ya data ni kuzingatia mazingira ya changamoto na mabadiliko ya hali ya mwanga, hali mbaya ya hewa, na vitu vya nguvu. Algorithms ya kisasa ya SLAM ina shida kufuatilia pose ya kamera katika seti yetu ya data na daima kupotea kwenye mfululizo fulani wenye changamoto. Sisi kupendekeza kipimo cha kutathmini nguvu ya algorithm. Kwa kuongezea, tunatengeneza bomba la kukusanya data moja kwa moja, ambalo linaturuhusu kusindika mazingira zaidi kwa uingiliaji mdogo wa binadamu. "Hizi ni ""mfano"" za data za kawaida za hali ya juu, ""mfano wa hali ya juu"" na ""mfano wa hali ya chini"" na ""mfano wa hali ya chini"" na ""mfano wa hali ya chini"" na ""mfano wa hali ya chini"" na ""mfano wa hali ya chini"" na ""mfano wa hali ya chini"" na ""mfano wa hali ya chini"" na ""mfano wa hali ya chini.""" Mazingira yanatupa aina mbalimbali za matukio ambayo yanafunika hali nyingi zenye kupendeza lakini zenye changamoto. Simulation matukio ni pamoja na - Ndani na nje matukio na vitu vya kina 3D. Tuna vyumba vingi vya ndani vilivyo na mapambo mengi. Kwa ajili ya uigizaji wa nje, kuna aina mbalimbali za majengo, miti, maeneo, na mandhari. - Vifaa vya kusudi maalum na mandhari ya kawaida ya nyumbani. - Mandhari za mashambani na za mijini. - Maisha halisi na mandhari ya sayansi ya uwongo. Matokeo ya kuona changamoto Katika baadhi ya simulations, sisi simulated aina nyingi ya athari za kuona changamoto. - Hali mbaya ya taa. Mchana na usiku hubadilishana. Nuru ya chini. Mwangaza unaobadilika haraka. - Athari za hali ya hewa. Hali ni nzuri, mvua inanyesha, theluji inanyesha, upepo unanyesha, na ukungu. - Mabadiliko ya majira. Katika kila mazingira simulated, sisi kukusanya data kwa kufuata njia nyingi na kufanya harakati na ngazi tofauti za unyanyasaji. Kamera virtual inaweza kusonga polepole na kwa urahisi bila vitendo ghafla jittering. Au inaweza kuwa na vitendo vya nguvu na vikali vilivyochanganywa na mwendo mkubwa wa kuteleza na kuteleza. Multimodal Ground Truth Labels: Kwa kutumia nguvu ya Unreal Engine na AirSim, tunaweza kutoa aina mbalimbali za Ground Truth Labels ikiwa ni pamoja na kina, semantic segmentation tag, na kamera pose. Kutokana na data ya awali iliyotolewa, sisi pia kuhesabu nyingine ardhi ukweli lebo kama vile mtiririko wa macho, stereo tofauti, simulated multi-line LiDAR pointi, na simulated IMU kusoma. Data upatikanaji bomba Sisi kuendeleza high automatiska bomba-mstari ili kuwezesha upatikanaji wa data. Kwa kila mazingira, sisi kujenga ramani ya ukaaji kwa ramani incremental. Msingi juu ya ramani, sisi kisha sampuli kundi la trajectories kwa kamera virtual kufuata. Seti ya kamera virtual kufuata trajectories kukamata data ghafi kutoka Unreal Engine na AirSim. Data mbichi ni kusindika kuzalisha lebo kama vile mtiririko wa macho, stereo tofauti, simulated pointi LiDAR, na simualated IMU kusoma. Uthibitisho wa data. Kuthibitisha data synchronization na usahihi wa lebo inayotokana.