TraffiX - A V2X Dataset for Multi-Modal Cooperative 3D Object Detection of Traffic Participants Using Onboard and Roadside Sensors

Development Kit Model 3D labeling tool Dataset

Overview

TraffiX is the first high-quality real-world V2X dataset for the cooperative 3D object detection and tracking task in autonomous driving.

It contains:
  • data collected by 9 sensors simultaneously from onboard and roadside sensors.
  • 1,600 labeled point clouds and 4,000 labeled images.
  • 25k 3D bounding boxes with track IDs.
  • Challenging scenarios: near-miss and traffic violation events, overtaking and U-turn maneuvers.
  • HD maps of the driving domains.
  • Labels in OpenLABEL standard.
  • A dev kit to load, preprocess, visualize, convert labels, and to evaluate perception methods.

Abstract

Cooperative perception offers several benefits for enhancing the capabilities of autonomous vehicles and improving road safety. Using roadside sensors in addition to onboard sensors increases reliability and extends the sensor range. They offer a higher situational awareness for automated vehicles and prevent occlusions. We propose CoopDet3D, a cooperative multi-modal fusion model, and TraffiX, a V2X dataset, for the cooperative 3D object detection and tracking task. Our dataset contains 1,600 labeled point clouds and 4,000 labeled images from five roadside and four onboard sensors. It includes 50k 3D boxes with track IDs and precise GPS and IMU data. We labeled eight categories and covered occlusion scenarios with challenging driving maneuvers, like traffic violations, near-miss events, overtaking, and U-turns. Through multiple experiments, we show that our CoopDet3D camera-LiDAR fusion model achieves an increase of +14.36 3D mAP compared to a vehicle camera-LiDAR fusion model. Finally, we make our model, dataset, labeling tool, and development kit publicly available to advance in connected and automated driving.

Sensor Setup

On the infrastructure, the following roadside sensors were used:                                                                                                                              
  • 1x Ouster LiDAR OS1-64 (generation 2), 64 vertical layers, 360° FOV,
    below horizon configuration, 10 cm accuracy @120 m range
  • 4x Basler ace acA1920-50gc, 1920×1200, Sony IMX174 with 8 mm lenses

On the vehicle, the following onboard sensors were used:
                                                                     
  • 1x Robosense RS-LiDAR-32, 32 vert. layers, 360° FOV, 3 cm accuracy @200 m range
  • 1x Basler ace acA1920-50gc, 1920×1200, Sony IMX174 with 16 mm lens
  • 1x Emlid Reach RS2+ multi-band RTK GNSS receiver
  • 1x XSENS MTi-30-2A8G4 IMU
  infrastructure_sensors

Visualization of roadside sensors used to record the TraffiX V2X Cooperative Dataset from infrastructure perspective.

Dataset Labeling

Dataset Statistics

Benchmark

Config BEV mAP 3D mAP
Domain Modality   Easy Moderate Hard Average
Vehicle Camera 46.83 31.47 37.82 30.77 30.36
Vehicle LiDAR 85.33 85.22 76.86 69.04 80.11
Vehicle Camera + LiDAR 84.90 77.60 72.08 73.12 76.40
Infrastructure Camera 61.98 31.19 46.73 40.42 35.04
Infrastructure LiDAR 92.86 86.17 88.07 75.73 84.88
Infrastructure Camera + LiDAR 92.92 87.99 89.09 81.69 87.01
Cooperative Camera 68.94 45.41 42.76 57.83 45.74
Cooperative LiDAR 93.93 92.63 78.06 73.95 85.86
Cooperative Camera + LiDAR 94.22 93.42 88.17 79.94 90.76