Abstract

Point cloud semantic segmentation plays an essential role in autonomous driving, providing vital information about drivable surfaces and nearby objects that can aid higher level tasks such as path planning and collision avoidance. While current 3D semantic segmentation net- works focus on convolutional architectures that perform great for well represented classes, they show a significant drop in performance for underrepresented classes that share similar geometric features. We propose a novel Detection Aware 3D Semantic Segmentation (DASS) framework that explicitly leverages localization features from an auxiliary 3D object detection task. By utilizing multitask training, the shared feature representation of the network is guided to be aware of per class detection features that aid tackling the differentiation of geometrically similar classes. We additionally provide a pipeline that uses DASS to generate high recall proposals for existing 2-stage detectors and demonstrate that the added supervisory signal can be used to improve 3D orientation estimation capabilities. Extensive experiments on both the SemanticKITTI and KITTI object datasets show that DASS can improve 3D semantic segmentation results of geometrically similar classes up to 37.8% IoU in image FOV while maintaining high precision bird's-eye view (BEV) detection results.

Detection Aware 3D Semantic Segmentation

Below is the network (left) and extension (right) overview. The network is trained on two partial datasets: (1) with pointwise semantic labels and (2) with 3D object annotations. Since the detection datasets provides annotations for just the camera field of view, the segmentation dataset is cropped to avoid introducing further domain shifts. A PointNet++ (Qi et al. NIPS 2017) feature extractor is trained on both supervisory signals on a multitask loss, followed by the individual task heads for the 3D semantic segmentation task and the auxiliary 3D proposal generation task. Additionally DASS can be used as the RPN of PointRCNN (Shi et al. CVPR 2019). To further improve object detection results, a novel semantic feature fusion (SFF) layer is applied before the proposal generation to overcome the capacity limitations of encoder-level multitask learning.

We do an extensive qualitative analysis of our proposed framework and compare its performance against other state-of-the-art methods for both point cloud semantic segmentation and 3D object detection.

DASS, combined with existing 2 stage detectors can produce 19 class semantic masks and high precision 3D bounding boxes in real time at an operating frequency of 11Hz with only a 0.15% additional memory requirement.