LiDARNet: A Boundary-Aware Domain Adaptation Model for Point Cloud Semantic Segmentation
Introduction
We present a boundary-aware domain adaptation model for LiDAR scan full-scene semantic segmentation (LiDARNet). Our model can extract both the domain private features and the domain shared features with a two branch structure. We embedded Gated-SCNN into the segmenter component of LiDARNet to learn boundary information while learning to predict full-scene semantic segmentation labels. Moreover, we further reduce the domain gap by inducing the model to learn a mapping between two domains using the domain shared and private features. Besides, we introduce a new dataset (SemanticUSL). The dataset has the same data format and ontology as SemanticKITTI. We conducted experiments on real-world datasets SemanticKITTI, SemanticPOSS, and SemanticUSL, which have differences in channel distributions, reflectivity distributions, diversity of scenes, and sensors setup. Using our approach, we can get a single projection-based LiDAR full-scene semantic segmentation model working on both domains. Our model can keep almost the same performance on the source domain after adaptation and get an 8%-22% mIoU performance increase in the target domain.
Updates:
The paper is released on arXiv
The code is released on Github
Approach
Generally, we expect a model that can complete a task for data from similar domains. However, feature difference between two similar domains causes a model, which learns from one domain (called source domain (S)), can not perform well on another domain (called target domain (T)). Therefore, we expect a method that can adapt a model from one domain to another domain. If the target domain does not provide ground truth, the problem is called unsupervised domain adaptation. In this paper, the task is full-scene semantic segmentation for Lidar scan. In this problem, we use (X_S) denotes source data, (Y_S) denotes source labels, and (X_T) denotes target data, but target labels are not accessible.
Based on the intuition that two similar domains should contain shared information across the two domains and private information to each domain. And the adaptable information should be contained in shared information of two domains. Therefore, we designed an end-to end trainable model that splits input data into domain shared and private features. The model then utilizes the extracted shared features to perform semantic segmentation. Fig.1 shows the components of the model and the information flow in the model. The model contains two extractors: a shared feature extractor (f_P) and a private feature extractor (f_D).
To induce the two extractors to produce such split information, we add a loss function that encourages the independence of these parts, and connect the private feature extractor to a classifier (f_C) to differentiate the data from two domains. Besides, we feed the output of the domain shared extractor to a segmenter to complete the same task (predict semantic labels (\hat{Y})). To ensure that the private features are still useful and to further reduce the domain gap, we introduce the CycleGAN mechanism \cite{Zhu2017} to induce the models to learn two mappings between two domains. The domain private and shared features are fed into domain converters to convert the data from one domain to another domain: ((f_{S\to T}) converts source data into target domain, (f_{T\to S}) converts target data into source domain). The conversion is learning through an adversarial learning procedure. Therefore, the converted data are separately fed into domain discriminators ((D_{T}) and (D_{S})). Meanwhile, we add Gated-SCNN \cite{Takikawa2019} on the side of the segmenter to extract boundary maps (B) while learning to predict semantic segmentation. We utilize the output boundaries to penalize the label predictions from the target domain. To further penalize the label output, we add a boundaries discriminator (D_{B}), and a labels discriminator (D_{Y}) to penalize the output label and boundary.
Results
Experiment I: From SemanticKITTI to SemanticPOSS and SemanticUSL
Experiment II: From SemanticPOSS to SemanticKITTI and SemanticUSL
Experiment III: From SemanticUSL to SemanticPOSS and SemanticKTTI
Citation
@misc{jiang2020lidarnet,
title={LiDARNet: A Boundary-Aware Domain Adaptation Model for Point Cloud Semantic Segmentation
author={Peng Jiang and Srikanth Saripalli},
year={2020},
eprint={2003.01174},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Related Work
SemanticUSL: A Dataset for Semantic Segmentation Domain Adaptation