Learning End-To-End 2d-3d Representations For Cross-Modality And Cross-Domain Shape Reconstruction And Processing

Author

Yifan Wang, Wayne State University

Access Type

WSU Access

Date of Award

January 2022

Degree Type

Dissertation

Degree Name

Ph.D.

Department

Computer Science

First Advisor

Zichun Zhong

Abstract

The rapid development of deep learning techniques paves a new way to explore the relationship not only between different 2D / 3D representations of objects, but also between 3D processing and 2D visual clues from our 3D real world. In this dissertation, we are going to introduce several interesting yet challenging real-world vision / graphics tasks, learning 2D-3D representations for cross-modality and cross-domain shape reconstruction and processing, (mostly) by using different medical testbeds. At first, we review the related works in different application field, i.e. computer vision and medical imaging analysis, which covers wide topics such as 2D-to-3D reconstructions, 3D image segmentation, image super resolution. Then we present our three major contributions in this dissertation, which can be categorized into three parts as follows:

2D-to-3D Learning: We present an end-to-end deep learning method with a lightweight but effective neural network to reconstruct multiple high-fidelity 3D organ meshes with a variety of geometric shapes from a single-view medical image with complicated background and noises. The proposed organ reconstruction network simultaneously learns the optimal selection and the best smooth deformation from multiple templates via a trivariate tensor-product deformation technique, i.e., free-form deformation (FFD), to match the query 2D image. Our deep learning framework is the first method to generate multiple 3D organ meshes (such as left and right lungs in our application) from a single-view medical image. The application and user study on IGRT demonstrate that the accurate on-the-fly tracking and reconstruction of 3D / 4D organ shapes facilitated by our method have the potential in improving the current IGRT procedure and practice.

Joint 3D-2D Learning - Part 1: We present an effective end-to-end deep learning method to segment and visualize high-fidelity 3D sparse microvascular structure with complicated geometry and topology variations from volumetric images with significant noise. Our multi-stream CNN framework is designed to effectively learn the feature vectors of 3D raw volume and multislice composited 2D MIP (volume rendering), respectively, and explore inter-dependencies between 3D and 2D embedded features in a joint volume-composition embedding space by unprojecting (inverse volume rendering) the 2D features, learned from MIP, into the 3D volume embedding space. To our knowledge, this is the first time that a deep learning framework is proposed to construct such a joint convolutional embedding space, where the computed joint vessel probabilities from 2D projection and 3D volume can be integrated synergistically.The application and experiments on the accurate in-vivo segmentation and visualization of sparse and complicated 3D microvascular structure facilitated by our method demonstrate the potential in a novel and powerful MR arteriogram and venogram (MRAV) diagnosis of vascular disease.

Joint 3D-2D Learning - Part 2: We proposes an effective end-to-end deep learning method to segment and visualize high-fidelity 3D SR microvascular structure with complicated geometry and topology variations and tiny sizes from LR volumetric images in the wild. Our multi-stream CNN framework is designed to effectively learn the feature vectors of 3D raw microstructured volume and multislice compound 2D super-resolution microstructured image, respectively, and explore inter-dependencies between 3D and 2D embedded features in the joint multi-level hybrid embedding spaces by aggregating deeply at different fusion stages. To our knowledge, this is the first time that a deep learning framework is proposed to construct the joint multi-level hybrid (imaging and processing) embedding spaces, where the computed joint micro vessel probabilities for the 3D volume processing (segmentation) can be synergistically adapted and augmented from the compound 2D super-resolution imaging acquisition. The application and experiments on the accurate in-vivo segmentation and visualization of complicated and tiny 3D microvascular structure in the midbrain regions facilitated by our method demonstrate the potential in a novel and powerful MR arteriogram and venogram (MRAV) diagnosis of vascular disease.

Recommended Citation

Wang, Yifan, "Learning End-To-End 2d-3d Representations For Cross-Modality And Cross-Domain Shape Reconstruction And Processing" (2022). Wayne State University Dissertations. 3611.
https://digitalcommons.wayne.edu/oa_dissertations/3611