Neural radiance field based 3D view reconstruction for gastrointestinal tract surgery planning

Gastrointestinal (GI) tract is a 9-meter-long food passage that transports food and nutrition to the human body. Due to the narrow space (~2cm in diameter) of the GI tract, a lesion removal surgery through endoscopic procedures faces limited view of the lesion, leading to the challenge of maximum removal of diseased lesion and minimum sacrifice of healthy tissues. This paper evaluates the application of the state-of-the-art AI technique of neural radiance field to reconstruct 3D models of a lesion based on 2D endoscopic videos. In this way, a full 3D view of concerned GI lesion can be presented to surgeons to allow comprehensive surgical planning. The advanced AI techniques, neural radiance field (NeRF), structure-from-motion (SfM) and multi-view stereo (MVS) are applied. This system is implemented based on COLMAP and Nerfstudio libraries. Results: The system is evaluated using two sets of video clips containing 2600 images. Initial results illustrate that this end-to-end deep learning architecture, i.e. from 2D video input to 3D model output, pre-sents considerable potential for reconstruction of GI lesions. The similarity measures of SSIM, PSNR and LPIPS between original (ground truth) and rendered images are of 19.46 ± 2.56, 0.70 ± 0.054, and 0.49 ± 0.05 respectively. Conclusion: This work contributes to the solution towards one of the long-standing challenges in computer vision field, which is to construct 3D views from 2D videos in the field of GI surgery planning and has achieved a promising performance. Future work includes enlarging datasets and removal of ghostly artefact from rendered images.

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.