3D Reconstruction in Endoscopy

Abstract:

Endoscopic optical systems have unique characteristics that differentiate from other optical imaging techniques. Especially, the near-light sources in capsule endoscopy cause the problem of inferring shape from shading (SfS) a complicated one. Moreover, specular and non-Lambertian nature of the intestinal tissues call for a non-traditional model. In this project, we study various approaches that can help reconstruct 3D shapes in an efficient manner for a variety of endoscopic systems:

I. Classical SfS models

II. Robust perspective SfS models 

III. Artificial intelligence (AI) and foundation models (FMs)

Reference:

[1]* V. B. S. Prasath, I. N. Figueiredo, P. N. Figueiredo, K. Palaniappan. Mucosal region detection and 3D reconstruction in wireless capsule endoscopic videos by active contours. 34th Annual International Conference IEEE EMBS (EMBC), San Diego, CA, 2012. Proc. IEEE, pp. 4014-4017.

Acknowledgement:

*Part of the work [1] was done while the first author was at the Department of Mathematics, University of Coimbra, PT. The first author also gratefully acknowledges Prof. Richard Tsai (UTAustin) for his advise and help.

*The work [1] was partially supported by the research project UTAustin/MAT/0009/2008 of the UT Austin|Portugal Program (http://www.utaustinportugal.org/) and by CMUC and FCT (Portugal), through European program COMPETE/FEDER.

II. Shape Reconstruction for Endoscopic Images with Robust Perspective Shape from Shading (RP-SfS)

Aim: Endoscopic optical systems have unique characteristics that differentiate from other optical imaging techniques. Especially, the near-light sources cause the problem of inferring shape from shading (SfS) a complicated one. Moreover, specular and non-Lambertian nature of the intestinal tissues call for a non-traditional model. In this work, we propose a robust perspective SfS model and show how it reconstructs shapes in an efficient manner for a variety of endoscopic systems. 

Example 3D reconstructions in Colonoscopy image with a peduncled polyp:

In this example, a Colonoscopy image, has a big peduncle polyp and our 3D reconstruction shows the amount of its protrusion out of lumen. This is before an illumination correction step (images to be added) and hence the small bumps seen are due to specularities of the mucosa surface. A preliminary specular removal step will eliminate such bumps.

Example 3D reconstruction in Colonoscopy image with a sessile polyp:

Note that, this example, a Colonoscopy image, contains a polyp and it is captured efficiently in our 3D reconstruction (after a specular removal step).

Comparison with related Perspective Shape from Shading schemes

References:

[1] V. B. S. Prasath, H. Kawanaka. Near-light perspective shape from shading for 3D visualizations in endoscopy systems. IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Kansas, USA. Proc. IEEE, pp. 2293-2295, Nov 2017. doi:10.1109/BIBM.2017.8218031. Download poster at figshare: 10.6084/m9.figshare.5526907

[2] V. B. S. Prasath. Mucosal surface reconstruction from endoscopic images with robust perspective shape from shading - From classical models to deep learning. In preparation, 2025.

Bibliography:

[3] M. Visentini-Scarzanella, D. Stoyanov, G.-Z. Yang, Metric Depth Recovery from Monocular Images Using Shape-from-Shading and Specularities, IEEE International Conference on Image Processing (ICIP), Orlando, FL, 2012.

Back Endoscopy Projects Main Page. Back to Research Page.

References:

[1] V. B. S. Prasath. More wiggle room – AI-based reconstruction of mucosal surfaces for 3D endoscopy visualization. In preparation, 2025. Preliminary work presented at 7th Annual IBDHorizons Midwest Symposium, Cincinnati, 2024.

[2] V. B. S. Prasath. More wiggle room – AI-based reconstruction of mucosal surfaces for 3D endoscopy visualization. Submitted, 2025.

Back Endoscopy Projects Main Page. Back to Research Page.