I. 3D Reconstruction in Capsule Endoscopy Videos using Shape from Shading (SfS) Techniques

Abstract:

Endoscopic optical systems have unique characteristics that differentiate from other optical imaging techniques. Especially, the near-light sources in capsule endoscopy cause the problem of inferring shape from shading (SfS) a complicated one. Moreover, specular and non-Lambertian nature of the intestinal tissues call for a non-traditional model. In this project, we study robust perspective SfS models and show how it helps reconstruct 3D shapes in an efficient manner for a variety of endoscopic systems.

Example reconstructions in wireless capsule endoscopy images:

The following examples illustrates the proposed shape from shading (SfS) based 3D reconstruction scheme [1] for different Wireless Capsule Endoscopy images:

3D Reconstructions using our scheme [1]

Note that, the first example, a Wireless Capsule Endoscopy image, contains villus structures outlining the mucosa surface, the second example contains a polyp, and both are captured efficiently in our 3D reconstruction.

Reference:

[1]* V. B. S. Prasath, I. N. Figueiredo, P. N. Figueiredo, K. Palaniappan. Mucosal region detection and 3D reconstruction in wireless capsule endoscopic videos by active contours. 34th Annual International Conference IEEE EMBS (EMBC), San Diego, CA, 2012. Proc. IEEE, pp. 4014-4017.

Acknowledgement:

*Part of the work [1] was done while the first author was at the Department of Mathematics, University of Coimbra, PT. The first author also gratefully acknowledges Prof. Richard Tsai (UTAustin) for his advise and help.

*The work [1] was partially supported by the research project UTAustin/MAT/0009/2008 of the UT Austin|Portugal Program (http://www.utaustinportugal.org/) and by CMUC and FCT (Portugal), through European program COMPETE/FEDER.

II. Shape Reconstruction for Endoscopic Images with Robust Perspective Shape from Shading (RP-SfS) and AI models

Aim: Endoscopic optical systems have unique characteristics that differentiate from other optical imaging techniques. Especially, the near-light sources cause the problem of inferring shape from shading (SfS) a complicated one. Moreover, specular and non-Lambertian nature of the intestinal tissues call for a non-traditional model. In this work, we propose a robust perspective SfS model and show how it reconstructs shapes in an efficient manner for a variety of endoscopic systems. 

Recently, Artificial intelligence (AI)-based techniques such as deep learning models offer better and robust reconstruction than traditional SfS methods. In this work, given a 2D endoscopy image, the AI model predicts the corresponding 3D surface of the mucosa. This reconstructed surface captures the subtle details and folds that are often missed in 2D images. Comparison of AI-based 3D reconstructions with SfS methods is undertaken on a variety of colonoscopy systems. Our experimental results demonstrate the promising potential of AI-based reconstruction for improving colonoscopy visualization. Compared to traditional 2D images, the reconstructed 3D surfaces offer (1) Enhanced Detail - subtle features like polyps and early-stage cancers become more visible, improving detection rates (2) Improved Depth Perception - the 3D view allows viewers to better understand the spatial relationships between different parts of the colon, leading to more accurate diagnoses (3) Reduced Examination Time - by providing a clearer picture, the 3D visualization can potentially shorten the examination time and improve patient comfort. Our AI-based approach shows significant potential for improving 3D colonoscopy visualization and potentially enhancing diagnostic accuracy. The reconstructed 3D surfaces can be visualized using advanced 3D rendering techniques. This allows clinicians to navigate the overall surface better, zoom in on specific areas of interest, and analyze the mucosa in greater detail. Further research is needed to validate these findings in a larger clinical setting and investigate the impact on clinical outcomes.

Example 3D reconstructions in Colonoscopy image with a peduncled polyp:

In this example, a Colonoscopy image, has a big peduncle polyp and our 3D reconstruction shows the amount of its protrusion out of lumen. This is before an illumination correction step (images to be added) and hence the small bumps seen are due to specularities of the mucosa surface. A preliminary specular removal step will eliminate such bumps.

Example 3D reconstruction in Colonoscopy image with a sessile polyp:

Note that, this example, a Colonoscopy image, contains a polyp and it is captured efficiently in our 3D reconstruction (after a specular removal step).

Comparison with related Perspective Shape from Shading schemes:

Sample result:

References:

[1] V. B. S. Prasath, H. Kawanaka. Near-light perspective shape from shading for 3D visualizations in endoscopy systems. IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Kansas, USA. Proc. IEEE, pp. 2293-2295, Nov 2017. doi:10.1109/BIBM.2017.8218031. Download poster at figshare: 10.6084/m9.figshare.5526907

[2] V. B. S. Prasath. More wiggle room – AI-based reconstruction of mucosal surfaces for 3D endoscopy visualization. In preparation, 2024. Preliminary work presented at 7th Annual IBDHorizons Midwest Symposium, Cincinnati, 2024.

[3] V. B. S. Prasath. Mucosal surface reconstruction from endoscopic images with robust perspective shape from shading - From classical models to deep learning. In preparation, 2024.

Bibliography:

[2] M. Visentini-Scarzanella, D. Stoyanov, G.-Z. Yang, Metric Depth Recovery from Monocular Images Using Shape-from-Shading and Specularities, IEEE International Conference on Image Processing (ICIP), Orlando, FL, 2012.

Back Endoscopy Projects Main Page. Back to Research Page.