Corneal Confocal Microscopy and Mosaicking

In the mid 2010s, corneal confocal microscopes could only see a very small window of the cornea at a time. To look at the subbasal nerve plexus across the entire cornea, an ophthalmologist would take dozens to hundreds of overlapping frames and stitch them into one image by manually dragging the images into place. This is an extremely slow and laborious process, and was one of the reasons that the Siemens machines we had access to were only used in research, and rarely in clinical practice.

One of my interns at Techlab (Stuart Rucker) refined a pipeline based on OpenCV's image stitcher, which we adapted to account for deformation in the cornea itself, and used rough edge detection to find overlaps between frames. I helped advise Stuart on image processing, the graph theory behind choosing the order that frames get stitched in, and some other technical details here and there. Stuart is named on the paper as second author due to the enormous amount of work he did as a high school student.

Stitch order is a graph problem#

Frames arrive as an unordered pile with no reliable position data. Treat each frame as a node and each plausible overlap as a weighted edge, and picking a stitch order becomes a question about that graph rather than a question about images. Composing along strong edges first keeps error from accumulating through a chain of weak matches.

We were able to stitch frames in minutes, for what used to be a multi-hour process. Looking back on this project around a decade later, the technology available to do this has advanced significantly. We now have deep learning models that can stitch images in real time, and even some research on how to do this in a way that is more robust to corneal deformation and saccadic motion.