KR-CNN Example

We present the first approach to automatically detect and track a needle in an endoscopic ultrasound (EUS) video. EUS is a fundamental medical procedure that is used extensively for biopsying targets in the upper gastro-intestinal tract. The approach enables various new applications for advancing EUS, including automatic EUS workflow analysis, automatic report generation, video summarization,  technical skills assessment and quality indicator automation. Our approach does not require annotations of needles in videos, which is a laborious and demanding task. We propose to solve this problem by combining a state-of-the-art multi-hypothesis single image detector using Convolutional Neural Networks (CNN), with Bayesian temporal filtering to incorporate dynamics constraints. The single image detector is trained using a small but diverse dataset, covering variations in device, needle type, resolution and medical centre. The dynamics filter is used to detect the presence of the needle and tracking it over time, using noisy observations provided by the single-image detector's bounding box distribution. The method has only two free parameters that have been experimentally set using only one video. We have evaluated the approach on a dataset of 15 EUS videos, consisting in various real-world challenges including artefacts, fast needle motion, partially visible needles caused by needle bending and Doppler artefacts. Compared to the current state-of-the-art for needle detection in ultrasound (US) images, we improve considerably the mean average precision performance from 0.27 to 0.42. We will make our datasets public available.