We present the first approach to automatically detect and track a needle in an endoscopic ultrasound (EUS) video. EUS is a fundamental medical procedure that is used extensively for biopsying targets in the upper gastro-intestinal tract. The approach enables various new applications for advancing EUS, including automatic EUS workflow analysis, automatic report generation, video summarization,  technical skills assessment and quality indicator automation. Our approach does not require annotations of needles in videos, which is a laborious and demanding task.