Visual tracking methods have achieved a successful development in recent years. Especially the Discriminative Correlation Filter (DCF) based methods have significantly advanced the state-of-the-art in tracking. The advancement in DCF tracking performance is predominantly attributed to powerful features and sophisticated online learning formulations. However, it would come to some troubles if the tracker learns the samples indiscriminately.