Visual Contrastive Self-Distillation
On policy self distillation (OPSD) is promising as it removes the external teacher required by on policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be removed, yieldin...