A Holistic Deep Learning Framework For Endoscopy