Advanced digital modulation formats for optical coherent transmission systems