The morphology and molecular mechanisms of animal photoreceptor cells and eyes reveal a complex pattern of duplications and co-option of genetic modules, leading to a number of different light-sensitive systems that share many components, in which clear-cut homologies are rare. On the basis of molecular and morphological findings, I discuss the functional requirements for vision and how these have constrained the evolution of eyes. The fact that natural selection on eyes acts through the consequences of visually guided behaviour leads to a concept of task-punctuated evolution, where sensory systems evolve by a sequential acquisition of sensory tasks. I identify four key innovations that, one after the other, paved the way for the evolution of efficient eyes. These innovations are (i) efficient photopigments, (ii) directionality through screening pigment, (iii) photoreceptor membrane folding, and (iv) focusing optics. A corresponding evolutionary sequence is suggested, starting at non-directional monitoring of ambient luminance and leading to comparisons of luminances within a scene, first by a scanning mode and later by parallel spatial channels in imaging eyes.