For color vision, retinal circuits separate information about intensity and wavelength. In vertebrates that use the full complement of four “ancestral” cone types, the nature and implementation of this computation remain poorly understood. Here, we establish the complete circuit architecture of outer retinal circuits underlying color processing in larval zebrafish. We find that the synaptic outputs of red and green cones efficiently rotate the encoding of natural daylight in a principal components analysis–like manner to yield primary achromatic and spectrally opponent axes, respectively. Blue cones are tuned to capture most remaining variance when opposed to green cones, while UV cone present a UV achromatic axis for prey capture. We note that fruitflies use essentially the same strategy. Therefore, rotating color space into primary achromatic and chromatic axes at the eye’s first synapse may thus be a fundamental principle of color vision when using more than two spectrally well-separated photoreceptor types.