Currently, image sensors are hybrid devices, combining semiconductor photodiodes with off-chip color filters of different materials to convert wavelength-selected light into useful photocurrent. Here we demonstrate a fully integrated, metal-semiconductor-metal (MSM) photodetector and plasmonic color filter fabricated entirely from Aluminum and silicon designed to detect light in selected wavelength bands across the visible spectrum. The device produces photocurrent gain by carrier accumulation, while exploiting the evanescent field of the surface plasmon for both wavelength selectivity and photocurrent enhancement. With a maximum responsivity of 12.54 A/W and a full-width-half-maximum (FWHM) spectral selectivity of ~100nm, this high performance photodetector has potential for immediate applications in color-selective low-light imaging and high pixel density imaging sensors.