Optimal detectors based on time-frequency/time-scale representations (TFRs/TSRs) admit a representation in terms of a bank of spectrograms/scalograms that yields a large class of detectors. These range from the conventional matched filter to the more complex higher-rank detectors offering a superior performance in a wider variety of detection situations. In this paper, we optimize this complexity versus performance tradeoff by characterizing TFR/TSR detectors that optimize performance (based on the deflection criterion) for any given fixed rank. We also characterize the gain in performance as a function of increasing complexity thereby facilitating a judicious tradeoff. Our experience with real data shows that, in many cases, relatively low-rank optimal detectors can provide most of the gain in performance relative to matched-filter processors.