Computer vision with a single-pixel camera is currently limited by a trade-off between reconstruction capability and image classification accuracy. If random projections are used to sample the scene, then reconstruction is possible but classification accuracy suffers, especially in cases with significant background signal. If data-driven projections are used, then classification accuracy improves and the effect of the background is diminished, but image recovery is not possible. Here, we employ a shallow neural network to nonlinearly convert from measurements acquired with random patterns to measurements acquired with data-driven patterns. The results demonstrate that this improves classification accuracy while still allowing for full reconstruction.