There has been a growing demand for integrating Convolutional Neural Networks (CNNs) powered functionalities into Internet-of-Thing (IoT) devices to enable ubiquitous intelligent "IoT cameras". However, there are two challenges limiting the application of Internet-of-Thing (IoT) devices powered by convolutional neural networks (CNNs) in real-world. First, some applications, especially medicine- and biology-related ones, impose strict requirements on camera size. Second, powerful CNNs often require a large number of parameters that correspond to considerable computing, storage, and memory bandwidth, whereas IoT devices only have limited resources. PhlatCam, due to its potentially orders-of-magnitude reduced form-factor, has provided a promising solution to the first aforementioned challenge, while the second one remains a bottleneck. To tackle this problem, existing compression techniques, focusing merely on the CNN algorithm itself, show some promise yet still limited. To this end, this work proposes SACoD, a Sensor Algorithm Co-Design framework to enable energy-efficient CNN-powered PhlatCam. In particular, the mask coded in the PhlatCam sensor and the CNN model in the algorithm is jointly optimized in terms of both parameters and architectures based on differential neural architecture search. Extensive experiments including both simulation and actual physical measurement on manufactured masks show that the proposed SACoD framework achieves aggressive model compression and energy savings while maintaining or even boosting the task accuracy, when benchmarking over two state-of-the-art (SOTA) designs with six datasets on two different tasks.