Deep neural networks have revolutionized static image understanding through their amazing performance in tasks such as classification, segmentation and style transfer. However, current architectures have yet to find as much success in video tasks mainly due to increased data dimension, higher information throughput and a sub-optimal frame-driven representation. Inspired by the early vision systems of mammals and insects, we propose an event-driven input representation (EDR) that models several major properties of early retinal circuits: 1. Output scales logrithmically with input intensity, 2. ON/OFF pathways for differentiating event types, 3. Relative change detection and event detection via thresholding. With UCF-101 video action recognition experiments, we show that neural network utilizing EDR as input performs near state-of-the-art in accuracy while achieving a 1,500x speedup in input representation processing (9K frames/sec), as compared to optical flow. EDR's fast processing speed enables real-time inference/learning in time sensitive video applications such as reinforcement learning. In this vein, we use EDR to demonstrate performance improvements over state-of-the-art reinforcement learning algorithms for Atari games, something that has not been possible with optical flow.