Colonoscopy is routinely performed for colorectal cancer screening but lacks the capability to accurately characterize precursor lesions and early cancers. High-resolution microendoscopy (HRME) is a low-cost imaging tool to visualize colorectal polyps with subcellular resolution. We present a computer-aided algorithm to evaluate HRME images of colorectal polyps and classify neoplastic from benign lesions. Using histopathology as the gold standard, clinically relevant features based on luminal morphology and texture are quantified to build the classification algorithm. We demonstrate that adenomatous polyps can be identified with a sensitivity and specificity of 100% and 80% using a two-feature linear discriminant model in a pilot test set. The classification algorithm presented here offers an objective framework to detect adenomatous lesions in the colon with high accuracy and can potentially improve real-time assessment of colorectal polyps.