Despite advances in therapy over the past decades, metastatic colorectal cancer (mCRC) remains a highly morbid disease. While the impact of MHC-I on immune infiltration in mCRC has been well studied, data on the consequences of MHC-II loss are lacking. Multiplex fluorescent immunohistochemistry (mfIHC) was performed on 149 patients undergoing curative intent resection for mCRC and stratified into high and low human leukocyte antigen isotype DR (HLA-DR) expressing tumors. Intratumoral HLA-DR expression was found in stromal bands, and its expression level was associated with different infiltrating immune cell makeup and distribution. Low HLA-DR expression was associated with increased intercellular distances and decreased population mixing of T helper cells and antigen-presenting cells (APC), suggestive of decreased interactions. This was associated with less co-localization of tumor cells and cytotoxic T lymphocytes (CTLs), which tended to be in a less activated state as determined by Ki67 and granzyme B expression. These findings suggest that low HLA-DR in the tumor microenvironment of mCRC may reflect a state of poor helper T-cell interactions with APCs and CTL-mediated anti-tumor activity. Efforts to restore/enhance MHC-II presentation may be a useful strategy to enhance checkpoint inhibition therapy in the future.