Tumor cell migration and invasion of body tissues are prerequisite mediators for lymphatic or hematogenous cancer dissemination. To date, there is insufficient understanding of what triggers the metastatic cascade, and of how the interplay among cell receptors, the cellular and acellular components of the extracellular matrix and proteolytic enzymes mediate cancer migration, invasion, proliferation and survival. In addition to the inherent complexity of each one of the aforementioned phenomena is the lack of an experimental technique capable of dissecting the mechanisms that mediate the dynamic invasive and migratory behavior at the cellular level and with respect to the properties of the cell environment. The goal of my thesis was to develop an automated system for cell tracking in three dimensions and use it to model the dynamics of cancer invasion and migration. Therefore the hardware and software were designed for a fully automated optical 3D cell tracking system that quantified long-term invasion and migration of cancer cells infiltrating 3D extracellular matrix analogs. The quantitative analysis of the cell trajectories employed a novel formulation of the continuous Markov model that evaluated the potential for invasive or lateral motion and cell stops. The infiltration of human HT1080 fibrosarcoma and human MDA-MB-231 adenocarcinoma cells, was monitored in plain or Matrigel-containing collagen type I gels. Parameters such as the speed subpopulations, the persistence of motion in certain directions, the turning frequency of the cells, the preferred directions of motion, and the invasion depth profiles over time quantified infiltration at the cellular level. Distinct migratory and invasive phenotypes significantly dependent on the gel composition were identified for the two cell types. The HT1080 cell line expressed a high motility phenotype and well-preserved lateral motion on the plain collagen gel surface. The basement membrane components transformed the HT1080 cells to robust invaders by significantly enhancing the matrix infiltration and the turning frequency. The low motility, slow invasion and low turning behavior of MDA-MB-231 cells indicated that their invasiveness may depend on matrix-degrading activity. To the best of my knowledge this is the first study employing a detailed set of quantitative descriptors to demonstrate that tumor invasion and migration are dynamic processes of individual cells that depend significantly on the cell type and the tumor microenvironment.