The atomic force microscopy (AFM) is a high-resolution measurement tool for measuring sample topography and material properties in micro-scale and nano-scale research. The dynamics of the cantilever probe in AFM is influenced by the intrinsically nonlinear interaction between the probe tip and the surface of the sample. Previous work has shown that in off-resonance excited intermittent-contact AFM, a period-doubling bifurcation can occur as a result of the nonlinearity. The amplitude of the resulting sub-harmonic frequency component of the response has been identified as a source of contrast to measure the Young's modulus of the sample. This dissertation details the continued work in this area and includes three parts. In the first part, the focus is to investigate the performance of a material characterization method, proposed to use the relationship between the sample modulus and the sub-harmonic frequency component, to study material property transitions for one-dimensional samples. In the second part, the focus is on the effect of the inclusion of the explicit dissipative interaction force in the system model on the numerical simulation on the AFM. Both resonant and off-resonant excitation conditions are discussed. In the third part, the focus is on the generation of the sub-harmonic amplitude for an unique dual-frequency excitation condition. The influence of this excitation condition is numerically investigated and experimental studies are conducted with a macro-scale constrained cantilevered beam system to qualitatively verify the numerically predicted behavior. The work in this dissertation brings a wider understanding for these nonstandard excitation methods and their applications in AFM.