Recent advances in nanotechnology have highlighted the need for precise and stable methods for manipulating both biological and non-biological particles. Among the available tools, atomic force microscopy (AFM) is considered a key instrument due to its high capability for controlled contact and measurement of extremely small displacements. However, maintaining stable contact between the probe tip and the particle and preventing undesired slippage, especially when dealing with complex geometries, remains a significant challenge. In this study, a dynamic modeling framework combined with sliding mode control (SMC) was proposed to enhance AFM performance during particle manipulation. Simulation results demonstrated that the designed controller could maintain the probe’s position and angle with high accuracy. Examination of three particle geometries—spherical, cylindrical, and chamfered cylindrical—revealed that slippage increased with surface complexity, with spherical particles exhibiting the least sliding (3.4%) and chamfered cylindrical particles the most (4.8%). Furthermore, a comparison of three cantilever types showed that the V-shaped cantilever achieved the best performance, with only 2.1% sliding and a significant reduction in angular fluctuations.These findings indicate that combining accurate dynamic modeling with sliding mode control provides an effective approach for developing advanced AFM systems and expanding their applications in biological studies and tissue engineering.