The entry of air into hydraulic power systems presents a key challenge for both stationary and mobile machinery. Given the performance limitations of conventional online deaeration methods, this research was conducted to design, fabricate, and optimize an air-oil separator unit. To simulate the turbulent flow and the physical separation mechanism, a three-dimensional, two-phase numerical model was developed, integrating the Volume of Fluid (VOF) and Discrete Phase Model (DPM) approaches. A laboratory-scale hydro-pneumatic test rig was designed and fabricated to validate the numerical model. A comparison between experimental and numerical results revealed a discrepancy of approximately 2.5% for separation efficiency and 5.6% for pressure drop, thereby confirming the model's validity for analyzing the variables affecting the separator's performance. The results indicated that separation efficiency exhibits a non-linear behavior, with an optimal inlet velocity range of 3.5 to 5.5 m/s establishing a balance between high efficiency (87.5–90%) and an acceptable pressure drop (735–1150 Pa). Furthermore, the geometric analysis identified optimal ranges for key parameters. Accordingly, optimal values for the conical section angle (5–6°), cylindrical diameter (36–40 mm), and air outlet diameter (18 mm) yielded separation efficiencies of 89%, 90%, and 90%, respectively. These findings provide a quantitative and reliable framework for the optimal design of air-oil separators, demonstrating that an effective balance between efficiency and energy consumption can be achieved by selecting appropriate geometric parameters. The implementation of this approach is expected to enhance the reliability and service life of hydraulic systems significantly.