The performance of millimeter wave (mmWave) line-of-sight multiple input multiple output (LOS MIMO) systems using hybrid arrays of planar subarrays was studied. We characterized the achievable maximum spatial multiplexing gain for such LOS MIMO systems by the measures of spectral efficiency and effective degree of freedom (EDoF). By proposing a joint plane-wave and spherical-wave-based general 3D channel model, we derived the optimal design parameters in the analog domain, i.e., the optimal subarray separation products, and analyzed their sensitivity on the system performance. We also gave analytical eigenvalue expressions of the equivalent LOS MIMO channel matrix, which are applicable to the case of a non-optimal design, as well as the upper and lower bounds of the EDoF for system performance evaluation. A piecewise uniform quantization codebook was further designed for quantizing phase shifter values in practical applications. The numerical and simulation results show that planar subarrays are superior to traditional arrays in terms of spectral efficiency and EDoF in Ricean fading channels because they are more robust to the change in the communication distance and the deviation from the optimal design. The use of hybrid arrays of planar subarrays effectively removes the limitation of mmWave LOS MIMO systems using traditional arrays, through which, the conventional Rayleigh distance criterion has to be satisfied to achieve the optimal performance.
