The incorporation of user-assisted cooperative relaying into beamspace massive multiple-input multiple-output (mMIMO) non-orthogonal multiple access (NOMA) system can extend the coverage area and improve the spectral and energy efficiency for millimeter wave (mmWave) communications when a dynamic cluster of mobile user terminals (MUTs) is formed within a beam. We propose threshold-based user-assisted cooperative relaying into a beamspace mMIMO NOMA system in a downlink scenario. Specifically, the intermediate MUTs between the next-generation base station (gNB) and the cell-edge MUT become relaying MUTs after the successful decoding of the signal of the cell-edge MUT only when they meet the predetermined signal-to-interference plus noise ratio (SINR) threshold. A zero forcing (ZF) precoder and iterative power allocation are used to minimize both inter- and intra-beam interferences to maximize the system sum rate. We then evaluate the performance of this system in a delay-intolerant cell-edge MUT scenario. Moreover, the outage probability of the cell-edge MUT of the proposed scheme is investigated and an analytic expression is derived. Simulation results confirm that the proposed threshold-based user-assisted cooperative relaying beamspace mMIMO NOMA system outperforms the user-assisted cooperative relaying in beamspace mMIMO NOMA, beamspace MIMO-NOMA, and beamspace MIMO orthogonal multiple access (OMA) systems in terms of spectrum efficiency, energy efficiency, and outage probability.
