臺灣博碩士論文加值系統

當兩個設備在通信時，可能會受到各種干擾，像是來自其他設備的無意干擾或者是惡意的信號攔截和阻塞。這些干擾會嚴重影響通信品質，導致信號衰減和數據傳輸錯誤。為了在這樣的干擾環境下保持一定的通信品質，我們需要有效的干擾抑制技術，這種技術在相控陣列中被稱為零位消除 (nulling)。相控陣列由多個天線元件組成，能夠通過調整每個天線元件的相位和幅度來控制信號能量的方向。當相控陣列將能量集中於目標方向時，該方向的信號強度會顯著增強。相反地，若在某一方向上集中的能量較低，則該方向的信號強度將會減弱。這種能量分配能力使相控陣列能有效地增強目標信號的強度或減少干擾，以實現高品質的通信。然而，在動態環境中，干擾和信號方向的迅速變化使得我們需要不斷調整相控陣列的權重，以在變動的環境中保持最佳的信號品質和干擾抑制。一般的最佳化演算法，如粒子群最佳化 (PSO) 或是遺傳算法 (GA)，依賴於透過迭代來搜索最佳解。這些演算法在面對動態環境時，因其迭代特性而無法迅速更新相控陣列的權重，導致難以有效應對信號方向的快速變化。此外，這些演算法通常需要預先計算好的最佳權重配置數據，這在變動環境中是難以獲得的。我們提出了一種名為 DRLNuller 的創新深度強化學習模型。該模型利用近端策略優化 (PPO) 演算法，通過與環境的持續互動來動態優化相控陣列的權重，無需依賴事先計算好的數據。DRLNuller 在訓練時適應環境的變化後，便能夠迅速且有效地調整相控陣列的權重。在實驗中，DRLNuller 在計算速度上顯著優於一般最佳化演算法超過 2.83x10^5 倍，並在不同的環境條件下仍然保持有效的通信品質，其平均信號干擾比 (SIR) 約為 25.06 dB。

When two devices communicate, they may encounter various types of interference, such as unintentional interference from other devices or malicious signal jamming. This interference can degrade communication quality, leading to signal attenuation and data errors. To maintain high communication quality in such environments, effective interference suppression techniques are needed, known as nulling in phased arrays. Phased arrays consist of multiple antenna elements that can control signal direction by adjusting each element's phase and amplitude. Focusing energy in a target direction enhances signal strength, while low energy focus weakens it. In dynamic environments, rapid changes in interference and signal directions require constant adjustment of phased array weights. Traditional optimization algorithms, such as Particle Swarm Optimization (PSO) and Genetic Algorithm (GA), struggle to quickly update weights due to their iterative nature and need for pre-computed configurations, which are difficult to obtain in changing conditions. We propose an innovative deep reinforcement learning model called DRLNuller. Using the Proximal Policy Optimization (PPO) algorithm, it dynamically optimizes phased array weights through continuous interaction with the environment, without relying on pre-computed data. After adapting to environmental changes during training, DRLNuller can quickly and effectively adjust weights. In experiments, DRLNuller outperformed traditional algorithms in computation speed by over 2.83x10^5 times faster and maintained effective communication quality under different conditions, with an average Signal-to-Interference Ratio (SIR) of approximately 25.06 dB.

摘要 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 Background and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1 Phased Array . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Heuristic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3 System Model and Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4 DRLNuller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.1 Solution Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 DRLNuller Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.1 Open Sources and Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.2 Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6.2 Issues and Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . 24
6.3 DRLNuller vs. DDPG vs. Heuristic Algorithm . . . . . . . . . . . . . . . . . 25
6.4 Phased Array Size on Performance . . . . . . . . . . . . . . . . . . . . . . . . 28
6.5 The Effect of Signal Direction Variation . . . . . . . . . . . . . . . . . . . . . 29
6.6 The Effect of Strict Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . 30
7 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
7.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
7.2 Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

[1] C.-H. Hsu, C.-H. Chen, W.-J. Shyr, K.-H. Kuo, Y.-N. Chung, and T.-C. Lin, “Optimizing
beam pattern of linear adaptive phase array antenna based on particle swarm optimiza-
tion,” in 2010 Fourth International Conference on Genetic and Evolutionary Computing,
IEEE, 2010, pp. 586–589.
[2] E. Aksoy and E. Afacan, “Planar antenna pattern nulling using differential evolution al-
gorithm,” AEU-International Journal of Electronics and Communications, vol. 63, no. 2,
pp. 116–122, 2009.
[3] W.-P. Liao and F.-L. Chu, “Array pattern synthesis with null steering using genetic algo-
rithms by controlling only the current amplitudes,” International journal of electronics,
vol. 86, no. 4, pp. 445–457, 1999.
[4] D. Mandal, N. T. Yallaparagada, S. P. Ghoshal, and A. K. Bhattacharjee, “Wide null
control of linear antenna arrays using particle swarm optimization,” in 2010 Annual IEEE
India Conference (INDICON), IEEE, 2010, pp. 1–4.
[5] S. Lu, S. Zhao, and Q. Shi, “Learning-based massive beamforming,” in GLOBECOM
2020-2020 IEEE Global Communications Conference, IEEE, 2020, pp. 1–6.
[6] S. Bianco, P. Napoletano, A. Raimondi, M. Feo, G. Petraglia, and P. Vinetti, “Aesa adap-
tive beamforming using deep learning,” in 2020 IEEE Radar Conference (RadarConf20),
IEEE, 2020, pp. 1–6.
[7] F. Zardi, P. Nayeri, P. Rocca, and R. Haupt, “Artificial intelligence for adaptive and recon-
figurable antenna arrays: A review,” IEEE Antennas and Propagation Magazine, vol. 63,
no. 3, pp. 28–38, 2020.
[8] R. Lovato and X. Gong, “Phased antenna array beamforming using convolutional neural
networks,” in 2019 IEEE International Symposium on Antennas and Propagation and
USNC-URSI Radio Science Meeting, IEEE, 2019, pp. 1247–1248.
[9] A. H. Sallomi and S. Ahmed, “Multi-layer feed forward neural network application in
adaptive beamforming of smart antenna system,” in 2016 Al-Sadeq International Con-
ference on Multidisciplinary in IT and Communication Science and Applications (AIC-
MITCSA), IEEE, 2016, pp. 1–6.
[10] P. Ramezanpour and M.-R. Mosavi, “Two-stage beamforming for rejecting interferences
using deep neural networks,” IEEE Systems Journal, vol. 15, no. 3, pp. 4439–4447, 2020.
[11] X. Xiao and Y. Lu, “Data-based model for wide nulling problem in adaptive digital beam-
forming antenna array,” IEEE Antennas and Wireless Propagation Letters, vol. 18, no. 11,
pp. 2249–2253, 2019.
[12] T. Sallam and A. M. Attiya, “Convolutional neural network for 2d adaptive beamforming
of phased array antennas with robustness to array imperfections,” International Journal
of Microwave and Wireless Technologies, vol. 13, no. 10, pp. 1096–1102, 2021.
[13] J. Wang, F. Wang, B. Ding, W. Luo, and S. Niu, “A personnel-free method for array
synthesis based on artificial intelligence,” in 2022 IEEE 10th Asia-Pacific Conference on
Antennas and Propagation (APCAP), IEEE, 2022, pp. 1–2.
[14] L. R. Rere, M. I. Fanany, and A. M. Arymurthy, “Simulated annealing algorithm for deep
learning,” Procedia Computer Science, vol. 72, pp. 137–144, 2015.
[15] J. Kennedy and R. Eberhart, “Particle swarm optimization,” in Proceedings of ICNN’95-
international conference on neural networks, ieee, vol. 4, 1995, pp. 1942–1948.
[16] M. Mitchell, An introduction to genetic algorithms. MIT press, 1998.
[17] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy opti-
mization algorithms,” arXiv preprint arXiv:1707.06347, 2017.
[18] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-
critic methods,” in International conference on machine learning, PMLR, 2018, pp. 1587–
1596.
[19] I. Mallioras, Z. D. Zaharis, P. I. Lazaridis, and S. Pantelopoulos, “A novel realistic ap-
proach of adaptive beamforming based on deep neural networks,” IEEE Transactions on
Antennas and Propagation, vol. 70, no. 10, pp. 8833–8848, 2022.
[20] I. Mallioras, Z. D. Zaharis, P. I. Lazaridis, V. Poulkov, N. V. Kantartzis, and T. V. Yioult-
sis, “An adaptive beamforming approach applied to planar antenna arrays using neural
networks,” in 2022 IEEE International Black Sea Conference on Communications and
Networking (BlackSeaCom), IEEE, 2022, pp. 293–297.
[21] Y. Zhang, T. Osman, and A. Alkhateeb, “Online beam learning with interference nulling
for millimeter wave mimo systems,” IEEE Transactions on Wireless Communications,
2023.
[22] K.-B. Yu and M. F. Fernandez, “Methods to combine deterministic nulling and adaptive
nulling,” in 2017 IEEE Radar Conference (RadarConf), IEEE, 2017, pp. 0123–0128.
[23] A. Paszke, S. Gross, F. Massa, et al., “Pytorch: An imperative style, high-performance
deep learning library,” Advances in neural information processing systems, vol. 32, 2019.
[24] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation
hyperparameter optimization framework,” in Proceedings of the 25th ACM SIGKDD in-
ternational conference on knowledge discovery & data mining, 2019, pp. 2623–2631.
[25] J. Blank and K. Deb, “Pymoo: Multi-objective optimization in python,” Ieee access,
vol. 8, pp. 89 497–89 509, 2020.
[26] M. L. Waskom, “Seaborn: Statistical data visualization,” Journal of Open Source Soft-
ware, vol. 6, no. 60, p. 3021, 2021.
[27] J. D. Hunter, “Matplotlib: A 2d graphics environment,” Computing in science & engi-
neering, vol. 9, no. 03, pp. 90–95, 2007.