臺灣博碩士論文加值系統

本研究探討了在聯邦學習（Federated Learning, FL）中如何透過模型相似度來選
擇最佳的客戶端，以提升訓練的通訊成本及準確度。聯邦學習是一種分散式的機器
學習方法，讓多個設備可以協作訓練模型而不需要共享原始數據。但在模型傳輸過
程會產生大量的通訊成本，以及在不同資料分佈的情況下，容易出現 non-iid 的情形，
因此我們想透過強化學習架構讓機器能自我學習，達到自動選擇選擇最佳客戶端組
合的研究目標。研究主要貢獻如下：設計了一種基於客戶端相似度的強化學習架構，
能自動選擇最佳的客戶端參與訓練，從而提升全域模型的準確度。透過相似度矩陣
和泛化性分析進行客戶端分群，並選擇優質的客戶端進行訓練，有效減少通訊成本
及傳輸時間。實驗結果顯示，在減少了 56% 通訊成本的情況下，模型依舊能維持和
FedAvg 相同水準的準確度。此方法在保護數據隱私的同時，提高了模型的訓練效率
和減少通訊成本，展示了聯邦學習在不同應用場景中的潛力和優勢。

The study explores how to enhance communication costs and accuracy in Federated
Learning (FL) by selecting the best clients based on model similarity. Federated Learning is a
decentralized machine learning approach that allows multiple devices to collaboratively train
a model without sharing raw data. However, the model transmission process generates significant communication costs, and the non-iid (non-independent and identically distributed)
situation can easily arise due to different data distributions. Therefore, we aim to leverage
a reinforcement learning framework to enable the machine to autonomously select the optimal combination of clients, achieving the research objective of automatic client selection.
The main contributions of this research are as follows: We designed a reinforcement learning
framework based on client similarity that can automatically select the best clients for training, thereby improving the accuracy of the global model. Through the use of a similarity
matrix and generalization analysis, clients are clustered, and high-quality clients are selected
for training, effectively reducing communication costs and transmission time. Experimental
results show that the model can maintain the same level of accuracy as FedAvg while reducing
communication costs by 56%. This method enhances training efficiency and reduces communication costs while protecting data privacy, demonstrating the potential and advantages
of Federated Learning in various application scenarios.

目錄
摘要. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
Abstract. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
誌謝. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
本文. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
圖目錄. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
表目錄. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
1簡介. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2先備知識與文獻探討. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1強化學習. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 FederatedLearning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3文獻探討. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3系統架構與設計. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.1深度Q學習（DeepQ-Network,DQN）. . . . . . . . . . . . . . . . . . . . 11
3.2相似度辨別. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.3泛化性辨別. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.4初始模型架構與DistanceMatrix生成和泛化性模型挑選流程. . . . . . . . 15
3.5客戶端分群. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.6 Action. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.7 Actiontransformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.8 Reward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.9 State. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.10建立DQN模型與Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.11整體架構流程. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4效能評估. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.1資料集. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.1.1 Client端資料分配. . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2模型架構. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.3實驗配置. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.4實驗評估. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5結論. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
參考文獻. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
圖目錄
圖1.1 FederatedLearning於金融領域應用圖例. . . . . . . . . . . . . . . . . 1
圖1.2傳統機器學習與FederatedLearning之比較圖. . . . . . . . . . . . . . 2
圖1.3 FederatedLearning架構圖. . . . . . . . . . . . . . . . . . . . . . . . . 3
圖1.4應用Client端選擇之通訊成本比較. . . . . . . . . . . . . . . . . . . . 3
圖2.1強化學習示意圖. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
圖2.2 FairFederatedLearning架構圖[1] . . . . . . . . . . . . . . . . . . . . . 8
圖2.3 FedDRL架構圖[2] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
圖2.4 Learning-basedClientSelections架構圖. . . . . . . . . . . . . . . . . . 9
圖3.1深度Q學習基本架構圖[3] . . . . . . . . . . . . . . . . . . . . . . . . . 11
圖3.2 DistanceMatrix示意圖. . . . . . . . . . . . . . . . . . . . . . . . . . . 13
圖3.3 DistanceMatrix生成架構流程圖. . . . . . . . . . . . . . . . . . . . . . 15
圖3.4 Actiontransformation示意圖. . . . . . . . . . . . . . . . . . . . . . . . 18
圖3.5 Actiontransformation示意圖. . . . . . . . . . . . . . . . . . . . . . . . 18
圖3.6 DNN模型. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
圖3.7本文系統架構圖. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
圖4.1 CIFAR-10資料集內容[4] . . . . . . . . . . . . . . . . . . . . . . . . . . 24
圖4.2 Datadistribution範例. . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
圖4.3 Datadistribution範例. . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
圖4.4 CNN模型架構圖. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
圖4.5加入泛化性模型準確率比較. . . . . . . . . . . . . . . . . . . . . . . . 29
圖4.6不同Cluster數量及γ數量準確度比較. . . . . . . . . . . . . . . . . . 30
圖4.7 α設置為0.3，準確度比較. . . . . . . . . . . . . . . . . . . . . . . . . 31
圖4.8 α設置為0.1，準確度比較. . . . . . . . . . . . . . . . . . . . . . . . . 31
圖4.9與Fedavg比較. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
圖4.10與Fedavg(12)比較. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
圖4.11與Random比較. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
圖4.12通訊成本比較. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

[1] Yaqi Sun, Shijing Si, Jianzong Wang, Yuhan Dong, Zhi Bo Zhu, and Jing Xiao. A
fair federated learning framework with reinforcement learning. In Proceedings of the
International Joint Conference on Neural Networks (IJCNN), 2022.
[2] Nang Hung Nguyen, Phi-Le Nguyen, Duc Long Nguyen, Trung Thanh Nguyen, ThuyDung Nguyen, Huy-Hieu Pham, and Truong Thao Nguyen. Feddrl: Deep reinforcement
learning-based adaptive aggregation for non-iid data in federated learning. In Proceedings of the International Conference on Parallel Processing (ICPP), 2022.
[3] Dilith Jayakody. Deep q-networks (dqn) – a quick introduction (with code). https://dilithjay.com/blog/
deep-q-networks-dqn-a-quick-introduction-with-code/.
[4] Alex Krizhevsky. Cifar-10 and cifar-100 datasets. https://www.cs.toronto.edu/
~kriz/cifar.html.
[5] H. B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera
y Arcas. Communication-efficient learning of deep networks from decentralized data.
In Proceedings of the International Conference on Artificial Intelligence and Statistics(AISTATS), 2016.
[6] Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang
Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. Federated learning in mobile
edge networks: A comprehensive survey. IEEE Communications Surveys Tutorials,
22(3):2031–2063, 2020.
[7] Robin C. Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning:
A client level perspective. https://arxiv.org/abs/1712.07557, 2018.
[8] Aaron Segal, Antonio Marcedone, Benjamin Kreuter, Daniel Ramage, H. Brendan
McMahan, Karn Seth, K. A. Bonawitz, Sarvar Patel, and Vladimir Ivanov. Practical
secure aggregation for privacy-preserving machine learning. In Proceedings of the 2017
ACM SIGSAC Conference on Computer and Communications Security, page 1175–
1191, New York, NY, USA, 2017. Association for Computing Machinery.
[9] Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S. Rellermeyer. A survey on distributed machine learning. ACM Computing
Surveys (CSUR), 53(2), 2020.
[10] HUI-ROU ZHOUG. Learning-based client selections considering client similarity in
federated learning. Master’s thesis, Yuan Ze University, 2023.
[11] S. A. Gokte. Most popular distance metrics used in knn and when to use them. KDnuggets, 2023.
[12] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT
Press, Cambridge, MA, 2nd edition, 2018.
[13] Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A
survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013.
[14] J. Moody and M. Saffell. Learning to trade via direct reinforcement. IEEE Transactions
on Neural Networks, 12(4):875–889, 2001.
[15] H. B. McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera y Arcas. Federated
learning of deep networks using model averaging. arXiv preprint arXiv:1602.05629,
2016.
[16] Jian Li, Tongbao Chen, and Shaohua Teng. A comprehensive survey on client selection
strategies in federated learning. Computer Networks, 251:110663, 2024.
[17] Zhipeng Cheng, Xuwei Fan, Ning Chen, Minghui Liwang, Lianfen Huang, and Xianbin Wang. Learning-based client selection for multiple federated learning services with
constrained monetary budgets. ICT Express (2023): n. pag., 9(6):1059–1064, 2023.
[18] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. https://arxiv.org/
abs/1611.03530, 2017.
[19] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From
Theory to Algorithms. Cambridge University Press, Cambridge, UK, 2014.
[20] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra.
Federated learning with non-iid data. https://arxiv.org/abs/1806.00582, 2018.
[21] Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques,
Yan Gao, Lorenzo Sani, Hei Li Kwing, Titouan Parcollet, Pedro PB de Gusmão, and
Nicholas D Lane. Flower: A friendly federated learning research framework. arXiv
preprint arXiv:2007.14390, 2020.