(3.238.173.209) 您好!臺灣時間:2021/05/16 21:13
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果

詳目顯示:::

我願授權國圖
: 
twitterline
研究生:許家繻
研究生(外文):Chia-Ju Hsu
論文名稱:在Kubernetes與Kafka上實作基於Consumer lag之微服務動態轉移機制
論文名稱(外文):Consumer lag aware microservice relocation mechanism over Kubernetes and Kafka
指導教授:王尉任王尉任引用關係
學位類別:碩士
校院名稱:國立中央大學
系所名稱:軟體工程研究所
學門:電算機學門
學類:軟體發展學類
論文出版年:2020
畢業學年度:108
語文別:中文
論文頁數:61
中文關鍵詞:KubernetesKafka微服務動態轉移
外文關鍵詞:KubernetesKafkamicroservicedynamic relocation
相關次數:
  • 被引用被引用:0
  • 點閱點閱:75
  • 評分評分:
  • 下載下載:0
  • 收藏至我的研究室書目清單書目收藏:0
近年來容器化技術發展迅速,許多軟體服務開始以容器和微服務的形式部署運行,而開源容器管理系統Kubernetes為容器自動部署和管理提供強大的功能,使容器部署變得更加彈性且可靠,許多雲端供應商也提供Kubernetes代管服務,使在雲端叢集中部署容器變得更加普遍。同時,一些物聯網和邊緣運算的應用情境也開始嘗試使用Kubernetes部署工作,這些工作可能會面臨邊緣端環境變化導致的性能下降、節點計算能力不均或工作量動態變化等問題導致工作量堆積。然而,Kubernetes的計算資源管理依賴管理者的估算,難以適應動態的變化。因此,本研究設計出一套Kubernetes擴充功能,以Kafka訊息串流平台作為容器之間的資料傳遞管道,透過監控其Consumer Lag以支援在Kubernetes上實現根據容器的工作進度對容器進行轉移,並設計轉移流程以保證轉移後不會有資料遺失或重複處理的狀況。
Container technology has developed rapidly in recent years, and many software services are deployed as containers and microservices. Kubernetes, an open-source container orchestration system that provides powerful features to automatically deploy and manage containerized applications, makes container deployment more flexible and reliable. So many cloud vendors provide managed Kubernetes service make it more popular to deploy containers on a cloud cluster. Some scenarios of the Internet of Things or edge computing also use Kubernetes to deploy their jobs. These jobs may face some problems, such as degradation of work node performance caused by edge environment variety, uneven computing power between work nodes, or dynamic workload. However, managing resources for containers in Kubernetes is rely on the manager’s estimation, and it is difficult to adjust to dynamic workload changes. Therefore, this thesis proposes a set of extension functions for Kubernetes, make it possible to relocate containers according to the work progress, which is achieved by using Kafka as message passing channel and monitoring Consumer Lag of each job. This thesis also proposes a relocation control flow to avoid data loss or data re-process after container relocation.
摘要 i
Abstract ii
目錄 iii
表目錄 v
圖目錄 v
一、 緒論 1
1-1 研究背景 1
1-2 研究動機與目的 2
1-3 論文架構 2
二、 背景知識 3
2-1 Kubernetes 3
2-2 Kubernetes Pod轉移機制 5
2-2-1 基於資源使用量 5
2-2-2 節點不可用 6
2-3 Prometheus 7
2-3-1 kube-prometheus 8
2-4 Apache Kafka 9
2-4-1 Kafka Offset Commit 機制 10
2-4-2 Kafka Consumer Group Rebalance機制 11
2-5 服務發現 12
2-5-1 Client-Side模式 12
2-5-2 Server-Side模式 12
2-5-3 Kubernetes中的服務發現 12
三、 相關研究與討論 15
3-1 Circuit Breaker 15
3-2 邊緣運算工作轉移相關研究 15
四、 系統設計 17
4-1 系統架構 17
4-2 Job Operator 22
4-2-1 ManagedJob 22
4-2-2 Region 24
4-2-3 ConsumerLagMonitor 26
4-3 自訂kube-prometheus 28
4-4 ConsumerLag Monitor 29
4-5 Consumer Helper 32
4-5-1 轉移情況討論 32
4-5-2 流程設計 33
五、 使用範例及實驗 35
5-1 實驗環境 35
5-2 模擬計算量之overhead量測 36
5-2-1 模擬中斷後重新啟動狀況 38
5-2-2 正常轉移狀況 40
5-3 Consumer數量變化 42
5-4 模擬車流量偵測應用情境之overhead量測 43
六、 結論與未來研究方向 48
參考文獻 49
[1] D. Bernstein, “Containers and Cloud: From LXC to Docker to Kubernetes,” IEEE Cloud Computing, vol. 1, no. 3, pp. 81–84, Sep. 2014, doi: 10.1109/MCC.2014.51.
[2] D. Jaramillo, D. V. Nguyen, and R. Smart, “Leveraging microservices architecture by using Docker technology,” in SoutheastCon 2016, Mar. 2016, pp. 1–5, doi: 10.1109/SECON.2016.7506647.
[3] F. Montesi and J. Weber, “Circuit Breakers, Discovery, and API Gateways in Microservices,” arXiv:1609.05830 [cs], Sep. 2016, Accessed: Apr. 26, 2020. [Online]. Available: http://arxiv.org/abs/1609.05830.
[4] “What is Kubernetes.” https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/ (accessed Mar. 26, 2020).
[5] B. Burns, J. Beda, and K. Hightower, “Kubernetes: Up and Running,” p. 1.
[6] “Managing Compute Resources for Containers.” https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ (accessed Apr. 11, 2020).
[7] “Configure Out of Resource Handling.” https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/ (accessed Apr. 11, 2020).
[8] “Overview | Prometheus.” https://prometheus.io/docs/introduction/overview/ (accessed Mar. 26, 2020).
[9] “prometheus/node_exporter.” https://github.com/prometheus/node_exporter (accessed Mar. 26, 2020).
[10] “coreos/kube-prometheus,” May 29, 2020. https://github.com/coreos/kube-prometheus (accessed May 29, 2020).
[11] “Jsonnet - The Data Templating Language.” https://jsonnet.org/ (accessed May 29, 2020).
[12] “Apache Kafka,” Apache Kafka. https://kafka.apache.org/intro (accessed Mar. 28, 2020).
[13] M. Kleppmann and J. Kreps, “Kafka, Samza and the Unix Philosophy of Distributed Data,” p. 7.
[14] “Kafka 0.9 Consumer Rewrite Design - Apache Kafka - Apache Software Foundation.” https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design#Kafka0.9ConsumerRewriteDesign-HeartbeatResponse (accessed Jun. 13, 2020).
[15] “Kafka Client-side Assignment Proposal - Apache Kafka - Apache Software Foundation.” https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal#KafkaClientsideAssignmentProposal-Phase1:JoiningtheGroup (accessed Jun. 13, 2020).
[16] C.-Y. Fan and S.-P. Ma, “Migrating Monolithic Mobile Application to Microservice Architecture: An Experiment Report,” in 2017 IEEE International Conference on AI Mobile Services (AIMS), Jun. 2017, pp. 109–112, doi: 10.1109/AIMS.2017.23.
[17] “Service.” https://kubernetes.io/docs/concepts/services-networking/service/ (accessed Apr. 30, 2020).
[18] Xuan-Qui Pham and Eui-Nam Huh, “Towards task scheduling in a cloud-fog computing system,” in 2016 18th Asia-Pacific Network Operations and Management Symposium (APNOMS), Oct. 2016, pp. 1–4, doi: 10.1109/APNOMS.2016.7737240.
[19] D. Haja, M. Szalay, B. Sonkoly, G. Pongracz, and L. Toka, “Sharpening Kubernetes for the Edge,” in Proceedings of the ACM SIGCOMM 2019 Conference Posters and Demos, Beijing, China, Aug. 2019, pp. 136–137, doi: 10.1145/3342280.3342335.
[20] “The confluent_kafka API — confluent-kafka 1.4.0 documentation.” https://docs.confluent.io/current/clients/confluent-kafka-python/ (accessed Jun. 02, 2020).
電子全文 電子全文(網際網路公開日期:20220731)
連結至畢業學校之論文網頁點我開啟連結
註: 此連結為研究生畢業學校所提供,不一定有電子全文可供下載,若連結有誤,請點選上方之〝勘誤回報〞功能,我們會盡快修正,謝謝!
QRCODE
 
 
 
 
 
                                                                                                                                                                                                                                                                                                                                                                                                               
第一頁 上一頁 下一頁 最後一頁 top