臺灣博碩士論文加值系統

雲端運算採用虛擬化技術將資源虛擬為服務，以滿足雲用戶的即時性工作需求，例如，Spark Streaming資料流處理、資料庫的讀取和互動查詢等。但雲端運算面臨著各雲用戶者間不同硬體需求、不同結構化資料，資料時效性等問題，當多個雲用戶同時請求資源時資源調度即成為很重要的議題。目前雲端系統管理者面對大量雲用戶請求資源，需進行雲端資源分配時，管理者會以維持雲戶間公平性為考量，但現有資源分配機制卻未考量資源的充分使用。
為克服此問題，本研究提出在雲環境中考慮工作的資源異質性及虛擬機資源異質性的資源配置策略。並針對雲端環境中建置Mesos叢集，探討Mesos的資源配置演算法，以改善資源使用率。首先，透過工作資源需求比例，將工作分類為CPU密集型或內存密集型的工作，再由TAHR演算法搜尋目前能達到資源使用率最大的子集合，確保資源能夠最大限度被利用以增加雲環境的效率。本研究採用對於能處理大數據資料的計算框架，Apache Spark，作為標竿程式，並部署到適當的虛擬機，提升系統的吞吐量。

Cloud computing adopts the virtualization technology to provide various resources as services to stratify the real-time jobs of cloud users, such as Spark Streaming data processing, the read/write actions in database and interactive queries. However, cloud computing would receive various requests with different hardware requirements, different structured data, and different real-time constraints by different cloud users. So, resource allocation becomes an important issue in improving the system performance of cloud computing when multiple cloud users request diverse resource requirements at the same time. In general, most of resource allocation mechanisms in cloud computing focuses on the allocation fairness among service requests. However, fairness usually could not meet better resource utilization in clouds.
To overcome the low-utilization problem, this work proposes a new resource allocation approach for considering both the resource heterogeneity and the job-demand heterogeneity. This work considers the available resources in the cloud system and tries to allocate CPU-intensive applications to the server with higher ratio of available number of CPUs to the available number of memory, or allocate Memory-intensive applications to the server with lower ratio of available number of CPUs to the available number of memory. The above allocation decision applies the TAHR algorithm to obtain a higher utilization with more stable performance. In order to show the performance of the proposed resource allocation approach in the cloud, this work builds an Mesos cluster in the cloud environment, and adopts one big data processing framework, Apache Spark, to demonstrate the performance improvement. In the experiments, this work could deploy the Spark applications to the appropriate available resources in virtual machines to increase system throughput.

致謝 I
摘要 II
Abstract III
圖目錄 VII
表目錄 IX
第 一 章 緒論 1
1.1背景 1
1.2 動機 2
1.3 議題 3
1.4 目標 4
1.5論文章節及內容 4
第 二 章 相關研究 5
2.1 Apache Mesos 5
2.2 Apache Spark 11
2.3 Mesosphere 1312
2.4 資源配置 13
第 三 章 研究方法 18
3.1 系統架構 18
3.2 Task Allocation for Heterogeneous Resources演算法 20
3.3 Heterogeneous Task Allocation for Heterogeneous Resources演算法 28
第 四 章 效能評估 31
4.1實驗環境 31
4.2 效能評估程式 34
4.3 實驗一 35
4.4實驗二 44
4.5實驗三 53
第 五 章 結論與未來展望 55
參考文獻 57

[1] Adam, O. Y., Lee, Y. C., & Zomaya, A. Y. “Constructing performance-predictable clusters with performance-varying resources of clouds,” IEEE Transactions on Computers, Vol. 65, No. 2016, pp. 2709 – 2724.
[2] Apache ZooKeeper, https://zookeeper.apache.org/ 2018/7/4.
[3] Beloglazov, A., Abawajy, J., & Buyya, R. “Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing,” Future Generation Computer Systems, Vol. 28, No. 5, pp. 755768, 2012
[4] Farahnakian, F., Pahikkala, T., Liljeberg, P., Plosila, J., & Tenhunen, H. “Utilization prediction aware VM consolidation approach for green cloud computing,” In Cloud Computing (CLOUD), IEEE 8th International Conference on, pp. 381-388, 2015
[5] Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S., & Stoica, I. “Dominant Resource Fairness: Fair Allocation of Multiple Resource Types, ” NSDI Symposium on Networked Systems Design and Implementation, Vol. 11, No. 2011, pp. 24-24.
[6] Ghodsi, A., Zaharia, M., Shenker, S., & Stoica, I. “Choosy: Max-min fair sharing for datacenter jobs with constraints, ” In Proceedings of the 8th ACM European Conference on Computer Systems, pp. 365-378, 2013.
[7] Ghribi, C., Hadji, M., & Zeghlache, D. “Energy e_cient vm scheduling for
cloud data centers: Exact allocation and migration algorithms,” In: Proceedings of the 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp. 671–678, 2013.
[8] Hadoop: Fair Scheduler. https://translate.google.com.tw/?hl=zh-TW#en/zh-TW/Hadoop%3A%20Fair%20Scheduler 2018/7/4.
[9] Hosseinalipour, S., & Dai, H. “Options-based sequential auctions for dynamic cloud resource allocation, ” IEEE International Conference on Communications, pp. 1-6, 2017
[10] Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A. D., Katz, R. H., ... & Stoica, I. “Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center, ” USENIX conference on Networked systems design and implementation, Vol. 11, No. 2011, pp. 295-308.
[11] Mesos , http://mesos.apache.org/ 2018/7/4.
[12] Mesos Architecture, http://mesos.apache.org/documentation/latest/architecture/ 2018/7/4.
[13] Mesos Shared Persistent Volumes, http://mesos.apache.org/documentation/latest/　shared-resources/ 2018/7/4.
[14] Marathon Constraints, https://mesosphere.github.io/marathon/docs/ constraints.html 2018/7/4.
[15] Mesosphere, https://mesosphere.com/product/ 2018/7/4.
[16] Mesosphere DC/OS, https://dcos.io/install/ 2018/7/4.
[17] Reiss, C., Tumanov, A., Ganger, G. R., Katz, R. H., & Kozuch, M. A. “Heterogeneity and dynamicity of clouds at scale: Google trace analysis,” In Proceedings of the Third ACM Symposium on Cloud Computing, pp. 7, 2012
[18] Spark Coarse-Grained. https://spark.apache.org/docs/latest/running-on-mesos.html
[19] Shi, W., Wu, C., & Li, Z. “An online auction mechanism for dynamic virtual cluster provisioning in geo-distributed clouds, ” IEEE Transactions on Parallel and Distributed Systems, Vol. 28, No. 2017, pp. 677-688.
[20] Scale like Twitter with Apache Mesos, https://opensource.com/business/14/8/　interview-chris-aniszczyk-twitter-apache-mesos 2018/7/4.
[21] Tang, L., & Chen, H. “ Joint pricing and capacity planning in the iaas cloud market, ” IEEE Transactions on Cloud Computing, Vol. 5 No. 2017, pp. 57-70.
[22] Verma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., & Wilkes, J. “Large-scale cluster management at Google with Borg, ” ACM Proceedings of the Tenth European Conference on Computer Systems, No. 2015, pp. 18.
[23] Wang, W., Liang, B., & Li, B. “ Multi-resource fair allocation in heterogeneous cloud computing systems, ” IEEE Transactions on Parallel and Distributed Systems, Vol. 26, 2015, pp. 2822-2835.
[24] Wei, L., Foh, C. H., He, B., & Cai, J. “Towards Efficient Resource Allocation for Heterogeneous Workloads in IaaS Clouds, ” IEEE Transactions on Cloud Computing, Vol. 6, No. 2018, pp. 264 - 275.
[25] Xu, J., & Palanisamy, B. “ Cost-aware Resource Management for Federated Clouds Using Resource Sharing Contracts, ” IEEE International Conference on Cloud Computing, pp. 238-245, 2017