大數據技術與管理

臺北科技大學 114-2 學期課程，課號 358371。

課程大綱

由於雲端運算的快速發展，大量資料收集與儲存技術已成瓶頸，大數據技術與管理提供一套具延展性與容錯能力的分散儲存與計算架構, 能有效處理大量複雜數據資料的儲存與運算。 本課程除介紹現有的大數據生態與趨勢，並針對目前軟硬體架構的處理瓶頸，提供系統組態與部署、服務層級協定、監控與管理等系統性規劃的解決方案，並導入跨領域的工程管理科學知識。課程簡扼論述大數據技術與管理的重要，並在Hadoop生態系統中，以Cloudera Hadoop為應用範例，講授說明其重要性。 課程內容包括：大數據的概念、應用生態與趨勢、Hadoop硬體規格與基礎架構規劃、叢集組態與部署、叢集服務層級協定、叢集監控與管理、相關生態技術。

課程進度

(Syllabus, preliminary) Friday. 19:20 - 22:00 (BCD) Week# Subject to change per needs and circumstances 01 2/27 Public Holiday - 228 Peace Memorial Day 02 3/06 Syllabus & organization of all mission cases from the class 03 3/13 Big Data Application and Trend 04 3/20 Hadoop Cluster Deployment 05 3/27 Core Technologies for Big Data Management 06 4/03 Public Holiday - Spring Festival 07 4/10 Hadoop Distributed File System (HDFS) 08 4/17 MapReduce 09 4/24 YARN Applications (No Mid-term Exam) ，Check Note Column below* 10 5/01 Public Holiday - Labour Day 11 5/08 Hadoop I/O & Hadoop Cluster Hardware and Software 12 5/15 Hadoop High Available, Failover, Fensing and Federation 13 5/22 Hadoop Ecosystem Project Presentation and Q&A -1 14 5/29 Hadoop Ecosystem Project Presentation and Q&A -2 15 6/05 Hadoop Ecosystem Project Presentation and Q&A -3 16 6/12 Resource & Scheduler Management + Hadoop Security 17 6/19 Public Holiday - Dragon Boat Festival 18 6/26 Final-Exam

評量方式

1.作業 - Assignments including Reports , Presentation & Q&A: 50% 2.期末考 - (Final-Term Exam): 50%

教材

【遵守智慧財產權觀念，請使用正版教科書，不得使用非法影印教科書】 使用外文原文書：是 Teacher's handouts, plus the references below 1. Hadoop: The Definitive Guide, Fourth Edition, April 2015, Published by O’Reilly Media, Inc., Editor: Tom White. 2. Hadoop Essentials, First Edition, April 2015, Published by O’Reilly Media, Inc., Editor: Shiva Achari. 3. Hadoop Operation, First Edition, September 2012, Published by O’Reilly Media, Inc., Editor: Eric Sammer. 4. 大數據大時代 - 新一代儲存技術及實作，作者: 查偉，發行所:佳魁資訊， 出版日期: 2017, 3月. 5. Yusuf Aytas, Designing Big Data Platform, How to Use, Deploy and Maintain Big Data Systems, Wiley, February 2021. 6. Jimmy Lin and Chris Dyer, Data-Intensive Text Processing with MapReduce, Morgan & Claypool Publishers, 2010. 7. Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia, Learning Spark: Lightning-Fast Big Data Analysis, O'Reilly Media, January 2015. Additional Reading References: 8. The Google File System; https://static.googleusercontent.com/media/research.google.com/zh-TW//archive/gfs-sosp2003.pdf 9. MapReduce: Simplified Data Processing on Large Clusters; https://static.googleusercontent.com/media/research.google.com/zh-TW//archive/mapreduce-osdi04.pdf 10. Big Data associated, To-Be-Advised in class