DENG-251: Building an Open Data Lakehouse Using Apache Iceberg

Course Modules

• Introduction to Data Lakehouse Architecture

  • Understanding the core concepts of Data Lakehouses and how they differ from traditional data lakes and warehouses
  • Key components and architecture of a modern data lakehouse
  • Use cases for building a data lakehouse with Apache Iceberg

• Getting Started with Apache Iceberg

  • Overview of Apache Iceberg and its role in the data lakehouse ecosystem
  • Installation and setup of Apache Iceberg on cloud-based platforms (AWS, Azure, GCP)
  • Iceberg’s architecture, metadata handling, and file formats

• Data Modeling and Partitioning in Apache Iceberg

  • Data modeling best practices for building efficient data lakehouses
  • Using partitioning and clustering techniques in Apache Iceberg to optimize query performance
  • Understanding schema evolution and how Iceberg manages schema changes over time

• Data Ingestion and Management with Apache Iceberg

  • Best practices for ingesting large-scale data into Iceberg tables
  • Incremental data loading strategies and handling streaming data
  • Managing data consistency and transactional operations in Apache Iceberg

• Querying Data in the Open Data Lakehouse

  • Performing fast, scalable queries using Apache Iceberg tables with Apache Spark, Presto, or Trino
  • Leveraging Iceberg’s time travel and versioning capabilities to query historical data
  • Optimizing query performance and cost in the data lakehouse

• Data Governance and Security in the Data Lakehouse

  • Implementing data governance policies using Apache Iceberg
  • Securing data in the lakehouse and managing user access control
  • Ensuring compliance and data auditing for analytical workloads

• Advanced Features of Apache Iceberg

  • Managing large-scale datasets efficiently with Iceberg's ACID properties
  • Using snapshot isolation and time travel for advanced analytical capabilities
  • Integration with other cloud-native tools and frameworks for advanced analytics

• Operationalizing and Scaling the Data Lakehouse

  • Scaling your Apache Iceberg setup for big data workloads
  • Monitoring and maintaining the Iceberg-based data lakehouse environment
  • Best practices for disaster recovery, backups, and cluster management