Data Engineering Knowledge Base
200 data engineering terms, each opening with a one-paragraph definition and then going deeper: how the idea works, where it shows up in a real pipeline, and which neighbouring concepts it connects to. Written for people who have to build the thing, not just name it.
Looking for something specific? Search the whole site, or jump to a letter below.
A
Active Data Governance
Active data governance is the modern approach that shifts governance from passive documentation to automated, programmatic enforcement of security, privacy, and quality rules directly within the data pipeline and serving layers.
Read DefinitionAgentic Analytics
The next evolution of enterprise data, where autonomous AI agents use semantic layers and the data lakehouse to reason, plan, and execute complex analytical workflows without human intervention.
Read DefinitionAnalytics Engineering
Analytics engineering is the discipline that sits between data engineering and data analysis, using software engineering best practices and tools like dbt to transform raw data into reliable, well-documented, business-ready analytical models.
Read DefinitionApache Airflow
Apache Airflow is the open-source workflow orchestration platform that schedules, monitors, and manages complex data pipeline DAGs in production data engineering environments.
Read DefinitionApache Arrow
Apache Arrow is the open-source cross-language columnar memory format that enables high-performance in-memory analytics and zero-copy data exchange between data systems.
Read DefinitionApache Avro
Apache Avro is the row-oriented serialization format with schema evolution support that serves as the standard format for Kafka event streaming and data pipeline message exchange.
Read DefinitionApache Calcite
Apache Calcite is the open-source query planning framework that provides SQL parsing, validation, relational algebra, and cost-based optimization capabilities used by Hive, Flink, Druid, Trino, and Dremio as the foundation for their query planners.
Read DefinitionApache Flink
Apache Flink is the open-source distributed stream processing engine that provides low-latency, stateful event processing for real-time analytical and operational pipelines.
Read DefinitionApache Gravitino
Apache Gravitino is the open-source unified metadata layer that provides a single catalog API over multiple heterogeneous data sources, enabling governed multi-engine data discovery and access.
Read DefinitionApache Hudi
Apache Hudi is the open-source data lakehouse storage format from Uber that pioneered incremental data processing and upsert capabilities for streaming workloads on object storage.
Read DefinitionApache Iceberg
The definitive open table format for the data lakehouse, enabling ACID transactions, hidden partitioning, and schema evolution at massive scale.
Read DefinitionApache Kafka
Apache Kafka is the distributed event streaming platform that serves as the central nervous system for real-time data pipelines, enabling high-throughput, durable, and scalable event streaming.
Read DefinitionApache Kafka Connect
Apache Kafka Connect is the scalable, resilient integration framework within the Kafka ecosystem designed to stream data reliably between Kafka and external databases, key-value stores, and cloud storage systems.
Read DefinitionApache Nessie
Project Nessie is the open-source transactional catalog for Apache Iceberg that provides Git-like branching and versioning semantics at the catalog level, enabling multi-table atomic transactions and full catalog history.
Read DefinitionApache ORC
Apache ORC (Optimized Row Columnar) is the columnar file format developed for the Hadoop ecosystem that pioneered many of the columnar storage optimizations later extended by Parquet and modern lakehouse formats.
Read DefinitionApache Parquet
Apache Parquet is the open-source columnar storage format that has become the foundational data file format for modern data lakehouses and analytical processing.
Read DefinitionApache Pinot
Apache Pinot is a real-time, distributed OLAP datastore purpose-built to deliver ultra-low latency analytics across massive, constantly updating event streams for user-facing applications.
Read DefinitionApache Polaris
Apache Polaris is the open-source, vendor-neutral Iceberg REST catalog that provides unified table governance across multiple compute engines and cloud environments.
Read DefinitionApache Spark
Apache Spark is the distributed computing engine that transformed large-scale data processing with its unified API for batch, streaming, SQL, ML, and graph analytics.
Read DefinitionApache Superset
Apache Superset is the open-source data exploration and visualization platform originally created at Airbnb, designed for fast dashboarding and SQL-based ad-hoc analytics at enterprise scale.
Read DefinitionAPI Gateway
An API gateway is the architectural component that acts as the single entry point and traffic cop for thousands of microservices, handling routing, security, rate limiting, and analytics.
Read DefinitionArrow Flight
Apache Arrow Flight is the high-performance data transport protocol built on gRPC and the Arrow columnar format that enables ultra-fast, network-saturating data transfer between analytical systems.
Read DefinitionAt-Least-Once Delivery
At-least-once delivery is the pragmatic streaming semantics pattern that guarantees no data loss at the risk of generating duplicates, requiring downstream idempotency to maintain data integrity.
Read DefinitionAttribute-Based Access Control (ABAC)
Attribute-Based Access Control is the fine-grained authorization model that makes access decisions based on attributes of the user, resource, environment, and action rather than static role assignments.
Read DefinitionAWS Glue Data Catalog
AWS Glue Data Catalog is the fully managed, serverless metadata repository that serves as the central catalog for AWS analytics services and provides an HMS-compatible API for Apache Iceberg and Hive-compatible tables.
Read DefinitionB
Backfilling
Backfilling in data engineering is the essential process of reprocessing historical data using new pipeline logic to ensure consistency across the entire dataset after a bug fix or feature addition.
Read DefinitionBias-Variance Tradeoff
The bias-variance tradeoff is the fundamental tension in machine learning between a model being too simple to capture patterns (high bias) and being so complex it memorizes noise (high variance).
Read DefinitionBitemporal Data
Bitemporal data modeling is the advanced technique of tracking data across two distinct timelines (valid time and transaction time) to accurately recreate historical states and track retroactive corrections.
Read DefinitionBloom Filters
Bloom filters are a probabilistic data structure used in Iceberg, Parquet, and query engines to dramatically accelerate point lookups by skipping data files that definitely cannot contain matching values.
Read DefinitionBloom Filters in Parquet
Parquet bloom filters are a probabilistic data structure embedded in Parquet files that enables point lookup queries to skip entire row groups containing no matching values for high-cardinality columns.
Read DefinitionC
Change Data Capture (CDC)
Change Data Capture (CDC) is the data integration technique that identifies and delivers row-level database changes in real time to downstream analytical systems.
Read DefinitionCloud Data Warehouse
Cloud data warehouses are the fully managed, scalable analytical database systems like Snowflake, BigQuery, and Redshift that reshaped analytics by decoupling storage from compute in the cloud era.
Read DefinitionColumn Masking
Column masking in data lakehouses is the data governance technique that dynamically replaces sensitive column values with masked representations based on the querying user's authorization level.
Read DefinitionColumnar Storage vs. Row-Oriented Storage
A comprehensive guide to the architectural difference between columnar and row-oriented storage, and why columnar storage is the foundation of high-performance analytical data platforms.
Read DefinitionCompaction
Compaction in Apache Iceberg and data lakehouses is the critical table maintenance operation that merges small files into optimally sized Parquet files to restore query performance degraded by high-frequency writes.
Read DefinitionContinuous Processing
Continuous processing is the true streaming architecture that processes data event-by-event with millisecond latency, as opposed to waiting for scheduled batches or micro-batch intervals.
Read DefinitionControl Plane vs Data Plane
A guide to the architectural separation of the Control Plane (the management and orchestration layer) and the Data Plane (the physical execution and storage layer) in modern distributed data systems.
Read DefinitionCopy-on-Write (CoW)
A guide to the Copy-on-Write storage strategy in Apache Iceberg and Apache Hudi, where write operations rewrite entire data files to produce clean, merged snapshots optimized for read-heavy analytical workloads.
Read DefinitionCost-Based Optimizer
The Cost-Based Optimizer (CBO) is the algorithmic engine within a query planner that uses statistical metadata to mathematically estimate and select the fastest, cheapest execution path for a SQL query.
Read DefinitionCQRS (Command Query Responsibility Segregation)
CQRS is the architectural pattern that completely separates the code and databases used for reading data (Queries) from the code and databases used for writing data (Commands) to maximize performance and scalability.
Read DefinitionCredential Vending
Credential vending in the Apache Iceberg ecosystem is the security pattern where a catalog service issues scoped, time-limited storage credentials to compute engines rather than distributing permanent broad-access credentials.
Read DefinitionD
Data Annotation
Data annotation is the labor-intensive process of manually labeling raw unstructured data (images, text, audio) so that supervised machine learning models can learn to identify patterns.
Read DefinitionData Architecture
Data architecture is the structural design of the data systems, pipelines, and storage layers that dictates how data is acquired, processed, stored, and distributed across an organization to support its data strategy.
Read DefinitionData Archiving
Data archiving is the strategic process of moving cold, infrequently accessed historical data off expensive, high-performance storage onto ultra-cheap, durable storage tiers to optimize costs.
Read DefinitionData as a Service (DaaS)
Data as a Service (DaaS) is the architectural pattern that treats curated data products as governed, API-accessible services with defined SLAs, ownership, and discoverability, enabling self-service data consumption across an organization.
Read DefinitionData Catalog
Enterprise data catalogs are the metadata management platforms that make data assets discoverable, understandable, and trustworthy for both human analysts and AI-powered analytical systems.
Read DefinitionData Classification
Data classification is the critical governance process of categorizing data based on its sensitivity, business value, and regulatory risk to apply appropriate security controls and retention policies.
Read DefinitionData Compliance
Data compliance covers the legal and regulatory frameworks governing data storage, retention, and deletion, and how data engineering architectures must adapt to meet GDPR, CCPA, and HIPAA requirements.
Read DefinitionData Contracts
Data contracts are the formal agreements between data producers and consumers that define schema, quality standards, SLAs, and ownership to prevent breaking changes and ensure reliable data pipelines.
Read DefinitionData Deduplication
Data deduplication covers the techniques used to identify and remove duplicate records that occur due to at-least-once delivery semantics, retry logic, or source system anomalies.
Read DefinitionData Democratization
Data democratization is the strategic initiative to make data accessible to all roles in an organization without requiring SQL or engineering expertise, using self-service BI tools, governed semantic layers, and AI-powered natural language query interfaces.
Read DefinitionData Discovery
Data discovery covers the tooling and processes that enable users to find, understand, and trust data assets across an organization through searchable catalogs, metadata enrichment, and automated lineage tracking.
Read DefinitionData Enrichment
Data enrichment in analytical pipelines is the process of augmenting internal datasets with external context, third-party data, or derived classifications to increase the analytical value and predictive power of the data.
Read DefinitionData Fabric
Data Fabric is the unified architecture that combines data integration, governance, and intelligent automation to connect distributed enterprise data sources into a coherent analytical fabric.
Read DefinitionData Governance
Data governance in the modern lakehouse is the framework of policies, processes, and technologies that ensures data is trustworthy, secure, and used appropriately across the enterprise.
Read DefinitionData Gravity
Data Gravity is the concept that as datasets grow massive, they become increasingly difficult to move, forcing applications, compute, and services to be built physically close to the data itself.
Read DefinitionData Integration
Data integration is the technical and business process of combining data from disparate sources into unified, consistent datasets that provide a complete view of the organization's operations for analytics and reporting.
Read DefinitionData Lake
An in-depth exploration of the data lake, from its origins in the Hadoop ecosystem to its role in modern cloud object storage, and its evolution into the governed data lakehouse.
Read DefinitionData Lakehouse
A comprehensive guide to the data lakehouse architecture, bridging the reliability of data warehouses with the scale and flexibility of data lakes, powered by open table formats like Apache Iceberg.
Read DefinitionData Lakehouse Architecture
Data lakehouse architecture is the modern analytical platform that combines open file formats on object storage with table format governance and SQL query engines to deliver warehouse performance with data lake flexibility.
Read DefinitionData Lakehouse vs. Data Warehouse
A guide comparing the data lakehouse and traditional data warehouse architectures, examining the trade-offs in openness, cost, flexibility, and performance that determine the right choice for different organizational contexts.
Read DefinitionData Lineage
Data lineage is the map of a data asset's lifecycle that traces its origins, transformations, and downstream consumption to ensure trust, simplify debugging, and enable impact analysis.
Read DefinitionData Mesh
Data Mesh is a decentralized socio-technical paradigm that shifts data ownership from centralized engineering bottlenecks to distributed business domains.
Read DefinitionData Mesh
Data Mesh is the decentralized sociotechnical approach to analytical data architecture that distributes data ownership to domain teams, treating data as a product with federated governance and shared platform infrastructure.
Read DefinitionData Migration
Data migration is the complex engineering process of securely and accurately transferring massive datasets from legacy systems (like on-premises data warehouses) to modern cloud architectures like the data lakehouse.
Read DefinitionData Modeling
Data modeling is the discipline of structuring and organizing data to accurately represent business processes and enable efficient analytical querying in data warehouses and lakehouses.
Read DefinitionData Observability
Data observability is the practice of continuously monitoring data pipelines and data assets for reliability, freshness, quality, and anomalies to ensure the trustworthiness of analytical outputs.
Read DefinitionData Pipeline Testing
Data pipeline testing strategies are the quality assurance practices for validating data transformations, schema integrity, business logic correctness, and pipeline idempotency before deploying changes to production.
Read DefinitionData Privacy
Data privacy in data engineering covers the practices and architectural patterns used to protect sensitive personally identifiable information (PII) from unauthorized access while maintaining analytical utility.
Read DefinitionData Products
The data product is the foundational concept of Data Mesh architecture, treating data as a product with defined owners, SLAs, discoverability, and quality standards rather than as a pipeline output.
Read DefinitionData Profiling
Data profiling is the automated analysis of datasets to understand their structure, content quality, and statistical characteristics before building pipelines or data models, enabling informed engineering and governance decisions.
Read DefinitionData Quality
Data quality in the modern data lakehouse covers the principles, dimensions, patterns, and tools that ensure analytical data is accurate, complete, consistent, and trustworthy.
Read DefinitionData Reliability
Data reliability is the engineering discipline focused on ensuring data pipelines consistently deliver accurate, fresh, and complete data to consumers, treating data downtime with the same urgency as software application downtime.
Read DefinitionData Retention Policies
Data retention policies in lakehouses are the governance rules that define how long different categories of data are stored, when data is deleted or archived, and how Apache Iceberg's snapshot expiration supports automated retention enforcement.
Read DefinitionData Serialization
Data serialization is the process of converting complex data structures into byte streams for efficient transmission across networks and storage, comparing formats like JSON, Avro, and Protobuf.
Read DefinitionData Sharing
Open data sharing in the lakehouse ecosystem covers the patterns and protocols that enable organizations to share Iceberg tables with external partners without data movement, duplication, or proprietary format lock-in.
Read DefinitionData Silos
Data silos are the isolated pockets of data controlled by individual departments or applications that prevent organizations from achieving a unified view of their business, and how modern lakehouse architectures break them down.
Read DefinitionData Skewness
Data skewness in distributed data engineering is the performance-killing imbalance where some partitions or tasks process dramatically more data than others, and how to detect and address it.
Read DefinitionData Skipping
Data skipping in Apache Iceberg and modern query engines is the collection of techniques including partition pruning, file-level statistics, row group statistics, and Bloom filters that minimize the data scanned to answer analytical queries.
Read DefinitionData Sovereignty
Data sovereignty in global data engineering is the regulatory and governance requirements that mandate certain data must remain within specific geographic boundaries and be governed by the laws of the jurisdiction where it was collected.
Read DefinitionData Strategy
A data strategy is the organizational roadmap that aligns technology investments, data governance, and analytics capabilities with core business objectives to drive competitive advantage.
Read DefinitionData Topology
Data topology is the structural mapping of how data physically and logically flows through an organization's systems, networks, and geographic regions to optimize performance and compliance.
Read DefinitionData Trust
Data trust is the qualitative measure of business confidence in an organization's data assets, built through reliable pipelines, transparent lineage, rigorous data quality metrics, and clear data ownership.
Read DefinitionData Vault Modeling
Data Vault modeling is the enterprise data warehouse methodology developed by Dan Linstedt that uses Hubs, Links, and Satellites to build scalable, auditable, and historically accurate analytical architectures.
Read DefinitionData Virtualization
Data virtualization is the architecture that enables querying and combining data across multiple disparate systems (databases, object storage, APIs) without copying or moving the data to a centralized repository.
Read DefinitionData Warehouse
The data warehouse is the foundational architecture for business intelligence, its history, its strict schema-on-write enforcement, and its evolution into the modern data lakehouse.
Read DefinitionDataOps
DataOps is the agile methodology for data engineering that applies DevOps principles of automation, CI/CD, version control, and monitoring to data pipelines, enabling faster, more reliable data delivery with improved quality and observability.
Read Definitiondbt (Data Build Tool)
dbt is the SQL-first transformation framework that brings software engineering best practices (version control, testing, documentation, modularity) to the data transformation layer.
Read DefinitionDelta Lake
Delta Lake is the open-source storage layer from Databricks that brings ACID transactions, scalable metadata handling, and data versioning to Apache Spark and the data lakehouse.
Read DefinitionDescriptive Analytics
Descriptive analytics is the foundational tier of data analysis that focuses on summarizing historical data to answer the question 'What happened?' using dashboards, standard reporting, and core KPIs.
Read DefinitionDiagnostic Analytics
Diagnostic analytics is the second tier of data analysis that goes beyond summarizing what happened to investigate the root causes and correlations to answer the question 'Why did it happen?'
Read DefinitionDimension Tables
Dimension tables in dimensional modeling are the contextual tables in a star schema that store the 'who, what, where, and when' attributes used to filter and group analytical metrics.
Read DefinitionDimensional Modeling (Star Schema & Snowflake Schema)
Dimensional modeling is the technique developed by Ralph Kimball for structuring analytical databases into Fact tables and Dimension tables for fast, intuitive business intelligence queries.
Read DefinitionDirected Acyclic Graph (DAG)
A directed acyclic graph (DAG) is the mathematical structure used by orchestration tools like Apache Airflow and dbt to define, execute, and monitor complex data pipelines.
Read DefinitionDremio
Dremio is the Intelligent Lakehouse Platform that provides a unified semantic layer, high-performance SQL query engine, and governed access layer for Apache Iceberg-based data lakehouses.
Read DefinitionDuckDB
DuckDB is the embeddable in-process analytical database engine that brings high-performance columnar SQL analytics to local workloads, notebooks, and serverless environments.
Read DefinitionE
Embedded Analytics
Embedded analytics is the integration of analytical capabilities, dashboards, and data visualizations directly into user-facing operational applications, bridging the gap between data exploration and operational workflows.
Read DefinitionETL Offloading
ETL offloading is the architectural strategy of moving heavy, resource-intensive data transformations out of expensive proprietary data warehouses and into the scalable, cost-effective data lakehouse.
Read DefinitionEvent Sourcing
Event sourcing is the architectural pattern where the state of an application is not stored as a single snapshot, but rather as an immutable sequence of historical events that can be replayed to derive the current state.
Read DefinitionExactly-Once Processing
Exactly-once processing is the hardest guarantee in streaming data architecture, ensuring that every event is processed and delivered to the final destination exactly one time, without duplicates or data loss.
Read DefinitionExtract, Load, Transform (ELT)
ELT is the modern inversion of traditional ETL that uses the computational power of cloud data warehouses and lakehouses to perform transformations after loading raw data.
Read DefinitionExtract, Transform, Load (ETL)
Extract, Transform, Load (ETL) is the foundational data integration pattern that has shaped enterprise data pipelines for decades and continues to evolve in the modern lakehouse era.
Read DefinitionF
Fact Tables
Fact tables in dimensional modeling are the central tables in a star schema that store quantitative measurements and metrics for business processes, forming the foundation of analytical reporting.
Read DefinitionFeature Engineering
Feature engineering is the critical data science and engineering process of transforming raw data into meaningful variables (features) that machine learning algorithms can actually understand and learn from.
Read DefinitionFeature Store
A feature store is the centralized ML infrastructure component that computes, stores, and serves machine learning features consistently across model training and real-time inference to eliminate training-serving skew.
Read DefinitionFinOps
FinOps (Cloud Financial Management) is the evolving cultural practice and engineering discipline that brings financial accountability to the highly variable, consumption-based spend of cloud computing and data architecture.
Read DefinitionG
Generative AI
Generative AI is the class of artificial intelligence models designed not to analyze or classify existing data, but to create entirely new, original content-including text, images, and code.
Read DefinitionGraph Data
A guide to graph data structures in analytics, used to model complex relationships and interconnected networks such as social graphs, fraud rings, and supply chains, where the connections are as important as the entities themselves.
Read DefinitionH
Hardware Acceleration
Hardware acceleration in data engineering is the use of specialized silicon like GPUs, FPGAs, and ASICs to execute massive data processing workloads exponentially faster than traditional CPUs.
Read DefinitionHeadless BI
Headless BI is the architectural pattern that decouples the semantic metric definition layer from the presentation layer, allowing consistent business logic to be consumed via API by any downstream application or tool.
Read DefinitionHidden Partitioning
Apache Iceberg's hidden partitioning is the feature that decouples physical data organization from analytical query semantics, eliminating partition-aware query requirements.
Read DefinitionHive Metastore (HMS)
The Hive Metastore is the foundational metadata catalog of the Hadoop ecosystem that tracks table schemas, partitions, and storage locations, and its evolving role in modern Iceberg lakehouses.
Read DefinitionHyperparameter Tuning
Hyperparameter tuning is the experimental process of adjusting the external configuration settings of a machine learning model to optimize its learning efficiency and final predictive accuracy.
Read DefinitionI
Iceberg Changelogs
The Iceberg changelog is the feature that exposes the precise row-level inserts, updates, and deletes between two snapshots, enabling incremental processing pipelines and downstream system synchronization.
Read DefinitionIceberg REST Catalog
The Apache Iceberg REST Catalog specification is the open standard HTTP API that enables any compute engine to interact with any catalog implementation through a common, vendor-neutral interface.
Read DefinitionIceberg Table Branching
Apache Iceberg table branching is the Git-like feature that creates isolated development branches within a single Iceberg table, enabling safe data experimentation, multi-team collaboration, and the Write-Audit-Publish quality workflow.
Read DefinitionIceberg Table Tags
Apache Iceberg table tags are the immutable named references to specific snapshots that enable point-in-time data access, release marking, audit checkpoints, and regulatory compliance snapshots in the lakehouse.
Read DefinitionIdempotency
Idempotency in data engineering is the critical property of a data pipeline ensuring that executing the same code multiple times yields the exact same result, preventing duplicate data during retries.
Read DefinitionIn-Memory Processing
In-memory processing is the performance architecture that loads massive datasets entirely into RAM to execute analytical queries at lightning speed, eliminating the bottleneck of reading from physical disks.
Read DefinitionIncremental Processing
Incremental processing patterns in data engineering are the techniques for processing only new or changed data rather than reprocessing entire datasets on every pipeline run.
Read DefinitionJ
K
L
Lakehouse Concurrency
Lakehouse concurrency covers the mechanisms that allow thousands of users and pipelines to read and write data simultaneously to object storage without locking, corruption, or reading partial data.
Read DefinitionLambda Architecture
Lambda Architecture is the dual-stream data processing pattern that separates batch and real-time processing into distinct layers to deliver both comprehensive historical accuracy and low-latency query results.
Read DefinitionLarge Language Models
Large Language Models (LLMs) are the massive neural networks built on the Transformer architecture that power modern AI by understanding, generating, and translating human language at very large scale.
Read DefinitionLate-Arriving Data
A guide to handling late-arriving data in streaming and batch pipelines, understanding how network lag and offline devices complicate time-based aggregations and the architectural patterns used to gracefully reconcile the past.
Read DefinitionLiquid Clustering
Liquid Clustering in Delta Lake and Databricks is the adaptive file organization technique that replaces static partitioning with flexible, incremental clustering that automatically optimizes file layout for the most common query patterns.
Read DefinitionM
Master Data Management (MDM)
Master Data Management is the discipline of creating a single, reliable source of truth for an organization's critical data entities (customers, products, employees) across fragmented source systems.
Read DefinitionMaterialized Views
A guide to materialized views in data engineering, pre-computed query results stored as physical tables that dramatically accelerate repeated analytical queries by eliminating redundant aggregation and join work.
Read DefinitionMedallion Architecture
The Medallion Architecture is a layered data design pattern used to logically organize data in a lakehouse, progressing from raw ingestion to business-ready aggregates.
Read DefinitionMerge-on-Read (MoR)
A guide to the Merge-on-Read storage strategy in Apache Iceberg and Apache Hudi, where write operations append delta records for low write latency, with merging deferred to read time or compaction.
Read DefinitionMetric Store
The metric store (or headless BI) is the architectural layer that centrally defines and computes business metrics, ensuring consistency across all downstream dashboards, AI agents, and applications.
Read DefinitionMicro-batching
Micro-batching is the hybrid data processing architectural pattern that achieves near-real-time streaming performance by rapidly executing tiny, high-frequency batch jobs, forming the foundation of systems like Spark Streaming.
Read DefinitionMicroservices
Microservices are an architectural design pattern that breaks massive, monolithic applications down into small, independent, loosely-coupled services that communicate over standard network protocols.
Read DefinitionModel Drift
Model drift is the phenomenon where a perfectly trained machine learning model slowly loses predictive accuracy in production because the real-world environment and underlying data have changed over time.
Read DefinitionMulti-Cloud Architecture
Multi-cloud data architecture is the strategy of distributing data storage and compute across multiple cloud providers (AWS, Azure, GCP) to avoid vendor lock-in, use best-of-breed services, and increase resilience.
Read DefinitionMulti-Table Transactions
Multi-table transactions in Apache Iceberg atomically commit changes across multiple Iceberg tables in a single transaction, ensuring cross-table consistency without distributed locking overhead.
Read DefinitionMulti-Tenant Architecture
Multi-tenant architecture is the software design pattern where a single instance of an application or database serves multiple distinct customers (tenants), heavily used in SaaS products and centralized data platforms.
Read DefinitionN
O
Object Storage
Object storage is the massively scalable, low-cost storage architecture that underlies modern data lakehouses, and how it differs fundamentally from block and file storage systems.
Read DefinitionOLAP Cubes
OLAP cubes are the pre-aggregated multidimensional data structures that enabled fast analytical queries in 1990s business intelligence systems, and how modern lakehouse materialized views and Data Reflections achieve equivalent performance without the rigidity of cube architectures.
Read DefinitionOpen Table Formats: Iceberg, Delta Lake, and Hudi
A comparison of the three leading open table formats for lakehouses: Apache Iceberg, Delta Lake, and Apache Hudi, covering their architectural differences, strengths, ecosystem compatibility, and when to choose each.
Read DefinitionOperational Analytics
Operational analytics is the practice of analyzing data in real-time or near-real-time to drive immediate, automated actions in front-line business systems rather than waiting for historical reporting.
Read DefinitionOptimistic Concurrency Control
Optimistic Concurrency Control (OCC) in Apache Iceberg is the conflict detection strategy that enables high-throughput parallel writes to the same table without distributed locking, detecting and resolving conflicts at commit time.
Read DefinitionOrchestration
Data pipeline orchestration is the practice of scheduling, sequencing, and monitoring complex multi-step data workflows using tools like Apache Airflow, Prefect, and Dagster to ensure reliable, observable pipeline execution.
Read DefinitionP
Partition Evolution
Apache Iceberg partition evolution is the capability that allows table partitioning to be changed without rewriting data, enabling partition strategies to adapt to changing query patterns and data volumes without downtime or costly migrations.
Read DefinitionPolars
Polars is the Rust-native DataFrame library that delivers very fast in-process analytical query performance in Python and Rust, becoming a high-performance alternative to pandas for data engineering workflows.
Read DefinitionPredicate Pushdown
Predicate pushdown is the query optimization technique that evaluates filter conditions as close to the data source as possible to minimize the volume of data read and transferred through the query pipeline.
Read DefinitionPredictive Analytics
Predictive analytics is the advanced tier of data analysis that uses historical data, statistical algorithms, and machine learning techniques to identify the likelihood of future outcomes to answer 'What will happen?'
Read DefinitionPrescriptive Analytics
Prescriptive analytics is the pinnacle of data analysis maturity that uses optimization algorithms and simulation to not only predict the future but to recommend specific actions to answer 'What should we do?'
Read DefinitionProject Nessie
Project Nessie is the open-source transactional catalog for data lakes that brings Git-like branching, tagging, and merging semantics to Apache Iceberg table management.
Read DefinitionPrompt Engineering
Prompt engineering is the practice of designing, refining, and structuring the text inputs given to Large Language Models to extract the most accurate, useful, and formatted outputs.
Read DefinitionProperty Graphs
The property graph is the data model used by graph databases to represent complex, highly connected networks of nodes and relationships, where both can contain rich metadata properties.
Read DefinitionPull-Based Ingestion
Pull-based ingestion is the traditional batch data integration pattern where the central data platform proactively extracts data from source databases at scheduled intervals, favored for its simplicity and reliability.
Read DefinitionPush-Based Ingestion
Push-based ingestion is the data integration pattern where source systems actively send data to a central platform via API or streaming, enabling true real-time event-driven architectures.
Read DefinitionPyIceberg
PyIceberg is the official Python library for Apache Iceberg that enables Python developers and data scientists to interact with Iceberg tables directly without requiring a JVM-based engine like Spark.
Read DefinitionQ
Query Caching
Query caching is the performance optimization technique that stores the results of complex database operations in fast memory to drastically reduce response times for subsequent, identical queries.
Read DefinitionQuery Federation
Query federation in data engineering is the architecture pattern that enables a single SQL query to join and aggregate data from multiple heterogeneous data sources without moving the data into a central system first.
Read DefinitionQuery Optimization
Query optimization in data lakehouses covers the techniques that reduce query execution time and resource usage through predicate pushdown, partition pruning, column pruning, join ordering, and pre-computed materialization.
Read DefinitionQuery Planner
The query planner is the intelligent software component within a database engine that translates a user's SQL string into the most efficient physical execution strategy.
Read DefinitionR
Real-Time Analytics
Real-time analytics is the capability to ingest, process, and query streaming data instantly, allowing businesses to react to events as they happen rather than waiting for overnight batch processing.
Read DefinitionRelational Databases
Relational databases (RDBMS) are the foundational technology of the data industry that stores information in highly structured tables linked by primary and foreign keys, enforcing strict data integrity.
Read DefinitionResult Set Caching
Result set caching is the specific optimization layer that intercepts exact-match SQL queries and returns pre-computed final outputs instantly, bypassing all underlying compute and network traversal.
Read DefinitionRetrieval-Augmented Generation (RAG)
RAG is the foundational AI architecture that grounds Large Language Models in verifiable, private corporate data, eliminating hallucinations and ensuring accurate, context-aware responses.
Read DefinitionRetrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation (RAG) is the AI architecture that grounds Large Language Models in private enterprise data, preventing hallucinations and enabling domain-specific conversational analytics without model fine-tuning.
Read DefinitionReverse ETL
Reverse ETL is the data pipeline pattern that syncs curated analytical data from the data warehouse or lakehouse back into operational business tools like CRMs, marketing platforms, and customer success systems.
Read DefinitionRole-Based Access Control (RBAC)
Role-Based Access Control (RBAC) in data lakehouses is the authorization model that assigns permissions to roles rather than individual users for scalable, auditable data access governance.
Read DefinitionRollup Tables
Rollup tables are the pre-aggregated summary tables used to accelerate analytical queries by storing high-level metrics instead of forcing the database to scan millions of raw transaction rows.
Read DefinitionRow-Level Security
Row-level security (RLS) in data lakehouses is the access control mechanism that automatically filters query results to return only the rows a querying user is authorized to see based on their identity attributes.
Read DefinitionRust in Data Engineering
A guide to Rust's growing role in the data engineering ecosystem, where its memory safety, zero-cost abstractions, and native performance are powering a new generation of high-performance data tools including DataFusion, Delta-rs, and iceberg-rust.
Read DefinitionS
Schema Evolution
Schema evolution is the critical capability of modern data platforms to safely alter the structure of a database table (adding, dropping, or renaming columns) without breaking existing data or pipelines.
Read DefinitionSchema Registry
The schema registry is the centralized governance component in streaming architectures that enforces data structure contracts and manages schema evolution across decoupled producers and consumers.
Read DefinitionSchema-on-Read
Schema-on-read is the foundational big data paradigm where raw data is stored exactly as it arrives without enforcement, and the structure is only applied later when a query is actually executed.
Read DefinitionSemantic Layer
The semantic layer in data engineering is the governed translation layer between raw data and business consumers that defines metrics, business logic, and access control centrally, ensuring consistent data definitions across all BI tools and AI agents.
Read DefinitionSemantic Search
Semantic search is the AI-driven methodology that retrieves information based on the contextual meaning and intent of a query, rather than relying on exact keyword matching.
Read DefinitionSemi-Structured Data
Semi-structured data covers the flexible formats like JSON and XML that don't adhere to a rigid relational schema, and how modern lakehouses enable scalable analytical querying over nested hierarchies.
Read DefinitionServerless Architecture
Serverless architecture is the cloud computing model where the cloud provider dynamically manages the allocation of machine resources, allowing data engineers to focus entirely on code and data rather than infrastructure.
Read DefinitionSlowly Changing Dimensions (SCD)
Slowly Changing Dimensions (SCD) are the data warehouse design patterns for tracking how dimension attribute values change over time, from simple overwrites to full historical preservation for accurate point-in-time analysis.
Read DefinitionSnapshot Expiration
Snapshot expiration in Apache Iceberg is the table maintenance operation that removes historical snapshots and their associated data files to reclaim storage space while preserving configurable data retention windows.
Read DefinitionSnowflake Schema
The snowflake schema is a dimensional modeling technique where dimension tables are normalized into multiple related tables, trading query simplicity for storage efficiency and data integrity.
Read DefinitionSpark Structured Streaming
Apache Spark Structured Streaming is the micro-batch and continuous streaming engine built on Spark SQL that enables fault-tolerant, stateful stream processing with exactly-once semantics and native Apache Iceberg sink support.
Read DefinitionStar Schema
The star schema dimensional model is the foundational data warehouse design pattern that organizes analytical data into fact tables surrounded by denormalized dimension tables for optimized query performance.
Read DefinitionStorage-Compute Separation
The separation of storage and compute is the foundational architectural principle of modern cloud data platforms that allows scaling processing power independently of data volume to minimize costs.
Read DefinitionStreaming Lakehouse
A guide to the streaming lakehouse architecture that unifies real-time streaming ingestion with ACID table format semantics, enabling sub-minute data freshness in Iceberg-based analytical platforms.
Read DefinitionSurrogate Keys
Surrogate keys in dimensional data modeling are the system-generated artificial identifiers used in data warehouse fact and dimension tables to replace natural business keys and enable efficient joins and slowly changing dimension management.
Read DefinitionT
Table Format Metadata
Table format metadata is the structured layer of snapshots, manifests, and statistics that open table formats like Apache Iceberg use to enable ACID transactions, time travel, and efficient query planning.
Read DefinitionTime Travel Queries
Time travel in Apache Iceberg is the capability to query historical snapshots of a table at any point in its version history for auditing, debugging, and reproducible analytics.
Read DefinitionTime-Series Data
Time-series data is the specialized data structure consisting of sequential measurements over time, requiring specific storage, indexing, and querying techniques for IoT, financial, and observability use cases.
Read DefinitionTime-to-Live (TTL)
Time-to-Live (TTL) is the automated data lifecycle mechanism that permanently deletes or archives records after a specified duration to enforce privacy compliance and manage storage costs.
Read DefinitionTrino
Trino (formerly PrestoSQL) is the open-source distributed SQL query engine designed for fast interactive analytics across multiple data sources including Iceberg lakehouses, relational databases, and object storage.
Read DefinitionU
Unity Catalog
Databricks Unity Catalog is the unified governance layer for the Databricks Lakehouse Platform that provides centralized access control, auditing, and data discovery across all Databricks workspaces.
Read DefinitionUnstructured Data
Unstructured data is the massive category of data (text, images, audio, video) that lacks a pre-defined schema, and how modern lakehouses and AI transform it into analyzable business value.
Read DefinitionV
Vector Databases
Vector databases are the specialized storage systems designed to store, index, and query high-dimensional vector embeddings, forming the retrieval backbone for generative AI and semantic search applications.
Read DefinitionVector Embeddings
Vector embeddings are the mathematical representations of unstructured data (text, images, audio) that allow machine learning models to understand and compute the conceptual similarities between complex objects.
Read DefinitionVectorized Execution
Vectorized execution in analytical query engines is the CPU optimization technique that processes batches of column values using SIMD instructions, delivering orders-of-magnitude query performance improvements over row-at-a-time processing.
Read DefinitionW
Window Functions
SQL window functions are an analytical feature that lets users to perform calculations across a defined set of rows related to the current row, enabling complex calculations like running totals and moving averages.
Read DefinitionWrite Amplification
Write amplification is the hidden performance penalty in analytical databases and lakehouses where a small logical update results in massive physical data being rewritten on disk.
Read DefinitionWrite-Audit-Publish (WAP)
The Write-Audit-Publish pattern in Apache Iceberg is the branch-based data quality workflow that writes new data to an isolated branch, validates it, and atomically publishes it to the main branch only when quality checks pass.
Read DefinitionZ
Z-Ordering and Data Skipping
Z-ordering and data skipping are the file-level data organization techniques that cluster related records together in Parquet files to enable dramatic I/O reduction for multi-column filter queries.
Read DefinitionZero-Copy Cloning
Zero-copy cloning is the powerful data lakehouse feature that allows engineers to create instant, functional copies of massive datasets without physically duplicating any of the underlying storage.
Read Definition