Marquez

Marquez is a highly scalable metadata server and visualization platform designed to aggregate, store, and visualize metadata about data production and consumption. Built as the reference implementation for the OpenLineage standard, Marquez provides a robust technical architecture for maintaining a complete history of dataset evolution and job execution. Its core architecture utilizes a relational backend (PostgreSQL) and exposes a comprehensive RESTful API for metadata ingestion and retrieval. By 2026, Marquez has solidified its position as the foundational layer for decentralized data mesh architectures, enabling data engineers to automate impact analysis and root cause identification across polyglot data stacks. It tracks job runs, versioning of both code and data schemas, and the physical location of datasets. Its design philosophy centers on late-binding metadata, allowing it to integrate seamlessly with various orchestrators like Apache Airflow and execution engines like Spark. As an LF AI & Data project, it benefits from a neutral governance model, ensuring its longevity and interoperability in the evolving AI and Data lifecycle management market.