What Is Data Lineage? A Practical Guide with Open-Source Tools

Data lineage is the record of where a dataset came from and which jobs and transformations produced it, so you can trace any output table back through every step to its inputs. In research, it answers a practical question: when a number in a figure changes, which upstream data or processing step caused it? This guide explains what lineage is, then walks through two openly documented tools you can run yourself: OpenLineage with its Marquez reference backend, and dbt Docs.

Lineage is related to, but narrower than, data provenance. Lineage describes the technical flow of data through jobs. Provenance also covers who did what and why. The FAQ below separates the two.

What data lineage is used for in research

• Tracing a result to its inputs. Start from an analysis table and follow it back to the raw files and the jobs that transformed them.
• Impact checks before a change. Look downstream from a dataset to see which jobs and outputs depend on it before you alter or retire it.
• Documenting a pipeline for others. A lineage graph is a readable companion to a FAIR-minded data package, because it shows how derived data relates to its sources.
• Debugging. A failed or stale run is easier to locate when each job run is recorded against its input and output datasets.

The open standard OpenLineage describes itself as “an open framework for data lineage collection and analysis.” It defines a generic model of three entities: a dataset, a job, and a run (a job in execution). Entities are identified by consistent naming strategies, and a facet is user-defined metadata that enriches an entity, for example a schema attached to an output dataset. This is the vocabulary used in the rest of this page.

Install or access (official sources only)

OpenLineage and Marquez

The OpenLineage getting-started walkthrough uses Marquez, a lineage backend, running in Docker. It lists two prerequisites: Docker 17.05 or later, and Docker Compose. OpenLineage’s documentation lists integrations for Apache Airflow, Apache Flink, Apache Hive, Apache Spark, dbt, Feast, Great Expectations, SQL and Trino.

dbt Docs

dbt documents its project docs site in its official documentation (docs.getdbt.com, “Documentation”). The self-hosted docs site is generated and previewed with two commands, shown below. The same page notes that adding a dbt docs generate step to a job does not produce a static site on the dbt platform; on the platform, the Catalog shows resources and their lineage (availability depends on plan).

Step-by-step task: record and view lineage with OpenLineage and Marquez

These steps follow the official OpenLineage getting-started page.

1. Confirm Docker (17.05 or later) and Docker Compose are installed.
2. Clone the Marquez repository and enter it: git clone https://github.com/MarquezProject/marquez.git && cd marquez
3. Start Marquez: ./docker/up.sh. On macOS, port 5000 can be reserved by the system; in that case use ./docker/up.sh --api-port 9000 and substitute port 9000 in the URLs below.
4. Open the web interface at http://localhost:3000. The backend database listens on port 5432.
5. Send a START event to the lineage endpoint: an HTTP POST to http://localhost:5000/api/v1/lineage with eventType set to START, a job named my-job and an input dataset named my-input. A successful request returns 201 CREATED.
6. Send a COMPLETE event to the same endpoint for the same job and run, naming an output dataset my-output and attaching a schema facet. This also returns 201 CREATED.

Step-by-step task: generate a lineage view with dbt Docs

1. From your dbt project, run dbt docs generate. Per dbt’s documentation this compiles the project, writes the index, and exports the documentation site in a single command.
2. Run dbt docs serve to preview the site locally.

dbt’s documentation describes this docs site (dbt Docs v2) as including column-level lineage.

Output and how to read it

In Marquez, use the search box at the top right to find my-job and click it. The lineage graph shows my-input as the input dataset and my-output as the output dataset. The RUN HISTORY tab shows the run as COMPLETED. Clicking my-output shows its columns and types, which come from the schema facet you attached in the COMPLETE event.

Read a graph from datasets to jobs to datasets: each job node is a transformation, and its edges show which datasets it consumed and which it produced. Column-level lineage, where a tool provides it, narrows this from whole tables to individual fields.

Troubleshooting

Symptom	Cause and fix (from the official documentation)
Marquez will not start on macOS	Port 5000 can be reserved on macOS. Start with ./docker/up.sh --api-port 9000 and use port 9000 in the API URLs.
No static docs site after adding dbt docs generate to a job	dbt states this does not produce a static site on the dbt platform. Use dbt docs generate and dbt docs serve on a self-hosted setup, or the platform’s Catalog.

Free alternatives

Tool	What the official documentation says it does
OpenLineage	An open framework and standard for collecting and analysing lineage, built on dataset, job and run entities, with integrations for Airflow, Flink, Hive, Spark, dbt, Feast, Great Expectations, SQL and Trino.
Marquez	The lineage backend and web interface used in the OpenLineage getting-started walkthrough, run with Docker and Docker Compose.
dbt Docs (self-hosted)	A documentation site generated by dbt docs generate and previewed with dbt docs serve, including column-level lineage.

For related reading, see data catalog vs data dictionary and the metadata and metadata schema entries; lineage records are themselves a kind of metadata. For documenting the variables in a dataset, see the data dictionary guide.

FAQ

What is data lineage?

Data lineage is the traceable record of how data moves and changes: which datasets feed which jobs, and which datasets those jobs produce. OpenLineage models it with dataset, job and run entities.

What is the difference between data lineage and data provenance?

Lineage focuses on the technical flow of data through jobs and datasets. Provenance is broader and also covers who was responsible and why steps were taken. See the data provenance guide for the research-reproducibility side.

What is the difference between table-level and column-level lineage?

Table-level lineage shows which datasets feed which. Column-level lineage traces individual fields. dbt Docs v2 includes column-level lineage per dbt’s documentation.

Do I need a commercial product to get lineage?

No. The OpenLineage standard with Marquez, and dbt Docs, are documented above with the commands to run them.

Sources (official docs)

• OpenLineage documentation: https://openlineage.io/docs/ (checked 2026-10-29)
• OpenLineage getting started (Marquez): https://openlineage.io/getting-started/ (checked 2026-10-29)
• dbt documentation, “Documentation”: https://docs.getdbt.com/docs/build/documentation (checked 2026-10-29)