What Is Data Integration? ETL, ELT and a First Run

Q: Where does integrated data go?

Often into a shared store such as a data hub or data lake ; for lab data specifically, see scientific data management systems .

Data integration is the process of combining and harmonizing data from multiple sources into a unified, coherent format (IBM’s definition). In a research setting it is what you do when instrument exports, clinical or survey records and public datasets have to end up in one analysis-ready place without being copied by hand. This guide explains the main integration methods, then walks through a first run with openly documented software: Apache NiFi, with Airbyte and Apache Hop as alternatives.

Integration moves and reshapes data. Recording what it did afterwards is a separate job, covered in what data lineage is and how to track it. For the places integrated data often ends up, see the dictionary entries for a data hub and a data lake.

What data integration is used for in research

Data integration brings records from separate systems together so they can be analysed as one dataset. IBM describes the method choices as a trade-off between where the data is transformed and whether it is physically moved at all. The table below lists the methods IBM names, with a short description of each, and what that means for where your data sits.

Method	How IBM describes it	Where the data ends up
ETL (extract, transform, load)	Data is transformed before it is loaded into the data storage system, in a separate staging area.	Copied into the target store after transformation
ELT (extract, load, transform)	Data is first loaded into a database or warehouse, then transformed.	Copied into the target store as-is, transformed there
Change data capture	Applies updates made to the data in source systems to data warehouses and other repositories.	Target is updated as sources change
Real-time and streaming integration	Captures and processes data as it becomes available in source systems.	Target receives data continuously
Data replication	Continuously replicates data from source systems to target systems, keeping them in sync (IBM lists replication among its data integration tool categories).	Copy of the source data
Data virtualization	A virtual layer that provides a unified view without physically moving the data.	Stays in the source systems
Federated integration	Data stays in the source systems and is queried in real time; IBM notes this can cause performance issues.	Stays in the source systems
Application integration	Integrates data between different software applications to ensure seamless data flow.	Not stated by IBM

Source: IBM, What is data integration?, checked 2026-10-29. This is a vendor’s description, not a standard, so treat the categories as a vocabulary rather than a specification.

What is the difference between ETL and ELT?

The difference is the order of the steps. In ETL the data is transformed before it is loaded into the storage system; in ELT it is loaded first and transformed inside the database or warehouse. IBM presents both as ways to combine data from several sources.

Install or access (official source only)

Three openly documented tools are covered here. Each is obtained from its own project site.

Apache NiFi: download from the NiFi Downloads page linked from nifi.apache.org/docs.html and extract the zip. The project describes NiFi as a dataflow system based on the concepts of flow-based programming, with a web UI for design, control and monitoring.
Airbyte: Airbyte Core is Airbyte’s open source product. It is deployed with abctl, Airbyte’s command-line tool, which needs Docker Desktop (or Docker Engine on headless Linux). Airbyte recommends 4 or more CPUs and 8 GB of memory, with a low-resource mode at 2 CPUs and 8 GB in which the Connector Builder is unavailable.
Apache Hop: an open source platform for data integration and orchestration at hop.apache.org. As shown on the site on 2026-10-29, version 2.19.0 requires Java 21.

Step-by-step: start Apache NiFi and sign in

Download NiFi from the official Downloads page and extract the zip.
Start it from the extracted folder. On Windows run nifi.cmd start from the bin directory. On Linux or macOS run bin/nifi.sh run to stay in the foreground, or bin/nifi.sh start to run in the background. Check a background instance with bin/nifi.sh status and stop it with bin/nifi.sh stop. With Homebrew the commands are nifi start and nifi stop.
Open https://localhost:8443/nifi in a browser.
Sign in with the generated credentials. They are written to logs/nifi-app.log, on lines labelled “Generated Username” and “Generated Password”.
To set your own login, run ./bin/nifi.sh set-single-user-credentials <username> <password>.

Commands as given in the Apache NiFi Getting Started guide, checked 2026-10-29: nifi.apache.org/nifi-docs/getting-started.html.

Alternative first run: Airbyte with abctl

Install Docker Desktop (or Docker Engine on headless Linux) and make sure it is running.
On Mac or Linux install abctl with curl -LsfS https://get.airbyte.com | bash -. This command does not work on Windows, where the Airbyte documentation points to GitHub releases. With Homebrew, run brew tap airbytehq/tap and then brew install abctl.
Confirm the install with abctl version.
Run abctl local install. The documentation says this can take up to 30 minutes. Airbyte then opens at http://localhost:8000.
Retrieve the login with abctl local credentials.
To remove it, run abctl local uninstall, which keeps your data; add --persisted to delete the data too.

Source: Airbyte OSS quickstart, checked 2026-10-29.

Output and how to read it

NiFi moves data as FlowFiles. A FlowFile has two parts: attributes, which are key-value pairs describing the data, and content, which is the data itself. The building block that acts on FlowFiles is the processor, which the NiFi documentation calls the most important building block: processors create, send, receive, transform, route, split and merge FlowFiles. NiFi also records fine-grained data provenance, so what happened to a FlowFile can be inspected afterwards.

When a flow runs, the attributes on a FlowFile describe what it is and the content is the data that was moved or changed. For the wider record of who did what and why, see data provenance.

Troubleshooting

Cannot sign in to NiFi. The username and password are generated at first start and written to logs/nifi-app.log. Search that file for “Generated Username” and “Generated Password”, or set your own with set-single-user-credentials.
Which Java does NiFi need? See the minimum system requirements in the NiFi Administration Guide before installing.
Airbyte install seems stuck. The documentation says abctl local install can take up to 30 minutes. Docker must be running before you start.
Airbyte is too heavy for the machine. Low-resource mode (abctl local install --low-resource-mode) runs on 2 CPUs and 8 GB of memory, but the Connector Builder is unavailable in that mode.
The Airbyte install one-liner fails on Windows. It is documented for Mac and Linux only; Windows uses GitHub releases.

Free alternatives

Tool	What its own documentation says	License stated by the project
Apache NiFi	A dataflow system based on flow-based programming; web UI for design, control and monitoring; fine-grained data provenance.	Apache License 2.0
Airbyte (Airbyte Core)	Airbyte’s open source product, deployed with the abctl command-line tool.	Open source (Airbyte Core)
Apache Hop	An open source platform for data integration and orchestration. Pipelines (transforms joined by hops) and workflows are built visually and can run locally, on Hop Server, or on Spark, Flink or Dataflow. 250+ pipeline transforms and 80+ workflow actions listed.	Apache License 2.0

Sources: Apache NiFi, Airbyte and Apache Hop, all checked 2026-10-29.

FAQ

What is data integration?

It is the process of combining and harmonizing data from multiple sources into a unified, coherent format, as IBM defines it.

Does data integration always copy the data?

No. ETL, ELT, replication and change data capture copy data into a target store. Data virtualization provides a unified view without physically moving the data, and federated integration queries data that stays in the source systems.