Data Warehouse
This tutorial shows you how to create and use a simple ETL. You can use this example as a starting point for further development.
As you go through the tutorial, imagine you are a researcher who wants to identify a group of participants for a research study. The participants must meet certain criteria to be included in the study, such as having a certain condition or diagnosis. You already have the following in place:
- You have a running installation of LabKey Server which includes the dataintegration module.
- You already have access to a large database of demographic information of candidate participants. This database is continually being updated with new data and new candidates for your study.
- You have an empty table called "Patients" on your LabKey Server which is designed to hold the study candidates.
How do you get the records from the outside database into your LabKey Server, especially those records that meet your study's criteria? In this tutorial, you will set up an ETL process to solve this problem. The ETL script will automatically query the source database for participants that fit your criteria. If it finds any such records, it will automatically copy them into your system. The ETL process will run on a schedule: every hour it will re-query the database looking for new, or updated, records that fit your criteria.
Tutorial Steps
- ETL Tutorial: Set Up - Set up a sample ETL workspace.
- ETL Tutorial: Run an ETL Process - Run an ETL process.
- ETL Tutorial: Create a New ETL Process - Add a new ETL process for a new query.
