This tutorial shows you how to create a simple ETL script or process as a starting point for further development.
As you go through the tutorial, imagine you are a researcher who wants to collect a group of participants for a research study. The participants must meet certain criteria to be included in the study, such as having a certain condition or diagnosis. You already have the following in place:
- You have a running installation of LabKey Server.
- You already have access to a large database of demographic information of candidate participants. This database is continually being updated with new data and new candidates for your study.
- You have an empty table called "Patients" on your LabKey Server which is designed to hold the study candidates.
So how do you get the records from the outside database into your LabKey Server, especially those records that meet your study's criteria? In this tutorial, you will set up an ETL process to solve this problem. The ETL script will automatically query the source database for participants that fit your criteria. If it finds any such records, it will automatically copy them into your system. The ETL process will run on a schedule: every hour it will re-query the database looking for new, or updated, records that fit your criteria.