Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.

Directory Structure of an ETL Module

The directory structure for including an ETL in a module is shown below. Depending on your ETL, you may need all or only part of the elements shown here, and depending on your module, the directory structure may contain additional directories and resources unrelated to the ETL.

Items shown in lowercase below are literal values (and file extensions) that should be preserved in the directory structure. Items shown in uppercase should be replaced with values that reflect the nature of your project.

MODULE_NAME
├───etls
│ ETL1.xml
│ ETL2.xml

├───queries
│ └───SCHEMA_NAME
│ QUERY_NAME.sql

└───schemas
│ SCHEMA_NAME.xml

└───dbscripts
├───postgresql
│ SCHEMA_NAME-X.XX-Y.YY.sql

└───sqlserver
SCHEMA_NAME-X.XX-Y.YY.sql

Files and Directories

  • ETL1.xml - The main config file for an ETL process. Defines the sources, targets, transformations, and schedules for the transfers. Any number of ETL processes and tasks can be added. For examples see ETL: Examples.
  • QUERY_NAME.sql - SQL queries for data sources and targets.
  • schemas - Optional database schemas.
    • May contain dbscripts - Optional sql scripts that will run automatically upon deployment of the module, in order to generate target databases for your ETL processes. Script names are formed from three components: (1) schema name, (2) previous module version (X.XX), and (3) current module version (Y.YY). Learn more in this topic: Modules: SQL Scripts.

ETL Configuration Reference

For details see ETL XML Reference.

Reference Links

  • etl.xml Reference - xml reference docs for ETL config files.
  • etl.xsd - The XSD schema file on which the XML config files are based.

Discussion

Was this content helpful?

Log in or register an account to provide feedback


previousnext
 
expand all collapse all