Directory Structure of an ETL Module

The directory structure for including an ETL in a module is shown below. Depending on your ETL, you may need all or only part of the elements shown here, and depending on your module, the directory structure may contain additional directories and resources unrelated to the ETL.

Items shown in lowercase below are literal values (and file extensions) that should be preserved in the directory structure. Items shown in uppercase should be replaced with values that reflect the nature of your project.

MODULE_NAME
├───etls
│       ETL1.xml
│       ETL2.xml
│
├───queries
│   └───SCHEMA_NAME
│           QUERY_NAME.sql
│
└───schemas
    │   SCHEMA_NAME.xml
    │
    └───dbscripts
        ├───postgresql
        │       SCHEMA_NAME-X.XX-Y.YY.sql
        │
        └───sqlserver
                SCHEMA_NAME-X.XX-Y.YY.sql

Files and Directories

• ETL1.xml - The main config file for an ETL process. Defines the sources, targets, transformations, and schedules for the transfers. Any number of ETL processes and tasks can be added. For examples see ETL: Examples.
• QUERY_NAME.sql - SQL queries for data sources and targets.
• schemas - Optional database schemas.
• May contain dbscripts - Optional sql scripts that will run automatically upon deployment of the module, in order to generate target databases for your ETL processes. Script names are formed from three components: (1) schema name, (2) previous module version (X.XX), and (3) current module version (Y.YY). Learn more in this topic: Modules: SQL Scripts.