Extract-Transform-Load functionality lets you encapsulate some of the most common database tasks, especially:

1. Extracting data from a database
2. Transforming it, and
3. Loading it into another database.

Some scenarios where ETLs are useful:

• Assembling data warehouses that integrate data from multiple data sources.
• Migration from one database schema to another, especially where the source schema is an external database.
• Coalescing many tables into one.
• Distributing (aka, provisioning) one table into many.
• Cloning the current state of a table.
• Normalizing data from different systems.
• Moving data in scheduled increments.
• When migration processes require logging and auditing.

ETL functionality can be encoded (1) in a custom module or (2) using the folder management UI on the server. The following topics will get you started developing ETL scripts and processes and packaging them as modules:

• Tutorial: Extract-Transform-Load (ETL) - Introduction to ETLs.
• ETL: Create a New ETL - Configure an ETL using the server UI.
• ETL: Schedules - Configure the refresh schedule.
• ETL: Column Mapping - Transform columns in an ETL process.
• ETL: Stored Procedures - Use a stored procedure as a transformation.
• ETL: SQL Scripts - Create table schemas with sql scripts.
• ETL: Logs and Error Handling - Logging and error reporting.
• ETL: Remote Connections - Extract data via a remote connection.
• ETL: Examples - Example ETL modules.
• ETL: Reference - File references, directory structure.
• Premium Resource: ETL Best Practices - (Premium Resource) Recommendations for ETL design and scheduling.

Related Topics

• External Schemas and Data Sources
• SQL Query Browser
• Methods for Loading Data Into LabKey