Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.

This topic will cover the main 'attributes' applied at the root level of ETL syntax. Some reference other topics, some described here directly.

Standalone (vs. Component)

ETL processes can be set as either "standalone" or "sub-component":

  • Standalone ETL processes:
    • Appear in the Data Transforms web part
    • Can be run directly via the user or via another ETL
  • Sub-Component ETL processes or tasks:
    • Not shown in the Data Transforms web part
    • Cannot be run directly by the user, but can be run only by another ETL process, as a sub-component of a wider job.
    • Cannot be enabled or run directly via an API call.
To configure as a sub-component, set the ETL "standalone" attribute to false. By default the standalone attribute is true.
<etl xmlns="http://labkey.org/etl/xml" standalone="false">
...
</etl>

Learn more here: ETL: Queuing ETL Processes

Load Referenced Files

To handle files generated during or otherwise involved in ETL processing (for example, if one ETL process outputs a file, and then queues another process that expects to use the file in a pipeline task), all ETL configurations in the chain must have the attribute loadReferencedFiles="true” in order for the runs to link up properly.

<etl xmlns="http://labkey.org/etl/xml" loadReferencedFiles="true">
...
</etl>

Site Scope

By default an ETL is container scoped. To let an ETL run at a site scope level, set the attribute siteScope to true.

Allow Multiple Queuing

By default, if a job for the ETL is already pending (waiting), we block adding another instance of the same ETL to the job queue. Set the allowMultipleQueuing flag to "true" override and allow multiple instances in the queue.

Transaction Wrapped ETLs

The attributes transactSourceSchema and transactDestinationSchema are used for wrapping a multi-step ETL into a single transaction.

Learn more in this topic: ETL: Transactions

Related Topics

Discussion

Was this content helpful?

Log in or register an account to provide feedback


previousnext
 
expand all collapse all