Premium Feature — Available in the Professional, Professional Plus, and Enterprise Editions. Learn more or contact LabKey.

ETL User Interface

The web part Data Transforms lists all of the ETL processes that are available in the current folder.

  • Columns:
    • Name - This column displays the name of the process.
    • Source Module - This column tells you module where the configuration file resides.
    • Schedule - This column shows you the reload schedule. In this case the ETL process is configured to run once every hour.
    • Enabled - This checkbox controls whether the automated schedule is enabled: when unchecked, the ETL process must be run manually.
    • Last Status, Successful Run, Checked - These columns record the latest run of the ETL process.
    • Set Range - (Available only in devMode) The Set Range column is displayed only in dev mode and is intended for testing purposes during ETL module development. The Run button is only displayed for ETL processes with a filter strategy of RunFilterStrategy or ModifiedSinceFilterStrategy; the button is not displayed for the filter strategy SelectAllFilterStrategy. Click Run to set a date or row version window range to use for incremental ETL filters, overriding any persisted or initial values.
    • Last Transform Run Log Error - Shows the last error logged, if any exists.
  • Buttons:
    • Run Now - This button immediately activates the ETL process.
    • Reset State - This button returns the ETL process to its original state, deleting its internal history of which records are, and are not, up to date. There are two options:
      • Reset
      • Truncate and Reset
    • View Processed Jobs - This button shows you a log of all previously run ETL jobs, and their status.

Run an ETL Process Manually

The Data Transforms web part lets you:

  • Run jobs manually. (Click Run Now.)
  • Enable/disable the recurring run schedule, if such a schedule has been configured in the ETL module. (Check or uncheck the column Enabled.)
  • Reset state. (Select Reset State > Reset resets an ETL transform to its initial state, as if it has never been run.)
  • See the latest error raised in the Last Transform Run Log Error column.

Cancel and Roll Back Jobs

While a job is running you can cancel and roll back the changes made by the current step by pressing the Cancel button.

The Cancel button is available on the Job Status panel for a particular job, as shown below:

To roll back a run and delete the rows added to the target by the previous run, view the Data Transforms web part, then select Reset State > Truncate and Reset. Note that rolling back an ETL which outputs to a file will have no effect, that is, the file will not be deleted or changed.

See Run History

The Data Transform Jobs web part provides a detailed history of all executed ETL runs, including the job name, the date and time when it was executed, the number of records processed, the amount of time spent to execute, and links to the log files.

To add this web part to your page, enter > Page Admin Mode, then scroll down to the bottom of the page and click the dropdown <Select Web Part>, select Data Transform Jobs, and click Add. When added to the page, the web part appears with a different title: "Processed Data Transforms". Click Exit Admin Mode.

Click Run Details for fine-grained details about each run, including a graphical representation of the run.


Was this content helpful?

Log in or register an account to provide feedback

expand all collapse all