Transform scripts are attached to assay designs, run before the assay data is imported, and can reshape the data file to match the expected import format. Several scripts can run sequentially to perform different transformations. The extension of the script file identifies the scripting engine that will be used to run the validation script. For example, a script named test.pl will be run with the Perl scripting engine.
Transform scripts (which are always attached to assay designs) are different from
trigger scripts, which are attached to a dataset (database table or query).
Scenarios
A wide range of scenarios can be addressed using transform scripts. For example:
- Input files might not be "parsable" as is. Instrument-generated files often contain header lines before the main data table, denoted by a leading #, !, or other symbol. These lines may contain useful metadata about the protocol, reagents, or samples tested which should either be incorporated into the data import or skipped over to find the main data to import.
- File or data formats in the file might not be optimized for efficient storage and retrieval. Display formatting, special characters, etc. might be unnecessary for import. Transformation scripts can clean, validate, and reformat imported data.
- During import, display values from a lookup column may need to be mapped to foreign key values for storage.
- You may need to fill in additional quality control values with imported assay data, or calculate contents of a new column from columns in the imported data.
- Inspect and change the data or populate empty columns in the data. Modify run- and batch-level properties. If validation only needs to be done for particular single field values, the simpler mechanism is to use a validator within the field properties for the column.
Scripting Prerequisites
Any scripting language that can be invoked via the command line and has the ability to read/write files is supported for transformation scripts, including:
Before you can run scripts, you must configure the necessary
scripting engine on your server. If you are missing the necessary engine, or the desired engine does not have the script file extension you are using, you'll get an error message similar to:
A script engine implementation was not found for the specified QC script (my_transformation_script.py). Check configurations in the Admin Console.
How Transformation Scripts Work
Transformation and validation scripts are invoked in the following
Script Execution Sequence:
1. A user imports assay result data and supplies run and batch properties.
2. The server uses that input to create:
- (1) A "runProperties.tsv" file.
- This file contains assay-specific properties, run and batch level fields, and generated paths to various resources, all of which can be used by your transformation script.
- Learn more in the topic: Run Properties Reference.
- (2) If the first 'non-comment' line of the imported result data contains fields that match the assay design, the server will infer/generate a "runDataFile" which is a TSV based on the result data provided by the user (which may be in another file format like Excel or CSV).
- Note that if the server cannot infer this file, either because the "relevant" part you want to read in is not at the top, column names are not recognizable to the assay mechanism, or generally isn't tabular/'rectangular', you cannot use the generated "runDataFile", but will be able to use the originally uploaded file in its original format (as "runDataUploadedFile").
- Regardless of whether the server can generate the "runDataFile", the path to the intended location is included in the runProperties.tsv file.
3. The server invokes the transform script, which can use the ${runInfo} substitution token to locate the "runProperties.tsv" file created in step 2. The script can then extract information and paths to input and output files from that file.
4. After script completion, the server checks whether any errors have been written by the transform script and whether any data has been transformed.
5. If transformed data is available, the server uses it for subsequent steps; otherwise, the original data is used.
6. If multiple transform scripts are specified, the server invokes the other scripts in the order in which they are defined.
7. Field-level validator and quality-control checks, including range and regular expression validation that are included in the assay definition are performed on the 'post-transformation' data.
8. If no errors have occurred, the run is loaded into the database.
Use Transformation Scripts
Each assay design can be associated with one or more validation or transform scripts which are run in the order listed in the assay design.
This section describes the process of using a transform script that has already been developed for your assay type. An example workflow for how to
create an assay transform script in perl can be found in
Example Workflow: Develop a Transformation Script.
Identify the Path to the Script File
It is convenient to upload the script file to the File Repository in the same folder as the assay design. The absolute path to the script file can be determined by concatenating the file root for the folder (available at
(Admin) > Folder > Management > Files tab) plus the path to the script file in the Files web part (for example, "@files\scripts\LoadData.R").
You can also find the absolute path to a script in the File Repository as follows:
- Go the > Go To Module > FileContent.
- Select your transform script.
- The absolute path is shown at the bottom of the panel.
In the file path, LabKey Server accepts either backslashes (the default Windows format) or forward slashes.
Example path to script:
/labkey/labkey/files/MyProject/MyAssayFolder/@files/MyTransformScript.R
When working on your own developer workstation, you can put the script file wherever you like, but putting it within the File Repository will make it easier to deploy to a production server. It also makes iterative development against a remote server easier, since you can use a Web-DAV enabled file editor to directly edit the same script file that the server is calling.
When you decide
where to locate your transform script file, consider that it is convenient to keep script files in the same location as the assay. You will enter the full path when you configure your assay design to run this script. Use the built-in substitution token "${srcDirectory}" which the server automatically fills in to be the directory where the called script file (the one identified in the
Transform Scripts field) is located.
Access and Use the Run Properties File
The primary mechanism for communication between the LabKey Assay framework and the Transform script is the Run Properties file. Again a substitution token ${runInfo} tells the script code where to find this file. The script file should contain a line like
run.props = labkey.transform.readRunPropertiesFile("${runInfo}");The run properties file contains three categories of properties:
1. Batch and run properties as defined by the user when creating an assay instance. These properties are of the format:
<property name> <property value> <java data type>
for example,
gDarkStdDev 1.98223 java.lang.Double
An example Run Properties file to examine:
runProperties.tsvWhen the transform script is called these properties will contain any values that the user has typed into the "Batch Properties" and "Run Properties" sections of the import form. The transform script can assign or modify these properties based on calculations or by reading them from the raw data file from the instrument. The script must then write the modified properties file to the location specified by the transformedRunPropertiesFile property.
2. Context properties of the assay such as assayName, runComments, and containerPath. These are recorded in the same format as the user-defined batch and run properties, but they cannot be overwritten by the script.
3. Paths to input and output files. These are absolute paths that the script reads from or writes to. They are in a <property name> <property value> format without property types. The paths currently used are:
- a. runDataUploadedFile: The assay result file selected and imported to the server by the user. This can be an Excel file (XLS, XLSX), a tab-separated text file (TSV), or a comma-separated text file (CSV).
- b. runDataFile: The file produced after the assay framework converts the user imported file to TSV format. The path will point to a subfolder below the script file directory, with a path value similar to <property value> <java property type>. The AssayId_22\42 part of the directory path serves to separate the temporary files from multiple executions by multiple scripts in the same folder.
C:\labkey\files\transforms\@files\scripts\TransformAndValidationFiles\AssayId_22\42\runDataFile.tsv
- c. AssayRunTSVData: This file path is where the result of the transform script will be written. It will point to a unique file name in an "assaydata" directory that the framework creates at the root of the files tree. NOTE: this property is written on the same line as the runDataFile property.
- d. errorsFile: This path is where a transform or validation script can write out error messages for use in troubleshooting. Not normally needed by an R script because the script usually writes errors to stdout, which are written by the framework to a file named "<scriptname>.Rout".
- e. transformedRunPropertiesFile: This path is where the script writes out the updated values of batch- and run-level properties that are listed in the runProperties file.
Choose the Input File for Transform Script Processing
Assuming that the system successfully infers a TSV file/runDataFile, the transform script developer can choose to use either the runDataFile or the runDataUploadedFile as its input. The runDataFile would be the right choice for an Excel-format raw file. By using the runDataFile, the assay framework does the Excel-to-TSV conversion and the script doesn't need to know how to parse Excel files. The runDataUploadedFile would be the right choice if, for example, the original file is already in TSV format or when the conversion process does not produce a useable TSV file.
A Python example that loads the original imported results file...
fileRunProperties = open(filePathRunProperties, "r")
for l in fileRunProperties:
row = l.split()
if row[0] == "runDataUploadedFile":
filePathIn = row[1]
if row[0] == "runDataFile":
filePathOut = row[3]
… and one that loads the inferred TSV file
fileRunProperties = open(filePathRunProperties, "r")
for l in fileRunProperties:
row = l.split()
if row[0] == "runDataFile":
filePathIn = row[1]
if row[0] == "runDataFile":
filePathOut = row[3]
Associate the Script with an Assay
To specify a transform script in an assay design, you enter the full path including the file extension in the
Transform Script field.
- Open the assay designer for a new assay, or edit an existing assay design.
- Click Add Script.
- Enter the full path to the script in the Transform Scripts field.
- You may enter multiple scripts by clicking Add Script again. Delete scripts by clicking the X to the right of the row.
- Confirm that other properties and fields required by your assay are correctly specified.
- Scroll down and click Save.
When you import (or re-import) run data using this assay design, the script will be executed.
There are two useful options presented as checkboxes in the Assay designer.
- Save Script Data for Debugging tells the framework to not delete the intermediate files such as the runProperties file after a successful run. This option is important during script development. It can be turned off to avoid cluttering the file space under the TransformAndValidationFiles directory that the framework automatically creates under the script file directory.
- Import In Background tells the framework to create a pipeline job as part of the import process, rather than tying up the browser session. It is useful for importing large data sets.
A few notes on usage:
- Client API calls are not supported in transform scripts.
- Columns populated by transform scripts must already exist in the assay definition.
- Executed scripts show up in the experimental graph, providing a record that transformations and/or quality control scripts were run.
- Transform scripts are run before field-level validators.
- The script is invoked once per run upload.
- Multiple scripts are invoked in the order they are listed in the assay design.
- Note that non-programmatic quality control remains available -- assay designs can be configured to perform basic checks for data types, required values, regular expressions, and ranges. Learn more in these topics: Field Editor and Dataset QC States - Admin Guide.
The general purpose assay tutorial includes another example use of a transform script in
Set up a Data Transform Script.
Pass Run Properties to Transform Scripts
Information on run properties can be passed to a transform script in two ways. You can put a substitution token into your script to identify the run properties file, or you can configure your scripting engine to pass the file path as a command line argument. See
Transformation Script Substitution Syntax for a list of available substitution tokens.
For example, using perl:
Option #1: Put a substitution token (${runInfo}) into your script and the server will replace it with the path to the run properties file. Here's a snippet of a perl script that uses this method:
# Open the run properties file. Run or upload set properties are not used by
# this script. We are only interested in the file paths for the run data and
# the error file.
open my $reportProps, '${runInfo}';
: Configure your scripting engine definition so that the file path is passed as a command line argument:
- Go to (Admin) > Site > Admin Console.
- Under Configuration, click Views and Scripting.
- Select and edit the perl engine.
- Add ${runInfo} to the Program Command field.
Related Topics
