Premium Feature — This feature is available in the Professional, Professional Plus, and Enterprise Editions.
Learn more or
contact LabKey
The File Watcher lets a site administrator configure a listener that will monitor any directory on the file system looking for files matching specified conditions to appear.
For example, if we configure the File Watcher in a container to listen at C:/test/listenHere for directories containing FCS files, then a new directory that appears at C:/test/listenHere/newRunDirectory will get noticed and the entire newRunDirectory will get moved (deleting the original) to the given container's pipeline root and a pipeline run will be initiated to process all the FCS files in the newRunDirectory directory.
Create a FileWatcher Listener
The listener configurations are stored in table of the pipeline schema called pipeline.TriggerConfigurations. To create a new entry, use the Schema Browser:
- Navigate to the container where you want the files to be imported, i.e. the destination in LabKey.
- Select (Admin) > Developer Links > Schema Browser.
- Choose the pipeline schema.
- Select the TriggerConfigurations table, and click View Data.
- Click (Insert new row).
- Enter a Name and optional Description.
- Select a Type: "pipeline-filewatcher"
- Enter the Configuration in JSON. Details are below.
- Choose the Pipeline Id. This dropdown is populated with all the pipeline tasks that can be run without user intervention, making them eligible to use during FileWatcher imports.
- Click Submit.
Configuration
The
Configuration column in the TriggerConfigurations table is required, and should be provided in JSON. For example:
{
"move" : "@pipeline/${now:date('YYYY-MM')}",
"parameterFunction" : "var userName = sourcePath.getNameCount() > 0 ? sourcePath.getName(0) : null;
var ret = {'pipeline, username': userName }; ret;",
"location" : "C:/dev/labkey/trunk/build/deploy/files/_WatchedRoot/",
"quiet" : "10000"
}
Parameters:
- move: Where the file should be moved. This can be absolute or relative to the current project/folder, and values from the filename can be substituted in the move path (ex. study).
- location: File drop location. An absolute path on the server's file system.
- filePattern: A Java regular expression that captures filenames of interest and can extract and use information from the filename to set other properties.
- quiet: number of seconds to wait after file activity before executing a job (min is 1000).
- recursive: A boolean indicating whether to seek uploadable files in subdirectories.
- parameterFunction: (optional) Include a JavaScript function to be executed during the move.
Using the FileWatcher to Upload Data
Example 1
Consider a process where FCS flow data is deposited in a common location by a number of users, with each data export placed into a subdirectory of the watched folder, perhaps in a separate subdirectory per user.
When the File Watcher finds these files, they are placed into a new location under the folder pipeline root based on the current user and date. Example: @pipeline/${username}/${date('YYYY-MM')}
LabKey then imports the FCS data to that container. All FCS files within a single directory are imported as a single experiment run in the flow module.
Example 2
Consider a set of data with original filenames matching a format like this: "sample_<timestamp>_<study_id>.xml", for example:
sample_2017-09-06_study20.xml
An example
filePattern regular expression that would capture such filenames would be:
sample_(.+)_(?<study>.+).xml
If the specified pattern matches a file placed in the watched
location, then the specified move and/or execute steps will be performed on that file. Nothing will happen to files in the watched location which do not match the pattern.
If the regular expression contains named capturing groups, such as the "(?<study>.+)" portion in the example above, then the corresponding value (in this example "study20" can be substituted into other property expressions. For instance, a
move setting of:
/studies/${study}/@pipeline/import/${now:date} would resolve into:
/studies/study20/@pipeline/import/2017-11-07 (or similar)
This substitution allows the administrator to configure the file watcher to automatically determine the destination folder based on the name, ensuring that the data is uploaded to the correct location.
