Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.

File Watchers let administrators set up the monitoring of directories on the file system and perform specific actions when desired files appear. This topic outlines the process of creating file watch triggers.

Create a File Watcher

  • Navigate to the folder where you want the files to be imported, i.e. the destination in LabKey.
  • Open the File Watcher management UI:
    • Select (Admin) > Folder > Management. Click the Import tab and scroll down.
    • In a study folder, you can instead click the Manage tab, then click Manage File Watchers.

Configure the Trigger

The two panels of the Create Pipeline Trigger wizard define a file watcher. Configuration options and on-screen guidance may vary for each task type.

Details

  • Name: A unique name for the trigger.
  • Description: A description for the trigger.
  • Type: Currently supports one value 'pipeline-filewatcher'.
  • Pipeline Task: The type of filewatcher task you want to create. By default the option you clicked to open this wizard is selected but you can change this selection from available options on the dropdown menu.
  • Run as username: The file watcher will run as this user in the pipeline. It is strongly recommended that this user has elevated permissions to perform updates, deletes, etc. For example, an elevated "service account" could be used.
  • Assay Provider: Use this provider for running assay import runs.
  • Enabled: Turns on detection and triggering.
  • Click Next to move to the next panel.

Configuration

  • Location to Watch: File location to watch for uploadable files.
    • This can be a path relative to the local container's pipeline root, beginning with "./". Use "." to indicate the pipeline root directory itself (or you could also enter the matching full path). Users with "Folder Admin" or higher can watch locations relative to the container root.
    • Site and Application Admins can watch an absolute path on the server's file system (beginning with a "/"), or a location outside the local filesystem, such as on the local machine or a networked drive.
    • Learn more about setting the file and/or pipeline roots for your folder.
  • Include Child Folders: A boolean indicating whether to seek uploadable files in subdirectories (currently to a max depth of 3) of the Location to Watch you specified.
  • File Pattern: A Java regular expression that captures filenames of interest and can extract and use information from the filename to set other properties. We recommend using a regex interpreter, such as https://regex101.com/, to test the behavior of your file pattern. Details are in this topic: File Watcher: File Name Patterns.
  • Quiet Period (Seconds): Number of seconds to wait after file activity before executing a job (minimum is 1). If you encounter conflicts, particularly when running multiple filewatchers monitoring the same location, try increasing the quiet period.
  • Move to Container: Move the file to this container before analysis. This must be a relative or absolute container path. If this field is blank, and the watched file is already underneath a pipeline root, then it will not be moved. If this field is blank but the watched file is elsewhere, it will be moved to the pipeline root of the current container. You must have at least Folder Administrator permissions in the folder where files are being moved to.
  • Move to Subdirectory: Move the file to this directory under the destination container's pipeline root. Leaving this blank will default to the pipeline root. You must have at least Folder Administrator permissions in the folder where files are being moved to.
  • Copy File To: Where the file should be copied to before analysis. This can be absolute or relative to the current project/folder. You must have at least Folder Administrator permissions in the folder where the files are being copied to.
  • Action: When importing data, the following import behaviors are available. Different options are available depending on the filewatcher task and the type of target table. Lists support 'merge' and 'replace' but not 'append', etc.
    • 'Merge' inserts new rows and updates existing rows in the target table.
    • 'Append' adds incoming data to the target table.
    • 'Replace' deletes existing rows in the target table and imports incoming data into the empty table.
  • Allow Domain Updates: When updating lists and datasets, by default, the target data structure will be updated to reflect new columns in the incoming list or dataset. Any columns missing from the incoming data will be dropped (and their data deleted). To override this behavior, uncheck the Allow Domain Updates box to retain the column set of the existing list or dataset.
  • Import Lookups by Alternate Key: If enabled, the server will try to resolve lookups by values other than the target's primary key. For details see Populate a List.
  • Show Advanced Settings: Click the symbol to add custom functions and parameters.
    • Parameter Function: Include a JavaScript function to be executed during the move. (See example here.)
    • Add Custom Parameter: These parameters will be passed to the chosen pipeline task for consumption in addition to the standard configuration. Some pipeline tasks have specific custom parameters available. For details, see File Watcher Tasks.
  • Click Save when finished.

Manage File Watchers

  • Navigate to the folder where you want to manage file watchers.
  • Select (Admin) > Folder > Management. Click the Import tab and scroll down.
    • In a study folder, you can instead click the Manage tab, then click Manage File Watchers.
  • Click Manage File Watcher Triggers.
  • All the file watchers defined in the current folder will be listed.
  • Hover to expose a (pencil) icon for editing.
  • Select a row and click the (delete) icon to delete.

Developing and Testing File Watchers

When iteratively developing a file watcher it is useful to reset the timestamp on the source data file so that the file watcher is triggered. To reset timestamps on all files in a directory, use the following Windows command like the following.

C:\testdata\datasets> copy /b *.tsv +,,

Related Topics

Discussion

Was this content helpful?

Log in or register an account to provide feedback


previousnext
 
expand all collapse all