File Watchers let administrators set up the monitoring of directories on the file system and perform specific actions when desired files appear. This topic outlines the process of creating file watch triggers.
Create a File Watcher
- Navigate to the folder where you want the files to be imported, i.e. the destination in LabKey.
- Open the File Watcher management UI:
- Select (Admin) > Folder > Management. Click the Import tab and scroll down.
- In a study folder, you can instead click the Manage tab, then click Manage File Watchers.
- Depending on your project's enabled module set, options for trigger creation may vary.
- Click one of the links below Create a trigger to... to create that type of trigger.
- Manage File Watcher Triggers: Click to see the table of all currently configured File Watchers.
Configure the Trigger
The two panels of the
Create Pipeline Trigger wizard define a File Watcher. Configuration options and on-screen guidance may vary for each
task type.
Details
- Name: A unique name for the trigger.
- Description: A description for the trigger.
- Type: Currently supports one value 'pipeline-filewatcher'.
- Pipeline Task: The type of File Watcher task you want to create. By default the option you clicked to open this wizard is selected but you can change this selection from available options on the dropdown menu.
- Run as username: The File Watcher will run as this user in the pipeline. It is strongly recommended that this user has elevated permissions to perform updates, deletes, etc. For example, an elevated "service account" could be used.
- Assay Provider: Use this provider for running assay import runs. Learn more here.
- Enabled: Turns on detection and triggering.
- Click Next to move to the next panel.
Configuration
- Location to Watch: File location to watch for uploadable files. Confirm that this directory exists before saving the File Watcher configuration.
- This can be a path relative to the local container's pipeline root, beginning with "./". Use "." to indicate the pipeline root directory itself (or you could also enter the matching full path). Users with "Folder Admin" or higher can watch locations relative to the container root.
- Site and Application Admins can watch an absolute path on the server's file system (beginning with a "/"), or a location outside the local filesystem, such as on the local machine or a networked drive.
- Learn more about setting the file and/or pipeline roots for your folder.
- Note that if you use the root location here, you should also set a Move to Container location to avoid a potential loop when the system tries to make a copy to the same location "before analysis".
- Include Child Folders: A boolean indicating whether to seek uploadable files in subdirectories (currently to a max depth of 3) of the Location to Watch you specified.
- File Pattern: A Java regular expression that captures filenames of interest and can extract and use information from the filename to set other properties. We recommend using a regex interpreter, such as https://regex101.com/, to test the behavior of your file pattern. Details are in this topic: File Watcher: File Name Patterns. If no pattern is provided, the default pattern is used:
(^\D*)\.(?:tsv|txt|xls|xlsx)
Note that this default pattern
does not match file names that include numbers. For example, "AssayFile123.xls" will not be matched.
- Quiet Period (Seconds): Number of seconds to wait after file activity before executing a job (minimum is 1). If you encounter conflicts, particularly when running multiple File Watchers monitoring the same location, try increasing the quiet period.
- The quiet period is respected (i.e. restarted) for each subdirectory created, which can lead to the perception of extra delays.
- Be sure that the quiet period you set will allow time for the upload of all content that should be acted upon. In the case of subdirectories or large uploads, you may need a longer quiet period to avoid missing files. Learn more here
- Move to Container: Move the file to this container before analysis. This must be a relative or absolute container path.
- If this field is blank, and the watched file is already underneath a pipeline root, then it will not be moved.
- If this field is blank but the watched file is elsewhere, it will be moved to the pipeline root of the current container.
- You must have at least Folder Administrator permissions in the folder where files are being moved to.
- Move to Subdirectory: Move the file to this directory under the destination container's pipeline root.
- Leaving this blank will default to the pipeline root.
- You must have at least Folder Administrator permissions in the folder where files are being moved to.
- Copy File To: Where the file should be copied to before analysis. This can be absolute or relative to the current project/folder. You must have at least Folder Administrator permissions in the folder where the files are being copied to. For example, an absolute file path to the Testing project:
/labkey/labkey/files/Testing/@files
- Action: When importing data, the following import behaviors are available. Different options are available depending on the File Watcher task and the type of target table. Lists support 'merge' and 'replace' but not 'append', etc.
- 'Merge' inserts new rows and updates existing rows in the target table.
- 'Append' adds incoming data to the target table.
- 'Replace' deletes existing rows in the target table and imports incoming data into the empty table.
- Allow Domain Updates: When updating lists and datasets, by default, the target data structure will be updated to reflect new columns in the incoming list or dataset. Any columns missing from the incoming data will be dropped (and their data deleted). To override this behavior, uncheck the Allow Domain Updates box to retain the column set of the existing list or dataset.
- Import Lookups by Alternate Key: If enabled, the server will try to resolve lookups by values other than the target's primary key. For details see Populate a List.
- Show Advanced Settings: Click the symbol to add custom functions and parameters.
- Parameter Function: Include a JavaScript function to be executed during the move. (See example here.)
- Add Custom Parameter: These parameters will be passed to the chosen pipeline task for consumption in addition to the standard configuration. Some pipeline tasks have specific custom parameters available. For details, see File Watcher Tasks.
- Click Save when finished.
Manage File Watchers
- Navigate to the folder where you want to manage File Watchers.
- Select (Admin) > Folder > Management. Click the Import tab and scroll down.
- In a study folder, you can instead click the Manage tab, then click Manage File Watchers.
- Click Manage File Watcher Triggers.
- All the File Watchers defined in the current folder will be listed.
- Hover to expose a (pencil) icon for editing.
- Select a row and click the (delete) icon to delete.
Developing and Testing File Watchers
When iteratively developing a File Watcher it is useful to reset the timestamp on the source data file so that the File Watcher is triggered. To reset timestamps on all files in a directory, use the following Windows command like the following.
C:\testdata\datasets> copy /b *.tsv +,,
Common Configuration Issues
Problem: File Watcher Does Not Trigger, But No Error Is Logged
When the user provides a file in the watched directory, nothing happens. The failure is silent, and no error is shown or logged.
If this behavior results, there are a few common culprits:
1. The provided file does not match the file pattern. Confirm that the file name matches the regex file pattern. Use a regex interpreter service such as
https://regex101.com/ to test the behavior.
2. The File Watcher was initially configured to watch a directory that does not exist. To recover from this: create a new directory to watch, edit
Location to Watch to point to this directory, and re-save the File Watcher configuration.
3. For an assay File Watcher, double check that the File Watcher is configured to use the desired
Assay Protocol and that the data matches the expectations of that assay design.
Related Topics
