Cloud Storage for File Watchers: /Documentation

Cloud Storage for File Watchers

Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Also available as an Add-on to the Starter Edition. Learn more or contact LabKey.

If you plan to use file watchers for files uploaded to your bucket, you need to supply an SQS Queue URL in your Storage Config on LabKey Server. This topic describes how to configure the bucket and queue on AWS to support this.

Create SQS Queue on AWS
Include SQS Queue URL in Storage Config
Configure S3 Event Notifications

You will be configuring an SQS Queue that LabKey Server can 'watch' for notifications, then setting up event notifications on the bucket that will add to that queue when certain files are added to certain bucket locations.

Create SQS Queue on AWS

First, you must configure a Simple Queue Service within AWS to which your bucket will be able to post notifications.

If you have not already set up your bucket, follow the instructions in this topic: Configure Cloud Storage

Take note of the region (like "us-west-2") for your bucket.

In your AWS account, click Create Queue.
Select Standard queue. (FIFO queues are not supported).
Accept the default configuration options.

Set the Access Policy for the queue to Advanced and provide for both of the following:

Allow users and IAM roles to manipulate the queue/messages (create, read, update, delete). The example below allows full control. This is what will allow the credentials supplied to LabKey Server to read/delete the messages from the Queue.
Allow S3 to send messages to queue.

Create the queue.

Take note of the SQS Queue URL that you will use in the Storage Config in the next section.

Access Policy Example

Replace the items in brackets "<>" as appropriate for your deployment.

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<AccountNumber>:<role>"
      },
      "Action": [
        "SQS:*"
      ],
      "Resource": "arn:aws:sqs:::<QueueName>"
    }, {
      "Sid": "allow-bucket",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:::<QueueName>",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "<AccountNumber>"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:*:*:*"
        }
      }
  ]
}

SQS Queue URL in Storage Config

Once you've enabled the queue on AWS, create or edit the LabKey Storage Config for this bucket, providing the SQS Queue URL.

Note that the region (like "us-west-2" in this URL) must match the region for the S3 Bucket specified in this storage config.

Configure S3 Event Notifications

Next, configure your bucket to send event notifications to your SQS Queue.

Log in to AWS and access the bucket you intend to monitor.
On the bucket's Properties tab, click Create Event Notification.
Under General configuration, name the event (i.e. MyFileDrop) and provide optional prefixes and suffixes to define which bucket contents should trigger notifications. Only objects with both the specified prefix and suffix will trigger notifications.

Prefix: This is typically the directory path within the bucket, and will depend on how you organize your buckets. For example, you might be using a "cloud_storage" main directory and then employ a LabKey directory prefix like "lk_" for each LabKey folder, then a "watched" subdirectory there. In this case the prefix would be something like "/cloud_storage/lk_2483/watched/". This prefix should either match, or be a subdirectory of, the Location to Watch you will set in your file watcher.
Suffix: A suffix for the files that should trigger notifications, such as ".list.tsv" if you want to look for files like "FirstList.list.tsv" and "SecondList.list.tsv" in the container specified in the prefix.

Event Types:

Select All Object Create Events.

Destination:

Select SQS Queue, then either:

Choose from your SQS Queues to use a dropdown selector or
Enter SQS queue ARN to enter it directly.

Click Save when finished.

Now when files in your bucket's Prefix location have the Suffix you defined, an event notification will be triggered on the queue specified. LabKey is "monitoring" this queue via the Storage Config, so now you will be able to enable file watchers on those locations.

Note that the files in your bucket that trigger notifications do not have to be the same as the files your file watcher will act upon. For example, you might want to reload a set of list files, but only trigger that file watcher when a master manifest was added. In such a scenario, your event notifications would be triggered by the manifest, and the filewatcher would then act on the files matching it's definition, i.e. the lists.

Troubleshooting Event Notifications

If you have multiple event notifications defined on your queue, note that they will be validated as a group. If you have existing notifications configured when you add a new one for your LabKey file watcher, you will not be able to save or edit new ones until you resolve the configuration of the prior ones. When in a state that cannot be validated, notifications will still be sent, but this is an interaction to keep in mind if you encounter problems defining a new one.

Create File Watcher for Cloud Files

Once cloud storage with queue notifications is configured, and enabled in your folder, you will be able to create a Files web part in the folder that "surfaces" the S3 location. For example, you can drop files into the bucket from the LabKey interface, and have them appear in the S3 bucket, or vice versa.

You will also be able to configure file watchers in that folder triggered by the event notifications on the bucket.

The Reload Lists Using Data File task is currently supported for this feature. Other file watcher types will support S3 cloud loading in future releases.

LabKey Support

LabKey Support