LabKey Server can integrate cloud storage for management of large data files using Amazon S3 (Simple Storage Service). Support for other storage providers will be considered in the future. For more information about this feature and possible future directions, please contact LabKey.

Currently, cloud storage services are best suited to providing an archive for large files. Some pipeline jobs can be run against files managed by cloud storage. Currently, the import of Skyline document files into a Panorama folder is supported.

• Cloud Data Storage
• AWS Identity
• Configure LabKey Server to use Cloud Storage
• Enable Cloud Storage
• Use Files from the Cloud
• S3 Permissions Required

Cloud Data Storage

Cloud Services offer the ability to upload and post large data files in the cloud, and LabKey Server can interface with this data allowing users to integrate it smoothly with other data for seamless use by LabKey analysis tools. In order to use these features, you must have installed the Cloud Module in your LabKey Server.

Cloud Storage services store data in buckets which are typically limited to a certain number by user account, but can contain unlimited files. LabKey server Cloud Storage uses a single bucket with a directory providing a pseudo-hierarchy so that multiple structured folders can appear as a multi-bucket storage system.

Learn more about Amazon S3 Buckets here: Working with Amazon S3 Buckets

AWS Identity

The identity and credential LabKey will use to access your S3 bucket are generated by creating an AWS Identity .

On the console click "Add User", provide a user name, select Programmatic Access, create a new group and give it AdministratorAccess. If AdministratorAccess is not possible, the detailed permissions required are listed later in this document.

At the end of the wizard, you will be given an "Access key id" and a "Secret access key". Enter these in the Identity and Credentials fields in LabKey.

Configure LabKey Server to use Cloud Storage

Before you can use your Cloud Storage account from within LabKey server, you must first create the bucket you intend to use and the user account must have "list" as well as "upload/delete" permissions on the bucket. It is possible to have multiple cloud store services per account. To configure your account for cloud storage:

Create Account

• Select (Admin) > Site > Admin Console.
• Under Premium Features, click Cloud Settings.
• If you do not see this option, you do not have the Cloud Module installed.
• Under Cloud Accounts, click Create Account.

• Select a Provider.
• Enter your Identity and Credential. See AWS Identity above.
• Click Create.

This feature uses the encrypted property store for credentials and requires an administrator to provide a master encryption key in the labkey.xml file. LabKey will refuse to store credentials if a key is not provided. For instructions, see: Installation: SMTP, Encryption, LDAP, and File Roots.

Create Storage Config

Next define a file storage alias pointing to a bucket available to your account. LabKey can create new subfolders there, or if you want to use a pre-existing S3 subdirectory within your bucket, you can specify it using the S3 Path option.

• Click Create Storage Config on the cloud account settings page.

• Provide a Config Name.
• Select the Account you just created from the pulldown.
• Provide the S3 Bucket name itself. Do not include "S3://" or other elements of the full URL with the bucket name in this field. Learn more about bucket naming rules here
• Select Enabled.
• S3 Path: (Optional) You can specify a path within the S3 bucket that will be the configuration root of any LabKey folder using this configuration. This enables use of an existing folder within the S3 bucket. If no path is specified, the root is the bucket itself.
• Directory Prefix: (Optional) Select whether to create a directory named <prefix><id> in the bucket or S3 path provided for this folder. The default prefix is "container".
• If you check the Directory Prefix box (default), LabKey will automatically create a subdirectory in the configuration root (the bucket itself or the S3 path provided above) for each LabKey folder using this configuration. Note that using this option means that the subdirectory and its contents will be deleted if the LabKey folder is deleted.
• If you do not check the box, all LabKey folders using this configuration will share the root location and LabKey will not delete the root contents when any folder is deleted.
• Click Create.

Enable Cloud Storage

In each folder where you want to access cloud data, configure the filesystem to use your cloud storage. If a cloud store is disabled at the site-level it will not be possible to enable it within a folder.

• Navigate to the folder where you want to enable cloud storage and open (Admin) > Folder > Management. (If you want to enable cloud storage at the project level, open (Admin) > Folder > Project Settings.)
• Select the Files tab.
• Under Cloud Stores, enable the desired cloud stores using the checkboxes. Note that this enables cloud storage at the folder level, not the site level.
• Click Save.

• Under File Root select Use cloud-based file storage and use the dropdown to select the desired cloud store. If you select this option before enabling the cloud store, you will see an empty dropdown.

• Existing Files: When you select a new file root for a folder, you will see the option Proposed File Root change from '<prior option>'. Select what you want to happen to any existing files in the root. Note that if you are not using directory containers, you will not be able to move files as they will not be deleted from the shared root. See Migrate Existing Files for details about file migration options.
• Click the Save button a second time.

• Go to the Files web part in your folder. (You may need to add the Files web part first.)
• Select Customize from the (triangle) menu in the web part header.
• Select the @cloud directory, or any of its sub-directories, in the File Root pane.
• You can give the webpart a descriptive title if you like.
• Click Submit.

Use Files from the Cloud

The Files web part will now display the cloud storage files as if they are local, as in the case of the .fcs file shown here:

The file is actually located in cloud storage as shown here:

When a download request for a cloud storage file comes through LabKey server, the handle is passed to the client so the client can download the file directly.

Files uploaded to the S3 bucket independently of LabKey will appear in the LabKey Files web part.

Deleting Files from Cloud Storage

If you have configured cloud storage in LabKey to create a new subdirectory (using Directory Prefix) for each new folder created on LabKey, the files placed within it will be associated with the LabKey folder. If you delete the LabKey folder, the associated subfolder of your S3 bucket (and all files within it) will also be deleted.

If you instead configured cloud storage to use an existing S3 folder, any files placed there will be visible from within the LabKey folder, but will NOT be deleted if the LabKey folder is deleted.

S3 Permissions Required

The detailed permissions required for S3 access are listed below. Substitute your bucket name where you see BUCKET_NAME.

{ 
  "Version":"2012-10-17", 
  "Statement":[ 
    { 
      "Effect":"Allow", 
      "Action":[ 
            "s3:GetAccountPublicAccessBlock",
            "s3:GetBucketAcl",
            "s3:GetBucketLocation",
            "s3:GetBucketPolicyStatus",
            "s3:GetBucketPublicAccessBlock",
            "s3:HeadBucket",
            "s3:ListAllMyBuckets"
      ], 
      "Resource":"*"
    }, 
    { 
      "Effect":"Allow", 
      "Action":[ 
            "s3:GetLifecycleConfiguration",
            "s3:GetBucketTagging",
            "s3:GetInventoryConfiguration",
            "s3:GetObjectVersionTagging",
            "s3:ListBucketVersions",
            "s3:GetBucketLogging",
            "s3:ReplicateTags",
            "s3:ListBucket",
            "s3:GetAccelerateConfiguration",
            "s3:GetBucketPolicy",
            "s3:ReplicateObject",
            "s3:GetObjectVersionTorrent",
            "s3:GetObjectAcl",
            "s3:GetEncryptionConfiguration",
            "s3:GetBucketObjectLockConfiguration",
            "s3:AbortMultipartUpload",
            "s3:PutBucketTagging",
            "s3:GetBucketRequestPayment",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectTagging",
            "s3:GetMetricsConfiguration",
            "s3:PutObjectTagging",
            "s3:DeleteObject",
            "s3:DeleteObjectTagging",
            "s3:GetBucketPublicAccessBlock",
            "s3:GetBucketPolicyStatus",
            "s3:ListBucketMultipartUploads",
            "s3:GetObjectRetention",
            "s3:GetBucketWebsite",
            "s3:PutObjectVersionTagging",
            "s3:PutObjectLegalHold",
            "s3:DeleteObjectVersionTagging",
            "s3:GetBucketVersioning",
            "s3:GetBucketAcl",
            "s3:GetObjectLegalHold",
            "s3:GetReplicationConfiguration",
            "s3:ListMultipartUploadParts",
            "s3:PutObject",
            "s3:GetObject",
            "s3:GetObjectTorrent",
            "s3:PutObjectRetention",
            "s3:GetBucketCORS",
            "s3:GetAnalyticsConfiguration",
            "s3:GetObjectVersionForReplication",
            "s3:GetBucketLocation",
            "s3:ReplicateDelete",
            "s3:GetObjectVersion"
      ], 
      "Resource":[ 
            "arn:aws:s3:::BUCKET_NAME",
            "arn:aws:s3:::BUCKET_NAME/*"
      ] 
    } 
  ] 
}

Additionally, if ACLs are defined on individual objects within a bucket, the user will need READ and READ_ACP permission to each object for read-only usage, and WRITE and WRITE_ACP for write usage.

See more information about S3 permissions in the AWS documentation.

What's Next?

If you are interested in learning more about the future directions for this functionality, please contact LabKey.