LabKey Server can integrate cloud storage for management of large data files using Amazon S3 (Simple Storage Service). Support for other storage providers will be considered in the future. For more information about this feature and possible future directions, please contact LabKey.

Cloud storage services are best suited to providing an archive for large files. Files managed by cloud storage can be exposed within LabKey as if they were in the local filesystem. You can add and remove files from the cloud via the LabKey interface and many pipeline jobs can be run directly on them. This section outlines the steps necessary to access data in an S3 bucket from the Files web part on your server.

• Cloud Data Storage Overview
• Set Up Cloud Storage and Configure LabKey Server
• Use Files from Cloud Storage

Cloud Data Storage Overview

Cloud Services offer the ability to upload and post large data files in the cloud, and LabKey Server can interface with this data allowing users to integrate it smoothly with other data for seamless use by LabKey analysis tools. In order to use these features, you must have installed the cloud module in your LabKey Server.

Contact your Account Manager for assistance if needed.

Buckets

Cloud Storage services store data in buckets which are typically limited to a certain number by user account, but can contain unlimited files. LabKey Server Cloud Storage uses a single bucket with a directory providing a pseudo-hierarchy so that multiple structured folders can appear as a multi-bucket storage system.

Learn more about Amazon S3 Buckets here: Working with Amazon S3 Buckets

Encryption and Security

The Cloud module supports AWS S3 "default" AES encryption. This can be configured when the bucket is provisioned. With "default" encryption S3 transparently encrypts/decrypts the files/objects when they are written to or read from the S3 bucket.

AWS S3 also supports unique KMS (Key Management System) encryption keys that are managed by the customer within AWS. Other security controls on the S3 bucket such as who can access those files via other AWS methods is in full control of the customer by using AWS IAM Policies.

Regardless of how the S3 buckets are configured, LabKey has a full authentication and authorization system built in to manage who can access those files within LabKey.

Set Up Cloud Storage

To set up to use Cloud Storage with LabKey Server, you need the targets on AWS/S3, and configuration at the site-level before you'll be able to use the storage.

1. Create the necessary AWS indentity credential.
2. Each bucket on S3 that you plan to access must be created on AWS first.
3. On LabKey Server, create a Cloud Account for accessing your buckets.
4. Each bucket is then defined as a Storage Config on LabKey Server, using that account.

These Storage Configs will then be available in your projects and folders as Cloud Stores.

Use Files from Cloud Storage

Once you have completed the setup steps above, you will be able to select which Cloud Stores to use on a per-folder basis as required. The enabling and usage are covered in this topic:

• Use Files from Cloud Storage

For example, once correctly configured, you will be able to "surface" files from your S3 bucket in a Files web part on your LabKey Server.

Related Topics

• File Root Options
• AWS Identity Credentials
• Configure Cloud Storage
• Use Files from Cloud Storage

What's Next?

If you are interested in learning more about, or contributing to, the future directions for this functionality, please contact LabKey.