LabKey Server is developing the ability to integrate
cloud storage for management of large data files. Prototype support for storage on Amazon S3 (Simple Storage Service) is available upon request. Support for other storage providers will be considered in the future. This feature is currently in development and not yet recommended for use on a production server. For more information about this feature and possible future directions, please
contact LabKey.
Cloud Data Storage
Cloud Services offer the ability to upload and post large data files in the cloud, and LabKey Server can interface with this data allowing users to integrate it smoothly with other data for seamless use by LabKey analysis tools. In order to use these features, you must have installed the
Cloud Module in your LabKey Server.
Cloud Storage services store data in
buckets which are typically limited to a certain number by user account, but can contain unlimited files. LabKey server Cloud Storage uses a single bucket with a directory providing a pseudo-hierarchy so that multiple structured folders can appear as a multi-bucket storage system.
Configure LabKey Server to use Cloud Storage
Before you can use your Cloud Storage account from within LabKey server, you must first create the bucket you intend to use and the user account must have "list" as well as "upload/delete" permissions on the bucket. It is possible to have multiple cloud store services per account. To configure your account for cloud storage:
- Navigate to Admin > Site > Admin Console.
- Select Cloud Settings. If you do not see this option, you do not have a cloud module installed.
- Configure Cloud Accounts.
- Select a Provider.
- Enter your Identity and Credential.
- Click Update.
This feature uses the new encrypted property store for credentials and requires an administrator to provide a master encryption key in the labkey.xml file. LabKey will refuse to store credentials if a key is not provided. For instructions, see:
labkey.xml Configuration File.
Next define a file storage alias pointing to a bucket available to your account.
- Select Create Storage Config.
- Provide a Config Name.
- Select the Account you just created from the pulldown.
- Provide the Bucket name.
- Select Enabled.
- Click Create.
Enable Cloud Storage
In each folder where you want to access cloud data, configure the filesystem to use your cloud storage. If a cloud store is disabled at the site-level it will not be possible to enable it within a folder.
- Open Admin > Folder Management.
- Select the Files tab.
- Under Enable Cloud Store, enable and disable the available cloud stores using the checkboxes. Note that this enables cloud storage at the folder level, not the site level.
- Click Save.
- Return to the Files web part.
- Select Customize from the pulldown arrow in the upper left.
- Select the cloud in File Root.
- Click Submit.
Use Files from the Cloud
The Files web part will now display the cloud storage files as if they are local, as in the case of the .fcs file shown here:
The file is actually located in cloud storage as shown here:
When a download request for a cloud storage file comes through LabKey server, the handle is passed to the client so the client can download the file directly.
Future Directions
This feature is currently under development and not yet suitable for deployment in a production environment. In addition to supporting production deployment, some other possible future directions include:
- An upload widget that can perform "multi-part" uploads directly to the cloud storage back end to support large files.
- Performance improvements.
- Mirror/replication between buckets for remote sites.
- Execute pipeline jobs via cloud compute/execute service.
If you are interested in learning more about the future directions and timeline for this functionality, please
contact LabKey.
