help setting up file uploads/downloads to an external service

General Server Forum (Inactive)
help setting up file uploads/downloads to an external service angel kennedy  2018-09-17 23:50
Status: Closed
 

Hi I'm hoping to get some advice on the best way to upload and download files to an external server via Labkey.

Here's the situation...
We are planning to upload large image datasets to labkey along with other trial data etc so that they can be shared amongst users.
We've been allocated a cloud based server with a small amount of storage on it that we can run LabKey from.
We've also been allocated a large amount of storage space on a separate system (it's for storage only) that at present cannot be mounted as a drive to our VM server.
We can however copy files between the 2 using pshell (a python based program that contains simple commands for file management).

I pretty new to LabKey and I haven't set up a file server before but the options I'm considering are
(a) set up a different file service that uses pshell beind the scenes to access the storage space and place a url link to the files in this service within rows of labkey.
(b) use trigger scripts within Labkey to trigger a pshell script to copy or retrieve files as needed.

How I think (b) might work
uploads: Use the normal labkey file upload system to upload the files onto the labkey server. A trigger script would be used to run a pshell script to move the files to the remote location. If it is not possible to attatch a trigger script to the file upload section then one could be attached to the “create” and “update” calls when an appropriate entry is created or edited in a table associating the files with patient data. The table would have a file link as per the “Linking Data Records with External Files” tutorial. A field indicating whether the file was locally stored or not (“local storage”) would be present and when the file transfer was complete the script could update this field to indicate that the file was no longer stored locally.

Downloads: The table mentioned above would also have a field indicating that the file should be retrieved. This would default to false. When a user wanted to download the file they would need to edit the row and change this field to true. Upon update a trigger script would then fetch the file and change the “local storage” field to true indicating that the user could now use the link to download the file.

One issue is how to know when to remove the file from local storage again. This could perhaps occurr after a specific interval or daily. At any rate it seems like it will be problematic.

My actual questions...

  • Can anyone suggest a good (hopefully easy to set up and use) file-server that could run pshell under the hood to access files?
  • Is my plan for getting labkey to use pshell reasonable? Is there a better and/or simpler method of doing this? Ideally I would like the process to be as user frinedly as possible and I feel the plan I've outlined above is not ideal in that respect.

thanks

 
 
Jon (LabKey DevOps) responded:  2018-10-01 20:59
Hi Angel,

Unfortunately, we currently do not have any recommendations regarding pshell. Also, we have no way of guaranteeing the file repository system within LabKey will work in the manner you're describing since we utilize WebDAV and at the moment, the only other external storage functionality we support is our Cloud Service (a Premium feature - https://www.labkey.org/Documentation/wiki-page.view?name=cloudStorage) that leverages Amazon Web Services S3 storage. But otherwise, if you have some kind of external storage, it needs to somehow be accessible through a mount on the LabKey server, such as setting up NFS or SMB/CIFS.

Regards,

Jon