• Sign In
  1. Messages

Automatically importing a study with lists and reports

LabKey Support Forum (Inactive)

View Message

view list print
Automatically importing a study with lists and reports gkericks  2017-04-24 14:54
Status: Closed
 
Hi there,

I have been working working on a system to facilitate using Labkey with an external data collection service. Our use case is as follows:

We have many different teams accessing various project on a Labkey server. They are mostly using the tool to share files and post wikis. For each project (or at least most of the projects) we will have a corresponding project in the aforementioned external service. The corresponding projects in the external service are the points of data collection and they will be constantly updated.

What I want to do:

I have been working to standardise and automate moving data out of the external service and in to labkey. I have wrote a simple program that requests data from the external service using an API. I want to also automate the importing of that data into Labkey. Now, I started using the study import/reload features. I have gotten pretty far with this approach and can automate the data reloading as a study. As I have tried to add more details to the import, this has gotten more difficult.

Current challenge:

The study reload feature doesn't include lists or reports. This is problematic for two reasons:

1) Lists are needed to encode complex data types where the raw representation is different from the labels.

2) Reports will change as the number of projects we support changes and we want to be able to easily export a a 'stock' report to many different projects.

I have been trying to get the technique described here (https://www.labkey.org/home/Support/Developer%20Forum/announcements-thread.view?rowId=11385) to work with no success. I am using Labkey 16.2. I am also using pycurl to give commands to curl. Does anyone know if this technique still works? Does the fact that I am using pycurl (python) matter?

I would really appreciate some feedback and discussion around this. The automation of the study export/import/reload went fairly smoothly, however trying to add in extra features (reports, data type encoding via lists, anything to do with folder structure) has been a headache. I am guessing these challenges arise because Labkey does not explicitly support reloading of arbitrary folder types so perhaps I am simply looking into the wrong features for our use case.
 
 
Jon (LabKey DevOps) responded:  2017-04-25 11:44
Hello,

Have you tried to do a Folder Reload rather than a Study Reload?

If you re-import the folder into the same place, it will overwrite everything, from the study to the lists and even the reports.

Regards,

Jon
 
gkericks responded:  2017-04-25 12:33
Hi Jon,

Is the "Folder Reload" something I can automate? I did not see something like that in the docs. Thanks.
 
Jon (LabKey DevOps) responded:  2017-04-26 17:35
Hello,

I've updated the thread you were responding to:

https://www.labkey.org/home/Support/Developer%20Forum/announcements-thread.view?rowId=11385

Regards,

Jon
 
gkericks responded:  2017-04-28 12:46
Thanks Jon,

I managed to figure it out from those instructions. For anyone curious, the part that threw me off is that the syntax is slightly different in pycurl. You need a line like the following:

curl_client.setopt(pycurl.HTTPPOST, [("folderZip", (pycurl.FORM_FILE, "C:\\Path\to\file"))])

So this method appears to work now. I still have some concerns about whether or not this is the best approach for my use case. We will have a separate team of people setting up projects that they are using for sharing files and wiki pages. They will be managing user access rights, creating new users etc. On the same labkey deployment we would like to have data from the external service I mention in my first post "streaming" in to labkey. I am planning on having this happen nightly and will use the folder reload solution to facilitate this (unless a better and officially supported option emerges). So does this seems like an okay route to go down considering what we are trying to achieve? Any feedback would be much appreciated and I am happy to answer an questions if you need more details.

Thanks