Hi Alex,

Although we have some automation with the Pipeline, I'm not finding anything to indicate that a Pipeline Job could take a zip file of a folder structure, then create a new folder or project, then import that zip file in. I'm still digging into this to be sure though.

Can you provide us with more information as to what you're trying to do here?

What is the purpose of having to have a new folder get auto-created? What type of data is the folder supposed to contain? Study Data? Assay Data? Collaborative Data? Where is the source information coming from (for an import to happen, there would have to be some kind of export to make this zip file)?

The more we can understand the workflow, we can find the best solution for your needs.

Regards,

Jon

Hey Jon,

Thank you for looking into this in such detail. Much appreciated :)

The folders types we have are all Collaboration.

The overall goal I am trying to achieve is automated build and deployment for our LabKey configuration. The configuration itself was done manually via the UI thus far and we need to change this as managing changes between DEV, TEST, UAT and LIVE environments gets very difficult when you're copy pasting queries and manually doing folder config.

Automated build means that I will have the folders, queries, assays (xar files?), html wiki pages inside source control (git) and I will craft a maven script to build a zip file.
Automated deploy means - I will do either a shell script, a java tool, a web service call or any combination to get that zip file deployed into LabKey.

I am experimenting on how to do all this and I am looking at two avenues. The folder import/export mechanism and the custom module mechanism.

Please note that we already have a custom module that we build and deploy automatically.

Here are my pain points with each avenue:

Folder import/export:
- cannot automatically import a folder (the issue I am asking about in this thread)
- cannot export a folder with identical query names in different schemas (asked in a separate thread -> issue was fixed in 15.1 we are using 14.3 )
- cannot export a folder with linked schemas if a query depends on said schema (under investigation in another thread -> https://www.labkey.org/announcements/home/Developer/Forum/thread.view?rowId=11366)
- cannot rename queries by re uploading a folder (discussed and confirmed in another thread -> https://www.labkey.org/announcements/home/Developer/Forum/thread.view?rowId=11386)

Custom module:
- cannot specify a folder structure inside one module - a module is added to a folder via it's folder type - we have many folders and doing a module for each would be unwieldy
- I do not know how to specify linked schemas in a module as this seems to be a specific type of config for a folder

Our configuration that we are trying to automate looks like this:

- a bunch of folders with queries in them
- lots of linked schemas as in most queries depend on linked schemas
- a number of query web parts
- a number of wiki web parts - HTML + Javascript
- a folder with assays in it (all our linked schemas refer these assays by the way)

Regarding assays. I was looking at the assay custom module that could potentially replace our assay folder and also get added in all the other folders that need them thereby replacing all the linked schemas with a folder type reference (the export of which works as I have tested). There is one major problem with that approach. The folder with assays we have in place is populated with a lot of data, especially live data that I do not know how to migrate ... or even if a migration is necessary. So as a tangent question -> can I replace my assay folder with an assay custom module and transfer the data over?

The ideal goal is to be able to do with lab key elements as we do with code source control -> build -> deploy. However depending on the technical limitations we may need to live with some manual steps.

Once again thank you for looking into this.

Hello Jon,

Any update on the above please?

I am getting close to having all the pieces I need for automation. After upgrading to 15.2 I can now export queries with the same name, I have a work around for linked schemas.

All that remains is automatic deploy (upload of a folder zip file).

I was looking around the lk server file system and I noticed that my uploaded file lands in an actual folder on the server side:
/usr/local/labkey/files/test_project/ext_sch_upload/@files/unzip

I'm wondering what would happen if I stopped the server, copied over files to that folder and then started the server? Is that a way to deploy?

Hi Alex,

This is not really an official solution, but we have used curl in a script to hit the folder import action:

curl -c myCookie.txt -u ${USERNAME} "${HOST}/login/${CONTAINER_PATH}/login.view"
TOKEN=`cat myCookie.txt | grep 'X-LABKEY-CSRF' | cut -f 7`

curl -b myCookie.txt --form X-LABKEY-CSRF=${TOKEN} --form folderZip="@${FILE}" --form tabId=import "${HOST}/admin/${CONTAINER_PATH}/folderManagement.view"
rm myCookie.txt

As noted, this isnt a public LK API, so I presume subject to change without notice. It will let you script folder import though.

That will do. I was thinking of drafting some sort of HTTP requests and this is exactly what I need.

Thank you.

I would like to confirm that I am able to automatically upload the folder zip file using the above script.

Very happy with it :)

Hi,

I'm trying to use this solution (but with pycurl in Python). I am encountering a bit of difficulty and have a couple of quick questions:

1) Does anyone know if this solution still works in 16.2?

2) How is the argument to folderZip referenced? What I am asking is how the file path should be written? (I have Labkey on a windows server - do I use windows style file paths? What are they relative to? What is the root directory?

If anyone knows any of these answers I will be very grateful!

Hi,

Our default URL patterns did change from 15.X to 16.X (https://www.labkey.org/Documentation/wiki-page.view?name=url), so technically this should still work provided the actual API that Ben mentioned is still constructed to work in this fashion. But again, since this is not a public API, this is something that isn't officially supported.

Regarding the folderZip="@${FILE}" part, the @ sign is apart of cURL, telling curl to read the file. The ${FILE} part indicates that it's a variable that is in this case pointing to an actual file.

If you were to hardcode the file path, since you would be using @/path/to/file for the format, you should be using something like:

folderZip="@C:\path\to\file"

Regards,

Jon