I have been experimenting with the upload of a folder structure as we are trying to fully automate our Labkey configuration.
The only way to import/export folders that I found is through the Admin -> Folder -> Management -> Import/Export in the UI. Please let me know if there is another way to programatically specify the folders, queries and web parts structure.
The thing with automatic deployment though is that we would like to make changes and re upload the structure as we develop for it. What I found when uploading folder zip files was that when I re upload queries with the exact same name as existing queries then the content gets replaced -> which is expected and desired.
However if I change the name of a query and upload then no replacement takes place, both the queries are there. Which one could say that it's somewhat expected.
But when I turn around and do the same thing with web parts inside the folder (wiki pages in my case) I see that the web parts get entirely replaced in all cases.
My question: Is there a way to properly clean a folder before uploading it's new configuration? Specifically for renaming a query. |
| |
| adam responded: |
2015-08-26 07:51 |
You should consider building one or more modules to manage your standard queries, reports, views, webparts, etc. With modules, you can develop your resources under source code control and then deploy standard versions to multiple servers. Here's a starting point: https://www.labkey.org/wiki/home/Documentation/page.view?name=moduleDirectoryStructuresWe don't have a formal API for exporting/importing folders. That would be a nice addition… and wouldn't be difficult. You might be able to automate exports by posting directly to the folderManagement action and automate imports using the pipeline, but this can be tricky and fragile. We've also considered generalizing the "study reload" feature to folders, to provide an option for (say) nightly auto-reloading from a folder archive. One other existing option that may be relevant: at the point you create a new folder in the UI you can specify a "template folder" and select the objects to copy from that template. As for the replacement rules, here are the guiding principles:
- Existing objects in the folder are left alone unless there's a replacement in the archive.
- Individual objects are replaced if they have a canonical name… and an object with the same name is present in both archive and folder. For example, query names, list names, and dataset ids are canonical names, so these objects get replaced if names match. A new object (name not present in the folder) simply gets inserted.
- We take a "best effort" approach to importing. For example, if an archive refers to a folder or user that doesn't exist in the target system, the import process will do its best to succeed by resolving to the current folder or ignoring unknown users.
As you've seen, renaming a query (changing its canonical name) via folder import isn't possible under these rules. Since the name identifies the query there's no way to detect a rename. Webparts, on the other hand, are replaced as a group because they don't have individual names; the assumption is that the archive represents the full list of webparts. I hope this helps... Adam |
| |
| alx dobre responded: |
2015-08-26 08:06 |
Thank you for the detailed response Adam. Much appreciated :) I have been looking at custom modules to see if they can be used to do folder configuration. Indeed we are already using a custom module and it's under source control and automated build + deploy. Where I ran aground with using modules was when I realised that there isn't a way to specify multiple folders inside a custom module. It seems to me a custom module just gets added to a folder via the folder type, it does not specify a multiple folder structure. Given that we have a lot of folders it would not make sense for us to do a custom module for each individual folder. Please let me know if I am missing something cause if there is actually a way to do folders in a custom module we may well go that route. |
| |
| Jon (LabKey DevOps) responded: |
2015-08-26 17:13 |
Hi Alex,
Based on what I'm seeing, if you wanted to have a custom module to actually create folders, it looks like you would have to write your code in Java and specifically leverage the functionality that is in the AdminController.java code that is located under org.labkey.core.admin
So is the workflow supposed to be where this custom module creates folders all by itself?
Regards,
Jon |
| |
| alx dobre responded: |
2015-08-27 02:20 |
Hey Jon,
The overall goal I am trying to achieve is automated build and deployment for our LabKey configuration, as detailed in my latest reply in this thread: https://www.labkey.org/announcements/home/Developer/Forum/thread.view?rowId=11385
I suspected we may be able to do a lot with a custom JAVA module, however I am not keen to explore that alternative at least not without paying you guys for expert assistance cause with the relatively sparse documentation on the JAVA module side and the interesting combination of Spring and GWT which I am very curious how you make work ... I guess they just run side by side as they have completely different mechanisms for doing web content ... I would be further curious if I could use GWT 2.7 inside a custom module as I saw you guys use an older version... I would really like to leverage the UI binder feature too...
Oh and let's not forget the change cycle as in how fast can I see my code change on my browser while developing for it. I'm spoiled by GWT applications and I like the cycle to be close to instantaneous.
But that's all neither here nor there, I probably won't get the chance to dive that deep. |
| |
| Jon (LabKey DevOps) responded: |
2015-08-28 12:11 |
|
| |
| alx dobre responded: |
2015-08-29 02:56 |
Well, regarding the initial thread question "The rules around what gets replaced and what gets added when reuploading a folder structure", the answer is pretty clear from this thread. We will have to take into account that if we rename a query and reupload, the old query will still be there.
A minor gripe in the grand scheme of things.
I am glad I got all the other critical pieces together. I can export my structure, I can modify and build it using a script, then I can upload it using the commands you gave me.
That will do :)
Thank you for all the help :) |
| |
|
|