Error when upgrading: /home/Support/LabKey Support Forum

Error when upgrading

LabKey Support Forum

View Message

Error when upgrading GrumpyBadger 2024-05-28 06:31

Status: Active

I'm trying to upgrade our prod Labkey docker instance from 23.3.2 to 23.11.11.

The upgrade went ok on our dev instance but the upgrade fails every time when we try on prod, leaving things in a broken state so we have to roll back the database. I've done the database check via the UI and no issues are found.

The issue seems to be related to the Experiment module :

INFO  ModuleLoader             2024-05-28T14:04:00,532           Module Upgrade : Starting module 'Pipeline'
INFO  SpringModule             2024-05-28T14:04:00,533           Module Upgrade : Loading Spring configuration for the Pipeline module from [/WEB-INF/pipelineContext.xml]
INFO  ModuleLoader             2024-05-28T14:04:00,875           Module Upgrade : Starting module 'plateWellViewer'
INFO  ModuleLoader             2024-05-28T14:04:00,876           Module Upgrade : Starting module 'prevalenceDashboard'
INFO  ModuleLoader             2024-05-28T14:04:00,877           Module Upgrade : Starting module 'Query'
172.17.0.1 - - [28/May/2024:14:04:01 +0100] "POST /admin-sql-getModuleStatus.view HTTP/1.1" 200 3357
WARN  RoleManager              2024-05-28T14:04:01,228           Module Upgrade : Could not resolve the role org.labkey.api.security.roles.TrustedAnalystRole! The role may no longer exist, or may not yet be registered.
INFO  ModuleLoader             2024-05-28T14:04:01,228           Module Upgrade : Starting module 'Experiment'
INFO  SpringModule             2024-05-28T14:04:01,229           Module Upgrade : Loading Spring configuration for the Experiment module from [/WEB-INF/experimentContext.xml]
172.17.0.1 - - [28/May/2024:14:04:02 +0100] "POST /admin-sql-getModuleStatus.view HTTP/1.1" 200 3358
172.17.0.1 - - [28/May/2024:14:04:03 +0100] "POST /admin-sql-getModuleStatus.view HTTP/1.1" 200 3358
172.17.0.1 - - [28/May/2024:14:04:04 +0100] "POST /admin-sql-getModuleStatus.view HTTP/1.1" 200 3358
127.0.0.1 - - [28/May/2024:14:04:04 +0100] "GET /_/health HTTP/2.0" 200 15
172.17.0.1 - - [28/May/2024:14:04:05 +0100] "POST /admin-sql-getModuleStatus.view HTTP/1.1" 200 3358
172.17.0.1 - - [28/May/2024:14:04:06 +0100] "POST /admin-sql-getModuleStatus.view HTTP/1.1" 200 3358
FATAL ModuleLoader             2024-05-28T14:04:06,554           Module Upgrade : Startup failure, terminating
java.lang.NullPointerException: Cannot invoke "org.labkey.api.exp.property.ValidatorKind.getName()" because the return value of "org.labkey.api.exp.property.PropertyService.getValidatorKind(String)" is null
        at org.labkey.experiment.api.property.DomainImpl$PropertyChangeAuditInfo.lambda$renderValidators$0(DomainImpl.java:1107) ~[experiment-23.11.11.jar:?]
        at java.base/java.lang.Iterable.forEach(Unknown Source) ~[?:?]
        at java.base/java.util.Collections$UnmodifiableCollection.forEach(Unknown Source) ~[?:?]
        at org.labkey.experiment.api.property.DomainImpl$PropertyChangeAuditInfo.renderValidators(DomainImpl.java:1106) ~[experiment-23.11.11.jar:?]
        at org.labkey.experiment.api.property.DomainImpl.save(DomainImpl.java:718) ~[experiment-23.11.11.jar:?]
        at org.labkey.experiment.api.property.DomainImpl.save(DomainImpl.java:521) ~[experiment-23.11.11.jar:?]
        at org.labkey.experiment.api.property.DomainImpl.save(DomainImpl.java:369) ~[experiment-23.11.11.jar:?]
        at org.labkey.api.audit.AbstractAuditTypeProvider.ensureProperties(AbstractAuditTypeProvider.java:252) ~[api-23.11.11.jar:?]
        at org.labkey.api.audit.AbstractAuditTypeProvider.initializeProvider(AbstractAuditTypeProvider.java:149) ~[api-23.11.11.jar:?]
        at org.labkey.api.audit.AbstractAuditTypeProvider.<init>(AbstractAuditTypeProvider.java:109) ~[api-23.11.11.jar:?]
        at org.labkey.experiment.samples.SampleTimelineAuditProvider.<init>(SampleTimelineAuditProvider.java:58) ~[experiment-23.11.11.jar:?]
        at org.labkey.experiment.ExperimentModule.startupAfterSpringConfig(ExperimentModule.java:509) ~[experiment-23.11.11.jar:?]
        at org.labkey.api.module.SpringModule.doStartup(SpringModule.java:122) ~[api-23.11.11.jar:?]
        at org.labkey.api.module.DefaultModule.startup(DefaultModule.java:229) ~[api-23.11.11.jar:?]
        at org.labkey.api.module.ModuleLoader.completeStartup(ModuleLoader.java:1668) ~[api-23.11.11.jar:?]
        at org.labkey.api.module.ModuleLoader.initiateModuleStartup(ModuleLoader.java:1607) ~[api-23.11.11.jar:?]
        at org.labkey.api.module.ModuleLoader.afterUpgrade(ModuleLoader.java:1824) ~[api-23.11.11.jar:?]
        at org.labkey.api.module.ModuleLoader.lambda$startNonCoreUpgradeAndStartup$13(ModuleLoader.java:1803) ~[api-23.11.11.jar:?]
        at org.labkey.api.module.ModuleUpgrader.lambda$upgrade$0(ModuleUpgrader.java:111) ~[api-23.11.11.jar:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
INFO  ContextListener          2024-05-28T14:04:06,587 gApplicationShutdownHook : Calling DefaultFileWatcher shutdownPre()
INFO  FileSystemWatcherImpl    2024-05-28T14:04:06,588       DefaultFileWatcher : FileSystemWatcherThread is terminating
INFO  ContextListener          2024-05-28T14:04:06,588 gApplicationShutdownHook : Calling PollingFileWatcher shutdownPre()
INFO  FileSystemWatcherImpl    2024-05-28T14:04:06,589       PollingFileWatcher : FileSystemWatcherThread is terminating

Anyone seen anything like this or have any advice?
(Full log attached)

Thanks,
Will

upgradelog.txt

adam responded:	2024-05-28 08:11
Hi Will, I think I know what's going on; I believe there's one or more corrupted rows in the validators table. Could you run this query directly against this instance's database (via psql, pgAdmin 4, or similar tool) and share the results? There should be a fairly small number of rows. SELECT * FROM exp.PropertyValidator Thanks, Adam

GrumpyBadger responded:	2024-05-28 09:05
That returns 873 rows, which sounds a little more than "fairly small".. I'm guessing lots of these are invalid.. Let me know if you need the full results.
Screenshot 2024-05-28 170430.png

adam responded:	2024-05-28 09:28
Okay, fair enough. :) In that case, let's try this: `SELECT DISTINCT TypeURI FROM exp.PropertyValidator` I expect we'll see 'urn:lsid:labkey.com:PropertyValidator:length' in there. Adam

GrumpyBadger responded:	2024-05-28 09:57
Yep, I get : urn:lsid:labkey.com:PropertyValidator:textchoice urn:lsid:labkey.com:PropertyValidator:regex urn:lsid:labkey.com:PropertyValidator:range urn:lsid:labkey.com:PropertyValidator:length urn:lsid:labkey.com:PropertyValidator:lookup W

adam responded:	2024-05-28 10:31
The solution is simple; execute this SQL to delete the invalid rows: `DELETE FROM exp.PropertyValidator WHERE TypeURI = 'urn:lsid:labkey.com:PropertyValidator:length'` The "length" validators were created long ago, but are no longer valid. These rows should have been deleted by the upgrade process back in 2021. Somehow, your server skipped that part of the upgrade; this could happen if the server was shut down part of the way through a past upgrade. Please let us know if this fixes the problem. Separately, I'll look at making the code resilient to unrecognized validators; we definitely shouldn't block server startup. Adam

GrumpyBadger responded:	2024-05-28 10:40
There's 16 such rows, I'll remove them and retry the upgrade.. Thanks!, W

GrumpyBadger responded:	2024-05-28 12:28
Yep, that did it! Thank you! W

GrumpyBadger responded:	2024-05-29 00:56
Spoke too soon! :( Checking Container Column References... Checking PropertyDescriptor and DomainDescriptor consistency... Checking Schema consistency with tableXML. core: OK prop: OK test: OK comm: OK pipeline: OK query: OK junit: OK exp ERROR: Table "List", column "EntireListTitleSetting" found in XML but not in database. ERROR: Table "List", column "EachItemTitleSetting" found in XML but not in database. ERROR: Table "Material", column "RecomputeRollup" found in XML but not in database. ERROR: Table "material", column "availablealiquotcount" missing from XML. ERROR: Table "material", column "availablealiquotvolume" missing from XML. ERROR: Table "material", column "storedamount" missing from XML. ERROR: Table "material", column "units" missing from XML. ERROR: Table "dataclass", column "dataparentimportaliasmap" missing from XML. ERROR: Table "ExperimentRunMaterialInputs", column "RecomputeRollup" found in XML but not in database. ERROR: Table "experimentrunmaterialinputs", column "availablealiquotcount" missing from XML. ERROR: Table "experimentrunmaterialinputs", column "availablealiquotvolume" missing from XML. ERROR: Table "experimentrunmaterialinputs", column "storedamount" missing from XML. ERROR: Table "experimentrunmaterialinputs", column "units" missing from XML. ERROR: Table "datatypeexclusion" missing from XML. assay ERROR: Table "well", column "sampleid" missing from XML. ERROR: Table "plateproperty" missing from XML. filecontent: OK issues: OK study: OK survey: OK Checking Consistency of Provisioned Storage... 916 domains use Storage Provisioner database table assayresult.c16d63_biacore_data_fields has column 'createdby' without a property descriptor database table assayresult.c16d63_biacore_data_fields has column 'created' without a property descriptor database table assayresult.c16d63_biacore_data_fields has column 'modified' without a property descriptor database table assayresult.c16d63_biacore_data_fields has column 'modifiedby' without a property descriptor See this page for more info: https://localhost:8443/Tutorials/Assay%20Tutorial/experiment-types-repair.view?domainUri=urn%3Alsid%3Alabkey.com%3AAssayDomain-Data.Folder-16%3ABiacore database table assayresult.c67d1621_qpcr_data_fields has column 'createdby' without a property descriptor database table assayresult.c67d1621_qpcr_data_fields has column 'created' without a property descriptor database table assayresult.c67d1621_qpcr_data_fields has column 'modified' without a property descriptor database table assayresult.c67d1621_qpcr_data_fields has column 'modifiedby' without a property descriptor See this page for more info: https://localhost:8443/Target%20Identification%20and%20Validation/experiment-types-repair.view?domainUri=urn%3Alsid%3Alabkey.com%3AAssayDomain-Data.Folder-67.b10d0342-e052-103c-b59a-dcb7e9d37af6%3Aqpcr database table assayresult.c179d1712_qpcr_data_fields has column 'createdby' without a property descriptor database table assayresult.c179d1712_qpcr_data_fields has column 'created' without a property descriptor database table assayresult.c179d1712_qpcr_data_fields has column 'modified' without a property descriptor database table assayresult.c179d1712_qpcr_data_fields has column 'modifiedby' without a property descriptor See this page for more info: https://localhost:8443/Molecular%20samples%20and%20assays/qPCR/experiment-types-repair.view?domainUri=urn%3Alsid%3Alabkey.com%3AAssayDomain-Data.Folder-179%3AqPCR database table assayresult.c57d471_cell_culture_data_fields has column 'createdby' without a property descriptor database table assayresult.c57d471_cell_culture_data_fields has column 'created' without a property descriptor database table assayresult.c57d471_cell_culture_data_fields has column 'modified' without a property descriptor database table assayresult.c57d471_cell_culture_data_fields has column 'modifiedby' without a property descriptor See this page for more info: https://localhost:8443/Tutorials/Tutorials%20Katy/Assay%20tutorial/experiment-types-repair.view?domainUri=urn%3Alsid%3Alabkey.com%3AAssayDomain-Data.Folder-57%3ACell%2BCulture database table assayresult.c16d1218_test_data_fields has column 'createdby' without a property descriptor database table assayresult.c16d1218_test_data_fields has column 'created' without a property descriptor database table assayresult.c16d1218_test_data_fields has column 'modified' without a property descriptor database table assayresult.c16d1218_test_data_fields has column 'modifiedby' without a property descriptor See this page for more info: https://localhost:8443/Tutorials/Assay%20Tutorial/experiment-types-repair.view?domainUri=urn%3Alsid%3Alabkey.com%3AAssayDomain-Data.Folder-16%3Atest database table assayresult.c16d66_cell_culture_modified_data_fields has column 'createdby' without a property descriptor database table assayresult.c16d66_cell_culture_modified_data_fields has column 'created' without a property descriptor database table assayresult.c16d66_cell_culture_modified_data_fields has column 'modified' without a property descriptor database table assayresult.c16d66_cell_culture_modified_data_fields has column 'modifiedby' without a property descriptor See this page for more info: https://localhost:8443/Tutorials/Assay%20Tutorial/experiment-types-repair.view?domainUri=urn%3Alsid%3Alabkey.com%3AAssayDomain-Data.Folder-16%3ACell%2BCulture%2BModified database table assayresult.c26d408_cloning_elispot_data_fields has column 'createdby' without a property descriptor database table assayresult.c26d408_cloning_elispot_data_fields has column 'created' without a property descriptor database table assayresult.c26d408_cloning_elispot_data_fields has column 'modified' without a property descriptor database table assayresult.c26d408_cloning_elispot_data_fields has column 'modifiedby' without a property descriptor See this page for more info: https://localhost:8443/Tutorials/Caroline%20Test/experiment-types-repair.view?domainUri=urn%3Alsid%3Alabkey.com%3AAssayDomain-Data.Folder-26%3ACloning%2BELISpot database table assayresult.c16d56_cel_culture_data_fields has column 'createdby' without a property descriptor database table assayresult.c16d56_cel_culture_data_fields has column 'created' without a property descriptor database table assayresult.c16d56_cel_culture_data_fields has column 'modified' without a property descriptor database table assayresult.c16d56_cel_culture_data_fields has column 'modifiedby' without a property descriptor See this page for more info: https://localhost:8443/Tutorials/Assay%20Tutorial/experiment-types-repair.view?domainUri=urn%3Alsid%3Alabkey.com%3AAssayDomain-Data.Folder-16%3ACel%2BCulture database table assayresult.c57d478_cell_culture_modified_data_fields has column 'createdby' without a property descriptor database table assayresult.c57d478_cell_culture_modified_data_fields has column 'created' without a property descriptor database table assayresult.c57d478_cell_culture_modified_data_fields has column 'modified' without a property descriptor database table assayresult.c57d478_cell_culture_modified_data_fields has column 'modifiedby' without a property descriptor See this page for more info: https://localhost:8443/Tutorials/Tutorials%20Katy/Assay%20tutorial/experiment-types-repair.view?domainUri=urn%3Alsid%3Alabkey.com%3AAssayDomain-Data.Folder-57%3ACell%2BCulture%2BModified Database Consistency checker complete

adam responded:	2024-05-29 15:19
Well, that's not good (obviously). It seems like you have a current deployment problem in addition to past upgrade problems. Normally, a Premium Edition subscription provides the avenue for us to rectify problems like this (and maximize the value of your deployment). I will reach out privately (via email) to see if we can get you up and running again with a minimal effort. Adam

adam responded:	2024-05-30 11:35
Status: Closed
This issue was resolved and the server is running correctly now. The OP's deployment was getting unintentionally "downgraded" to an earlier LabKey Server version by an automated nightly process. This explains the schema inconsistencies that were appearing every morning.