Tomcat homepage at http://chembiolims.pnl.gov:8080 is correctly served: 130.20.181.90 - - [07/Apr/2017:13:07:06 -0700] "GET / HTTP/1.1" 200 11250 130.20.181.90 - - [07/Apr/2017:13:07:06 -0700] "GET /tomcat.css HTTP/1.1" 304 - 130.20.181.90 - - [07/Apr/2017:13:07:07 -0700] "GET /tomcat.png HTTP/1.1" 304 - 130.20.181.90 - - [07/Apr/2017:13:07:07 -0700] "GET /bg-nav.png HTTP/1.1" 304 - 130.20.181.90 - - [07/Apr/2017:13:07:07 -0700] "GET /asf-logo-wide.svg HTTP/1.1" 304 - 130.20.181.90 - - [07/Apr/2017:13:07:07 -0700] "GET /bg-upper.png HTTP/1.1" 304 - 130.20.181.90 - - [07/Apr/2017:13:07:07 -0700] "GET /bg-button.png HTTP/1.1" 304 - 130.20.181.90 - - [07/Apr/2017:13:07:07 -0700] "GET /bg-middle.png HTTP/1.1" 304 - 130.20.181.90 - - [07/Apr/2017:13:07:12 -0700] "GET / HTTP/1.1" 200 11250 130.20.181.90 - - [07/Apr/2017:13:07:12 -0700] "GET /favicon.ico HTTP/1.1" 200 21630 130.20.181.90 - - [07/Apr/2017:13:07:13 -0700] "GET / HTTP/1.1" 200 11250 130.20.181.90 - - [07/Apr/2017:13:07:13 -0700] "GET /favicon.ico HTTP/1.1" 200 21630 Requesting Labkey homepage at http://chembiolims.pnl.gov:8080/labkey/ returns error: 130.20.181.90 - - [07/Apr/2017:13:12:54 -0700] "GET /labkey HTTP/1.1" 302 - 130.20.181.90 - - [07/Apr/2017:13:12:55 -0700] "GET /labkey/ HTTP/1.1" 500 6084 HTTP output: 500: Unexpected server error A failure occurred during LabKey Server startup. org.labkey.api.data.SqlScriptRunner$SqlScriptException: exp-15.10-15.20.sql : SqlExecutor.execute(); bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:220) at org.labkey.api.data.SqlScriptRunner.runScripts(SqlScriptRunner.java:82) at org.labkey.api.module.DefaultModule.versionUpdate(DefaultModule.java:332) at org.labkey.api.module.ModuleUpgrader.upgrade(ModuleUpgrader.java:82) at org.labkey.api.module.ModuleUpgrader.lambda$upgrade$0(ModuleUpgrader.java:94) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.jdbc.BadSqlGrammarException: SqlExecutor.execute(); bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:97) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:37) at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:31) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:129) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:75) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:70) at org.labkey.api.data.SqlScriptExecutor$Block.execute(SqlScriptExecutor.java:208) at org.labkey.api.data.SqlScriptExecutor.lambda$execute$0(SqlScriptExecutor.java:101) at java.util.ArrayList.forEach(ArrayList.java:1249) at org.labkey.api.data.SqlScriptExecutor.execute(SqlScriptExecutor.java:98) at org.labkey.api.data.dialect.SqlDialect.runSql(SqlDialect.java:581) at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:215) ... 5 more -----------------labkey.log entries since tomcat reboot this morning at 11:38-------------------------------- 1788 INFO ContextListener 2017-04-07 11:38:45,001 localhost-startStop-2 : Calling HeartBeatThread shutdownPre() 1789 INFO ContextListener 2017-04-07 11:38:45,001 localhost-startStop-2 : Calling FileSystemWatcherThread shutdownPre() 1790 INFO ContextListener 2017-04-07 11:38:45,001 localhost-startStop-2 : Calling QueryProfilerThread shutdownPre() 1791 INFO ContextListener 2017-04-07 11:38:45,001 localhost-startStop-2 : Calling BreakpointThread shutdownPre() 1792 INFO ContextListener 2017-04-07 11:38:45,001 localhost-startStop-2 : Calling ThumbnailGeneratingThread shutdownPre() 1793 INFO ContextListener 2017-04-07 11:38:45,001 localhost-startStop-2 : Calling QuerySnapshotDependencyThread shutdownPre() 1794 INFO ContextListener 2017-04-07 11:38:45,001 localhost-startStop-2 : Calling Job Runner (Default) shutdownPre() 1795 INFO FileSystemWatcherImpl 2017-04-07 11:38:45,001 FileSystemWatcherThread : FileSystemWatcherThread is terminating 1796 INFO ContextListener 2017-04-07 11:38:45,001 localhost-startStop-2 : Calling Job Runner (Mothership Reporting) shutdownPre() 1797 INFO ContextListener 2017-04-07 11:38:45,002 localhost-startStop-2 : Calling HeartBeatThread shutdownStarted() 1798 INFO DatasetSnapshotProvider 2017-04-07 11:38:45,002 SnapshotDependencyThread : QuerySnapshotDependencyThread is terminating due to interruption 1799 INFO ContextListener 2017-04-07 11:38:45,002 localhost-startStop-2 : Calling FileSystemWatcherThread shutdownStarted() 1800 INFO ContextListener 2017-04-07 11:38:45,002 localhost-startStop-2 : Calling QueryProfilerThread shutdownStarted() 1801 INFO QueryProfiler 2017-04-07 11:38:45,002 localhost-startStop-2 : Starting to log statistics for queries prior to web application shut down 1802 INFO QueryProfiler 2017-04-07 11:38:45,018 localhost-startStop-2 : Completed logging statistics for queries prior to web application shut down 1803 INFO ContextListener 2017-04-07 11:38:45,018 localhost-startStop-2 : Calling BreakpointThread shutdownStarted() 1804 INFO ContextListener 2017-04-07 11:38:45,018 localhost-startStop-2 : Calling ThumbnailGeneratingThread shutdownStarted() 1805 INFO ContextListener 2017-04-07 11:38:45,018 localhost-startStop-2 : Calling QuerySnapshotDependencyThread shutdownStarted() 1806 INFO ContextListener 2017-04-07 11:38:45,018 localhost-startStop-2 : Calling Job Runner (Default) shutdownStarted() 1807 INFO ContextListener 2017-04-07 11:38:45,018 localhost-startStop-2 : Calling Job Runner (Mothership Reporting) shutdownStarted() 1808 INFO EhCacheProvider 2017-04-07 11:38:45,029 localhost-startStop-2 : Shutting down Ehcache 1809 INFO ModuleLoader 2017-04-07 11:39:05,179 localhost-startStop-1 : 1810 __ 1811 || | _ |_ |/ _ (¯ _ _ _ _ 1812 (__) |_(_||_)|\(/_\/ _)(/_| \/(/_| 1813 / 1814 INFO DbScope 2017-04-07 11:39:07,496 localhost-startStop-1 : Initializing DbScope with the following configuration: 1815 DataSource Name: labkeyDataSource 1816 Server URL: jdbc:postgresql://localhost/labkey 1817 Database Product Name: PostgreSQL 1818 Database Product Version: 9.3.16 1819 JDBC Driver Name: PostgreSQL Native Driver 1820 JDBC Driver Version: PostgreSQL 9.4.1212 1821 SQL Dialect: PostgreSql93Dialect 1822 INFO DbScope 2017-04-07 11:39:07,612 localhost-startStop-1 : Loading DbSchema "labkey.core" (Module) 1823 INFO DbScope 2017-04-07 11:39:08,234 localhost-startStop-1 : Loading DbSchema "labkey.exp" (Module) 1824 INFO DbScope 2017-04-07 11:39:11,564 localhost-startStop-1 : Loading DbSchema "labkey.prop" (Module) 1825 INFO DbScope 2017-04-07 11:39:11,645 localhost-startStop-1 : Loading DbSchema "labkey.test" (Module) 1826 INFO DbScope 2017-04-07 11:39:11,651 localhost-startStop-1 : Loading DbSchema "labkey.temp" (Provisioned) 1827 INFO DbScope 2017-04-07 11:39:11,654 localhost-startStop-1 : Loading DbSchema "labkey.pipeline" (Module) 1828 INFO ModuleLoader 2017-04-07 11:39:11,665 localhost-startStop-1 : Modules requiring upgrade: [Experiment, Audit, DataIntegration, List, ELISpotAssay, Microarray, Query, Flow, MS2, MS1, Search, Survey, TargetedMS, Wiki, Announcements, FileContent, Issues, Study, Elisa, Luminex, Nab, Visualization] 1829 INFO ModuleLoader 2017-04-07 11:39:11,678 localhost-startStop-1 : LabKey Server startup is complete; modules are being upgraded and initialized in the background 1830 INFO ViewServlet 2017-04-07 11:39:11,681 localhost-startStop-1 : ViewServlet initialized 1831 INFO ViewServlet 2017-04-07 11:39:11,681 localhost-startStop-1 : WEBAPP: /usr/local/labkey/labkeywebapp/ 1832 INFO ViewServlet 2017-04-07 11:39:11,681 localhost-startStop-1 : PATH: /sbin:/bin:/usr/sbin:/usr/bin 1833 INFO DbScope 2017-04-07 11:39:12,122 Module Upgrade : Loading DbSchema "labkey.expdataclass" (Provisioned) 1834 INFO SqlScriptRunner 2017-04-07 11:39:12,141 Module Upgrade : Running [exp-15.10-15.20.sql, exp-15.30-16.10.sql, exp-16.10-16.20.sql, exp-16.30-17.10.sql] 1835 INFO SqlScriptManager 2017-04-07 11:39:12,150 Module Upgrade : start running script : exp-15.10-15.20.sql 1836 ERROR Table 2017-04-07 11:39:12,255 Module Upgrade : SQL Exception 1837 org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist 1838 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455) 1839 at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155) 1840 at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288) 1841 at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430) 1842 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356) 1843 at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303) 1844 at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289) 1845 at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266) 1846 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:262) 1847 at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) 1848 at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) 1849 at org.labkey.api.data.dialect.StatementWrapper.execute(StatementWrapper.java:1908) 1850 at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:156) 1851 at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:144) 1852 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:122) 1853 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:75) 1854 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:70) 1855 at org.labkey.api.data.SqlScriptExecutor$Block.execute(SqlScriptExecutor.java:208) 1856 at org.labkey.api.data.SqlScriptExecutor.lambda$execute$0(SqlScriptExecutor.java:101) 1857 at java.util.ArrayList.forEach(ArrayList.java:1249) 1858 at org.labkey.api.data.SqlScriptExecutor.execute(SqlScriptExecutor.java:98) 1859 at org.labkey.api.data.dialect.SqlDialect.runSql(SqlDialect.java:581) 1860 at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:215) 1861 at org.labkey.api.data.SqlScriptRunner.runScripts(SqlScriptRunner.java:82) 1862 at org.labkey.api.module.DefaultModule.versionUpdate(DefaultModule.java:332) 1863 at org.labkey.api.module.ModuleUpgrader.upgrade(ModuleUpgrader.java:82) 1864 at org.labkey.api.module.ModuleUpgrader.lambda$upgrade$0(ModuleUpgrader.java:94) 1865 at java.lang.Thread.run(Thread.java:745) 1866 ERROR Table 2017-04-07 11:39:12,258 Module Upgrade : SQL [30767] 1867 SELECT core.fn_dropifexists('material', 'exp', 'INDEX', 'idx_material_AK'); 1868 1869 1870 UPDATE exp.material SET name='NULL' WHERE name IS NULL; 1871 ALTER TABLE exp.material ALTER COLUMN name SET NOT NULL; 1872 1873 WITH nonunique AS ( 1874 SELECT container, cpastype, name 1875 FROM exp.material 1876 WHERE cpastype IS NOT NULL 1877 GROUP BY container, cpastype, name 1878 HAVING COUNT(*) > 1) 1879 1880 UPDATE exp.material M 1881 SET Name = Name || ' - ' || RowId 1882 WHERE EXISTS (SELECT * FROM nonunique NU WHERE M.container=NU.container AND M.cpastype=NU.cpastype AND M.name=NU.name); 1883 1884 CREATE UNIQUE INDEX idx_material_AK ON exp.material (container, cpastype, name) WHERE cpastype IS NOT NULL; 1885 1886 1887 1888 ALTER TABLE exp.PropertyDescriptor RENAME COLUMN KeyVariable TO RecommendedVariable; 1889 1890 1891 1892 ALTER TABLE exp.Data ADD COLUMN Generated BOOLEAN NOT NULL DEFAULT FALSE; 1893 1894 org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:126) 1895 org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:75) 1896 org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:70) 1897 org.labkey.api.data.SqlScriptExecutor$Block.execute(SqlScriptExecutor.java:208) 1898 org.labkey.api.data.SqlScriptExecutor.lambda$execute$0(SqlScriptExecutor.java:101) 1899 ERROR ModuleUpgrader 2017-04-07 11:39:12,403 Module Upgrade : Failure during module upgrade 1900 org.labkey.api.data.SqlScriptRunner$SqlScriptException: exp-15.10-15.20.sql : SqlExecutor.execute(); bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist 1901 at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:220) 1902 at org.labkey.api.data.SqlScriptRunner.runScripts(SqlScriptRunner.java:82) 1903 at org.labkey.api.module.DefaultModule.versionUpdate(DefaultModule.java:332) 1904 at org.labkey.api.module.ModuleUpgrader.upgrade(ModuleUpgrader.java:82) 1905 at org.labkey.api.module.ModuleUpgrader.lambda$upgrade$0(ModuleUpgrader.java:94) 1906 at java.lang.Thread.run(Thread.java:745) 1907 Caused by: org.springframework.jdbc.BadSqlGrammarException: SqlExecutor.execute(); bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist 1908 at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:97) 1909 at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) 1910 at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) 1911 at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) 1912 at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:37) 1913 at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:31) 1914 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:129) 1915 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:75) 1916 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:70) 1917 at org.labkey.api.data.SqlScriptExecutor$Block.execute(SqlScriptExecutor.java:208) 1918 at org.labkey.api.data.SqlScriptExecutor.lambda$execute$0(SqlScriptExecutor.java:101) 1919 at java.util.ArrayList.forEach(ArrayList.java:1249) 1920 at org.labkey.api.data.SqlScriptExecutor.execute(SqlScriptExecutor.java:98) 1921 at org.labkey.api.data.dialect.SqlDialect.runSql(SqlDialect.java:581) 1922 at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:215) 1923 ... 5 more 1924 Caused by: org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist 1925 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455) 1926 at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155) 1927 at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288) 1928 at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430) 1929 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356) 1930 at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303) 1931 at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289) 1932 at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266) 1933 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:262) 1934 at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) 1935 at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) 1936 at org.labkey.api.data.dialect.StatementWrapper.execute(StatementWrapper.java:1908) 1937 at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:156) 1938 at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:144) 1939 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:122) 1940 ... 13 more 1941 ERROR ExceptionUtil 2017-04-07 13:12:54,673 http-nio-8080-exec-10 : Unhandled exception: 500: Unexpected server error 1942 org.labkey.api.data.SqlScriptRunner$SqlScriptException: exp-15.10-15.20.sql : SqlExecutor.execute(); bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist 1943 at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:220) 1944 at org.labkey.api.data.SqlScriptRunner.runScripts(SqlScriptRunner.java:82) 1945 at org.labkey.api.module.DefaultModule.versionUpdate(DefaultModule.java:332) 1946 at org.labkey.api.module.ModuleUpgrader.upgrade(ModuleUpgrader.java:82) 1947 at org.labkey.api.module.ModuleUpgrader.lambda$upgrade$0(ModuleUpgrader.java:94) 1948 at java.lang.Thread.run(Thread.java:745) 1949 Caused by: org.springframework.jdbc.BadSqlGrammarException: SqlExecutor.execute(); bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist 1950 at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:97) 1951 at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) 1952 at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) 1953 at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) 1954 at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:37) 1955 at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:31) 1956 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:129) 1957 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:75) 1958 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:70) 1959 at org.labkey.api.data.SqlScriptExecutor$Block.execute(SqlScriptExecutor.java:208) 1960 at org.labkey.api.data.SqlScriptExecutor.lambda$execute$0(SqlScriptExecutor.java:101) 1961 at java.util.ArrayList.forEach(ArrayList.java:1249) 1962 at org.labkey.api.data.SqlScriptExecutor.execute(SqlScriptExecutor.java:98) 1963 at org.labkey.api.data.dialect.SqlDialect.runSql(SqlDialect.java:581) 1964 at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:215) 1965 ... 5 more 1966 Caused by: org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist 1967 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455) 1968 at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155) 1969 at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288) 1970 at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430) 1971 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356) 1972 at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303) 1973 at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289) 1974 at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266) 1975 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:262) 1976 at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) 1977 at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) 1978 at org.labkey.api.data.dialect.StatementWrapper.execute(StatementWrapper.java:1908) 1979 at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:156) 1980 at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:144) 1981 at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:122) 1982 ... 13 more [root@chembiolims logs]# Caused by: org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303) at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:262) at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) at org.labkey.api.data.dialect.StatementWrapper.execute(StatementWrapper.java:1908) at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:156) at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:144) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:122) ... 13 more -----------labkey-errors.log from today---------------------------------- [root@chembiolims logs]# tail -n 400 labkey-errors.log | less [root@chembiolims logs]# cat labkey-errors.log | less [root@chembiolims logs]# [root@chembiolims logs]# clear [root@chembiolims logs]# cat labkey-errors.log ERROR Table 2017-04-07 11:39:12,255 Module Upgrade : SQL Exception org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303) at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:262) at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) at org.labkey.api.data.dialect.StatementWrapper.execute(StatementWrapper.java:1908) at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:156) at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:144) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:122) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:75) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:70) at org.labkey.api.data.SqlScriptExecutor$Block.execute(SqlScriptExecutor.java:208) at org.labkey.api.data.SqlScriptExecutor.lambda$execute$0(SqlScriptExecutor.java:101) at java.util.ArrayList.forEach(ArrayList.java:1249) at org.labkey.api.data.SqlScriptExecutor.execute(SqlScriptExecutor.java:98) at org.labkey.api.data.dialect.SqlDialect.runSql(SqlDialect.java:581) at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:215) at org.labkey.api.data.SqlScriptRunner.runScripts(SqlScriptRunner.java:82) at org.labkey.api.module.DefaultModule.versionUpdate(DefaultModule.java:332) at org.labkey.api.module.ModuleUpgrader.upgrade(ModuleUpgrader.java:82) at org.labkey.api.module.ModuleUpgrader.lambda$upgrade$0(ModuleUpgrader.java:94) at java.lang.Thread.run(Thread.java:745) ERROR Table 2017-04-07 11:39:12,258 Module Upgrade : SQL [30767] SELECT core.fn_dropifexists('material', 'exp', 'INDEX', 'idx_material_AK'); UPDATE exp.material SET name='NULL' WHERE name IS NULL; ALTER TABLE exp.material ALTER COLUMN name SET NOT NULL; WITH nonunique AS ( SELECT container, cpastype, name FROM exp.material WHERE cpastype IS NOT NULL GROUP BY container, cpastype, name HAVING COUNT(*) > 1) UPDATE exp.material M SET Name = Name || ' - ' || RowId WHERE EXISTS (SELECT * FROM nonunique NU WHERE M.container=NU.container AND M.cpastype=NU.cpastype AND M.name=NU.name); CREATE UNIQUE INDEX idx_material_AK ON exp.material (container, cpastype, name) WHERE cpastype IS NOT NULL; ALTER TABLE exp.PropertyDescriptor RENAME COLUMN KeyVariable TO RecommendedVariable; ALTER TABLE exp.Data ADD COLUMN Generated BOOLEAN NOT NULL DEFAULT FALSE; org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:126) org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:75) org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:70) org.labkey.api.data.SqlScriptExecutor$Block.execute(SqlScriptExecutor.java:208) org.labkey.api.data.SqlScriptExecutor.lambda$execute$0(SqlScriptExecutor.java:101) ERROR ModuleUpgrader 2017-04-07 11:39:12,403 Module Upgrade : Failure during module upgrade org.labkey.api.data.SqlScriptRunner$SqlScriptException: exp-15.10-15.20.sql : SqlExecutor.execute(); bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:220) at org.labkey.api.data.SqlScriptRunner.runScripts(SqlScriptRunner.java:82) at org.labkey.api.module.DefaultModule.versionUpdate(DefaultModule.java:332) at org.labkey.api.module.ModuleUpgrader.upgrade(ModuleUpgrader.java:82) at org.labkey.api.module.ModuleUpgrader.lambda$upgrade$0(ModuleUpgrader.java:94) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.jdbc.BadSqlGrammarException: SqlExecutor.execute(); bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:97) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:37) at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:31) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:129) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:75) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:70) at org.labkey.api.data.SqlScriptExecutor$Block.execute(SqlScriptExecutor.java:208) at org.labkey.api.data.SqlScriptExecutor.lambda$execute$0(SqlScriptExecutor.java:101) at java.util.ArrayList.forEach(ArrayList.java:1249) at org.labkey.api.data.SqlScriptExecutor.execute(SqlScriptExecutor.java:98) at org.labkey.api.data.dialect.SqlDialect.runSql(SqlDialect.java:581) at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:215) ... 5 more Caused by: org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303) at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:262) at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) at org.labkey.api.data.dialect.StatementWrapper.execute(StatementWrapper.java:1908) at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:156) at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:144) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:122) ... 13 more ERROR ExceptionUtil 2017-04-07 13:12:54,673 http-nio-8080-exec-10 : Unhandled exception: 500: Unexpected server error org.labkey.api.data.SqlScriptRunner$SqlScriptException: exp-15.10-15.20.sql : SqlExecutor.execute(); bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:220) at org.labkey.api.data.SqlScriptRunner.runScripts(SqlScriptRunner.java:82) at org.labkey.api.module.DefaultModule.versionUpdate(DefaultModule.java:332) at org.labkey.api.module.ModuleUpgrader.upgrade(ModuleUpgrader.java:82) at org.labkey.api.module.ModuleUpgrader.lambda$upgrade$0(ModuleUpgrader.java:94) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.jdbc.BadSqlGrammarException: SqlExecutor.execute(); bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:97) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:37) at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:31) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:129) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:75) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:70) at org.labkey.api.data.SqlScriptExecutor$Block.execute(SqlScriptExecutor.java:208) at org.labkey.api.data.SqlScriptExecutor.lambda$execute$0(SqlScriptExecutor.java:101) at java.util.ArrayList.forEach(ArrayList.java:1249) at org.labkey.api.data.SqlScriptExecutor.execute(SqlScriptExecutor.java:98) at org.labkey.api.data.dialect.SqlDialect.runSql(SqlDialect.java:581) at org.labkey.api.data.SqlScriptManager.runScript(SqlScriptManager.java:215) ... 5 more Caused by: org.postgresql.util.PSQLException: ERROR: column "keyvariable" does not exist at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303) at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:262) at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) at org.apache.tomcat.dbcp.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:294) at org.labkey.api.data.dialect.StatementWrapper.execute(StatementWrapper.java:1908) at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:156) at org.labkey.api.data.SqlExecutor$NormalStatementExecutor.execute(SqlExecutor.java:144) at org.labkey.api.data.SqlExecutor.execute(SqlExecutor.java:122) ... 13 more [root@chembiolims logs]#