In case of errors or other problems when installing and running LabKey Server, first review installation basics and options linked in the topic:
Install LabKey. This topic provides additional troubleshooting suggestions if the instructions in that topic do not resolve the issue.
LabKey Logs
From time to time Administrators may need to review the LabKey Tomcat logs to troubleshoot system startup or other issues. The LabKey Tomcat logs are located at:
The logs of interest are:
- catalina.out: Contains the log file entries from Tomcat as it is starting the LabKey application.
- labkey.log: Once the application starts, logging of INFO, ERROR, WARN and other messages is recorded here.
- labkey-errors.log: Contains ERROR messages.
Useful commands:
# monitor the catalina.out log file while LabKey/Tomcat starts
sudo tail -f /usr/local/labkey/apps/tomcat/logs/catalina.out
# monitor the labkey.log file while LabKey/Tomcat starts
sudo tail -f /usr/local/labkey/apps/tomcat/logs/labkey.log
Log Rotation and Storage
To avoid massive log files and retain recent information with some context, log files are rotated (or "rolled-over") into new files to keep them manageable. These rotated log files are named with a trailing number and shuffled such that .1 will be the most recently rolled over log file, and .7 or .3 the oldest log of its type.
labkey.log: Rotated when the file reaches 10MB. Creates labkey.log.1, labkey.log.2, etc. Up to 7 archived files (80MB total).
labkey-errors.log: Rotated on server startup or at 100MB. Up to 3 labkey-errors log files are rotated (labkey-errors.log.1, etc.) If the server is about to delete the first labkey-errors file from the current session (meaning it has generated hundreds of megabytes of errors since it started up), it will retain that first log file as labkey-errors-YYYY-MM-DD.log. This can be useful in determining a root cause of the many errors.
To save additional disk space, older log archives can be compressed for storage. A script to regularly store and compress them can be helpful.
Developer Mode
Running a server in development mode ("devmode") provides enhanced logging and enables the
MiniProfiler, and has the server prevent the browser from caching resources like JavaScript and CSS files as aggressively. You do not want to run your production server in devmode, but might set up a shared development server to support future work, or have each developer work on their own local dev machine.
To check whether the server is running in devmode:
- Go to (Admin) > Site > Admin Console.
- Under Diagnostics, click System Properties.
- Check the value of the devmode property.
Set -Ddevmode=true
If you are using a
local development machine, include the following in your VM options as part of your intelliJ configuration:
If you are not using a development machine, you can set this option follow these steps. The
supported version of tomcat will change over time; replace the # in our examples with the actual version number.
- Open a command prompt.
- Go to the <CATALINA_HOME>/bin directory, for example:
C:\labkey\apps\apache\apache-tomcat-#.#.##\bin
- Execute the tomcat#w.exe program:
tomcat#w.exe //ES//LabKeyTomcat#
- The command will open a program window. Click the Java tab.
- In the Java Option box, scroll to the bottom of the properties. Add the following property at the bottom of the list:
- Close the program window and restart the server.
JVM Caching
Note that the "caching" JVM system property can also be used to control just the caching behavior, without all of the other devmode behaviors. To disallow the normal caching, perhaps because files are being updated directly on the file system, add -Dcaching=false to the JVM arguments.
Diagnostic Information
Which Version of LabKey Server Is Running?
- Find your version number at (Admin) > Site > Admin Console.
- At the top of the Server Information panel, you will see the release version.
Learn more about other diagnostic information on the admin console in this topic:
Common Issues
Compatible Component Versions
Confirm that you are using the supported versions of the required components, as detailed in the
Supported Technologies Roadmap. It is possible to have multiple versions of some software, like Java, installed at the same time. Check that LabKey and other applications, such as Tomcat, are configured to use the correct versions.
For example, if you see an error similar to "...this version of the Java Runtime only recognizes class file versions up to ##.0..." it likely means that Tomcat is running under an older version of the JDK than is supported.
Connection Pool Size
If your server becomes unresponsive, it could be due to the depletion of available connections to the database. Watch for a
Connection Pool Size of 8, which is the Tomcat connection pool default size and insufficient for a production server. To see the connection pool size for the LabKey data source, select
(Admin) > Site > Admin Console and check the setting of
Connection Pool Size on the
Server Information panel. The connection pool size for every data source is also logged at server startup.
To set the connection pool size, edit your labkey.xml/ROOT.xml configuration file and change the "maxTotal" setting for your LabKey data source to at least 20. Depending on the number of simultaneous users and the complexity of their requests, your deployment may require a larger connection pool size.
You should also consider changing this setting for external data sources to match the usage you expect. Learn more in this topic:
External Schemas and Data Sources.
Deleted or Deactivated Users
Over time, you may have personnel leave the organization. Such user accounts should be deactivated and not deleted. If you encounter problems with accessing linked schemas, external schemas, running ETLs, or similar, check the logs to see if the former user may have "owned" these long term resources.
Tomcat Failure to Start
If installation fails to start tomcat (such as with an error like "The specified service already exists."), you may need to manually stop or delete a failing Tomcat service.
To stop the service on Windows, open
Control Panel > Administrative Tools > Services. Select the relevant service, such as
LabKey Server Apache Tomcat #.0 and click
Stop.
To delete, run the following from the command line as an administrator, substituting the major version number for the #:
Properties in Config Files
Check that the labkeyDataSource and any external data source properties in your config file (labkey.xml or ROOT.xml) match those expected of the current component versions. Watch the catalina.out log for warning messages like "Ignoring unknown property..." or "...is being ignored".
For example, older versions of tomcat used "maxActive" and "maxWait" and newer versions use "maxTotal" and "maxWaitMillis". Error messages like this would be raised if labkey.xml/ROOT.xml used the older versions:
17-Feb-2022 11:33:42.415 WARNING [main] java.util.ArrayList.forEach Name = labkeyDataSource Property maxActive is not used in DBCP2, use maxTotal instead. maxTotal default value is 8. You have set value of "20" for "maxActive" property, which is being ignored.
17-Feb-2022 11:33:42.415 WARNING [main] java.util.ArrayList.forEach Name = labkeyDataSource Property maxWait is not used in DBCP2 , use maxWaitMillis instead. maxWaitMillis default value is PT-0.001S. You have set value of "120000" for "maxWait" property, which is being ignored.
Enabling SSL is done by including "SSLEnabled=true" in the server.xml config file for tomcat. If you see this error, you may have tried to set the nonexistent "ssl" property in the labkey.xml or ROOT.xml configuration file.
19-Feb-2022 15:28:29.624 INFO [main] java.util.ArrayList.forEach Name = labkeyDataSource Ignoring unknown property: value of "true" for "ssl" property
SSL Errors
An error like "ERR_CONNECTION_CLOSED" during startup, particularly after upgrading your version of Tomcat, may indicate that necessary files are missing. Check that you copied the server.xml and web.xml as well as the SSL directory containing the keys. Check the logs for "SEVERE" to find any messages that might look like:
19-Feb-2022 15:28:13.312 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8443]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
...
org.apache.catalina.LifecycleException: The configured protocol [org.apache.coyote.http11.Http11AprProtocol] requires the APR/native library which is not available
Disable sslRequired
If you configure "SSL Required" and then cannot access your server, you will not be able to disable this requirement in the user interface. To resolve this, you can directly reset this property in your database.
Confirm that this query will select the "sslRequired" property row in your database:
SELECT * FROM prop.Properties WHERE Name = 'sslRequired' AND Set = (SELECT Set FROM prop.PropertySets WHERE Category = 'SiteConfig')
You should see a single row with a name of "sslRequired" and a value of "true". To change that value to false, run this update query:
UPDATE prop.Properties SET Value = FALSE WHERE Name = 'sslRequired' AND Set = (SELECT Set FROM prop.PropertySets WHERE Category = 'SiteConfig')
You can then verify with the SELECT query above to confirm the update.
Once the value is changed you'll need to restart the server to force the new value into the cache.
If you are using SQL Server, you'll need to double quote every reference to "Set" because that database considers Set a keyword.
Conflicting Applications
If you have problems during installation, retry after shutting down all other running applications. Specifically, you may need to try temporarily shutting down any virus scanning application, internet security applications, or other applications that run in the background to see if this resolves the issue.
PostgreSQL Installation
You may need to remove references to Cygwin from your Windows system path before installing LabKey, due to conflicts with the PostgreSQL installer. The PostgreSQL installer also conflicts with some antivirus or firewalls. (see
http://wiki.postgresql.org/wiki/Running_%26_Installing_PostgreSQL_On_Native_Windows for more information).
Restart Installation from Scratch
If you have encountered prior failed installations, don't have any stored data you need to keep, and want to clean up and start completely from scratch, the following process may be useful:
- Delete the Tomcat service (if it seems to be causing the failure).
- Uninstall PostgreSQL using their uninstaller.
- Control Panel > Programs and Features
- Select PostgreSQL program.
- Click Uninstall.
- Delete the entire LabKey installation directory.
- Install LabKey again.
Graphviz
If you encounter the following error, or a similar error mentioning Graphviz:
Unable to display graph view: cannot run dot due to an error.
Cannot run program "dot" (in directory "./temp/ExperimentRunGraphs"): CreateProcess error=2, The system cannot find the file specified.
then install Graphviz according to the following instructions:
Installation: Third-Party Components
Support Forum
You can search for problems resolved through community support in the
LabKey Support Forum.
If you don't see your issue listed there, you can post a new question. If the install seems successful, it is often helpful to submit
debugging logs for diagnosis.
If the install failed to complete, please include the install.log and install-debug.log from your selected LabKey install directory.
PostgreSQL logs its installation process separately. If PostgreSQL installation/upgrade fails, please locate and include the PostgreSQL install logs as well.
Related Topics
