In case of errors or other problems when installing and running LabKey Server, first review installation basics and options linked in the topic: Install LabKey
. This topic provides additional troubleshooting suggestions if the instructions in that topic do not resolve the issue.
From time to time Administrators may need to review the LabKey Tomcat logs to troubleshoot system startup or other issues. The LabKey Tomcat logs are located in the <CATALINA_HOME>/logs directory. CATALINA_HOME is the location where you installed Tomcat. The path might be similar to one of these
The logs of interest are:
- catalina.out: Contains the log file entries from Tomcat as it is starting the LabKey application.
- labkey.log: Once the application starts, logging of INFO, ERROR, WARN and other messages is recorded here.
- labkey-errors.log: Contains ERROR messages.
# monitor the catalina.out log file while LabKey/Tomcat starts
sudo tail -f /usr/local/labkey/apps/tomcat/logs/catalina.out
# monitor the labkey.log file while LabKey/Tomcat starts
sudo tail -f /usr/local/labkey/apps/tomcat/logs/labkey.log
Log Rotation and Storage
To avoid massive log files and retain recent information with some context, log files are rotated (or "rolled-over") into new files to keep them manageable. These rotated log files are named with a trailing number and shuffled such that .1 will be the most recently rolled over log file, and .7 or .3 the oldest log of its type.labkey.log
: Rotated when the file reaches 10MB. Creates labkey.log.1, labkey.log.2, etc. Up to 7 archived files (80MB total).labkey-errors.log
: Rotated on server startup or at 100MB. Up to 3 labkey-errors log files are rotated (labkey-errors.log.1, etc.) If the server is about to delete the first labkey-errors file from the current session (meaning it has generated hundreds of megabytes of errors since it started up), it will retain that first log file as labkey-errors-YYYY-MM-DD.log. This can be useful in determining a root cause of the many errors.
To save additional disk space, older log archives can be compressed for storage. A script to regularly store and compress them can be helpful.
Running a server in development mode ("devmode") provides enhanced logging and enables some features and behavior useful in development and troubleshooting. These options include (but are not limited to):
- Logging is enhanced, providing more verbose output from some areas
- The MiniProfiler is enabled
- Module Editing is enabled. Note that the devmode server must also have access to the source path for module editing to work. You can tell if your server can access the source for a module by viewing the Module Information on the admin console. Expand the module of interest and if the Source Path is inaccessible, it will be shown in red.
- Asserts are on
- Performance may be slower due to the additional logging and lack of caching
- Validation of XML is performed which may uncover badly formed XML files, such as those with elements in the wrong order.
- A variety of test options that aren't appropriate for production use will be available. A few examples:
- An "expire passwords every five seconds" option for testing the user password reset experience.
- An insecure "TestSSO" authentication provider.
- The ability to test ETL filter strategies with a Set Range option.
You do not want to run your production server in devmode, but might set up a shared development server to support future work, or have each developer work on their own local dev machine.
To check whether the server is running in devmode:
- Go to (Admin) > Site > Admin Console.
- Under Diagnostics, click System Properties.
- Check the value of the devmode property.
If you are using a local development machine
, include the following in your VM options as part of your intelliJ configuration:
If you are not using a development machine, you can set this option by following these steps. The supported version of Tomcat
will change over time; replace the # in our examples with the actual version number.
- Open a command prompt.
- Go to the <CATALINA_HOME>/bin directory, for example:
- Execute the tomcat#w.exe program:
- The command will open a program window. Click the Java tab.
- In the Java Option box, scroll to the bottom of the properties. Add the following property at the bottom of the list:
- Close the program window and restart the server.
Security Implications of Using Devmode
Setting a server to "devmode" doesn't impact security settings or validation by itself. When you set up a shared development server, use the same security settings/authentication controls/etc. as you do on your production system. You only need to set up user accounts for the developers who need to use it.
Note that the "caching" JVM system property can also be used to control just the caching behavior, without all of the other devmode behaviors. To disallow the normal caching, perhaps because files are being updated directly on the file system, add -Dcaching=false to the JVM arguments.
Which Version of LabKey Server Is Running?
- Find your version number at (Admin) > Site > Admin Console.
- At the top of the Server Information panel, you will see the release version.
Learn more about other diagnostic information on the admin console in this topic:
Compatible Component Versions
Confirm that you are using the supported versions of the required components, as detailed in the Supported Technologies Roadmap
. It is possible to have multiple versions of some software, like Java, installed at the same time. Check that LabKey and other applications, such as Tomcat, are configured to use the correct versions.
For example, if you see an error similar to "...this version of the Java Runtime only recognizes class file versions up to ##.0..." it likely means that Tomcat is running under an older version of the JDK than is supported.
Connection Pool Size
If your server becomes unresponsive, it could be due to the depletion of available connections to the database. Watch for a Connection Pool Size
of 8, which is the Tomcat connection pool default size and insufficient for a production server. To see the connection pool size for the LabKey data source, select (Admin) > Site > Admin Console
and check the setting of Connection Pool Size
on the Server Information
panel. The connection pool size for every data source is also logged at server startup.
To set the connection pool size, edit your labkey.xml/ROOT.xml configuration file and change the "maxTotal" setting for your LabKey data source to at least 20. Depending on the number of simultaneous users and the complexity of their requests, your deployment may require a larger connection pool size.
You should also consider changing this setting for external data sources to match the usage you expect. Learn more in this topic: External Schemas and Data Sources
Deleted or Deactivated Users
Over time, you may have personnel leave the organization. Such user accounts should be deactivated and not deleted. If you encounter problems with accessing linked schemas, external schemas, running ETLs, or similar, check the logs to see if the former user may have "owned" these long term resources.
Properties in Config Files
Check that the labkeyDataSource and any external data source properties in your config file (labkey.xml or ROOT.xml)
match those expected of the current component versions.
1. Watch the catalina.out log for warning messages like "Ignoring unknown property..." or "...is being ignored".
For example, older versions of tomcat used "maxActive" and "maxWait" and newer versions use "maxTotal" and "maxWaitMillis". Error messages like this would be raised if labkey.xml/ROOT.xml used the older versions:
17-Feb-2022 11:33:42.415 WARNING [main] java.util.ArrayList.forEach Name = labkeyDataSource Property maxActive is not used in DBCP2, use maxTotal instead. maxTotal default value is 8. You have set value of "20" for "maxActive" property, which is being ignored.
17-Feb-2022 11:33:42.415 WARNING [main] java.util.ArrayList.forEach Name = labkeyDataSource Property maxWait is not used in DBCP2 , use maxWaitMillis instead. maxWaitMillis default value is PT-0.001S. You have set value of "120000" for "maxWait" property, which is being ignored.
2. Enabling SSL is done by including "SSLEnabled=true"
in the server.xml config file for Tomcat
. If you see the following error, you may have tried to set the nonexistent "ssl" property in the labkey.xml or ROOT.xml configuration file.
19-Feb-2022 15:28:29.624 INFO [main] java.util.ArrayList.forEach Name = labkeyDataSource Ignoring unknown property: value of "true" for "ssl" property
If you have problems during installation, retry after shutting down all other running applications. Specifically, you may need to try temporarily shutting down any virus scanning application, internet security applications, or other applications that run in the background to see if this resolves the issue.
In order for files to be uploaded and logs to be written, the LabKey user account must have the ability to write to the underlying file system locations. For example, if the "Users" group on a windows installation has read but not write access to the site file root, an error message like this will be shown upon file upload:
Couldn't create file on server. This may be a server configuration problem. Contact the site administrator.
Browser Refresh for UI Issues
If menus, tabs, or other UI features appear to display incorrectly after upgrade, particularly if different browsers show different layouts, you may need to clear your browser cache to clear old stylesheets.
Tomcat Failure to Start
If installation fails to start tomcat (such as with an error like "The specified service already exists."), you may need to manually stop or delete a failing Tomcat service.
To stop the service on Windows, open Control Panel > Administrative Tools > Services
. Select the relevant service, such as LabKey Server Apache Tomcat #.0
and click Stop
To delete, run the following from the command line as an administrator, substituting the major version number for the #:
JDBC Connection Pool is Not Supported
The Tomcat JDBC Connection pool is not supported. Learn more in this issue report
Check the logs for messages similar to:
Failed to load DataSource "labkeyDataSource" defined in labkey.xml. This could be caused by an attempt to use the Tomcat JDBC connection pool, which is not supported. Please remove the "factory" attribute from this DataSource definition.
Check your config file
for this attribute, which needs to be removed if it is present. If it is not present there, confirm that your Tomcat installation is not using this connection pool.
ERR_CONNECTION_CLOSED during Startup
An error like "ERR_CONNECTION_CLOSED" during startup, particularly after upgrading your version of Tomcat, may indicate that necessary files are missing. Check that you copied the server.xml and web.xml as well as the SSL directory containing the keys. Check the logs for "SEVERE" to find any messages that might look like:
19-Feb-2022 15:28:13.312 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8443]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
org.apache.catalina.LifecycleException: The configured protocol [org.apache.coyote.http11.Http11AprProtocol] requires the APR/native library which is not available
If you configure "SSL Required" and then cannot access your server, you will not be able to disable this requirement in the user interface. To resolve this, you can directly reset this property in your database.
Confirm that this query will select the "sslRequired" property row in your database:
SELECT * FROM prop.Properties WHERE Name = 'sslRequired' AND Set = (SELECT Set FROM prop.PropertySets WHERE Category = 'SiteConfig')
You should see a single row with a name of "sslRequired" and a value of "true". To change that value to false, run this update query:
UPDATE prop.Properties SET Value = FALSE WHERE Name = 'sslRequired' AND Set = (SELECT Set FROM prop.PropertySets WHERE Category = 'SiteConfig')
You can then verify with the SELECT query above to confirm the update.
Once the value is changed you'll need to restart the server to force the new value into the cache.
If you are using SQL Server, you'll need to double quote every reference to "Set" because that database considers Set a keyword.
SSL/TLS Handshake Issues
When accessing the server through APIs, including RStudio, rcurl, Jupyter, etc. one or more errors similar to these may be seen either in a client call or command line access:
SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed)
Peer reports incompatible or unsupported protocol version.
Timeout was reached: [SERVER:PORT] Operation timed out after 10000 milliseconds with 0 out of 0 bytes received
This may indicate that your server is set to use a more recent sslProtocol (such as TLSv1.3) than your client tool(s).
Client programs like RStudio, curl, and Jupyter may not have been updated to use the newer TLSv1.3 protocol which has timing differences from the earlier TLSv1.2 protocol. Check to see which protocol version your server.xml is set to accept. To cover most cases, edit the server.xml to accept both TLSv1.2 and TLSv1.3, and make the default TLSv1.2. This line applies those settings:
SSLEnabled="true" sslEnabledProtocols="TLSv1.2, TLSv1.3" sslProtocol="TLSv1.2"
You may need to remove references to Cygwin from your Windows system path before installing LabKey, due to conflicts with the PostgreSQL installer. The PostgreSQL installer also conflicts with some antivirus or firewalls. (see http://wiki.postgresql.org/wiki/Running_%26_Installing_PostgreSQL_On_Native_Windows
for more information).
Detect Attempt to Connect to In-Use Database
If you attempt to start the server with a database that is already in use by another LabKey Server as primary data source, serious complications may arise, including data corruption. For example, this might arise if a developer attempts to start a new development machine without changing the reference from a copied configuration file.
To prevent this situation, the server will check for existing application connections at startup. If there is a collision, the server will not start and the administrator will see a message similar to this:
There is 1 other connection to database [DATABASE_NAME] with the application name [APPLICATION_NAME]! This likely means another LabKey Server is already using this database.
If the check for this situation is unsuccessful for any reason, the message:
Attempt to detect other LabKey Server instances using this database failed.
For either error message, confirm that each LabKey Server will start up on a unique database in order to proceed. Learn more in the issue description here
Restart Installation from Scratch
If you have encountered prior failed installations, don't have any stored data you need to keep, and want to clean up and start completely from scratch, the following process may be useful:
- Delete the Tomcat service (if it seems to be causing the failure).
- Uninstall PostgreSQL using their uninstaller.
- Control Panel > Programs and Features
- Select PostgreSQL program.
- Click Uninstall.
- Delete the entire LabKey installation directory.
- Install LabKey again.
If you encounter the following error, or a similar error mentioning Graphviz:
Unable to display graph view: cannot run dot due to an error.
Cannot run program "dot" (in directory "./temp/ExperimentRunGraphs"): CreateProcess error=2, The system cannot find the file specified.
then install Graphviz according to the following instructions:Installation: Third-Party Components
Load Balancers and IP Addresses
If you are using a load balancer, including on a server in the LabKey cloud, be aware that AWS may shift over time what specific IP address your server is "on". Note that this includes LabKey's support site, www.labkey.org. This makes it difficult to answer questions like "What IP address am I using?" with any long term reliability.
In order to maintain a stable set of allowlist traffic sources/destinations, it is more reliable to pin these to the domain (i.e. labkey.org) rather than than to the IP address which may change within the AWS pool.
Users of Premium Editions of LabKey Server can obtain support with installation and other issues by opening a ticket on their private support portal
. Your Account Manager will be happy to help resolve the problem.
All users can can search for issues resolved through community support in the LabKey Support Forum
If you don't see your issue listed in the community support forum, you can post a new question.
If the install seems successful, it is often helpful to submit debugging logs
If the install failed to complete, please include the install.log and install-debug.log from your selected LabKey install directory.
PostgreSQL logs its installation process separately. If PostgreSQL installation/upgrade fails, please locate and include the PostgreSQL install logs as well.