Troubleshoot Installation and Configuration

Documentation
In case of errors or other problems when installing and running LabKey Server, first review installation basics and options linked in the topic: Install LabKey. This topic provides additional troubleshooting suggestions if the instructions in that topic do not resolve the issue.

LabKey Logs

From time to time Administrators may need to review the logs to troubleshoot system startup or other issues. The logs are located in the <LABKEY_HOME>/logs directory.

The logs of interest are:

  • labkey.log: Once the application starts, logging of INFO, ERROR, WARN and other messages is recorded here.
    • You can jump to times the server started up by searching the log for the string (__) which is part of the LabKey Server ASCII art startup banner.
  • labkey-errors.log: Contains ERROR messages.

Log Rotation and Storage

To avoid massive log files and retain recent information with some context, log files are rotated (or "rolled-over") into new files to keep them manageable. These rotated log files are named with a trailing number and shuffled such that .1 will be the most recently rolled over log file, and higher numbers for any older logs of that type.

labkey.log: Rotated when the file reaches 10MB. Creates labkey.log.1, labkey.log.2, etc. Up to 7 archived files (80MB total).

labkey-errors.log: Rotated on server startup or at 100MB. Up to 3 labkey-errors.log files are rotated (labkey-errors.log.1, etc.) If the server is about to delete the first labkey-errors file from the current session (meaning it has generated hundreds of megabytes of errors since it started up), it will retain that first log file as labkey-errors-YYYY-MM-DD.log. This can be useful in determining a root cause of the many errors.

To save additional disk space, older log archives can be compressed for storage. A script to regularly store and compress them can be helpful.

Service Startup

LabKey Server is typically started and stopped with a service on either Linux or Windows. The methods differ for each platform, both for how to configure the service and how to run it.

For an error message like "The specified service already exists.", you may need to manually stop or even delete a failing service.

Here are some platform-specific things to check:

Linux

On Linux, if the service fails to start, the OS will likely tell you to review the output of:

systemctl status SERVICENAME.service output
...and also the logged info from:
journalctl -xe

Most of the time, if a service fails to start after you create it, it's likely that your configuration in your service file requires edits and a reload of the service daemon.

Windows

Stop and Delete a Service

If you need to stop a running service on Windows, such as to retry or if you see a message like "The specified service already exists", follow these steps:

Open Control Panel > Administrative Tools > Services. Select the service and click Stop.

To delete, run the following from the command line as an administrator:

sc delete LabKeyServer

Service Access Denied Exception An error on service startup like the following may indicate that the labkeyServer.jar file was previously unpacked in the <LABKEY_HOME> location, possibly by a previous attempt or command-line run as a different user.

Caused by: java.nio.file.AccessDeniedException: C:labkeylabkeylabkeywebapp

To resolve this, delete these directories that will be recreated by the service, then try the service again.

<LABKEY_HOME>/labkeywebapp
<LABKEY_HOME>/modules
<LABKEY_HOME>/logs

Version of prunsrv If you install the service using the wrong version of prunsrv, it will successfully install but will not run. Running it will generate a "commons-daemon.*.log" file which will likely help identify this problem. Specifically, if you use the 32-bit version of prunsrv on a 64-bit machine, you may see errors like:

You must delete the incorrect service, and return to copy the 64-bit version of prunsrv.exe. This is in the AMD64 subdirectory of the downloaded daemon bundle. Close and reopen your command window to pick up this change, then install using the correct version:

install_service.bat install

Development Mode

Running a server in development mode ("devmode") may change the behavior, provides additional logging, and can be helpful in troubleshooting some issues. DO NOT run your production server in devmode; use this mode on a staging, test, or development server only.

To check whether the server is running in devmode:

  • Go to (Admin) > Site > Admin Console.
  • Under Diagnostics, click System Properties.
  • Check the value of the devmode property.
Learn more about setting and using devmode in this topic:

Configuration Issues

Application.Properties

1. Review your application.properties file to confirm that all properties you expect to be in use are active. A leading # means the property is commented out and will not be applied. Also, double check that you have replaced any placeholder values in the file.

2. Check that the labkeyDataSource and any external data source properties match those expected of the current component versions.

3. Watch the <LABKEY_HOME>/logs for warning messages like "Ignoring unknown property..." or "...is being ignored". For example, older versions of Tomcat used "maxActive" and "maxWait" and newer versions use "maxTotal" and "maxWaitMillis". Error messages like this would be raised if your datasources used the older versions:

17-Feb-2022 11:33:42.415 WARNING [main] java.util.ArrayList.forEach Name = labkeyDataSource Property maxActive is not used in DBCP2, use maxTotal instead. maxTotal default value is 8. You have set value of "20" for "maxActive" property, which is being ignored.
17-Feb-2022 11:33:42.415 WARNING [main] java.util.ArrayList.forEach Name = labkeyDataSource Property maxWait is not used in DBCP2 , use maxWaitMillis instead. maxWaitMillis default value is PT-0.001S. You have set value of "120000" for "maxWait" property, which is being ignored.

Compatible Component Versions

Confirm that you are using the supported versions of the required components, as detailed in the Supported Technologies Roadmap. It is possible to have multiple versions of some software, like Java, installed at the same time. Check that LabKey and other applications in use are configured to use the correct versions.

For example, if you see an error similar to "...this version of the Java Runtime only recognizes class file versions up to ##.0..." it likely means you are using an unsupported version of the JDK.

Connection Pool Size

If your server becomes unresponsive, it could be due to the depletion of available connections to the database. Watch for a Connection Pool Size of 8, which is the Tomcat connection pool default size and insufficient for a production server. To see the connection pool size for the LabKey data source, select (Admin) > Site > Admin Console and check the setting of Connection Pool Size on the Server Information panel. The connection pool size for every data source is also logged at server startup.

To set the connection pool size, edit your application.properties file and change the "maxTotal" setting for your LabKey data source to at least 20. Depending on the number of simultaneous users and the complexity of their requests, your deployment may require a larger connection pool size. LabKey uses a default setting of 50.

You should also consider changing this setting for external data sources to match the usage you expect. Learn more in this topic: External Schemas and Data Sources.

Increase Header Size Limits

Tomcat sets a size limit for the GET header. If your API calls hit this limit, or you see errors like "Request Header is too large" in the logs, you can increase the default header size for GETs.

To increase the allowable header size, edit your application.properties file, adding this property set to a suitably high value:

server.max-http-request-header-size=65536

Deleted or Deactivated Users

Over time, you may have personnel leave the organization. Such user accounts should be deactivated and not deleted. If you encounter problems with accessing linked schemas, external schemas, running ETLs, or similar, check the logs to see if the former user may have "owned" these long term resources.

Conflicting Applications

If you have problems during installation, retry after shutting down all other running applications. Specifically, you may need to try temporarily shutting down any virus scanning application, internet security applications, or other applications that run in the background to see if this resolves the issue.

Filesystem Permissions

In order for files to be uploaded and logs to be written, the LabKey user account (i.e. the account that runs the service, often named 'labkey') must have the ability to write to the underlying file system locations.

For example, if the "Users" group on a windows installation has read but not write access to the site file root, an error message like this will be shown upon file upload:

Couldn't create file on server. This may be a server configuration problem. Contact the site administrator.

Browser Refresh for UI Issues

If menus, tabs, or other UI features appear to display incorrectly after upgrade, particularly if different browsers show different layouts, you may need to clear your browser cache to clear old stylesheets.

Diagnostics

Want to know which version of LabKey Server Is running?

  • Find your version number at (Admin) > Site > Admin Console.
  • At the top of the Server Information panel, you will see the release version.
Learn more and find more diagnostics to check in this topic:

HTTPS (TLS/SSL) Errors

Use Correct Property

Enabling HTTPS is done by including "server.ssl.enabled=true" in the application properties. If you see the following error, you may have tried to set the nonexistent "ssl" property instead.

19-Feb-2022 15:28:29.624 INFO [main] java.util.ArrayList.forEach Name = labkeyDataSource Ignoring unknown property: value of "true" for "ssl" property

ERR_CONNECTION_CLOSED during Startup

An error like "ERR_CONNECTION_CLOSED" during startup, particularly after upgrading, may indicate that necessary files are missing. Check that you have the right configuration in application.properties, particularly the HTTPS directory containing the keys. Check the logs for "SEVERE" to find any messages that might look like:

19-Feb-2022 15:28:13.312 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8443]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
...
org.apache.catalina.LifecycleException: The configured protocol [org.apache.coyote.http11.Http11AprProtocol] requires the APR/native library which is not available

Disable sslRequired

If you configure "SSL Required" and then cannot access your server, you will not be able to disable this requirement in the user interface. To resolve this, you can directly reset this property in your database.

Confirm that this query will select the "sslRequired" property row in your database:

SELECT * FROM prop.Properties WHERE Name = 'sslRequired' AND Set = (SELECT Set FROM prop.PropertySets WHERE Category = 'SiteConfig')

You should see a single row with a name of "sslRequired" and a value of "true". To change that value to false, run this update query:

UPDATE prop.Properties SET Value = FALSE WHERE Name = 'sslRequired' AND Set = (SELECT Set FROM prop.PropertySets WHERE Category = 'SiteConfig')

You can then verify with the SELECT query above to confirm the update.

Once the value is changed you'll need to restart the server to force the new value into the cache.

If you are using SQL Server, you'll need to double quote every reference to "Set" because that database considers Set a keyword.

HTTPS Handshake Issues

When accessing the server through APIs, including RStudio, rcurl, Jupyter, etc. one or more errors similar to these may be seen either in a client call or command line access:

SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed)
or
Peer reports incompatible or unsupported protocol version.
or
Timeout was reached: [SERVER:PORT] Operation timed out after 10000 milliseconds with 0 out of 0 bytes received

This may indicate that your server is set to use a more recent sslProtocol (such as TLSv1.3) than your client tool(s).

Client programs like RStudio, curl, and Jupyter may not have been updated to use the newer TLSv1.3 protocol which has timing differences from the earlier TLSv1.2 protocol. Check to see which protocol version your server is set to accept. To cover most cases, edit the application.properties section for HTTPS to accept both TLSv1.2 and TLSv1.3, and make the default TLSv1.2. These lines apply those settings:

server.ssl.enabled=true
server.ssl.enabled-protocols=TLSv1.3,TLSv1.2
server.ssl.protocol=TLSv1.2

Load Balancers

HTTPS Conflicts with Load Balancer or Proxy Use

Note that if you are using a Load Balancer or Proxy "in front of" LabKey, also enabling HTTPS may create conflicts as the proxy is also trying to handle the HTTPS traffic. This is not always problematic, as we have had success using Amazon's Application Load Balancer, but we have observed conflicts with versions from Apache, Nginx, and HA-Proxy. Review also the section about WebSockets below.

If you configure HTTPS and then cannot access the server, you may need to disable it in the database directly in order to resolve the issue. Learn more above.

WebSockets

LabKey Server uses WebSockets to push notifications to browsers. This includes a dialog telling the user they have been logged out (either due to an active logout request, or a session timeout). As WebSockets use the same port as HTTP and HTTPS, there is no additional configuration required if you are using Tomcat directly or embedded in LabKey Server.

If you have a load balancer or proxy in front of LabKey/Tomcat, like Apache or NGINX, be sure that it is configured in a way that supports WebSockets. LabKey Server uses a "/_websocket" URL prefix for these connections.

Note that while this is recommended for all LabKey Server installations, it is required for full functionality of the Sample Manager and Biologics products.

When WebSockets are not configured, or improperly configured, you will see a variety of errors related to the inability to connect and to trigger notifications and alerts. LabKey Server administrators will see a banner reading "The WebSocket connection failed. LabKey Server uses WebSockets to send notifications and alert users when their session ends."

IP Addresses May Shift

If you are using a load balancer, including on a server in the LabKey cloud, be aware that AWS may shift over time what specific IP address your server is "on". Note that this includes LabKey's support site, www.labkey.org. This makes it difficult to answer questions like "What IP address am I using?" with any long term reliability.

In order to maintain a stable set of allowlist traffic sources/destinations, it is more reliable to pin these to the domain (i.e. labkey.org) rather than than to the IP address which may change within the AWS pool.

Database Issues

PostgreSQL Installation

You may need to remove references to Cygwin from your Windows system path before installing LabKey, due to conflicts with the PostgreSQL installer. The PostgreSQL installer also conflicts with some antivirus or firewalls. (see http://wiki.postgresql.org/wiki/Running_%26_Installing_PostgreSQL_On_Native_Windows for more information).

Detect Attempt to Connect to In-Use Database

If you attempt to start the server with a database that is already in use by another LabKey Server as primary data source, serious complications may arise, including data corruption. For example, this might arise if a developer attempts to start a new development machine without changing the reference from a copied configuration file.

To prevent this situation, the server will check for existing application connections at startup. If there is a collision, the server will not start and the administrator will see a message similar to this:

There is 1 other connection to database [DATABASE_NAME] with the application name [APPLICATION_NAME]! This likely means another LabKey Server is already using this database.

If the check for this situation is unsuccessful for any reason, the message:

Attempt to detect other LabKey Server instances using this database failed.

For either error message, confirm that each LabKey Server will start up on a unique database in order to proceed. Learn more in the internal issue description (account required).

Restart Installation from Scratch

If you have encountered prior failed installations, don't have any stored data you need to keep, and want to clean up and start completely from scratch, the following process may be useful:

  • Delete the LabKey service (if it seems to be causing the failure).
  • Uninstall PostgreSQL using their uninstaller.
  • Delete the entire LabKey installation directory.
  • Install LabKey again from the original binaries.

Graphviz

If you encounter the following error, or a similar error mentioning Graphviz:

Unable to display graph view: cannot run dot due to an error.

Cannot run program "dot" (in directory "./temp/ExperimentRunGraphs"): CreateProcess error=2, The system cannot find the file specified.

...you need to install it. Learn more at these links:

Support Forum

Users of Premium Editions of LabKey Server can obtain support with installation and other issues by opening a ticket on their private support portal. Your Account Manager will be happy to help resolve the problem.

All users can can search for issues resolved through community support in the LabKey Support Forum.

If you don't see your issue listed in the community support forum, you can post a new question.

Supporting Materials

If the install seems successful, it is often helpful to submit debugging logs for diagnosis.

If the install failed to complete, please include the install.log and install-debug.log from your selected LabKey install directory.

PostgreSQL logs its installation process separately. If PostgreSQL installation/upgrade fails, please locate and include the PostgreSQL install logs as well. Instructions for locating PostgreSQL install logs can be found here:

Related Topics

Was this content helpful?

Log in or register an account to provide feedback


previousnext
 
expand allcollapse all