This topic covers troubleshooting some problems you may encounter building LabKey Server from source. Roughly divided into categories, there is some overlap among sections, so searching this page may be the most helpful way to find what you need.

• IntelliJ: Building and starting LabKey Server within IntelliJ
• Gradle: Using gradle tasks, plugins, etc.
• Log4j
• Tomcat
• Database State
• System PATH
• Environment Variables
• Ext Libraries
• Renaming Files and Directories

IntelliJ Troubleshooting

IntelliJ Version

If you are using IntelliJ and experience errors building or upgrading LabKey from source, check to see if your IntelliJ IDEA is up to date. You can download the the latest version here:

• http://www.jetbrains.com/idea

Build model 'org.jetbrains.plugins.gradle.model.GradleExtensions' for root project 'labkey'

IntelliJ Warnings and Errors

• Warning: Class "org.apache.catalina.startup.Bootstrap" not found in module "LabKey": You may ignore this warning in the Run/Debug Configurations dialog in IntelliJ.
• Error: Could not find or load main class org.apache.catalina.startup.Bootstrap on OSX (or Linux): you might see this error in the console when attempting to start LabKey server. Update the '-classpath' VM option for your Run/Debug configuration to have Unix (:) path separators, rather than Windows path separators (;).
• Can't find workspace.template.xml? On older enlistments of LabKey, for example version 15.3, copy <LABKEY_HOME>/server/LabKey.iws.template to LabKey.iws instead.
• On Windows, if you are seeing application errors, you can try resetting the winsock if it has gotten into a bad state. To reset:
• Open a command window in administrator mode
• Type into the command window: netsh winsock reset
• When you hit enter, that should reset the winsock and your application error may be resolved. You might need to restart your computer for the changes to take effect.

IntelliJ Slow

You can help IntelliJ run faster by increasing the amount of memory allocated to it. To increase memory:

• Go to C:\Program Files\JetBrains\IntelliJ IDEA <Version Number>\bin, assuming that your copy of IntelliJ is stored in the default location on a Windows machine.
• Right click on the idea.exe.vmoptions file and open it in notepad.
• Edit the first two lines of the file to increase the amount of memory allocated to IntelliJ. For example, on a 2 Gig machine, it is reasonable to increase memory from 32m to 512m. The first two lines of this file then read:

-Xms512m
-Xmx512m

• Save the file
• Restart IntelliJ

Server Not Starting Using a LabKey-Provided Run/Debug Configs

Problem: Server does not start in IntelliJ using one of the LabKey-provided Run/Debug configurations. In the console window the following error appears:

/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java 
-agentlib:jdwp=transport=dt_socket,address=127.0.0.1:59311,suspend=y,server=n -Dcatalina.base=./ 
-Dcatalina.home=./ -Djava.io.tmpdir=./temp -Ddevmode=true -ea -Dsun.io.useCanonCaches=false -Xmx1G 
-XX:MaxPermSize=160M -classpath "./bin/*;/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar" 
-javaagent:/Users/susanh/Library/Caches/IntelliJIdea2016.3/groovyHotSwap/gragent.jar 
-Dfile.encoding=UTF-8 org.apache.catalina.startup.Bootstrap start
Connected to the target VM, address: '127.0.0.1:59311', transport: 'socket'
Error: Could not find or load main class org.apache.catalina.startup.Bootstrap
Disconnected from the target VM, address: '127.0.0.1:59311', transport: 'socket'

Cause: This is most likely caused by an incorrect path separator in the Run/Debug configuration's classpath argument.

Solution: Edit the Run/Debug configuration and change the separator to the one appropriate to your platform (semicolon for Windows; colon for Mac/Linux).

Fatal Error in Java Runtime Environment

Error: When starting LabKey or importing data to a new server, you might see a virtual machine crash similar to this:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=23893, tid=39779
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  0x0000000000000000
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

Cause: These are typically bugs in the Java Virtual Machine itself.

Solution: Ensuring that you are on the latest patched release for your preferred Java version is best practice for avoiding these errors. If you have multiple versions of Java installed, be sure that JAVA_HOME and other configuration is pointing at the correct location. If you are running through the debugger in IntelliJ, check the JDK configuration. Under Project Structure > SDKs check the JDK home path and confirm it points to the newer version.

Gradle task already exists

Cause: A somewhat recent version of IntelliJ has started to put output files in an 'out' directory in each Gradle project's directory (https://youtrack.jetbrains.com/issue/IDEA-175172), so if you run a Gradle build command from IntelliJ these directories will be created. Some of the tasks in LabKey's custom gradlePlugins jar are a little too greedy when looking for input files and will pick up files from this out directory as well.

Solution: This bug for the antlr plugin has been fixed in version 1.1 of the gradlePlugins jar so you can update to that version of the plugins for LabKey 17.2 or later. Also, you can work around this by removing the out directories that Intellij creates and running the build again from the command line.

Cannot find dependencies for compressClientLibs task

Cause: A somewhat recent version of IntelliJ has started to put output files in an 'out' directory in each Gradle project's directory (https://youtrack.jetbrains.com/issue/IDEA-175172), so if you run a Gradle build command from IntelliJ these directories will be created. Some of the tasks in LabKey's custom gradlePlugins jar are a little too greedy when looking for input files and will pick up files from this out directory as well.

Solution: This bug has been fixed in version 1.2 of the gradlePlugins jar, so you can update to that version of the plugins for LabKey 17.2 or later. Also, you can work around this by removing the out directories that Intellij creates and running the build again from the command line.

Exception in compressClientLibs task

If you encounter this error when building in production mode:

* What went wrong:
Execution failed for task ':server:modules:platform:query:compressClientLibs'.
> org.xml.sax.SAXException: java.lang.NullPointerException
  java.lang.NullPointerException

It may indicate a parsing error from yuicompressor which handles javascript and css minification. The problem is usually that the code you added is using elements of ES6 (like the spread operator) that the parser doesn't support.

Solution: Fall back to older javascript syntax.

Cannot find class X when hot-swapping

Cause: IntelliJ puts its build output files, which it uses when doing hot-swapping, in different directories than the command-line Gradle build does, so if you have not built the entire project (or at least the jars that the file you are attempting to swap in depends on) within IntelliJ, IntelliJ will not find them.

Solution: Open the Gradle window in IntelliJ and run the root-level "build" task from that window.

XML Classes not found after Gradle Refresh

Problem: After doing a Gradle Refresh, some of the classes with package names like org.labkey.SOME_SCOPE.xml (e.g., org.labkey.security.xml) are not found.

Cause: This is usually due to the relevant schema jars not having been built yet.

Solution: Run the deployApp command and then do a Gradle Refresh within IntelliJ. Though you can use a more targeted Gradle task schemaJar to build just the jar file that is missing, you might find yourself running many of these to resolve all missing classes, so it is likely most efficient to simply use deployApp

Gradle Troubleshooting

Understanding how gradle builds the server, and the relationships between building and cleaning can help you diagnose many build problems. See these related topics:

• Build LabKey from Source
• Gradle Cleaning

Update version of gradlePlugins

We maintain release notes for the gradlePlugins that describe changes and bug fixes to the plugins. Check these notes to find out if the problem you are having has already been addressed. Be sure to pay attention to the "Earliest compatible LabKey version" indicated with each release to know if it is compatible with your current LabKey version. If you want to update to a new version of the gradle plugins to pick up a change, you need only update the gradlePluginsVersion property in the root-level gradle.properties file:

gradlePluginsVersion=1.3.6

Gradle Refresh

First, check that you are using a current version of IntelliJ

Problem: Gradle Refresh in IntelliJ has no effect on project structure after the settings.gradle file is updated.

Cause: Name conflict between the gel_test project and the gel_test IntelliJ module that would be created from the gel project’s ‘test’ source set. (IDEA-168284)

Workarounds: Do one of the following

Remove gel_test from the test of projects you are including in your settings.gradle file and then do the Gradle Refresh. Within IntelliJ do the following Preferences -> Build, Execution, Deployment -> Build Tools -> Gradle Uncheck the setting “Create separate module per source set” Click “OK” Do the Gradle Refresh in the Gradle window Preferences -> Build, Execution, Deployment -> Build Tools -> Gradle Check the setting “Create separate module per source set” Click “OK” Do the Gradle Refresh in the Gradle window

Could not resolve all files for configuration

Problem: When compiling a module, an error such as the following appears

Execution failed for task ':server:modules:myModule:compileJava'.
> Could not resolve all files for configuration ':server:modules:myModule:compileClasspath'.
   > Could not find XYZ-api.jar (project :server:modules:XYZ).

<ij_msg_gr>Project resolve errors<ij_msg_gr><ij_nav>/Development/labkey/labkeyHome/build.gradle<ij_nav><i><b>root project 'labkeyHome': Unable to resolve additional project configuration.</b><eol>Details: org.gradle.api.artifacts.ResolveException: Could not resolve all dependencies for configuration ':server:modules:myModule:compileClasspath'.<eol>Caused by: org.gradle.internal.resolve.ArtifactNotFoundException: Could not find XYZ-api.jar (project :server:modules:XYZ).</i>

Cause: The settings.gradle file has included only a subset of the dependencies within the transitive dependency closure of your project and Gradle is unable to resolve the dependency for the api jar file. In particular, though your settings file likely includes the :server:modules:XYZ project, it likely does not include another project that myModule depends on that also depends on XYZ. That is, you have the following dependencies:

• myModule depends on otherModule
• myModule depends on XYZ
• otherModule depends on XYZ

include ':server:modules:myModule'
include ':server:modules:XYZ'

Solution: Include the missing project in your settings.gradle file

include ':server:modules:myModule'
include ':server:modules:otherModule'
include ':server:modules:XYZ'

Module not deployed

gradlew projects

Module plugins not applied or missing properties or methods

FAILURE: Build failed with an exception.

* Where:
Build file '/path/to/Development/labkey/root/server/modules/myModule/build.gradle' line: 6

* What went wrong:
A problem occurred evaluating project ':server:modules:myModule'.
> Could not find method implementation() for arguments com.sun.mail:jakarta.mail:1.6.5 on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

Cause: As of LabKey Server version 21.3.0 we no longer apply the LabKey gradle plugins for all modules under the server/modules directory by default. Each module is responsible for applying the plugins it requires. When running a deployApp command, if we find a gradle project that contains a module.properties file but does not have a task named 'module', a message like the following will be shown:

The following projects have a 'module.properties' file but no 'module' task. These modules will not be included in the deployed server. You should apply either the 'org.labkey.build.fileModule' or 'org.labkey.build.module' plugin in each project's 'build.gradle' file.
        :server:modules:myModule
        :server:modules:otherModule

Solution: Apply the appropriate plugin in the module's build.gradle file by including either

plugins {
   id 'org.labkey.build.fileModule'
}

plugins {
   id 'org.labkey.build.module'
}

Starting Over with Gradle + IntelliJ

If you get into a situation where the IntelliJ configuration has been corrupted or created in a way that is incompatible with what we expect (which can happen if, for example, you say 'Yes' when IntelliJ asks if you want to have it link an unlinked Gradle project), these are the steps you can follow to start over with a clean setup for Gradle + IntelliJ:

• Shut down IntelliJ.
• Revert the <LABKEY_HOME>/.idea/gradle.xml file to the version checked in to VCS.
• Remove the <LABKEY_HOME>/.idea/modules directory.
• Remove the <LABKEY_HOME>/.idea/modules.xml file.
• Start up IntelliJ.
• Open the Gradle window (View > Tool Windows > Gradle) and click the Refresh icon, as described above.

Log4J

All LabKey-created code now uses Log4J version 2. If you are developing LabKey modules, upgrade to use Log4J version 2.

1. The main package is org.apache.logging.log4j replacing the older package org.apache.log4j. All the java classes using log4j should now import the new package and remove the old package.
2. The older way of getting the logger was : Logger.getLogger({name}). Now Java classes should get the logger through LogManager LogManager.getLogger.
3. There are updates to some commonly used methods:

The following methods are not supported in the new log4J version:

• org.apache.log4j.LogManager.shutdown() - similar functionality can be defined in log4j2.xml in the appender config i.e. OnStartupTriggeringPolicy
• org.apache.log4j.Logger.setLevel() - use org.apache.logging.log4j.core.config.Configurator.setLevel()

log4j.xml Appender

Problem: The server seems to have started fine, but there are no log messages in the console after server start up.

Cause: The log4j.xml file that controls where messages are logged does not contain the appropriate appender to tell it to log to the console. This appender is added when you deploy the application in development mode (as a consequence of running the deployApp command).

In log4j2, a custom appender is defined by creating a plugin. The log4j documentation has an explanation with an example for a custom appender here:

• http://logging.apache.org/log4j/2.x/manual/extending.html#Appenders

Tomcat

Tomcat Fails to Start

If Tomcat fails to start successfully, check the steps above to ensure that you have configured your JDK and development environment correctly. Some common errors you may encounter include:

org.postgresql.util.PSQLException: FATAL: password authentication failed for user "<username>" or java.sql.SQLException: Login failed for user "<username>"

These error occurs when the database user name or password is incorrect. If you provided the wrong user name or password in the .properties file that you configured above, LabKey will not be able to connect to the database. Check that you can log into the database server with the credentials that you are providing in this file.

java.net.BindException: Address already in use: JVM_Bind:<port x>:

This error occurs when another instance of Tomcat or another application is running on the same port. Specifically, possible causes include:

• Tomcat is already running under IntelliJ.
• Tomcat is running as a service.
• Microsoft Internet Information Services (IIS) is running on the same port.
• Another application is running on the same port.

• Shut down the instance of Tomcat or the application that is running on the same port.
• Change the port for the other instance or application.
• Edit the Tomcat server.xml file to specify a different port for your development installation of Tomcat.

In certain developer configurations, you will need to add an IntelliJ utility JAR file to your classpath.

• Edit the Debug Configuration in IntelliJ.
• Under the "VM Options" section, find the "-classpath" argument.
• Find your IntelliJ installation. On Windows machines, this is typically "C:\Program Files\JetBrains\IntelliJ IDEA <Version Number>" or similar. On OSX, this is typically "/Applications/IntelliJ IDEA <Version Number>.app" or similar.
• The required JAR file is in the IntelliJ installation directory, and is ./lib/idea_rt.jar. Add it to the -classpath argument value, separating it from the other values with a ":" on OSX and a ";" on Windows.
• Save your edits and start Tomcat.

Database State Troubleshooting

If you build the LabKey source yourself from the source tree, you may need to periodically delete and recreate your LabKey database. The daily drops often include SQL scripts that modify the data and schema of your database.

Database Passwords Not Working

Problem: My passwords to PostgreSQL and MS SQL Server aren't working.

Solution: Unlike Ant, the Gradle build system will automatically escape any special XML characters, such as quotes and ampersand symbols in the pg.properties / mssql.properties files. When migrating these files from Ant to Gradle, replace any escaped ampersands (&amp;) with plain text ampersands (&).

System PATH

LabKey Server will automatically load certain binaries and tools (graphviz, dot, proteomics binaries) from Artifactory and expects to find them on the users's PATH. Be sure to include the following location in your system PATH. This directory won't exist before you build your server, but add it to the path anyway.

<LABKEY_HOME>\build\deploy\bin

For example, C:\dev\labkey\labkeyHome\build\deploy\bin.

Environment Variables Troubleshooting

JAVA_TOOL_OPTIONS

Most users will not have this problem. However, if you see a build error something like the following:

error: unmappable character for encoding ASCII

then setting this environment variable may fix the problem

export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8

Ext Libraries

Problem: Ext4 is not defined.

The page or a portion of the page is not rendering appropriately. Upon opening the browser console you see an error stating "Ext4 is not defined". This usually occurs due to a page not appropriately declaring client-side code dependencies.

Example of the error as seen in Chrome console.

Solution

Declare a dependency on the code/libraries you make use of on your pages. Depending on the mechanism you use for implementing your page/webpart we provide different hooks to help ensure dependent code is available on the page.

• Wiki: In a wiki you can use LABKEY.requiresScript() with a callback.
• File-based module view: It is recommended that you use file-scoped dependencies by declaring a view.xml and using the <dependencies> attribute.
• Java: In Java you can use ClientDependency.fromPath(String p) to declare dependencies for your view. Note, be sure to declare these before the view is handed off to rendering otherwise your dependency will not be respected.
• JSP: Override JspBase.addClientDependencies(ClientDependencies dependencies) in the .jsp. Here is an example.

Renaming Files and Directories

Occasionally a developer decides a module or file (and possibly associated classes) should be renamed for clarity or readability.

Most file/directory name changes are handled transparently: rename locally through Intellij, verify your build and that tests pass, and commit.

The big exception to this is doing a case-only rename. (e.g., "myfile.java" -> "MyFile.java"). Because Windows file systems are case-insensitive, this causes major headaches. Don't do it as described above.

But what if I really need to do a case-only rename?

Do a renaming to only change casing (ex. Mymodule -> myModule) in two rename steps:

1. First rename to an intermediate name that changes more than the casing: (ex. Mymodule -> myModule2)
2. Commit that change.
3. Then rename to the target final name (ex. myModule2 -> myModule)
4. Commit again.
5. Push your commits.
6. Squash-merge to develop.

Related Topics

• Set Up a Development Machine
• Troubleshoot LabKey Server
• Troubleshoot the Enterprise Pipeline
• Troubleshoot Rlabkey
• Troubleshoot Server Installation and Configuration