This topic covers troubleshooting some problems you may encounter setting up a development machine and building LabKey Server from source. Roughly divided into categories, there is some overlap among sections, so searching this page may be the most helpful way to find what you need.
Building and Starting LabKey Server
Build Errors
When encountering build errors when first starting a dev machine, double check that you have carefully followed the
setup instructions. Check the output of this command to confirm that your distribution is set up correctly:
Build errors after switching branches or updating can be due to a mismatch in module versions. You can see a report of all the versions of your modules by running:
Note that you should also confirm, using IntelliJ, that the root <LK_ENLISTMENT> location is also on the correct branch.
Linux Service Executable Name
If you are on Linux and see an error similar to this:
labkey_server.service - LabKey Server Application
Loaded: bad-setting (Reason: Unit labkey_server.service has a bad unit file setting.)
Active: inactive (dead)
<DATE><SERVER DETAILS> /etc/systemd/system/labkey_server.service:19: Neither a valid executable name>
It is likely because you need to provide the full path to Java at the head of the "ExecStart" line in the
labkey_server.service file. The default we send shows a <JAVA_HOME> variable, and you need to update this line to use the full path to $JAVA_HOME/bin/java instead.
Unable to Start Gradle Daemon
If you are unable to start any Gradle daemon (i.e. even "./gradlew properties" raises an error), check to confirm that you are using the 64-bit version of the JDK. If you accidentally try to use the 32-bit version, that VM has a max heap size of about 1.4GB, which is insufficient for building LabKey.
Note that "java -version" will specify "64-bit" if it is, but will not specify "32-bit" for that (incorrect) version. Instead the version line reads "Client VM" where you need to see "64-bit". At present the correct result of "java -version" should be:
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7)
OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)
An error like this may indicate that you are using the 32 bit version, as it :
FAILURE: Build failed with an exception.
* What went wrong:
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used. For more details on the daemon, please refer to https://docs.gradle.org/8.6/userguide/gradle_daemon.html in the Gradle documentation.
Process command line: C:labkeyappsjdk-17binjava.exe -XX:+UseParallelGC --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED -Xmx2048m -Dfile.encoding=windows-1252 -Duser.country=US -Duser.language=en -Duser.variant -cp C:Usersmohara.gradlewrapperdistsgradle-8.6-binafr5mpiioh2wthjmwnkmdsd5wgradle-8.6libgradle-launcher-8.6.jar -javaagent:C:Usersmohara.gradlewrapperdistsgradle-8.6-binafr5mpiioh2wthjmwnkmdsd5wgradle-8.6libagentsgradle-instrumentation-agent-8.6.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 8.6
Please read the following process output to find out more:
-----------------------
Error occurred during initialization of VM
Could not reserve enough space for 2097152KB object heap
Server Not Starting Using a LabKey-Provided Run/Debug Configs
Problem: Server does not start in using one of the LabKey-provided Run/Debug configurations. In the console window the following error appears:
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java
-agentlib:jdwp=transport=dt_socket,address=127.0.0.1:59311,suspend=y,server=n -Dcatalina.base=./
-Dcatalina.home=./ -Djava.io.tmpdir=./temp -Ddevmode=true -ea -Dsun.io.useCanonCaches=false -Xmx1G
-XX:MaxPermSize=160M -classpath "./bin/*;/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar"
-javaagent:/Users/susanh/Library/Caches/IntelliJIdea2016.3/groovyHotSwap/gragent.jar
-Dfile.encoding=UTF-8 org.apache.catalina.startup.Bootstrap start
Connected to the target VM, address: '127.0.0.1:59311', transport: 'socket'
Error: Could not find or load main class org.apache.catalina.startup.Bootstrap
Disconnected from the target VM, address: '127.0.0.1:59311', transport: 'socket'
Cause: This is most likely caused by an incorrect path separator in the Run/Debug configuration's classpath argument.
Solution: Edit the Run/Debug configuration and change the separator to the one appropriate to your platform (semicolon for Windows; colon for Mac/Linux).
Fatal Error in Java Runtime Environment
Error: When starting LabKey or importing data to a new server, you might see a virtual machine crash similar to this:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000000000000, pid=23893, tid=39779
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C 0x0000000000000000
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
Cause: These are typically bugs in the Java Virtual Machine itself.
Solution: Ensuring that you are on the latest patched release for your preferred Java version is best practice for avoiding these errors. If you have multiple versions of Java installed, be sure that JAVA_HOME and other configuration is pointing at the correct location. If you are running through the debugger in IntelliJ, check the JDK configuration. Under
Project Structure > SDKs check the
JDK home path and confirm it points to the newer version.
Exception in compressClientLibs task
If you encounter this error when building in production mode:
* What went wrong:
Execution failed for task ':server:modules:platform:query:compressClientLibs'.
> org.xml.sax.SAXException: java.lang.NullPointerException
java.lang.NullPointerException
It may indicate a parsing error from yuicompressor which handles javascript and css minification. The problem is usually that the code you added is using
elements of ES6 (like the spread operator) that the parser doesn't support.
Solution: Fall back to older javascript syntax.
Cannot find class X when hot-swapping
Cause: IntelliJ puts its build output files, which it uses when doing hot-swapping, in different directories than the command-line Gradle build does, so if you have not built the entire project (or at least the jars that the file you are attempting to swap in depends on) within IntelliJ, IntelliJ will not find them.
Solution: Open the Gradle window in IntelliJ and run the root-level "build" task from that window.
IntelliJ Troubleshooting
IntelliJ Version
If you are using IntelliJ and experience errors building or upgrading LabKey from source, check to see if your IntelliJ IDEA is up to date. You can download the the latest version here:
For example, you might see an error like this when using gradle
refresh from within intelliJ:
Build model 'org.jetbrains.plugins.gradle.model.GradleExtensions' for root project 'labkey'
IntelliJ Warnings and Errors
- Warning: Class "org.apache.catalina.startup.Bootstrap" not found in module "LabKey": You may ignore this warning in the Run/Debug Configurations dialog in IntelliJ.
- Error: Could not find or load main class org.apache.catalina.startup.Bootstrap on OSX (or Linux): you might see this error in the console when attempting to start LabKey server. Update the '-classpath' VM option for your Run/Debug configuration to have Unix (:) path separators, rather than Windows path separators (;).
- On Windows, if you are seeing application errors, you can try resetting the winsock if it has gotten into a bad state. To reset:
- Open a command window in administrator mode
- Type into the command window: netsh winsock reset
- When you hit enter, that should reset the winsock and your application error may be resolved. You might need to restart your computer for the changes to take effect.
IntelliJ Slow
You can help IntelliJ run faster by increasing the amount of memory allocated to it. The specific method may vary depending on your version of IntelliJ and OS. Learn more here:
XML Classes not found after Gradle Refresh
Problem: After doing a Gradle Refresh, some of the classes with package names like org.labkey.SOME_SCOPE.xml (e.g., org.labkey.security.xml) are not found.
Cause: This is usually due to the relevant schema jars not having been built yet.
Solution: Run the deployApp command and then do a Gradle Refresh within IntelliJ. Though you can use a more targeted Gradle task schemaJar to build just the jar file that is missing, you might find yourself running many of these to resolve all missing classes, so it is likely most efficient to simply use deployApp
Gradle Troubleshooting
Understanding how gradle builds the server, and the relationships between building and cleaning can help you diagnose many build problems. See these related topics:
Update version of gradlePlugins
We maintain
release notes for the gradlePlugins that describe changes and bug fixes to the plugins. Check these notes to find out if the problem you are having has already been addressed. Be sure to pay attention to the "Earliest compatible LabKey version" indicated with each release to know if it is compatible with your current LabKey version. If you want to update to a new version of the gradle plugins to pick up a change, you need only update the gradlePluginsVersion property in the root-level gradle.properties file:
gradlePluginsVersion=1.3.6
Could not Resolve All Files for Configuration
Problem: When compiling a module, an error such as the following appears
Execution failed for task ':server:modules:myModule:compileJava'.
> Could not resolve all files for configuration ':server:modules:myModule:compileClasspath'.
> Could not find XYZ-api.jar (project :server:modules:XYZ).
Similarly, after doing a Gradle Refresh, some of the files that should be on the classpath are not found within IntelliJ. If you look in the Build log window, you see messages such as this:
<ij_msg_gr>Project resolve errors<ij_msg_gr><ij_nav>/Development/labkeyEnlistment/build.gradle<ij_nav><i><b>root project 'labkeyEnlistment': Unable to resolve additional project configuration.</b><eol>Details: org.gradle.api.artifacts.ResolveException: Could not resolve all dependencies for configuration ':server:modules:myModule:compileClasspath'.<eol>Caused by: org.gradle.internal.resolve.ArtifactNotFoundException: Could not find XYZ-api.jar (project :server:modules:XYZ).</i>
Cause: The settings.gradle file has included only a subset of the dependencies within the transitive dependency closure of your project and Gradle is
unable to resolve the dependency for the api jar file. In particular, though your settings file likely includes the :server:modules:XYZ project, it likely does not include another project that myModule depends on that also depends on XYZ. That is, you have the following dependencies:
- myModule depends on otherModule
- myModule depends on XYZ
- otherModule depends on XYZ
And in your settings file you have:
include ':server:modules:myModule'
include ':server:modules:XYZ'
(Notice that 'otherModule' is missing here.)
When gradle resolves the dependency for otherModule, it will pick it up from the artifact repository. That published artifact contains a reference to the API jar file for XYZ and Gradle will then try to resolve that dependency using your :server:modules:XYZ project, but referencing an artifact with a classifier (the -api suffix here) is
not something currently supported by Gradle, so it will fail. This may cause IntelliJ (particularly older versions of IntelliJ) to mark some dependencies within this dependency tree as "Runtime" instead of "Compile" dependencies, which will cause it to not resolve some references to classes within the Java files.
Solution: Include the missing project in your settings.gradle file
include ':server:modules:myModule'
include ':server:modules:otherModule'
include ':server:modules:XYZ'
Module Not Deployed
If you notice that your deployed server does not contain all the modules you expected, the first thing to verify is that your settings.gradle file includes all the modules you intend to build. You do this by running the following command
If the project is listed as expected in this output, the next thing to check is that the project has the appropriate plugins applied. See the
tip about plugins below for more information.
Module Plugins Not Applied or Missing Properties orMethods
FAILURE: Build failed with an exception.
* Where:
Build file '/path/to/Development/labkey/root/server/modules/myModule/build.gradle' line: 6
* What went wrong:
A problem occurred evaluating project ':server:modules:myModule'.
> Could not find method implementation() for arguments com.sun.mail:jakarta.mail:1.6.5 on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.
Cause: As of LabKey Server version 21.3.0 we no longer apply the LabKey gradle plugins for all modules under the server/modules directory by default. Each module is responsible for applying the plugins it requires. When running a deployApp command, if we find a gradle project that contains a module.properties file but does not have a task named 'module', a message like the following will be shown:
The following projects have a 'module.properties' file but no 'module' task. These modules will not be included in the deployed server. You should apply either the 'org.labkey.build.fileModule' or 'org.labkey.build.module' plugin in each project's 'build.gradle' file.
:server:modules:myModule
:server:modules:otherModule
Solution: Apply the appropriate plugin in the module's build.gradle file by including either
plugins {
id 'org.labkey.build.fileModule'
}
OR
plugins {
id 'org.labkey.build.module'
}
depending on whether the project houses a file-based module or a Java module, respectively. See the documentation on creating
Gradle modules for more information.
Starting Over with Gradle + IntelliJ
If you get into a situation where the IntelliJ configuration has been corrupted or created in a way that is incompatible with what we expect (which can happen if, for example, you say 'Yes' when IntelliJ asks if you want to have it link an unlinked Gradle project), these are the steps you can follow to start over with a clean setup for Gradle + IntelliJ:
- Shut down IntelliJ.
- Revert the <LK_ENLISTMENT>/.idea/gradle.xml file to the version checked in to VCS.
- Remove the <LK_ENLISTMENT>/.idea/modules directory.
- Remove the <LK_ENLISTMENT>/.idea/modules.xml file.
- Start up IntelliJ.
- Open the Gradle window (View > Tool Windows > Gradle) and click the Refresh icon, as described above.
This may also be required if the version of LabKey has been updated in your current enlistment but you find that your IntelliJ project still refers to jar files created with a previous version.
Log4J
log4j2.xml Appender
Problem: The server seems to have started fine, but there are no log messages in the console after server start up.
Cause: The log4j2.xml file that controls where messages are logged does not contain the appropriate appender to tell it to log to the console. This appender is added when you deploy the application in development mode (as a consequence of running the deployApp command).
In log4j2, a custom appender is defined by creating a plugin. The log4j documentation has an explanation with an example for a custom appender here:
LabKey Under Tomcat
LabKey Fails to Start
If LabKey fails to start successfully, check the steps above to ensure that you have configured your JDK and development environment correctly. Some common errors you may encounter include:
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "<username>" or
java.sql.SQLException: Login failed for user "<username>"These error occurs when the database user name or password is incorrect. If you provided the wrong user name or password in the
.properties file that you configured above, LabKey will not be able to connect to the database. Check that you can log into the database server with the credentials that you are providing in this file.
java.net.BindException: Address already in use: JVM_Bind:<port x>:
This error occurs when another instance of LabKey, Tomcat, or another application is running on the same port. Specifically, possible causes include:
- LabKey is trying to double bind the specified port.
- LabKey (or Tomcat) is already running (as a previous service, under IntelliJ, etc.
- Microsoft Internet Information Services (IIS) is running on the same port.
- Another application is running on the same port.
In any case, the solution is to ensure that your development instance of LabKey is running on a free port. You can do this in one of the following ways:
- Shut down the instance of LabKey or the application that is running on the same port.
- Change the port for the other instance or application.
- Edit the application.properties file to specify a different port for your installation of LabKey.
It is also possible to have misconfigured LabKey to double bind a given port. Check to see if your application.properties file has these two properties set to the same value:
server.port
context.httpPort
java.lang.NoClassDefFoundError: com/intellij/rt/execution/application/AppMain:
or
Error: Could not find or load main class com.intellij.rt.execution.application.AppMain:
In certain developer configurations, you will need to add an IntelliJ utility JAR file to your classpath.
- Edit the Debug Configuration in IntelliJ.
- Under the "VM Options" section, find the "-classpath" argument.
- Find your IntelliJ installation. On Windows machines, this is typically "C:\Program Files\JetBrains\IntelliJ IDEA <Version Number>" or similar. On OSX, this is typically "/Applications/IntelliJ IDEA <Version Number>.app" or similar.
- The required JAR file is in the IntelliJ installation directory, and is ./lib/idea_rt.jar. Add it to the -classpath argument value, separating it from the other values with a ":" on OSX and a ";" on Windows.
- Save your edits and start LabKey.
Database State Troubleshooting
If you build the LabKey source yourself from the source tree, you may need to periodically delete and recreate your LabKey database. The daily drops often include SQL scripts that modify the data and schema of your database.
Database Passwords Not Working
If your password contains a backslash, it may not get copied correctly from your properties file (i.e. pg.properties) to application.properties during the build and deploy process. Either use a password without a backslash, or double-escape the backslash within the properties file: (e.g. '\\\\').
System PATH
LabKey Server will automatically load certain binaries and tools (graphviz, dot, proteomics binaries) from Artifactory and expects to find them on the user's PATH. Be sure to include the following location in your system PATH. This directory won't exist before you build your server, but add it to the path anyway.
<LK_ENLISTMENT>\build\deploy\bin
For example, C:\labkeyEnlistment\build\deploy\bin.
Environment Variables Troubleshooting
JAVA_TOOL_OPTIONS
Most users will not have this problem. However, if you see a build error something like the following:
error: unmappable character for encoding ASCII
then setting this environment variable may fix the problem
export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8
Ext Libraries
Problem: Ext4 is not defined.
The page or a portion of the page is not rendering appropriately. Upon opening the browser console you see an error stating "Ext4 is not defined". This usually occurs due to a page not appropriately declaring client-side code dependencies.
Example of the error as seen in Chrome console.
Solution
Declare a dependency on the code/libraries you make use of on your pages. Depending on the mechanism you use for implementing your page/webpart we provide different hooks to help ensure dependent code is available on the page.
- Wiki: In a wiki you can use LABKEY.requiresScript() with a callback.
- File-based module view: It is recommended that you use file-scoped dependencies by declaring a view.xml and using the <dependencies> attribute.
- Java: In Java you can use ClientDependency.fromPath(String p) to declare dependencies for your view. Note, be sure to declare these before the view is handed off to rendering otherwise your dependency will not be respected.
- JSP: Override JspBase.addClientDependencies(ClientDependencies dependencies) in the .jsp. Here is an example.
Background: In the 17.3 release we were able to move away from ExtJS 4.x for rendering menus. This was the last “site-wide” dependency we had on ExtJS, however, we still continue to use it throughout the server in different views, reports, and dialogs. To manage these usages we use our dependency framework to ensure the correct resources are on the page. The framework provides a variety of mechanisms for module-based views, JavaServer Pages, and Java.
Renaming Files and Directories
Occasionally a developer decides a module or file (and possibly associated classes) should be renamed for clarity or readability.
Most file/directory name changes are handled transparently: rename locally through Intellij, verify your build and that tests pass, and commit.
The big exception to this is doing a case-only rename. (e.g., "myfile.java" -> "MyFile.java"). Because Windows file systems are case-insensitive, this causes major headaches.
Don't do it as described above.
But what if I really need to do a case-only rename?
Do a renaming to only change casing (ex. Mymodule -> myModule) in two rename steps:
- First rename to an intermediate name that changes more than the casing: (ex. Mymodule -> myModule2)
- Commit that change.
- Then rename to the target final name (ex. myModule2 -> myModule)
- Commit again.
- Push your commits.
- Squash-merge to develop.
Be sure to verify the build after each step. Depending upon what you are renaming, you may also have to rename classes.
Related Topics