Developers who build LabKey Server from source will not be able to use the Ant build targets starting in 17.2 (July 2017). Instead, the Gradle build framework will be used. This topic is intended for developers who want to migrate their already existing LabKey Server development environment from Ant to Gradle. If you want to build a new development environment from scratch use the following topic instead: Set Up a Development Machine.

For the latest news on the migration to Gradle, see Gradle Developer Discussion Thread.

Summary

It is assumed in what follows that you have a standard LabKey SVN enlistment.  We use LABKEY_ROOT to refer to the directory into which you check out your enlistment (i.e., the parent of the server directory). (You may also have git modules checked out, but the first step will be to get a build using SVN modules going.)

General Setup

Create a gradle properties file to contain information about global settings for your gradle build

  • Create a .gradle directory in your home directory (e.g., /Users/<you>/.gradle on a Mac or C:\Users\<you>\.gradle on Windows). Note: the windows file explorer may not allow you to create a folder beginning with a period. To solve this navigate to C:\Users\<you>\ in the command prompt and type mkdir .gradle.
  • Create a gradle.properties file in the .gradle directory using the template given below (or you can copy this from the LABKEY_ROOT/gradle/global_gradle.properties_template file)
  • Substitute your tomcat home directory for the value after systemProp.tomcat.home
    On Windows, use forward slashes, not backslashes, for the Tomcat path. For example:
    systemProp.tomcat.home=C:/apache/apache-tomcat-8.5.11

Your First Gradle Commands

1. Clean out the ant build directories and files

ant clean

2. Execute a gradle command to show you the set of currently configured projects (modules).  You do not need to install gradle and should resist the urge to do so.  We use the gradle wrapper to make updating version numbers easier. The gradle wrapper script (either gradlew or gradlew.bat) is included in the SVN sync and is already in the <LABKEY_ROOT> directory. 

On the command line, type ./gradlew projects (or gradlew.bat projects)

3. Execute a gradle command to build and deploy the application

./gradlew :server:deployApp

This will take some time as it needs to pull down many resources to initialize the local caches.  Subsequent builds will be much faster.

Changing the Set of Projects

Gradle uses the <LABKEY_ROOT>/settings.gradle file to determine which projects (modules) are included in the build.  (This is an analog of the various .modules files used in the ant build.)  To include a different set of projects in your build, you will need to edit this file.  By default, only modules in the server/modules directory and the server/test and server/test/modules directories are included in the build.  See the file for examples of different ways to include different subsets of the modules.  

Command Line

Ant-to-Gradle Command Line Mapping

The following table provides a mapping between the most popular (I assume) ant commands and their Gradle counterparts. Note that while a mapping is provided for the ‘ant clean’ and ‘ant rebuild’ commands to get you oriented, these commands should no longer be used automatically with each new day or each new sync from SVN.  Gradle handles checking for needed updates much better and so you do not need to proactively clean as often.

Ant Command

Gradle Command

Directory

ant build

gradlew deployApp

trunk or server

ant clean

gradlew cleanBuild

trunk or server

ant rebuild

gradlew cleanBuild deployApp

trunk or server

ant pick_pg

gradlew pickPg

trunk or server

ant pick_mssql

gradlew pickMSSQL

trunk or server

ant dist

gradlew distribution

trunk

ant wiki

gradlew :server:modules:wiki:deployModule

Any directory

ant wiki

gradlew deployModule

server/modules/wiki

ant test

gradlew :server:test:uiTest

Any directory

ant drt

gradlew :server:test:uiTest -Psuite=DRT

Any directory

ant production

gradlew deployApp -PdeployMode=prod

trunk or server

Tips

Here we include a few bits of information that may be useful for learning to use the Gradle command line.

  • Use gradlew -h to see the various options available for running gradle commands
  • If working offline, you will want to use the --offline option to prevent it from contacting the artifact server (You won’t have success if you do this for your first build.)
  • By default, gradle outputs information related to the progress in building and which tasks it considers as up to date or skipped.  If you don’t want to see this, or any other output about progress of the build, you’ll want to add the -q flag:

gradlew -q projects

Now might be a good time to set up an alias if you’re a command-line kind of person who can’t abide output to the screen.

  • If doing development in a single module, it is most efficient to do builds from the module’s directory as that will cause Gradle to do the least amount of building and configuration.  There is a command available to you that can be sort of a one-stop shopping experience:

gradlew deployModule

This will build the jar files, and the .module file and then copy it to the build/deploy/modules directory, which will cause Tomcat to refresh.

  • Gradle commands can (generally) be executed from anywhere within the project directory structure.  You need only provide the gradle path (Gradle paths use colons as separators instead of slashes in either direction) to the project as part of the target name. For example, from the server/modules directory, you can build the announcements module using the command

gradlew announcements:module

And from the server/modules/announcements directory, you can build the wiki module using

gradlew :server:modules:wiki:module

One exception to this is that in directories that have subprojects declared in a build.gradle file, (e.g., the root directory or the server directory), you can run commands such as “deployModule” and “clean” and it will run the corresponding tasks in all the subdirectories.  These global tasks are not executable from other directories.

  • Gradle provides many helpful tasks to advertise the capabilities and settings within the build system. Start with this and see where it leads you

gradlew tasks

  • Gradle automatically understands shortcuts and when you mistype a target name it will suggest possible targets nearby.  For example, you can build the announcements module with this command:

gradlew :se:m:an:b

And when you switch back to ant task mode momentarily and type

gradlew pick_pg

Gradle responds with:

* What went wrong:

Task 'pick_pg' not found in project ':server'. Some candidates are: 'pickPg'.

Cleaning

First, you should know that Gradle is generally very good about keeping track of when things have changed and so you can, and should, get out of the habit of wiping things clean and starting from scratch because it just takes more time.  If you find that there’s some part of the process that does not recognize when its inputs have changed or its outputs are up-to-date, please file a bug or post to the [developer support board|https://www.labkey.org/home/Support/Developer%20Forum/project-begin.view?] so we can get that corrected. 

The gradle tasks also provide much more granularity in cleaning.  Generally, for each task that produces an artifact, we try to have a corresponding cleaning task that removes that artifact.  This leads to a plethora of cleaning tasks, but there are only a few that you will probably ever want to use.  We summarize the most commonly useful ones here.

Module Cleaning

The two most important tasks for cleaning modules are:

  • undeployModule - removes all artifacts for this module from the staging and deploy directories.  This should always be used when switching between feature branches if you have set the includeVcs property.
  • reallyClean - removes the build directory for the module as well as all artifacts for this module from the staging and deploy directories.  Use this to remove all evidence of your having built a module.

undeployModule - This is the opposite of deployModule.  deployModule copies artifacts from the build directories into the staging (LABKEY_ROOT/build/staging) and then the deployment (LABKEY_ROOT/build/deploy) directories, so undeployModule removes the artifacts for this module from the staging and deployment directories.  This will cause a restart of a running server since tomcat will recognized that the deployment directory is changed.  This command should always be used when switching between feature branches because the artifacts created in a feature branch will have the feature branch name in their version number and thus will look different from artifacts produced from a different branch.  If you don’t do the undeployModule, you’ll likely end up with multiple versions of your .module file in the deploy directory and thus in the classpath, which will cause confusion.

directory

gradle command

result

trunk

clean

All build directories for all projects are removed

any

:server:modules:wiki:undeployModule

LABKEY_ROOT/build/staging/modules/wiki* and /build/deploy/modules/wiki* will be removed

server/modules/wiki

undeployModule

LABKEY_ROOT/build/staging/modules/wiki* and /build/deploy/modules/wiki* will be removed

clean - This task comes from the standard Gradle lifecycle.  Its purpose, generally, is to remove the build directory for a project.  For our modules, these are the directories under LABKEY_ROOT/build/modules.  Note that this will have little to no effect on a running server instance.  It will simply cause gradle to forget about all the building it has previously done so the next time it will start from scratch.

directory

gradle command

result

trunk

clean

All build directories for all projects are removed

any

:server:modules:wiki:clean

LABKEY_ROOT/build/modules/wiki is removed.

server/modules/wiki

clean

LABKEY_ROOT/build/modules/wiki is removed

reallyClean - combines undeployModule and clean to remove the build, staging and deployment directories for a module.

Application Cleaning

cleanBuild - Removes the build directory entirely.  This will also stop the tomcat server if it is running.  This is the big hammer that you should avoid using unless there seems to be no other way out.   

cleanDeploy - Removes the build/deploy directory.  This will also stop the tomcat server if it is running.

cleanStaging - Removes the build/staging directory.  This does not affect the running server.

IntelliJ Setup

Follow these steps in order to make IntelliJ able to find all the source code and elements on the classpath as well as be able to run tests.

  • Upgrade to the latest version of IntelliJ. Version 2016.1.x has issues with Gradle. 2016.3.4 has been tested and works.
  • Be sure that IntelliJ has enough heap memory. The default max is OK if you’re just dealing with the core modules, but you will likely need to raise the limit if you’re adding in customModules, optionalModules, etc. 3GB seems sufficient.
  • Enable the Gradle plugin in IntelliJ
    • Go to File -> Preferences/Settings -> Plugins and choose “Gradle”
    • Gradle is enabled by default, so this may not be necessary.
  • Create the workspace.xml file for the IntelliJ project in LABKEY_ROOT (NOTE: this is a different project than the one in the LABKEY_ROOT/server directory, which will be removed after transition has happened.  If you want to use the workspace.xml file that currently resides in the LABKEY_ROOT/server/.idea directory, you will need to edit this file and adjust the paths that refer to $PROJECT_DIR$ such that they are accurate for the root being at LABKEY_ROOT instead of LABKEY_ROOT/server)
    • Copy the file LABKEY_ROOT/.idea/workspace.template.xml. Rename the copy to create a file called LABKEY_ROOT/.idea/workspace.xml
    • This file configures the debug information for LabKey project. To review the debug settings go to Run > Edit Configurations in IntelliJ.
  • Close the current IntelliJ project
  • If your IntelliJ install is brand new, you will see the "Welcome to IntelliJ" pop up screen. Click Open. If you have previously installed IntelliJ, select File > Open.
  • Select the LabKey IntelliJ project directory, <LABKEY_HOME>
  • If asked about an “Unlinked Gradle project”, DO NOT “Import Gradle project” in the default way from IntelliJ. (If you do this your IntelliJ files will be in a somewhat strange stage, but you can follow the steps below to undo this and start again.)
  • If your project is new, make sure your Project SDK is set correctly. This is under “Project Structure”.
  • Open the Gradle tool window (using icon shown on the right) Screen Shot 2016-11-26 at 1.54.53 PM.png
  • Then click the Refresh icon in that window.  (This will take a while, perhaps 15-30 minutes. After a few minutes, you should start seeing messages about its progress. If not, something is probably hung up.)

  • After your Gradle Sync is done, edit the Run / Debug Configuration:
    • Go to Run -> Edit Configurations. Select LabKey Dev.
    • VM options: Confirm that the path separators are appropriate for your operating system. On Windows, ensure that the paths to the jar files are separated by semicolons. For example: "./bin/bootstrap.jar;./bin/tomcat-juli.jar;C:/Program Files (x86)/JetBrains/IntelliJ IDEA 2016.3.3/lib/idea_rt.jar".  For Macs, the paths should be separated by a colon.
    • Confirm that ‘api_main’ is chosen for the setting “Use classpath of module”

Starting Over

If you get into a situation where the IntelliJ configuration has been corrupted or created in a way that is incompatible with what we expect (which can happen if, for example, you say 'Yes' when IntelliJ asks if you want to have it link an unlinked Gradle project) , these are the steps you can follow to start over with a clean setup for Gradle + IntelliJ:

  • Shut down IntelliJ
  • Revert the LABKEY_ROOT/.idea/gradle.xml file to the version checked in to VCS
  • Remove the LABKEY_ROOT/.idea/modules directory
  • Remove the LABKEY_ROOT/.idea/modules.xml file
  • Start up IntelliJ
  • Open the Gradle window and click the Refresh icon, as described above

Troubleshooting

Problem: Gradle Sync in IntelliJ has no effect on project structure after the settings.gradle file is updated.

Cause: Name conflict between the gel_test project and the gel_test IntelliJ module that would be created from the gel project’s ‘test’ source set. (IDEA-168284)

Workarounds: Do one of the following

  • Remove gel_test from the test of projects you are including in your settings.gradle file and then do the Gradle Sync.
  • Within IntelliJ do the following
    • Preferences -> Build, Execution, Deployment -> Build Tools -> Gradle
    • Uncheck the setting “Create separate module per source set”
    • Click “OK”
    • Do the Gradle Sync in the Gradle window
    • Preferences -> Build, Execution, Deployment -> Build Tools -> Gradle
    • Check the setting “Create separate module per source set”
    • Click “OK”
    • Do the Gradle Sync in the Gradle window

Problem: My passwords to PostgreSQL and MS SQL Server aren't working.

SolutionUnlike Ant, the Gradle build system will automatically escape any special XML characters, such as quotes and ampersand symbols in the pg.properties / mssql.properties files. When migrating these files from Ant to Gradle, replace any escaped ampersands (&amp;) with plain text ampersands (&).

Problem: Server does not start in IntelliJ using one of the LabKey-provided Run/Debug configurations.  In the console window the following error appears:

 /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java 
-agentlib:jdwp=transport=dt_socket,address=127.0.0.1:59311,suspend=y,server=n -Dcatalina.base=./ 
-Dcatalina.home=./ -Djava.io.tmpdir=./temp -Ddevmode=true -ea -Dsun.io.useCanonCaches=false -Xmx1G 
-XX:MaxPermSize=160M -classpath "./bin/*;/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar" 
-javaagent:/Users/susanh/Library/Caches/IntelliJIdea2016.3/groovyHotSwap/gragent.jar 
-Dfile.encoding=UTF-8 org.apache.catalina.startup.Bootstrap start
Connected to the target VM, address: '127.0.0.1:59311', transport: 'socket'
Error: Could not find or load main class org.apache.catalina.startup.Bootstrap
Disconnected from the target VM, address: '127.0.0.1:59311', transport: 'socket'

Cause: This is most likely caused by an incorrect path separator in the Run/Debug configuration's classpath argument.

Solution: Edit the Run/Debug configuration and change the separator to the one appropriate to your platform (semicolon for Windows; colon for Mac/Linux).

Global Properties Template

deployMode=dev
# When set to true, Gradle will run the build with remote debugging enabled, listening on port 5005.
# Note that this is the equivalent of adding 
# -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 to the JVM command line
# and will suspend the virtual machine until a debugger is attached.
#org.gradle.debug=true
#We need to know the tomcat home directory for some of the dependencies in our build process.  IntelliJ does not
#pick up the CATALINA_HOME environment variable, so if working with the IDE, you need to set the tomcat.home
#system property either here as shown below on the command line with -Dtomcat.home=/path/to/tomcat/home
#Regardless of OS, use the forward slash (/) as a file separator in the path (yes, even on Windows)
systemProp.tomcat.home=/path/to/tomcat/installation
# uncomment to enable artifactory user; this should not generally be required
#artifactory_user=<your user>
# the encrypted password or API key;uncomment when supplying an artifactory_user
#artifactory_password=<your password or API key>
# uncomment to enable population of VCS revision number and url in module.properties file
# for localhost installation
#includeVcs
#svn_user=<your svn user name>
#svn_password=<your svn password>

Related Topics

discussion

Discussion

Search 

Pages 

previousnext
 
expand all collapse all