Developers who build LabKey Server from source will not be able to use the Ant build targets starting in 17.2 (July 2017). Instead, the Gradle build framework will be used. This topic is intended for early adopters who want to begin the process of migrating from Ant to Gradle early, before it is required.

For the latest news on the migration to Gradle, see Gradle Developer Discussion Thread.

• Summary
• General Setup
• Artifactory Setup
• Your First Gradle Commands
• Changing the Set of Projects
• Building from Source (or not)
• Command Line
• Ant-to-Gradle Command Line Mapping
• Tips
• Cleaning
• Module Cleaning
• Application Cleaning
• IntelliJ Setup
• Troubleshooting
• Global Properties Template
• Resources

Summary

It is assumed in what follows that you have a standard LabKey SVN enlistment.  We use LABKEY_ROOT to refer to the directory into which you check out your enlistment (i.e., the parent of the server directory). (You may also have git modules checked out, but the first step will be to get a build using SVN modules going.)

General Setup

Create a gradle properties file to contain information about global settings for your gradle build

• Create a .gradle directory in your home directory (e.g., /Users/<you>/.gradle on a Mac or C:\Users\<you>\.gradle on Windows). Note: the windows file explorer may not allow you to create a folder beginning with a period. To solve this navigate to C:\Users\<you>\ in the command prompt and type mkdir .gradle.
• Create a gradle.properties file in the .gradle directory using the template given below (or you can copy this from the LABKEY_ROOT/global_gradle.properties_template file)
• Substitute your tomcat home directory for the value after systemProp.tomcat.home
  On Windows, use forward slashes, not backslashes, for the Tomcat path. For example:
  systemProp.tomcat.home=C:/apache/apache-tomcat-8.5.11

Artifactory Setup (LabKey Staff Only)

This step is only for LabKey Software internal staff. If you are not a LabKey employee, skip this step.

1. Get your password or API key for the Artifactory server.

EITHER

• Log in to the Artifactory server using your GitHub credentials
• Let Susan or Brian know so your permissions can be set up to be able to read all necessary artifacts
• Click on your display name in the “Welcome, <user>” text in the upper right to go to your profile.
• Click the gear next to the API key input field to generate your API key
• Capture the API key so you can paste it in your properties file

OR

• Request an internal account on the Artifactory server from either Susan or Brian
• Log in to the Artifactory server
• Click on your display name in the “Welcome, <user>” text in the upper right to go to your profile.  
• Enter your password there to unlock the page and then scroll down to the Authentication Settings section.  The icons next to the Encrypted Password box allow you to see the encrypted value or copy it to the clipboard.

2. Modify the gradle properties file

• Substitute your (case-sensitive) artifactory user name  (e.g., labkey-susanh) and either your API key (if you logged in to artifactory with GitHub) or your encrypted password (if you used an internal Artifactory account) within this file.  Note that the text shown in the “Welcome, <user>” is a display name for your artifactory account, which is not the same as your user name, at least as far as casing is concerned.  You want to use your user name, not the display name in the properties file. This value is likely to be all lower case (e.g., for GitHub, this will be something like labkey-susanh).

Your First Gradle Commands

1. Clean out the ant build directories and files

ant clean

2. Put the Gradle script file in your operating system path.  This file (either gradlew or gradlew.bat) is included in the SVN sync and is already in the <LABKEY_ROOT> directory.
• For Windows, do the following:
• Open the start menu and type system environment variables then enter to open the system properties window.
• Click the button at the bottom labelled “Environment Variables...:”
• This will open another menu with two lists of variables. Find the variable “path” in the “System Variables” list.
• Clicking Edit will open a new menu, which enumerates all the items on your path.
• Click New then browse. Navigate to the <LABKEY_ROOT> directory.
• For Mac, add a line similar to the following in your ~/.bash_profile file

export PATH=<LABKEY_ROOT>:$PATH and then open a new terminal window.

3. Execute a gradle command to show you the set of currently configured projects (modules)
• On the command line, type gradlew projects (or gradlew.bat projects)
4. Execute a gradle command to build and deploy the application

gradlew deployApp

This will take some time as it needs to pull down many resources to initialize the local caches.  Subsequent builds will be much faster.

Changing the Set of Projects

Gradle uses the <LABKEY_ROOT>/settings.gradle file to determine which projects (modules) are included in the build.  (This is an analog of the various .modules files used in the ant build.)  To include a different set of projects in your build, you will need to edit this file.  By default, only modules in the server/modules directory and the server/test and server/test/modules directories are included in the build.  See the file for examples of different ways to include different subsets of the modules.  

Building from Source (or not)

This feature is currently only for LabKey Software internal staff. If you are not a LabKey employee, skip this section.

N.B.  As of 12 January 2017, there is a bug related to the buildFromSource parameter in conjunction with a dev build having to do with the .lib.xml files and client libraries, which means this probably won’t work as advertised here just yet.

Which modules are built from source is controlled by the buildFromSource property.  This property can be set at the LABKEY_ROOT level, at the project level, at the global (user) level, or on the command line (with overrides happening in the order given).

The default properties file in LABKEY_ROOT/gradle.properties has the following property defined

buildFromSource=true

This setting will cause a build command to construct all the jars and the .module files that are necessary for a server build.  But if you are not changing code, you do not need to build everything from source.  The following table illustrates how you can specify the properties to build just the source code you need to build.  

Command Line

Ant-to-Gradle Command Line Mapping

The following table provides a mapping between the most popular (I assume) ant commands and their Gradle counterparts. Note that while a mapping is provided for the ‘ant clean’ and ‘ant rebuild’ commands to get you oriented, these commands should no longer be used automatically with each new day or each new sync from SVN.  Gradle handles checking for needed updates much better and so you do not need to proactively clean as often.

Tips

Here we include a few bits of information that may be useful for learning to use the Gradle command line.

• Use gradlew -h to see the various options available for running gradle commands
• If working offline, you will want to use the --offline option to prevent it from contacting the artifact server (You won’t have success if you do this for your first build.)
• By default, gradle outputs information related to the progress in building and which tasks it considers as up to date or skipped.  If you don’t want to see this, or any other output about progress of the build, you’ll want to add the -q flag:

./gradlew -q projects

Now might be a good time to set up an alias if you’re a command-line kind of person who can’t abide output to the screen.

• If doing development in a single module, it is most efficient to do builds from the module’s directory as that will cause Gradle to do the least amount of building and configuration.  There is a command available to you that can be sort of a one-stop shopping experience:

gradlew deployModule

This will build the jar files, and the .module file and then copy it to the build/deploy/modules directory, which will cause Tomcat to refresh.

• Gradle commands can (generally) be executed from anywhere within the project directory structure.  You need only provide the gradle path (Gradle paths use colons as separators instead of slashes in either direction) to the project as part of the target name. For example, from the server/modules directory, you can build the announcements module using the command

gradlew announcements:module

And from the server/modules/announcements directory, you can build the wiki module using

gradlew :server:modules:wiki:module

One exception to this is that in directories that have subprojects declared in a build.gradle file, (e.g., the root directory or the server directory), you can run commands such as “deployModule” and “clean” and it will run the corresponding tasks in all the subdirectories.  These global tasks are not executable from other directories.

• Gradle provides many helpful tasks to advertise the capabilities and settings within the build system. Start with this and see where it leads you

gradlew tasks

• Gradle automatically understands shortcuts and when you mistype a target name it will suggest possible targets nearby.  For example, you can build the announcements module with this command:

gradlew :se:m:an:b

And when you switch back to ant task mode momentarily and type

gradlew pick_pg

Gradle responds with:

* What went wrong:

Task 'pick_pg' not found in project ':server'. Some candidates are: 'pickPg'.

Cleaning

First, you should know that Gradle is generally very good about keeping track of when things have changed and so you can, and should, get out of the habit of wiping things clean and starting from scratch because it just takes more time.  If you find that there’s some part of the process that does not recognize when its inputs have changed or its outputs are up-to-date, please file a bug or post to the [developer support board|https://www.labkey.org/home/Support/Developer%20Forum/project-begin.view?] so we can get that corrected.  (As of Feburary 2017, there is one known issue with the GWT compiler up-to-date check.)

The gradle tasks also provide much more granularity in cleaning.  Generally, for each task that produces an artifact, we try to have a corresponding cleaning task that removes that artifact.  This leads to a plethora of cleaning tasks, but there are only a few that you will probably ever want to use.  We summarize the most commonly useful ones here.

Module Cleaning

The two most important tasks for cleaning modules are:

• undeployModule - removes all artifacts for this module from the staging and deploy directories.  This should always be used when switching between feature branches if you have set the includeVcs property.
• reallyClean - removes the build directory for the module as well as all artifacts for this module from the staging and deploy directories.  Use this to remove all evidence of your having built a module.

undeployModule - This is the opposite of deployModule.  deployModule copies artifacts from the build directories into the staging (LABKEY_ROOT/build/staging) and then the deployment (LABKEY_ROOT/build/deploy) directories, so undeployModule removes the artifacts for this module from the staging and deployment directories.  This will cause a restart of a running server since tomcat will recognized that the deployment directory is changed.  This command should always be used when switching between feature branches because the artifacts created in a feature branch will have the feature branch name in their version number and thus will look different from artifacts produced from a different branch.  If you don’t do the undeployModule, you’ll likely end up with multiple versions of your .module file in the deploy directory and thus in the classpath, which will cause confusion.

clean - This task comes from the standard Gradle lifecycle.  Its purpose, generally, is to remove the build directory for a project.  For our modules, these are the directories under LABKEY_ROOT/build/modules.  Note that this will have little to no effect on a running server instance.  It will simply cause gradle to forget about all the building it has previously done so the next time it will start from scratch.

reallyClean - combines undeployModule and clean to remove the build, staging and deployment directories for a module.

Application Cleaning

cleanBuild - Removes the build directory entirely.  This will also stop the tomcat server if it is running.  This is the big hammer that you should avoid using unless there seems to be no other way out.   

cleanDeploy - Removes the build/deploy directory.  This will also stop the tomcat server if it is running.

cleanStaging - Removes the build/staging directory.  This does not affect the running server.

IntelliJ Setup

Follow these steps in order to make IntelliJ able to find all the source code and elements on the classpath as well as be able to run tests.

• Upgrade to the latest version of IntelliJ. Version 2016.1.x has issues with Gradle. 2016.3.4 has been tested and works.
• Be sure that IntelliJ has enough heap memory. The default max is OK if you’re just dealing with the core modules, but you will likely need to raise the limit if you’re adding in customModules, optionalModules, etc. 3GB seems sufficient.
• Enable the Gradle plugin in IntelliJ
• Go to File -> Preferences/Settings -> Plugins and choose “Gradle”
• Gradle is enabled by default, so this may not be necessary.
• Create the workspace.xml file for the IntelliJ project in LABKEY_ROOT (NOTE: this is a different project than the one in the LABKEY_ROOT/server directory, which will be removed after transition has happened.  If you want to use the workspace.xml file that currently resides in the LABKEY_ROOT/server/.idea directory, you will need to edit this file and adjust the paths that refer to $PROJECT_DIR$ such that they are accurate for the root being at LABKEY_ROOT instead of LABKEY_ROOT/server)
• Copy the file LABKEY_ROOT/.idea/workspace.template.xml. Rename the copy to create a file called LABKEY_ROOT/.idea/workspace.xml
• This file configures the debug information for LabKey project. To review the debug settings go to Run > Edit Configurations in IntelliJ.
• Close the current IntelliJ project
• If your IntelliJ install is brand new, you will see the "Welcome to IntelliJ" pop up screen. Click Open. If you have previously installed IntelliJ, select File > Open.
• Select the LabKey IntelliJ project directory, <LABKEY_HOME>
• If asked about an “Unlinked Gradle project”, DO NOT “Import Gradle project” in the default way from IntelliJ.
• If your project is new, make sure your Project SDK is set correctly. This is under “Project Structure”.
• Open the Gradle tool window (using icon shown on the right)
• Then click the Refresh icon in that window.  (This will take a while, perhaps 15-30 minutes. After a few minutes, you should start seeing messages about its progress. If not, something is probably hung up.)

• After your Gradle Sync is done, edit the Run / Debug Configuration:
• Go to Run -> Edit Configurations. Select LabKey Dev.
• VM options: Confirm that the path separators are appropriate for your operating system. On Windows, ensure that the paths to the jar files are separated by semicolons. For example: "./bin/bootstrap.jar;./bin/tomcat-juli.jar;C:/Program Files (x86)/JetBrains/IntelliJ IDEA 2016.3.3/lib/idea_rt.jar"
• Confirm that ‘api_main’ is chosen for the setting “Use classpath of module”.

Troubleshooting

Problem: Gradle Sync in IntelliJ has no effect on project structure after the settings.gradle file is updated.

Cause: Name conflict between the gel_test project and the gel_test IntelliJ module that would be created from the gel project’s ‘test’ source set. (IDEA-168284)

Workarounds: Do one of the following

• Remove gel_test from the test of projects you are including in your settings.gradle file and the do the Gradle Sync.
• Within IntelliJ do the following
• Preferences -> Build, Execution, Deployment -> Build Tools -> Gradle
• Uncheck the setting “Create separate module per source set”
• Click “OK”
• Do the Gradle Sync in the Gradle window
• Preferences -> Build, Execution, Deployment -> Build Tools -> Gradle
• Check the setting “Create separate module per source set”
• Click “OK”
• Do the Gradle Sync in the Gradle window

Problem: Running tests within IntelliJ for the first time, you will likely encounter a problem related to not being able to find apache httpmime classes.  

Cause: A bug in IntelliJ, related to our project structure which marks some dependencies as “Runtime” instead of “Compile”.

Solution: To fix this, you will need to go to File-> Project Structure -> Modules -> remoteapi -> java -> java_main -> Dependencies.  You will see that there are a few dependencies designated as “Runtime” instead of “Compile”.  Choose “Compile” and then click “OK”.  After that your tests should be runnable.  Unfortunately, you’ll need to make this modification every time you manually sync from the Gradle window, but fortunately that shouldn’t be needed all that often.

Problem: My passwords to PostgreSQL and MS SQL Server aren't working.

Solution: Unlike Ant, the Gradle build system will automatically escape any special XML characters, such as quotes and ampersand symbols in the pg.properties / mssql.properties files. When migrating these files from Ant to Gradle, replace any escaped ampersands (&amp;) with plain text ampersands (&).

Global Properties Template

deployMode=dev
# When set to true, Gradle will run the build with remote debugging enabled, listening on port 5005.
# Note that this is the equivalent of adding -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 to the JVM command line
# and will suspend the virtual machine until a debugger is attached.
#org.gradle.debug=true
#We need to know the tomcat home directory for some of the dependencies in our build process.  IntelliJ does not
#pick up the CATALINA_HOME environment variable, so if working with the IDE, you need to set the tomcat.home
#system property either here as shown below on the command line with -Dtomcat.home=/path/to/tomcat/home
#Regardless of OS, use the forward slash (/) as a file separator in the path (yes, even on Windows)
systemProp.tomcat.home=/path/to/tomcat/installation
artifactory_user=<your user>
# the encrypted password or API key
artifactory_password=<your password or API key>
# uncomment to enable population of VCS revision number and url in module.properties file
# for localhost installation
#includeVcs
#svn_user=<your svn user name>
#svn_password=<your svn password>

Resources

• Gradle documentation - A basic introduction to the command line.
• Gradle Developer Discussion Thread - Current news on the migration to Gradle.