In some cases, very large pipeline jobs may run long enough that they prevent smaller job queues like nightly maintenance tasks from running on schedule. In some cases, it may be possible to designate different pools of threads to these two sets of tasks in order to ensure that smaller tasks are not blocked.
Note that this is not currently supported for all types of long running pipeline jobs. Please contact your Account Manager if you are interested in learning whether parallel pipelines may be suitable for your use case.
Scenario Example: Assay Import with File Watchers
This topic highlights an example scenario. File watchers use a pipeline job to process new files, and so do background imports of assay data. The steps described in this topic tell how an administrator can enable use of parallel pipelines for both of these job types, allowing the large imports to run in the background and permit smaller jobs to complete.
Setup Steps
1. Deploy LabKey Server.
2. Find the <LABKEY_HOME>/config directory; a sibling of your deployment's <LABKEY_HOME>/modules directory. You can also specify an alternative location in your application.properties file via:
context.pipelineConfig=/path/to/pipeline/config/dir
3. Copy these three XML files into that config directory:
Customize these files as needed:
- Edit webserverMuleConfig.xml to define additional "locations" that can independently do work. See comments including the text "******" in the XML file. You need a pair of <connector> and <mule-descriptor> elements to create a new thread pool.
- Edit assayConfig.xml and/or premiumConfig.xml to refer to your desired location.
- As uploaded, the files define a single new thread pool that allows up to 2 parallel jobs, and assigned both file-watcher and assay import jobs to that pool/location.
4. Edit the
JMS configuration section of the application.properties file to enable the
in-process ActiveMQ queue via these five properties:
## JMS configuration for remote ActiveMQ message management for distributed pipeline jobs
context.resources.jms.ConnectionFactory.type=org.apache.activemq.ActiveMQConnectionFactory
context.resources.jms.ConnectionFactory.factory=org.apache.activemq.jndi.JNDIReferenceFactory
context.resources.jms.ConnectionFactory.description=JMS Connection Factory
context.resources.jms.ConnectionFactory.brokerName=LocalActiveMQBroker
## Use an in-process ActiveMQ queue
context.resources.jms.ConnectionFactory.brokerURL=vm://localhost?broker.persistent=false&broker.useJmx=false
5. Restart LabKey Server. You should see a number of new logging outputs in labkey.log that indicate your configuration is in place, including the equivalent of these lines (where <LABKEY_HOME> is the root of your enlistment):
INFO SpringModule 2022-03-24T18:43:47,896 Module Upgrade : Loading Spring configuration for the Assay module from [/WEB-INF/assayContext.xml, <LABKEY_HOME>/build/deploy/config/assayConfig.xml]
INFO SpringModule 2022-03-24T18:43:48,427 Module Upgrade : Loading Spring configuration for the Premium module from [/WEB-INF/premiumContext.xml, <LABKEY_HOME>/build/deploy/config/premiumConfig.xml]
INFO ModuleLoader 2022-03-24T18:43:49,278 Module Upgrade : Running startup listener: EPipeline
INFO MuleListenerHelper 2022-03-24T18:43:49,279 Module Upgrade : Found Mule configuration file override at <LABKEY_HOME>/build/deploy/config/webserverMuleConfig.xml
Validate the Connection
Open the Admin Console via
> Site > Admin Console.
Under
Management, click
Pipeline. You should see "You are running the Enterprise Pipeline" under the overview heading. Your new location name(s) should be shown here too.
Under
Diagnostics, click
Pipelines and Tasks. You can click through on the tasks you configured via assayConfig.xml and premiumConfig.xml (org.labkey.api.assay.pipeline.AssayUploadPipelineTask$Factory and org.labkey.premium.trigger.FileWatcherMoveTask$Factory, respectively) and click on them to see their target locations.
Under
Diagnostics, click
Running Threads. You should see thread(s) with names that correspond to your Mule configuration names. When a job is actively running, you should see the thread doing work.
In a job's log file, it should tell you what location it’s using for a task. Example output:
Starting to run task 'org.labkey.api.assay.pipeline.AssayUploadPipelineTask' at location 'newLocation'
Related Topics
