• Sign In
  1. Start Page
  2. Administration
  3. Install LabKey
  4. Example Setups and Configurations

Set up a JMS-Controlled Remote Pipeline Server

_Documentation

Set up a JMS-Controlled Remote Pipeline Server

Print

The topic provides step by step instructions for setting up the following configuration:

  • a stand-alone LabKey Server installation
  • a Remote Pipeline Server (running on a different machine)
  • a JMS service for communication between these servers

Steps:

  1. Set Up a Shared File System
  2. Set Up a JMS Queue
  3. Install LabKey Server
  4. Install Remote Pipeline Service

 

#1. Set up a Shared File System

  • Decide on the location where the source data files will reside.
  • Share out this directory as a network share so that the Remote Pipeline Server machine can mount it.
  • Mount the directory, mapping it to a drive.
  • Record the network share and the mapped drive for use later in the Remote Pipeline Server installation wizard. In the wizard, these paths are referred to as:
    • LabKey Server webserver path to data files
    • Pipeline Remote Server path data files

 

#2. Set up a JMS Queue

As you install the JMS queue, record these values for use later in the Remote Pipeline Server installation wizard:

  • Host
  • Port

The pipeline requires a JMS Queue to transfer messages between the different pipeline services. The LabKey Server currently supports the ActiveMQ JMS Queue from the Apache Software Foundation.

JMS: Installation Steps

  1. Choose a server on which to run the JMS Queue
  2. Install the Java Runtime Environment
  3. Install and Configure ActiveMQ
  4. Test the ActiveMQ Installation

Choose a server to run the JMS Queue

ActiveMQ supports all major operating systems (including Windows, Linux, Solaris and Mac OSX). (For example, the Fred Hutchinson Cancer Research Institute runs ActiveMQ on the same linux server as the GRAM Server.) For this documentation we will assume you are installing on a Linux based server.

Install the Java Runtime Environment

  1. Download the Java Runtime Environment (JRE) from http://java.sun.com/javase/downloads/index.jsp
  2. Install the JRE to the chosen directory.
  3. Create the JAVA_HOME environmental variable to point at your installation directory.

Install and Configure ActiveMQ

Note: LabKey currently supports ActiveMQ 5.1.0 only.

Download and Unpack the distribution

  1. Download ActiveMQ from ActiveMQ's download site
  2. Unpack the binary distribution from into /usr/local
    1. This will create /usr/local/apache-activemq-5.1.0
  3. Create the environmental variable <ACTIVEMQ_HOME> and have it point at /usr/local/apache-activemq-5.1.0

Configure logging for the ActiveMQ server

To log all messages sent through the JMSQueue, add the following to the <broker> node in the config file located at <ACTIVEMQ-HOME>/conf/activemq.xml

<plugins>
      <!-- lets enable detailed logging in the broker -->
      <loggingBrokerPlugin/>
</plugins>

During the installation and testing of the ActiveMQ server, you might want to show the debug output for the JMS Queue software. You can enable this by editing the file <ACTIVEMQ-HOME>/conf/log4j.properties

uncomment

#log4j.rootLogger=DEBUG, stdout, out

and comment out

log4j.rootLogger=INFO, stdout, out

Authentication, Management and Configuration

  1. Configure JMX to allow us to use Jconsole and the JMS administration tools monitor the JMS Queue
  2. We recommend configuring Authentication for your ActiveMQ server. There are number of ways to implement authentication. See http://activemq.apache.org/security.html
  3. We recommend configuring ActiveMQ to create the required Queues at startup. This can be done by adding the following to the configuration file <ACTIVEMQ-HOME>/conf/activemq.xml
<destinations>
     <queue physicalName="job.queue" />
     <queue physicalName="status.queue" />
</destinations>

Start the server

To start the ActiveMQ server, you can execute the command below. This command will start the ActiveMQ server with the following settings
    • Logs will be written to <ACTIVEMQ_HOME>/data/activemq.log
    • StdOut will be written to /usr/local/apache-activemq-5.1.0/smlog
    • JMS Queue messages, status information, etc will be stored in <ACTIVEMQ_HOME>/data
    • job.queue Queue and status.queue will be durable and persistant. (I.e., messages on the queue will be saved through a restart of the process.)
    • We are using AMQ Message Store to store Queue messages and status information
To start the server, execute

<ACTIVEMQ_HOME>/bin/activemq-admin start xbean:<ACTIVEMQ_HOME>/conf/activemq.xml > <ACTIVEMQ_HOME>/smlog 2>&1 &

Monitoring JMS Server, Viewing JMS Queue configuration and Viewing messages on a JMS Queue.

Using the ActiveMQ management tools

Browse the messages on queue by running

<ACTIVEMQ_HOME>/bin/activemq-admin browse --amqurl tcp://localhost:61616 job.queue

View runtime configuration, usage and status of the server information by running

<ACTIVEMQ_HOME>/bin/activemq-admin query

Using Jconsole

Here is a good quick description of using Jconsole to test your ActiveMQ installation. Jconsole is an application that is shipped with the Java Runtime. The management context to connect to is

service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi

#3. Install LabKey Server (on Linux)

This Wiki web part is not configured to display content.

#4. Install Remote Pipeline Server (on Windows)

These instructions explain how to install a Remote Pipeline Server using the graphical installer for Windows.

A Remote Pipeline Server works in co-ordination with an existing LabKey Server installation, especially to execute long-running, data-intensive pipeline jobs assigned to it by the LabKey Server machine. You cannot install LabKey Server and a Remote Pipeline Server on the same machine: install the Remote Pipeline Server on a physically different machine than the one running LabKey Server. LabKey Server (i.e., the web server and database server) is not included in the Remote Pipeline Server installation.

Steps to Install: Remote Pipeline Server

  • Download and run the Windows installer: free download after registration.
  • On the Installation Type page, select Remote Pipeline Server.
  • Accept the license agreement.
  • To use the wizard-based installation, select Config Wizard.
  • Specify the JMS Server to communicate with the Remote Pipeline Server:
    • host
    • port.
  • Specify the pipeline file locations.
    • LabKey Server webserver path to data files
    • Pipeline Remote Server path to data files
  • Specify how to mount the network file share, either automatically or externally.
  • If you choose automatically, you'll also be prompted to specify information for mapping a network drive where LabKey Server can access files on a remote server:
    • Drive Letter
    • Network Drive Path (the UNC path to the remote server)
    • DOMAIN\username and password for accessing that share. User name and password can be left blank if they are not required by the file share.
  • Accept the MSFileReader agreement. MSFileReader is a 3rd party tool that converts binary mass spectrometry files to the mzXML format.
  • Specify the directory location for the Remote Pipeline Server installation.

Search

 

Pages

 

previousnext
 
expand allcollapse all