Get Started With LabKey Server 9.1

• Administrators + Potential Adopters
• Install the Server
• Access the Server
• Set Up a Folder & its Tools
• Learn User Basics
• Learn How to Administer your Server, particularly Security.
• Users
• Access the Server
• Learn User Basics
• Developers
• Extend LabKey Server
• Use LabKey APIs to Create Views

Version 9.1 Improvements

• What's New in v9.1 (released April 2, 2009)
  
• Items completed in 9.1

Training Materials

• Webinars and Videos
• Tutorials and Online Demos

Still Have Questions?

• Search the documentation. Use the Search box in the upper right corner of this page.
• Search the community forums. Each forum has a search box on its upper right side.
• Obtain commercial support. LabKey Corporation provides consulting services to users who need assistance installing, enhancing and maintaining the LabKey Server platform in a production setting. Email info@labkey.com for further information.
• Review documentation archive. See Documentation for LabKey Versions 1.1-8.3

Future Directions for LabKey Server

• Roadmap for the Future

Most LabKey projects are secured to protect the data they contain, so you will want to log in to access your projects. Depending on how LabKey is set up for your organization, you may be able to log in using your network user name and password, or you have have to request a LabKey account. If you're not sure, ask your administrator. He or she can create an account for you if you don't already have one, and also grant you project permissions as needed.

Once you've logged in, you can edit your account information by clicking on the My Account link in the upper right corner of any page.

Supported Browsers

LabKey is a web application that runs in your web browser. To access LabKey, you must use a web browser that LabKey supports.

• On Windows, you can use either Microsoft Internet Explorer or Mozilla Firefox.
• On Unix-based systems, use Firefox. The older Mozilla browser may also work, but it is not technically supported for use with LabKey.
• On the Macintosh, you must use Firefox to access LabKey. Other popular Mac browsers like Safari and Internet Explorer have serious problems with JavaScript, which is required for some key features of LabKey.

1. Access the Server
2. Add Users
3. Create Project or Folder. For further background, see Projects and Folders and the Application & Module Inventory.
4. Set Permissions
5. Add Web Parts

Basic Activities

• Access the Server
• Navigate the Project/Folder Hierarchy
• Selecting, Sorting & Filtering (Please do not skip this section.)
• Locate the Community Forums
• Sign out

Specialized Activities

• Proteomics
• Flow Cytometry
• Study
• Collaboration

• Issues
• Query
• Wiki
• BioTrue

Advanced Activities

• Manage your account settings via the "My Account" link.
• Custom Grid Views

• Support leading medical research institutions using the system as as a data integration platform to reduce the time it takes for laboratory discoveries to become treatments for patients
• Provide rapid to deploy software infrastructure for communities pursing collaborative clinical research efforts
• Deliver a secure data repository for managing and sharing laboratory data with colleagues, such as for proteomics, microarray, flow cytometry or other assay-based data.

Download LabKey Server v 9.1.

Quality Control

• Field-level quality control. Data managers can now set and display the quality control (QC) status of individual data fields. Data coming in via text files can contain the special symbols Q and N in any column that has been set to allow quality control markers. “Q” indicates a QC has been applied to the field, “N” indicates the data will not be provided (even if it was officially required).
• Programmatic quality control for uploaded data. Programmatic quality control scripts (written in R, Perl, or another language of the developer's choice) can now be run at data upload time. This allows a lab to perform arbitrary quality validation prior bringing data into the database, ensuring that all uploaded data meets certain initial quality criteria. Note that non-programmatic quality control remains available -- assay designs can be configured to perform basic checks for data types, required values, regular expressions, and ranges in uploaded data.
• Default values for fields in assays, lists and datasets. Dataset schemas can now be set up to automatically supply default values when imported data tables have missing values. Each default value can be the last value entered, a fixed value or an editable default.

Assay/Study Data Integration

• Display of assay status. Assay working folders now clearly display how many samples/runs have been processed for each study.
• Improved study integration. Study folders provide links to view source assay data and designs, as well as links to directly upload data via appropriate assay pipelines.
• Hiding of unnecessary "General Purpose" assay details. Previously, data for this type of assay had a [details] link displayed in the copied dataset. This link is now suppressed because no additional information is available in this case.
• Easier data upload. Previously, in order to add data to an assay, a user needed to know the destination folder. Now users are presented with a list of appropriate folders directly from the upload button either in the assay runs list or from the dataset.
• Improved copy to study process. It is now easier to find and fix incorrect run data when copying data to a study. Improvements:
• Bad runs can now be skipped.
• The run details page now provides a link so that run data can be examined.
• There is now an option to re-run an assay run, pre-populating all fields, including the data file, with the previous run. On successful import, the previous run will be deleted.

Proteomics and Microarrays

• Protein Search Allows Peptide Filtering. When performing a protein search, you can now filter to show only proteins groups that have a peptide that meets a PeptideProphet probability cutoff, or specify an arbitrarily complex peptide filter.
• Auto-derivation of samples during sample set import. Automated creation of derivation history for newly imported samples eases tracking of sample associations and history. Sample sets now support an optional column that provides parent sample information. At import time, the parent samples listed in that column are identified within LabKey Server and associations between samples are created automatically.
• Microarray bulk upload.
• When importing MageML files into LabKey Server, users can now include a TSV file that supplies run-level metadata about the runs that produced the files. This allows users to reuse the TSV metadata instead of manually re-entering it.
• The upload process leverages the Data Pipeline to operate on a single directory at a time, which may contain many different MageML files. LabKey Server automatically matches MageML files to the correct metadata based on barcode value.
• An Excel template is provided for each assay design to make it easier to fill out the necessary information.
• Microarray copy-to-study. Microarray assay data can now be copied to studies, where it will appear up as an assay-backed dataset.

Assays

• Support for saving state within an assay batch/run upload. Previously, once you started upload of assay data, you had to finish at one point in time. Now you can start by uploading an assay batch, then upload the run data later.
• NAb improvements:
• Auto-complete during NAb upload. This is available for specimen, visit, and participant IDs.
• Re-run of NAb runs. After you have uploaded a NAb run and you wish to make an edit, you can redo the upload process with all the information already pre-filled, ready for editing.

Specimens

• Specimen shopping cart. When compiling a specimen request, you can now perform a specimen search once, then build a specimen request from items listed in that search. You can add individual vials one-at-a-time using the "shopping cart" icon next to each vial. Alternatively, you can add several vials at once using the checkboxes next to each vial and the actions provided by the "Request Options" drop-down menu. After adding vials to a request of your choice, you return to your specimen search so that you can add more.
• Auditing for specimen comments. Specimen comments are now logged, so they can be audited.
• Specimen reports can now be based on filtered vial views. This increases the power of reporting features.

Views

• Enhanced interface for managing views. The same interface is now used to manage views within a study and outside of a study.
• Container filters for grid views. You can now choose whether the list of "Views" for a data grid includes views created within the current folder or both the current folder and subfolders.
• Ability to clear individual columns from sorts and filters for grid views. The "Clear Sort" and "Clear Filter" menu items area available in the sort/filter drop-down menu available when you click on a grid view column header. For example, the "Clear Sort" menu item is enabled when the given column is included in the current sort. Selecting that item will remove just that column from the list of sorted columns, leaving the others intact.
• More detailed information for the "Remember current filter" choice on the Customize View page. When you customize a grid view that already contains sorts and filters, these sorts and filters can be retained with that custom view, along with any sorts and filters added during customization. The UI now explicitly lists the pre-existing sorts and filters that can be retained.
• Stand-alone R views. You do not need to associate every R view with a particular grid view. R views can be created independently of a particular dataset through the "Manage Views" page.
• Improved identification of views displayed in the Reports web part. The Reports web part now can accept string-based form of report ID (in addition to normal integer report ID) so that you can refer to a report defined within a module.

Flow Cytometry

• Ability to download a single FCS file. A download link is now available on the FCS File Details page.
• New Documentation: Demo, Tutorial and additional Documentation
• Richer filter UI for "background column and value." Available in the ICS Metadata editor. This provides support for "IN" and multiple clauses. Example: Stim IN ('Neg Cont', 'negctrl') AND CD4_Count > 10000 AND CD8_Count > 10000
• Performance improvements. Allow loading larger FlowJo workspaces than previously possible.
• UI improvements for FlowJo import. Simplify repeated uploading of FlowJo workspaces.

Development: Client API

• New SAS Client API. The LabKey Client API Library for SAS makes it easy for SAS users to load live data from a LabKey Server into a native SAS dataset for analysis, provided they have permissions to read those data. It also enables SAS users to insert, update, and delete records stored on a LabKey Server, provided they have appropriate permissions to do so. All requests to the LabKey Server are performed under the user's account profile, with all proper security enforced on the server. User credentials are obtained from a separate location than the running SAS program so that SAS programs can be shared without compromising security.
• Additions to the Java, JavaScript, R and SAS Client Libraries:
• Quality Control support. Clients can request quality control values when selecting rows or executing SQL. For further documentation, see the Java or JavaScript libraries. For example, in the JavaScript library, see: config.requiredVersion in LABKEY.Query#selectRows and LABKEY.Query.ExtendedSelectRowsResults#rows in LABKEY.Query.ExtendedSelectRowsResults#constructor
• Container (folder) filtering. Allows you to filter the views displayed for an assay based on the containing folder. For further details, see config.containerFilter in LABKEY.Query#selectRows in the Javascript Library.
• Additions to the Javascript API:
• Callback to indicate that a web part has loaded. Provides a callback after a LABKEY.WebPart has finished rendering.
• Information on the current user (LABKEY.user). The LABKEY.Security.currentUser API exposes limited information on the current user.
• API/Ext-based management of specimen requests. See: LABKEY.Specimen.
• Sorting and filtering for NAb run data retrieved via the LabKey Client APIs. For further information, see: LABKEY.Assay#getNAbRuns
• Ability to export tables generated through the client API to Excel. This API takes a JavaScript object in the same format as that returned from the Excel->JSON call and pops up a download dialog on the client. See LABKEY.Utils#convertToExcel.
• Improvements to the Ext grid.
• Quality control information available.
• Performance improvements for lookup columns.
• Documentation for R Client API. Available here on CRAN.

Development: Modules

• File-based modules. File-based modules provide a simplified way to include R reports, custom queries, custom query views, HTML views, and web parts in your modules. You can now specify a custom query view definition in a file in a module and it will appear alongside the other grid views for the given schema/query. These resources can be included either in a simple module with no Java code whatsoever, or in Java-based modules. They can be delivered as a unit that can be easily added to an existing LabKey Server installation. Documentation: Overview of Simplified Modules and Queries, Views and Reports in Modules.
• File-based assays. A developer can now create a new assay type with a custom schema and custom views without having to be a Java developer. A file-based assay consists of an assay config file, a set of domain descriptions, and view html files. The assay is added to a module by placing it in an assay directory at the top-level of the module. For information on the applicable API, see: LABKEY.Experiment#saveBatch.

Development: Custom SQL Queries

• Support for additional SQL functions:
• UNION and UNION ALL
• BETWEEN
• TIMESTAMPDIFF
• Cross-container queries. You can identify the folder containing the data of interest during specification of the schema. Example: Project."studies/001/".study.demographics.
• Query renaming. You can now change the name of a query from the schema listing page via the “Edit Properties” link.
• Comments. Comments that use the standard SQL syntax ("--") can be included in queries.
• Metadata editor for built-in tables. This editor allows customization of the pre-defined tables and queries provided by LabKey Server. Users can change number or date formats, add lookups to join to other data (or query results), and change the names and description of columns. The metadata editor shows the metadata associated with a table of interest and allows users to override default values. Edits are saved in the same XML format used to describe custom queries.

Collaboration

• Version comparison tool for wiki pages. Differences between older and newer versions of wiki pages can now be easily visualized through the "History"->"Compare Versioned Content"->"Compare With" pathway.
• Attachments can now be downloaded from the "Edit" page. Also, if an attachment is an image, clicking on it displays it in a new browser tab.

Administration

• Tomcat 5.5.27 is now supported.
• Upgrade to PostgresSQL 8.3 is now strongly encouraged. For anyone running PostgreSQL 8.2.x or earlier, you will now see a yellow warning message in the header when logged in as a system admin. Upgrade to PostgreSQL 8.3 to eliminate the message. The message can also be hidden. Upgrade documentation.

PostgreSQL 8.3 Upgrade Tip for Custom SQL Queries

Problem. After upgrading to PostgreSQL 8.3, some custom SQL queries may generate errors instead of running. An example of an error message you might observe:

Query 'Physical Exam Query' has errors
java.sql.SQLException: ERROR: operator does not exist: character varying = integer

Solutions: Two Options.

1. Use the Query Designer. If your query is simple enough for viewing in the Query Designer:

• View your query in the Query Designer.
• Save your query. The Query Designer will make the adjustments necessary for compatibility with PostgreSQL 8.3 automatically.
• Your query will now run instead of generating an error message.

• Open it in the Source Editor.
• In the query editor, add single quotes around numbers so that they will be saved appropriately. For example, change

WHERE "Physical Exam".ParticipantId.ParticipantId=249318596

to:

WHERE "Physical Exam".ParticipantId.ParticipantId='249318596'

• Your query will now run instead of generating an error message.

Overview

LabKey Server v 9.2 has not yet been released. This feature list provides a preview of the release.

Version 9.2 represents a important step forward in the ongoing evolution of the open source LabKey Server. Enhancements in this release are designed to:

• Support leading medical research institutions using the system as as a data integration platform to reduce the time it takes for laboratory discoveries to become treatments for patients
• Provide rapid to deploy software infrastructure for communities pursing collaborative clinical research efforts
• Deliver a secure data repository for managing and sharing laboratory data with colleagues, such as for proteomics, microarray, flow cytometry or other assay-based data.

After 9.2 is released: Download LabKey Server v 9.2.

User administration and security

Finer-grained permissions settings for administrators

• Tighter security. Admins can now receive permissions tightly tailored to the subset of admin functions that they will perform. This allows site admins to strengthen security by reducing the number of people who possess broad admin rights. For example, "Specimen Requesters" can receive sufficient permissions to request specimens without being granted folder administration privileges.
• New roles. LabKey Server v9.2 includes four entirely new roles: "Site Admin," "Assay Designer," "Specimen Coordinator" and "Specimen Requester." This spreadsheet shows a full list of the new admin roles and the permissions they hold. It also shows roles that may be added in future releases of LabKey Server.

• Brief list of roles instead of long list of groups. Previously, the permissions management interface displayed a list of groups and allowed each group to be assigned a role. This list became hard to manage when the list of groups grew long. Now security roles are listed instead of groups, so the list is brief. Groups can be assigned to these listed roles or moved between roles.
• Rapid access to users, groups and permission settings. Clicking on a group or user brings up a floating window that shows the assigned roles of that group or user across all folders. You can also view the members of multiple groups by switching to the groups tab.

• Now individual users, not just groups, can be assigned to security roles. This allows admins to avoid creating groups with single members in order to customize permissions.

• This allows customization and export of the view.

• Administrators can create custom lists to store metadata about groups by joining a list with groups data. Any number of fields can be added to information about a each user or group. These lists can be joined to:
• Built in information about the user (name, email etc)
• Built in information about the group (group, group members)
• The results can also be combined with built-in information about roles assigned to each user & group in each container. From this information a variety of reports can be created, including group membership for every user and permissions for every group in every container.
• These reports can be generated on the client and exported as Excel Spreadsheets

• Deactivate/Re-Activate buttons are now on the user details page as well as the user list. When clicked on the user list, a confirmation page is shown listing all the selected users (users that are already activate/inactive are filtered out if action is deactivate/re-activate).
• Clicking Delete on the user list now takes you to a confirmation page much like the deactivate/re-activate users command. If at least one of the selected users is active, it will also include a note and button that encourages the admin to deactivate the user(s) rather than permanently delete them.

Study

Study export, import and reload

• Studies can be reloaded onto the same server or onto a different LabKey Server. This makes it easy to transfer a study from a staging environment to a live LabKey platform.
• You can populate a brand new study with the exported contents of an existing study. For similar groups of studies, this helps you leverage your study setup efforts.
• Studies can be set up to reloaded data from a data depot nightly. This allows regular transfer of updates from a remote, master database to a local LabKey Server. It keeps the local server up-to-date with the master database automatically.

• Field-Level Missing Value (MV) Indicators allow individual data fields to be flagged. Previously, only two MV values were allowed (N and Q). Administrators can now customize which MV values are available. A site administrator can customize the MV values at the site level and project administrators can customize the MV values at the folder level. If no custom MV values are set for a folder, they will be inherited from their parent folder. If no custom values are set in any parent folders, then the MV values will be read from the server configuration.
• MV value customization consists of creating or deleting MV values, plus editing their descriptions.
• A new API allows programmatic configuration of MV values for a folder. This allows study import/export to include MV values in its data and metadata.

• MV values are now displayed with a pop-up and a MV indicator on an item’s detail page.
• When inserting or updating an item with a MV-enabled field, possible MV values are now offered in a drop-down, along with the ability to set a raw value for the field. Currently a user is only able to specify one or the other on the update page.

Specimens

Import of specimen data allowed before completion of quality control (QC)

• Specimen import is now more lenient in the conflicts it allows in imported specimen data. Previously, import of the entire specimen archive was disallowed if conflicts were detected between transaction records for any individual vial. In 9.2, all fields with conflicts between vials are marked "NULL" and the upload is allowed to complete.
• Use a saved, custom view that filters for vials with the "Quality Control Flag" marked "True" in order to identify and manage vials that imported with conflicts.

• Vial events with conflicting information are flagged. Conflicts are differentiated by the presence of an "unknown" value for the conflicting columns, plus color highlighting. For example, you would see a flag when an imported specimen's globalUniqueID is associated with more than one primary type, as could occur if a clinic and repository entered different vial information pre- and post-shipment.
• Vial events that indicate a single vial is simultaneously at multiple locations are flagged. This can occur in normal operations when an information feed from a single location is delayed, but in other cases may indicate an erroneous or reused globalUniqueID on a vial.
• Vials or primary specimens that meet user-specified protocol-specific criteria are flagged. Examples of QC problems that could be detected with this method include:
• A saliva specimen present in a protocol that only collects blood (indicating a possibly incorrect protocol or primary type).
• Primary specimen aliquoted into an unexpectedly large number of vials, based on protocol expectations for specimen volume (indicating a possibly incorrect participantID, visit, or type for one or more subset of vials).

• The new "specimencheck" module identifies mismatched specimens and displays them in a grid view. It identifies specimens whose participantID, sequenceNum and/or visit dates fail to match, then produces a report that can be used to perform quality control on these specimens. For developers, the "specimencheck" module also provides an example of a simple file-based module.

• This allows specimen managers to indicate that a particular quality control problem has been investigated and resolved without modification of the underlying specimen data.
• A specimen manager can also manually flag vials as questionable even if they do not meet any of the previously defined criteria.
• Records of manual flagging/unflagging are preserved over specimen imports, in the same manner as specimen comments.

• When exported to Excel, individual worksheets of specimen reports may include blank columns. This is due to the fact that columns are included for all visits that have specimens of any kind, rather than for just those visits with specimens matching the current worksheet’s filter. Exported Excel files now display a minimal set of visit columns per report worksheet.

• Additional columns can be optionally presented in vial view and exported via Excel. These include the number of sibling vials currently available, locked in requests, currently at a repository and expected to become available, plus the total number of sibling vials.
• These columns are available via the ‘customize view’ user interface, so different named/saved views can be created. The built-in ability to save views per user enables specimen coordinators to see in-depth detail on available counts, while optionally presenting other users with a more minimal set of information.

• Faster loading of specimen queries. Please review the 9.2 Upgrade Tips to determine whether any of your queries will need to be updated to work with the refactored specimen tables.

• New filter options are available for specimen reports. You can now filter on the presence or absence of a completed request.

Assays

Validation and Transform Scripts

• Both transformation and validation scripts (written in Perl, R or Java) can now be run at the time of data upload. A validation script can reject data before acceptance into the database if the data do not meet initial quality control criteria. A data transformation script can to inspect an uploaded data file and modify the data or populate empty columns that were not provided in the uploaded data. For example, you can populate a column calculated from other columns or flag out-of-range values.
• Validation support has been extended to NAb, Luminex, Microarray, ELISpot and file-based assay types. Validation is not supported for MS2 and Flow assays.
• A few notes on usage:
• Columns populated by transform scripts must already exist in the assay definition.
• Executed scripts show up in the experimental graph, providing a record that transformations and/or quality control scripts were run.
• Transform scripts are run before field-level quality control. Sequence: Transform, field-level quality control, programmatic quality control
• A sample script and details on how to write a script are currently available in the specification.

• For an assay, a specimenID that doesn't appear in a study is displayed with a red highlight to show the mismatch in specimenID and participantID. GlobalUniqueIDs are matched within a study, not between studies.

• The columns included in the "Run Summary" section of the NAb "Details" page can be customized. If there is a custom run view named "CustomDetailsView", the column set and order from this view will apply to NAb run details view.
• Significant performance enhancements. For example, switching from a run to a print view is much faster.
• Users with read permissions on a dataset that has been copied into the study from a NAb assay now see an [assay] link that leads to the "Details" view of a NAb assay.

• Tutorial: Import Microarray Data

Proteomics

Proteomics metadata collection

• The way that users enter proteomics run-level metadata has been improved and bulk-import capabilities have been added. The same approach used for specifying expected properties for other LabKey assays is now used for proteomics.

• It is now possible to copy proteomics run-level data to a study dataset, allowing the proteomics data to be integrated with other study datasets. Note that the study dataset links back to the run that contains the metadata, not the search results.

• A new utility on the protein administration page allows you to test parsing a FASTA header line

Views

Filter improvements

• A filter notification bar now appears above grid views and notes which filters that have been applied to the view.
• The links above an assay remember your last filter. This helps you avoid reapplying the filter. For example, if you have applied a filter to the view, the filter is remembered when you switch between batches, runs and results. The filter notification bar above the view shows the filters that remain with the view as you switch between batches, runs and results.

File management

WebDAV UI enhancements provide a user-friendly experience

• Users can browse the repository in a familiar fashion similar to the Windows Explorer, upload files, rename files, and delete files. All these actions are subject to permission checking and auditing. Drag and drop from desktop and multi-file upload with progress indicator are supported. Additional information about the files is displayed, such as the date of file creation or records of file import into experiments.

Flow

Flow Dashboard UI enhancements

• These changes provide a cleaner set of entry points for the most common usages of Flow. The advanced features of the current Flow Dashboard remain easily accessible. Changes include:
• More efficient access to flow runs
• Ability to upload FCS files and import FlowJo workspaces from a single page.

• Tutorial: Perform a LabKey Analysis

Custom SQL Queries

New SQL functions supported

• COUNT(*)
• SELECT Table.*
• HAVING
• UNION in subqueries
• Parentheses in UNION and FROM clauses

Client API

New Tutorial and Demo for LabKey JavaScript APIs

• Tutorial
• Demo

• LABKEY.Query.exportSql. Accepts a SQL statement and export format and returns an exported Excel or TSV file to the client. The result set and the export file are generated on the server. This allows export of result sets over over 15,000 rows, which is too much for JavaScript to parse into objects on the client.
• LABKEY.QueryWebPart. Supports filters, sort, and aggregates (e.g., totals and averages). Makes it easier to place a Query Web Part on a page.
• LABKEY.Form. Utility class for tracking the dirty state of an HTML class
• LABKEY.Security Expanded. LABKEY.Security provides a range of methods for manipulating and querying security settings. A few of the new APIs:
• LABKEY.Security.getGroupsForCurrentUser. Reports the set of groups in the current project that includes the current user as a member.
• LABKEY.Security.ensureLogin. A client-side function that makes sure that the user is logged in. For example, you might be calling an action that returns different results based on the user's permissions, like what folders are available or setting a container filter.
• Enhanced LABKEY.Security.getUsers. Now includes users' email addresses as the "email" property in the response.

• The Java library now includes programmatic access to NAb data.

• A new menu option under the "Export" button above a grid view will generate a valid script that can recreate the grid view. For example, you can copy-and-paste generated JavaScript into a wiki page source or an HTML file to recreate the grid view. Filters that have been applied to the grid view that are shown in the filter bar above the view are included in the script.

Collaboration

Customization of the “Issues” label

• The issues module provides a convenient tracking service, but some of the things one might want to track with this service are best described by titles other than “issues.” For example, one might use the issues module to track “requests,” “action items,” or “tickets.”
• Administrator can now modify the label displayed in the issue module’s views. The admin can specify a singular and plural form of the new label on a per-container basis. In most places in the UI where either term "Issue" or "Issues" is used, these configured values are used instead. The only exceptions to this are the name of the issues module when displayed in the admin console and folder customization, and the name of the controller in URLs.

• Attachments
• A new option to hide the list of page attachments is available. Files attached to wiki pages are currently displayed below the page content, even if those attachments. This is undesirable in cases where the attachments are simply images used within the page content itself.
• When wiki attachments are displayed, a file attachment divider is shown by default. CSS allows the text associated with the divider to be hidden.
• HTML Editor
• The wiki HTML editor has been updated to a newer version.
• The button for manipulating images is now enabled in the Visual Editor.
• Spellcheck is enabled on Firefox (but not IE).
• Print. You can now print a subtree of a wiki page tree.

• Forms where you enter code and want to format it nicely. This includes the Wiki and query SQL editors.
• Forms where you enter TSV. This includes sample set, list, dataset, and custom protein annotation uploads.
• Support for simple tab entry, as well as multi-line indent and outdent with shift-tab.

• Expiration of messages is now "Off" by default for newly created message boards. Existing message boards remain as they are.

Administration

PostgreSQL

• Support for PostgreSQL 8.4 Beta 1.

Specimen Queries

The "Specimens" table has been split into two new tables, "Vials" and "Specimens," to enhance query speed. This means that you will need to reference one additional table when you use the raw specimen tables perform a lookup.

Queries that use the raw specimen tables will need to be updated. However, queries that use the special, summary tables (Specimen Detail and Specimen Summary) are unaffected and do not need to be modified.

Example: A 9.1 query would have referenced the PrimaryType of a vial as follows:

SpecimenEvent.SpecimenId.PrimaryType

A 9.2 version of the same query would reference the PrimaryType using "VialId," a column in the new "Vials" table:

SpecimenEvent.VialId.SpecimenId.PrimaryType

The Vial table contains: rowID (of the specimen transaction record), globalUniqueID (of the vial), volume and specimenID. The Specimen table contains: participantID, visit number, date, primary type and rowIDs (of the vials generated from this specimen).

Upgrade Note: If you have changed your specimen database using PgAdmin, you may have problems during upgrade. Please see a member of the LabKey team for assistance if this is the case.

Specimen Import

Specimen import is now more lenient in the conflicts it allows in imported specimen data. Previously, import of the entire specimen archive was disallowed if conflicts were detected between transaction records for any individual vial. In 9.2, all fields with conflicts between vials are marked "NULL" and the upload is allowed to complete.

Use a saved, custom view that filters for vials with the "Quality Control Flag" marked "True" in order to identify and manage vials that imported with conflicts.

Example: In 9.1, a vial with a single globalUniqueSpecimenID was required to have the same type (blood, saliva, etc.) for all transactions. Vials that listed different types in different reaction records prevented upload of the entire archive. In 9.2, the conflicting type fields would be marked "NULL" such that these vials and their problematic fields can be reviewed and corrected after upload.

PostgreSQL 8.3

PostgreSQL 8.2 and 8.1 are unsupported on LabKey Server 9.2 and beyond, so you will need to Upgrade PostgreSQL.

Security Model

Extensive changes have been made to the security model in LabKey Server 9.2. Please see the Permissions and Roles spreadsheet for a detailed mapping of permissions under the old model to permissions under the new.

View Management

For 9.2, the "Manage Views" page is accessible to admins only. This means that nonadmins cannot delete or rename views of their own creation, as they could previously. Delete/rename ability will be restored for nonadmins in a future milestone.

MS2 Metadata Collection

The metadata collection process for mass spec files has been replaced. It is now based on the assay framework.

Wiki Attachments

Authors of wiki pages now have the option to show or hide the list of attachments that is displayed at the end of a wiki page. If displayed, the list of attachments will now appear under a bar that reads "File Attachments." This bar helps distinguish the attachment list from the page list. For portal pages where display of this bar is undesirable, you can use CSS to hide the bar.

Quality Control (QC)

The "QC Indicator" field is now called the "Missing Value" field.

Folder/Project Administration UI

The "Manage Project" menu under the "Admin" dropdown on the upper right (and on the left navigation bar) has changed. The new menu options available under "Manage Project" are:

• Permissions (For the folder or project-- you can navigate around the project/folder tree after you get there)
• Project Users (Equivalent to the old "Project Members" option)
• Folders (Same as the current "Manage Folders," focused on current folder)
• Project Settings (Same as existing option of the same name, always available for the project)
• Folder Settings (Available if the container of interest is a folder. Equivalent to the old "Customize Folder." Allows you to set the folder type and choose missing value indicators)

Tutorials and Online Demos

Proteomics (CPAS): Tutorial and Demo

Flow: Tutorials (Import a FlowJo Workspace and Perform a LabKey Analysis) and Demo

Study: Tutorial and Demo

Microarray: Tutorial and Demo

Collaboration: Demo

JavaScript API: Tutorial and Demo

See also: Webinars and Videos.

See the demo reports and custom user interfaces from the webinar

Tutorial Video: Building Views and Custom User Interfaces

Proteomics New Features Webinar for v8.1

CPAS/Proteomics Tutorial Video

R Tutorial Video for v8.1

See also: Tutorials and Online Demos

LabKey Roadmap

What that means to us

• LabKey Server should be the first choice for data storage, sharing and integration for any lab looking to move beyond simple file-based storage and analysis.
• LabKey Server should be scalable to any organization with large quantities of assay data.
• LabKey server should be extensible to new experimental and analysis techniques.

Where we need to go

• Improved depth and breadth of assay support.
• Improved study support with an emphasis on data integration and analysis.
• Improved Ease of use.
• Easy extensibility.
• CFR 21 Part 11b compliance

Improved Depth and Breadth of Assay Support

• Improvements to the core MS2 and flow assays
• Improvements to general purpose assay toolkit (GPAT)
• Support for specific assays based on GPAT

Continued improvement in core assays

Flow

• Flow File Repository. A key use-case for Flow Customers is simply organizing, archiving and finding a large number of flow analyses. These could be new analyses or ones performed previously. This comprises the following features.
• Define drop-points with the ability to organize experiments based on administrator-defined rules.
• Automatic import and/or indexing of FCS data from file system
• Rich search across flow files.
• Improved FlowJo integration. Display full information including graphs for imported FlowJo workspaces. Open workspaces stored in LabKey in FlowJo. Funding: CAVD, Canary?
• Improved per-run/per-well gating. Improved user interface for creating, moving and redefining gates to be used in LabKey-based analysis. Funding: ITN
• Integrate with General Purpose Assay Framework, including support for sample resolution and publish to study. Funding: CAVD

MS2

• Better integration of Protein Databases with the core functionality.
• Move to a more mature and extensible processing pipeline. This will enhance reliability, improve throughput and support inserting custom analysis tools in the pipeline.
• Integrate MS2 results with Study analysis tools.
• Enable new analysis techniques.
• Label free quantitation
• Plug in tools that read CPAS data analyze it and return results that can be stored or displayed.
• Support new scoring engines as they become available.

Improvements to General Purpose Assay Toolkit

• General purpose dilution and plate based assay support. The General Purpose Assay Toolkit and the Plate Designer are extensible, pluggable tools, but we have not yet made it easy for labs to combine them to use on any plate-based dilution assay. The goal here is to allow labs to design their own plate designs and analysis to produce a set of results appropriate to their lab.
• Easier extensibility to new assay types. While the core LabKey team will do the work to import files for In particular it should be relatively easy for a programmer to write an extension to the assay toolkit that knows how to parse laboratory specific file types. These extensions would need minimal programming to get the full other benefits of the assay toolkit.
• Better consistency and sharing of core assay types. As MS2 and Flow Cytometry assay support predates the General Purpose Assay Toolkit. These assays don’t have an integrated “publish to study” capability and have slightly different customization profiles. We would like to make all supported assays support the same basic extensibility, tagging and publishing features.

Support for specific common assays based on GPAT

• ELISpot. ELISpot is a plate-based assay that we will provide custom support for. In particular we want to integrate plate layouts with sequence information. Support: CHAVI, CAVD.
• SoftMAX Pro. SoftMAX Pro is a popular data acquisition and analysis tool. The core LabKey team will be doing work to integrate the tool. Support: CHAVI

Improved Study Support for Data Integration and Analysis

• Study building and maintenance. The study framework relies on import of externally defined data structures. User interface for building and maintaining studies is marginal. This should be integrated in a rich user interface similar to the Vaccine study design tool. Support: CAVD, IAVI.
• Direct Data Entry For human studies we have relied on external tools to gather and. For animal studies, users do not want to enter data into an external system or spreadsheet before getting data into LabKey. LabKey will provide a data entry system Support: IAVI.
• Support for common analysis scenarios. The data analysis tools in can be applied to typical study problems, but they do not offer enough help in building common views & graphs. In particular, the system should be aware of cohorts and offer help in generating views that compare cohorts, for example charts with separate series for each cohort as well as simplified filtering & grouping by cohort.
• Cross-server data transfer and integration. We have several situations where servers area

Ease of Use

• Overall Navigation and UI Framework. A few standard metaphors for navigation need to be enforced throughout the product.
• Data grids should have a consistent UI and consistent customization and reporting capabilities available to them.
• Admin pages should have an integrated and consistent UI
• Support for common scenarios. Work on the user interface often stops when it is possible to perform some task rather than being easy or obvious. For example, just about all studies have the notion of cohorts, but the study structure and reporting tools don’t recognize this important concept, so building reports and graphs on the common case (cohorts) is no easier than building reports and graphs based on any other data structure.
• Reporting and analysis. LabKey incorporates a powerful query builder that allows integration. This power is obscured through inconsistent user interface and the need for scripting in R. We would like to make it easier to create standardized reports and to generalize R based reports so that they can be parameterized reused by people who do not know R.

Ease of Extensibility

• Improved web-based customization for non-programmers. The LabKey server already allows building custom schemas via the Lists feature, and custom pages that can include web parts. There are several
• Improved support for Lists including custom forms and validation for list data.
• Improved support for including web-based data in wiki pages. (Currently web-parts can be included, but they cannot be parameterized.
• Easy to build Java extensions. The current API is huge. We would like to make it easy to write a Java extension with minimal code to create and lay out pages.
• Extensions written in other languages. There is currently limited CGI support via a cgi servlet that passes some security and context information to the CGI script. This could be extended to create support for “Perl Modules” that integrate with the rest of the UI.

CFR 21 Part 11b compliance

Overview

[Community Forum]

Administrative features provided by LabKey Server include:

• Project organization, using a familiar folder hierarchy
• Role-based security and user authentication
• Dynamic web site management
• Backup and maintenance tools

Documentation Topics

Set Up Your Server

• Install or Upgrade your Server.
• Set up a workspace
• Projects and Folders. Create a hierarchical structure for workspaces. You will select Modules, Applications and Web Parts for your workspaces.
• Users and Groups. Allow authenticated accessed to workspaces based on user and group permissions.
• Learn basic user tools

• Use the Admin Console
• Site Settings. Configure basic system settings.
• Look & Feel Settings. Customize the "Look and Feel" of your site and/or projects.
• Authentication. View, enable, disable and configure the installed authentication providers (e.g., OpenSSO and LDAP).
• Email Customization. Customize auto-generated emails sent to users.
• Diagnostics. Review memory usage, errors and threads, then run diagnostic tests.
• Jobs. Manage Pipeline runs and MS2 jobs.
• System Configuration. View installed modules, JAR files, executables and many other items.
• Manage Security. Manage user and group permissions, plus authentication.
• Perform backups maintenance

Topics:

• Before You Install
• Install LabKey via Installer (Windows Only)
• Install LabKey Manually
• Install Required Components
• Configure the Web Application
• Modify the Configuration File
• Supported Tomcat Versions
• Third-Party Components and Licenses
• Upgrade LabKey
• Via Installer. See Install LabKey via Installer (Windows Only)
• Manual Upgrade
• Configure LDAP
• Set Up MS2 Search Engines
• Set Up Mascot
• Set Up Sequest
• Install SequestQueue
• Install the Enterprise Pipeline
• Prerequisites for the Enterprise Pipeline
• Configure LabKey Server to use the Enterprise Pipeline
• Install the Perl-Based MS2 Cluster Pipeline (Deprecated, will be removed in future release. Replaced with Enterprise Pipeline.)
• Install the mzXML Conversion Service
• Run the MS2 Cluster Pipeline
• Example Setups and Configurations
• Set up file transfer via FTP
• Linux Example: Configure FTP on Linux
• Set up R on a LabKey Server
• Linux Example: Configure R on Linux
• Enable R to display graphics on a server that lacks the X Windows display system (a.k.a. a headless server)
• Linux Example: Configure the Virtual Frame Buffer on Linux
• Install CPASS
• Linux Example: Install CPAS on Linux
• Set Up R
• Set Up OpenSSO
• Set Up the FTP Server (included as part of the Pipeline documentation)
• Customize "Look and Feel"
• Troubleshooting

Do I Need to Contact LabKey?

Install Manually or Use the Installer?

If your installation needs are more complex, you can install LabKey manually using our step-by-step instructions. To install LabKey manually, see Install LabKey Manually.

How Do I Upgrade?

What Happens When I Install LabKey?

When you install LabKey, the following components are installed on your computer:

• The Apache Tomcat web server, version 5.5.20
• The PostgreSQL database server, version 8.3 (unless you install manually and choose to run LabKey against Microsoft SQL Server instead)
• The Java Runtime Environment (JRE), version 1.6.0-10
• The LabKey web application components
• Additional third-party components, installed to the /bin directory of your LabKey installation.

Troubleshooting:

• The LabKey installer attempts to install PostgreSQL on your computer. You can only install one instance of PostgreSQL on your computer at a time. If you already have PostgreSQL installed, LabKey can use your installed instance; however, you will need to install LabKey manually. See Install LabKey Manually for more information.
• You may need to disable your antivirus or firewall software before running the LabKey installer, as the PostgreSQL installer conflicts with some antivirus or firewall software programs. (see http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html for more information).
• On Windows you may need to remove references to Cygwin from your Windows system path before installing LabKey, due to conflicts with the PostgreSQL installer (see http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html for more information).
• If you uninstall and reinstall LabKey, you may need to manually delete the PostgreSQL data directory in order to reinstall.

What System Resources are Required for Running LabKey?

LabKey is a web application that runs on Tomcat and accesses a PostgreSQL or Microsoft SQL Server database server. The resource requirements for the web application itself are minimal, but the computer on which you install LabKey must have sufficient resources to run Tomcat and the database server (unless you are connecting to a remote database server, which is also an option). The performance of your LabKey system will depend on the load placed on the system, but in general a modern server-level system running Windows or a Unix-based operating system should be sufficient.

We recommend the following resources as the minimum for running LabKey:

• Processor: a high-performing processer such as a Pentium 4, or, preferably, a dual-processor machine.
• Physical memory: at least 1 gigabyte RAM, preferably 2 GB.
• Disk space: 1 gigabyte hard drive space free

LabKey is supported on computers running Windows XP or later, with up-to-date service packs. LabKey may run on other versions of Windows as well, but only these versions are supported.

To install LabKey on a PC computer running Windows, you can download and run the LabKey installer, available from LabKey Corporation for free download after free registration. You can choose between one of two installers, depending on whether you have an existing installation of the Java Runtime Environment (JRE) on your computer. For more information on what components are installed on your computer with LabKey, see Before You Install.

When you run the installer, you will be prompted to choose between express and advanced installation. If you are installing LabKey on your local computer to try it out, the express installation, which installs the minimum features required for LabKey to work, may be sufficient for you. If you are installing LabKey for your organization to use, you'll want to perform an advanced installation, or install LabKey manually.

Express Installation

If you choose the Express installation option, the Windows installer will prompt you to take the following steps, in addition to standard software installation configuration options:

1. Indicate that you understand that when you install LabKey, your computer becomes a web server and a database server.
2. Provide connection information for an outgoing (SMTP) mail server. The mail server is used to send email generated by the LabKey system, including email sent to new users when they are given accounts on LabKey. The installer will prompt you to specify an SMTP host, port number, user name, and password, and an address from which automated emails are sent. Note that if you are running Windows and you don't have an SMTP server available, you can set one up on your local computer. For more information, see the SMTP Settings section in Modify the Configuration File.
3. Provide a user name and password for the database superuser for PostgreSQL, the database server which is installed by the installer. In PostgreSQL, a superuser is a user who is allowed all rights, in all databases, including the right to create users. You can provide the account information for an existing superuser, or create a new one. You may want to write down the user name and password you provide. This password is the first of the three discrete types of passwords used on LabKey Server
4. Provide a user name and password for the Windows service user. LabKey is installed as a Windows service, and must run under a unique Windows user account; you cannot specify an existing user account. This password is the second of the three discrete types of passwords used on LabKey Server.

Advanced Installation

If you choose the Advanced installation option, you'll be prompted to set up a connection to an outgoing (SMTP) mail server, as described above for the Express Installation.

You'll also be prompted to specify information for mapping a network drive in the case that LabKey needs to access files on a remote server. Specify a drive letter, the UNC path to the remote server, and a user name and password for accessing that share; these can be left blank if no user name or password is required.

Finally, if your organization has an LDAP server, you can optionally specify that LabKey should connect to the LDAP server for authenticating users. If you specify that LabKey should use the LDAP server, then any user listed by the LDAP server can log onto LabKey with the same user name and password that is managed by the LDAP server. By default any user specified by LDAP is a member of the Users group on the LabKey system, and has the same permissions as other members of the Users group.

Setting Up Your Account

At the end of the installation process, the LabKey installer will automatically launch your default web browser and launch LabKey if you have left checked the default option Open Browser to LabKey Home Page. Otherwise, open your web browser and navigate to http://localhost:8080/labkey.

Once you launch LabKey, you'll be prompted to set up an account by entering your email address and a password. This password is the third of the three discrete types of passwords used on LabKey Server. When you enter your name and password, you are added to the global administrators group for this LabKey installation. For more information on the role of the global (a.k.a. site) administrator, see Site Administrator.

You'll then be prompted to install the LabKey modules. For most users, the Express Install is recommended. LabKey will install all modules and then give you the choice of viewing the home page, or further customizing the installation by setting properties for the LabKey application. For more information on this option, see Site Settings.

The Advanced Install is for users who want to selectively upgrade modules and may be confusing unless you are familiar with the underlying architecture of the LabKey system. If you click the Advanced Install button and find yourself confronted by a confusing array of options, you can successfully finish the LabKey installation by clicking the Run Recommended Scripts and Finish button for each page displayed until the installation is complete.

Customize the Installation

After you've installed LabKey, you'll be prompted to customize your installation. See Site Settings for more information.

Installer Troubleshooting

Note that the LabKey installer installs PostgreSQL on your computer. You can only have one PostgreSQL installation on your computer at a time, so if you have an existing installation, the LabKey installer will fail. Try uninstalling PostgreSQL, or perform a manual installation of LabKey instead. See Install LabKey Manually for more information.

Before you install LabKey, you should shut down all other running applications. If you have problems during the installation, try additionally shutting down any virus scanning application, internet security applications, or other applications that run in the background.

On Windows you may need to remove references to Cygwin from your Windows system path before installing LabKey, due to conflicts with the PostgreSQL installer (see http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html for more information).

Securing the LabKey Configuration File

Important: The LabKey configuration file contains user name and password information for your database server, mail server, and network share. For this reason you should secure this file within the file system, so that only designated network administrators can view or change this file. For more information on this file, see Modify the Configuration File.

Reasons to install LabKey Server manually include:

• You're installing LabKey Server on a Linux- or Unix-based computer or a Macintosh.
• You're installing LabKey Server in a production environment and you want fine-grained control over file locations.
• You have an existing PostgreSQL installation on your Windows computer. Only one instance of PostgresSQL can be installed per computer, so the Windows installer will fail if there is an existing PostgreSQL installation.
• You have an existing Tomcat installation on your Windows computer and you want LabKey Server to use it, rather than installing a new instance. Note that Tomcat can be installed multiple times on the same machine.

LabKey Server can also reserve a network file share for the data pipeline, and use an outgoing (SMTP) mail server for sending system emails. LabKey Server may optionally connect to an LDAP server to authenticate users within an organization.

If you are manually installing LabKey Server, you need to download, install, and configure all of its components yourself. The following topics explain how to do this in a step-by-step fashion. If you are installing manually on Unix, Linux, or Macintosh, the instructions assume that you have super-user access to the machine, and that you are familiar with unix commands and utilities such as wget, tar, chmod, and ln.

• Install Required Components
• Configure the Web Application
• Modify the Configuration File
• Supported Tomcat Versions

Before you begin, register with LabKey Corporation if you haven't done so already such that you can download the installable LabKey Server files provided by LabKey Corporation. Note that you'll still need to download the third-party components required by LabKey Server separately, as described below.

Before installing these components, think about where you want them to reside in the file system. For example, you may want to create a LabKey Server folder at the root level and install all components there, or on unix systems, you may want to install them to /usr/local/labkey or some similar place.

Note: The only restriction on where you can install LabKey Server components is that you cannot put the LabKey Server web application files beneath the <tomcat-home>/webapps directory.

Note: We provide support only for the versions listed for each component, and so we strongly recommend that you install that version. These are the versions that have proven themselves over many months of testing and deployment. Some of these components may have more recent releases, but we have not tested or configured the system to work with them.

Install the Java Runtime Environment

1. Download the Java Runtime Environment (JRE) 1.6 from http://java.sun.com/javase/downloads/index.jsp.
2. Install the JRE to the chosen directory. On Windows the default installation directory is C:\Program Files\Java. On Linux a common place to install the JRE is /usr/local/jre<version>. We suggest creating a symbolic link from /usr/local/java to /usr/local/jre<version>. This will make upgrading the JRE easier in the future.

• The JDK includes the JRE, so if you have already installed the JDK, you don't need to also install the JRE.
• If you are planning on building the LabKey Server source code, you should install the JDK 1.6 and configure JAVA_HOME to point to the JDK. For more information, see Building the Source Code.
• If you are installing LabKey on a Mac, you do not need to install the JRE. The JRE comes with the operating system. You should check to make sure that the JRE version included with the OS is a sufficiently recent version of the JRE. For example, Tiger 10.4.10 comes with the JRE 1.5, which is fine.

Install the Apache Tomcat Web Server, Version 5.5.x

LabKey Server supports Tomcat versions 5.5.9 through 5.5.25 and version 5.5.27. Tomcat 5.5.27 is the recommended version of Tomcat for LabKey Server 9.1. For details on supported Tomcat versions, see Supported Tomcat Versions.

1. Download Tomcat 5.5.x from http://tomcat.apache.org/download-55.cgi. Note that this link leads you to the most recent version of Tomcat. For version 5.5.27, see http://tomcat.apache.org/download-55.cgi#5.5.27.
2. Install Tomcat. On Linux, install to /usr/local/apache-tomcat<version>, then create a symbolic link from /usr/local/tomcat to /usr/local/apache-tomcat<version>. We will call this directory <tomcat-home>.
3. Configure Tomcat to use the JRE installed in the first step. You can do this either by creating a JAVA_HOME environment variable under the user account that will be starting tomcat, or by adding that variable to the tomcat startup scripts, <tomcat-home>/bin/startup.sh on Linux or startup.bat on Windows. For example, on Linux add this line to the beginning of the tomcat's startup.sh file: Export JAVA_HOME=/usr/local/java.
4. Start Tomcat. On Linux run <tomcat-home>/bin/startup.sh. If you want Tomcat to start up automatically when you restart your computer see the Tomcat documentation.
5. Test your Tomcat installation by entering http://<machine_name or localhost or IP_address>:8080 in a web browser. If your Java and Tomcat installations are successful you will see the Tomcat success page.

Install the Database Server

You can run LabKey Server against the following database servers:

• PostgreSQL 8.3.x
• Microsoft SQL Server 2005 or 2008

Install PostgreSQL on Windows

1. Download and run the Windows PostgreSQL installer (http://www.postgresql.org/ftp/binary/).
2. Install PostgreSQL as a Windows service. Keep track of the Postgres Windows service account name and password. LabKey Server doesn't really care what this password is set to, but we need to ask for it so that we can pass it along to the Postgres installer. This password is one of the three password types used on LabKey Systems.
3. Also keep track of the database superuser name and password. You'll need these to configure LabKey Server. For more information, see Modify the Configuration File. LabKey Server uses this password to authenticate itself to Postgres. It is one of three types of passwords used on LabKey Server.
4. Select the PL/pgsql procedural language for installation when prompted by the installer.
5. We recommend that you install the graphical tool pgAdminIII for easy database administration. Leave the default settings as they are on the "Installation Options" page to include pgAdminIII.
6. If you have chosen to install pgAdminIII, enable the Adminpack contrib module when prompted by the installer.
7. Please read the notes below to forestall any difficulties with the PostgreSQL installation.

• You can only install one instance of PostgreSQL on your computer. If you already have PostgreSQL installed, LabKey Server can use your installed instance.
• You may need to disable your antivirus or firewall software before installing PostgreSQL, as the PostgreSQL installer conflicts with some antivirus or firewall software programs. (see http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html for more information).
• On Windows you may need to remove references to Cygwin from your Windows system path before installing PostgreSQL (see http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html for more information).
• If you uninstall and reinstall PostgreSQL, you may need to manually delete the data directory in order to reinstall. By default the data directory on a Windows computer is C:\Program Files\PostgreSQL\8.x\data.
• On Vista, you may need to run 'cmd.exe' as administrator and run the installer .msi from the command line.

Install PostgreSQL on Linux, Unix or Macintosh

1. From http://www.postgresql.org/ftp/ download the PostgreSQL binary RPM package if your system supports RPM, or download and build the source otherwise. If you download a source package ending in .gz, unpack it with the command tar xfz <download_file>. Follow the instructions in the INSTALL file.
2. Please read the notes below to forestall any difficulties with the PostgreSQL installation.

• You can only install one instance of PostgreSQL on your computer. If you already have PostgreSQL installed, LabKey Server can use your installed instance.
• If you uninstall and reinstall PostgreSQL, you may need to manually delete the data directory in order to reinstall.

• Increase the join collapse limit.

# join_collapse_limit = 8

to

join_collapse_limit = 10

If you do not do this step, you may see the following error when running complex queries: org.postgresql.util.PSQLException: ERROR: failed to build any 8-way joins

Or

Install Microsoft SQL Server 2005 or 2008

1. If you don't have a licensed version of Microsoft SQL Server, you can download SQL Server 2008 Express for free from http://www.microsoft.com/express/sql/download/. You will likely want to download a version that includes the SQL Server Management Studio graphical database management tool.
2. Keep track of the user name and password you specify for the administrative account. You have now specified the password for the database superuser. LabKey Server uses this password to authenticate itself to SQL Server. It must be provided in plaintext in labkey.xml and is one of three types of passwords used on LabKey Server.
3. To run LabKey Server against SQL Server, you'll need to edit the LabKey Server configuration file. See Modify the Configuration File for instructions.
4. After you've installed SQL Server, you'll need to configure it to use TCP/IP. Follow these steps:

• Launch the SQL Server Configuration Manager.
• Under the SQL Server Network Configuration node, select Protocols for <servername>.
• In the right pane, right-click on TCP/IP and choose Enable.
• Right-click on TCP/IP and choose Properties.
• Switch to the IP Addresses tab.
• Under the IPAll section, clear the value next to "TCP Dynamic Ports" and set the value for "TCP Port" to 1433 and click OK. By default, SQL Server will choose a random port number each time it starts, but the JDBC driver expects SQL Server to be listening on port 1433.
• Restart the service by selecting the "SQL Server Services" node in the left pane, selecting "SQL Server <edition name>" in the right pane, and choosing Restart from the Action menu (or use the Restart button on the toolbar).

• LabKey Server must be configured to use the jTDS JDBC driver for Microsoft SQL Server, which is included in the LabKey Server archive distribution. The template configuration for running against SQL Server with the jTDS driver is included in the LabKey Server configuration file. Documentation for this driver is available on SourceForge. Other JDBC drivers for Microsoft SQL Server have not been tested.
• If you are installing LabKey Server to run against an existing SQL Server database, you may want to set up a new login for LabKey Server to use:
• Run SQL Server Management Studio. Under Security->Logins, add a new login, and type the user name and password.
• Edit the database resource in the LabKey Server configuration file and specify the new user name and password (see Modify the Configuration File).

Install the LabKey Server System Components

1. Download the current binary zip distribution if you are installing on a Windows system, or the current binary tar.gz distribution file if you are installing on a Unix-based system.
2. Unzip the LabKey Server components to a directory on your computer. On Unix-based systems, the command tar xfz LabKey Server-bin.tar.gz will unzip and untar the archive. You will move these components later, so the directory you unpack them to is unimportant. After unpacking the directory should contain these files and directories:

• bin: binary files required by LabKey Server
• common-lib: required common library jars
• labkeywebapp: the LabKey Server web application
• modules: LabKey Server modules
• server-lib: required server library jars
• labkey.xml: LabKey Server configuration file
• README.txt: a file pointing you to this documentation.
• upgrade.sh: Linux upgrade script

Configure Tomcat to Run the LabKey Server Web Application

Follow these steps to run LabKey Server on Tomcat:

Move the LabKey Server Libraries

The LabKey Server binary distribution, available on the LabKey Corporation download page, includes four jar files which must be moved to your Tomcat installation. These jar files can be found in the common-lib directory in the binary distribution. The files are:

• activation.jar
• jtds.jar
• mail.jar
• postgresql.jar

You will also need to copy a library from the server-lib directory in the distribution. Currently, the only file required is:

• labkeyBootstrap.jar

Configure your LabKey Server home directory

Pick a location for your LabKey Server program files. On Windows the default is C:/Program Files/LabKey Server. On Unix the default is /usr/local/labkey. We will call this <labkey_home>.

Next, move the the /labkeywebapp and /modules directories to <labkey_home>.

Notes:

• Make sure that you do not move the /labkeywebapp directory to the /<tomcat-home>/webapps folder.
• The user who is executing the Tomcat process must have write permissions for the /labkeywebapp and /modules directories.

The Windows LabKey Server binary distribution includes a /bin directory that contains a number of pre-built Windows executable files required by LabKey Server. On Windows, simply move this directory to <labkey_home>. On Unix you must download and either install or build these components for your system, and install them to <labkey_home>/bin. For more information see Third-Party Components and Licenses.

Once the components are in place, add a reference to this directory to your system path, or to the path of the user account that will be starting Tomcat.

Move the LabKey Server Configuration File

The LabKey Server configuration file, named labkey.xml by default, contains a number of settings required by LabKey Server to run. This file must be moved into the <tomcat-home>/conf/Catalina/localhost directory.

Modify the LabKey Server Configuration File

The LabKey Server configuration file contains basic settings for your LabKey Server application. When you install manually, you need to edit this file to provide these settings. The parameters you need to change are surrounded by "@@", for example, @@docBase@@, @@jdbcUser@@, @@jdbcPassword@@, etc. For more information on modifying this file, see Modify the Configuration File.

Note: Some settings that were available in the LabKey Server configuration file in previous versions can now be set from the web application. For more information, see Site Settings.

Configure LabKey Server to Run Under SSL (Optional, Recommended)

You can configure LabKey Server to run under SSL (Secure Sockets Layer). We recommend that you take this step if you are setting up a production server to run over a network or over the Internet, so that your passwords and data are not passed over the network in clear text.

To configure Tomcat to run LabKey Server under SSL:

• Edit the <tomcat-home>/conf/server.xml file.
• Follow the directions given in the section titled "Define Tomcat as a Stand-Alone Service" in server.xml.
• Note that Tomcat's default SSL port is 8443, while the standard port for SSL connections recognized by web browsers is 443. To use the standard port, change this port number in the server.xml file.
• For more detailed information, see the SSL Configuration How-To in the Tomcat documentation.

• In the LabKey Server Admin Console, click the Customize Site button.
• Check Require SSL connections.
• Enter the SSL port number that you configured in the previous step in the SSL Port field.

Tomcat's session timeout specifies how long a user remains logged in after their last session activity. By default, Tomcat's session timeout is set to 30 minutes.

To increase session timeout, edit the web.xml file in the <tomcat-home>/conf directory. Locate the <session-timeout> tag and set the value to the desired number of minutes.

Configure Tomcat to Display Extended Characters (Optional)

If you originally installed LabKey using the graphical installer, Tomcat is automatically configured to display extended characters.

If you installed Tomcat manually, it does not by default handle extended characters in URL parameters. To configure Tomcat to handle extended characters:

• Edit the <tomcat-home>/conf/server.xml file.
• Add the following two attributes to the Tomcat connector via which users are connecting to LabKey Server:
• useBodyEncodingForURI="true"
• URIEncoding="UTF-8"

<!-- Define a non-SSL HTTP/1.1 Connector on port 8080 -->
<Connector port="8080" maxHttpHeaderSize="8192"
          maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
          enableLookups="false" redirectPort="8443" acceptCount="100"
          connectionTimeout="20000" disableUploadTimeout="true"
          useBodyEncodingForURI="true" URIEncoding="UTF-8"/>

For more information on configuring Tomcat HTTP connectors, see the Tomcat documentation at http://tomcat.apache.org/tomcat-5.5-doc/config/http.html .

Start the Server

Once you've configured LabKey Server, you can start the Tomcat server using the startup scripts in the <tomcat-home>/bin directory. After you start the server, point your web browser at http://localhost:8080/labkey/ if you have installed LabKey Server on your local computer, or at http://<server-name>:8080/labkey/ if you have installed LabKey Server on a remote server. If all has gone well, you should see the LabKey Server Home page in your browser.

Under Linux, we recommend adding -Djava.awt.headless=true to the tomcat command line. You can do this by adding this line this line to setenv.sh.
   export CATALINA_OPTS=-Djava.awt.headless=true
Without this line you might see this error in some configurations
   java.lang.InternalError: Can't connect to X11 window server using 'localhost:10.0' as the value of the DISPLAY variable.

Configure the Tomcat Default Port

Note that in the addresses list above, the port number 8080 is included in the URL. Tomcat uses port 8080 by default, and to load any page served by Tomcat, you must either specify the port number as shown above, or you must configure the Tomcat installation to use a different port number. To configure the Tomcat HTTP connector port, edit the server.xml file in the <tomcat-home>/conf directory. Find the entry that begins with <Connector port="8080" .../> and change the value of the port attribute to the desired port number. In most cases you'll want to change this value to "80", which is the default port number used by web browsers. If you change this value to "80", users will not need to include the port number in the URL to access LabKey Server.

You can only run two web servers on the same machine if they use different port numbers, so if you have two web servers running you may need to reconfigure one to avoid conflicts.

If you have an existing installation of Tomcat, you can configure LabKey Server to run on that installation. Alternately, you can install a separate instance of Tomcat for LabKey Server; in that case you will need to configure each instance of Tomcat to use a different port. If you have another web server running on your computer that uses Tomcat's default port of 8080, you will also need to configure Tomcat to use a different port.

If you receive a JVM_BIND error when you attempt to start Tomcat, it means that the port Tomcat is trying to use is in use by another application. The other application could be another instance of Tomcat, another web server, or some other application. You'll need to configure one of the conflicting applications to use a different port. Note that you may need to reconfigure more than one port setting. For example, in addition to the default HTTP port defined on port 8080, Tomcat also defines a shutdown port at 8005. If you are running more than one instance of Tomcat, you'll need to change the value of the shutdown port for one of them as well.

Start Tomcat as a Service

If you are using LabKey Server only on your local computer, you can start and stop Tomcat manually using the scripts in <tomcat-home>/bin. In most cases, however, you'll probably want to run Tomcat automatically, so that the operating system manages the server's availability. Running Tomcat as a service is recommended on Windows, and the LabKey Server installer configures Tomcat to start as a service automatically when Windows starts. You can call the service.bat script in the <tomcat-home>/bin directory to install or uninstall Tomcat as a service running on Windows. After Tomcat has been installed as a service, you can use the Windows service management utility to start and stop the service.

If you are installing on a different operating system, you will probably also want to configure Tomcat to start on system startup.

Important: Tomcat versions 5.5.17 through 5.5.23 contain a bug which renders the web server unable to send mail from any mail server other than one running on localhost (the computer on which Tomcat is installed). Apache has provided a patch for this bug, which is available at http://issues.apache.org/bugzilla/show_bug.cgi?id=40668. Please download this patch if you are running Tomcat 5.5.17 or later. The patch is a zip file containing .class files in a package structure starting at a folder named "org". Unzip these folders and files under the <tomcat-home>/common/classes/ directory, maintaining the patch's directory structure. You should find five .class files within "<tomcat-home>/common/classes/org/apache/naming/factory" if the patch is successfully applied. After verifying that the files are in the correct location, restart Tomcat.

The Configuration File Name

The name of the LabKey Server configuration file determines the URL address of your LabKey Server application. This means that the default URL for your LabKey Server installation is http://<servername>/labkey. You can change the name of the configuration file from labkey.xml to something else if you wish to access your LabKey Server application with a URL other than the default. It's best to do this when you first install LabKey Server, rather than on subsequent upgrades, as changing the name of the configuration file will cause any external links to your application to break. Also, since Tomcat treats URLs as case-sensitive, external links will also break if you change the case of the configuration file name.

Note that if you name the configuration file something other than labkey.xml, you will also need to edit the context path setting within the configuration file, described below.

If you wish for your LabKey Server application to run at the server root, you can rename labkey.xml to ROOT.xml. In this case, you should set the context path to be "/". You would then access your LabKey Server application with an address like http://<servername>/.

Securing the LabKey Configuration File

Important: The LabKey configuration file contains user name and password information for your database server, mail server, and network share. For this reason you should secure this file within the file system, so that only designated network administrators can view or change this file.

Modifying Configuration File Settings

You can edit the configuration file with your favorite text or XML editor. You will need to modify the LabKey Server configuration file if you are manually installing or upgrading LabKey Server, or if you want to change any of the following settings.

• The path attribute, which specifies the application context path used in the application's URL address
• The docbase attribute, which indicates the location of the web application in the file system
• Database settings, including server type, server location, username, and password for the database superuser.
• SMTP settings, for specifying the mail server LabKey Server should use to send email to users
• Mapped network drive settings

The path Attribute

The path attribute of the Context tag specifies the context path for the application URL. The context path identifies this application as a unique application running on Tomcat. The context path is the portion of the URL that follows the server name and port number. By default, the context path is set to "labkey".

Note that the name of the configuration file must match the name of the context path, including case, so if you change the context path, you must also change the name of the file.

The docBase Attribute

The docBase attribute of the Context tag must be set to point to the directory where you have extracted or copied the labkeywebapp directory. For example, if the directory where you've copied labkeywebapp is C:\Program Files\LabKey Server on a Windows machine, you would change the initial value to "C:\Program Files\LabKey Server\labkeywebapp".

Database Settings

The username and password attributes must be set to a user name and password with admin rights on your database server. The user name and password that you provide here can be the ones that you specified during the PostgreSQL installation process for the database superuser. Th database superuser password is one of three types of passwords used by LabKey Server. Both the name and password attribute are found in the Resource tag named "jdbc/labkeyDataSource". If you are running a local version of PostgreSQL as your database server, you don't need to make any other changes to the database settings in labkey.xml, since PostgreSQL is the default database choice.

If you are running LabKey Server against Microsoft SQL Server, you should comment out the Resource tag that specifies the PostgreSQL configuration, and uncomment the tag which provides the Microsoft SQL Server configuration. Then replace the default attribute values with your SQL Server user name and password.

Note: LabKey Server does not use Windows authentication to connect to Microsoft SQL Server; you must configure Microsoft SQL Server to accept SQL Server authentication.

If you are running LabKey Server against a remote installation of a database server, you will also need to change the url attribute to point to the remote server; by default it refers to localhost.

SMTP Settings (Optional)

LabKey Server uses an SMTP mail server to send messages from the system. Configuring LabKey Server to connect to the SMTP server is optional; if you don't provide a valid SMTP server, LabKey Server will function normally, except it will not be able to send mail to users.

The SMTP settings are found in the Resource tag named "mail/Session". The mail.smtp.host attribute should be set to the name of your organization's SMTP mail server. The mail.smtp.user specifies the user account to use to log onto the SMTP server. The mail.smtp.port attribute should be set to the SMTP port reserved by your mail server; the standard mail port is 25.

When LabKey Server sends administrative emails, as when new users are added or a user's password is reset, the email is sent with the address of the logged-in user who made the administrative change in the From header. The system also sends emails from the Issue Tracker and Announcements modules, and these you can configure using the mail.from attribute so that the sender is an aliased address. The mail.from attribute should be set to the email address from which you want these emails to appear to the user; this value does not need to correspond to an existing user account. For example, you could set this value to "labkey@mylab.org".

Notes:

• If you do not configure an SMTP server for LabKey Server to use to send system emails, you can still add users to the site, but they won't receive an email from the system. You'll see an error indicating that the email could not be sent that includes a link to an HTML version of the email that the system attempted to send. You can copy and send this text to the user directly if you would like them to be able to log into the system.
• If you are running on Windows XP or a later version of Windows and you don't have a mail server available, you can configure the SMTP service. This service is included with Internet Information Server to act as your local SMTP server. Follow these steps:
• From the Start menu, navigate to Control Panel | Add or Remove Programs, and click the Add/Remove Windows Components button on the left toolbar.
• Install Internet Information Services (IIS).
• From Start | Programs | Administrative Tools, open the Windows Services utility, select World Wide Web Publishing (the name for the IIS service), display the properties for the service, stop the service if it is running, and set it to start manually.
• From Start | Programs | Administrative Tools, open the Internet Information Services utility.
• Navigate to the Default SMTP Virtual Server on the local computer and display its properties.
• Navigate to the Access tab, click Relay, and add the address for the local machine (127.0.0.1) to the list of computers which may relay through the virtual server.
• Tomcat versions 5.5.17 through 5.5.23 contain a bug which renders the web server unable to send mail from any mail server other than one running on localhost (the computer on which Tomcat is installed). Apache has provided a patch for this bug, which is available at http://issues.apache.org/bugzilla/show_bug.cgi?id=40668. Please download this patch if you are running Tomcat 5.5.17 or later. The patch is a zip file containing .class files in a package structure starting at a folder named "org". Unzip these folders and files under the <tomcat-home>/common/classes/ directory, then restart Tomcat.

Version Notes for v5.5.20 Through Current Version

If you are upgrading your LabKey Server installation to use version 5.5.20, you must make a change to the LabKey Server configuration file. Edit the file and change the line

<Loader loaderClass="org.fhcrc.labkey.bootstrap.LabkeyBootstrapClassLoader" />

to:

<Loader loaderClass="org.labkey.bootstrap.LabkeyBootstrapClassLoader" useSystemClassLoaderAsParent="false" />

If you do not make this change, Tomcat will fail to start, and you'll see the following error in the Tomcat log:

SEVERE: Error listenerStart
SEVERE: Context [/labkey] startup failed due to previous errors

You'll also see an error page with the following text if you try to access the LabKey Server webapp.

HTTP Status 404 - /labkey/Project/home/home.view
type Status report
message /labkey/Project/home/home.view
description The requested resource (/labkey/Project/home/home.view) is not available.
Apache Tomcat/5.5.20

Version Notes for v5.5.17 Through v5.5.24

Tomcat versions 5.5.17 through 5.5.24 contain a bug which renders the web server unable to send mail from any mail server other than one running on localhost (the computer on which Tomcat is installed). Apache has provided a patch for this bug, which is available at http://issues.apache.org/bugzilla/show_bug.cgi?id=40668. Please download this patch if you are running Tomcat 5.5.17 or later. The patch is a zip file containing .class files in a package structure starting at a folder named "org". Unzip these folders and files under the <tomcat-home>/common/classes/ directory, then restart Tomcat.

Note: If you are installing LabKey Server for the first time using the Windows graphical installer, this change will have already been made for you. You need to install the patch only if you are upgrading an existing installation of LabKey Server, or if you are installing manually.

Version Notes for v5.5.26

LabKey does not recommend using Tomcat v5.5.26 due to the following Tomcat bug: https://issues.apache.org/bugzilla/show_bug.cgi?id=44494. This bug truncates posts from ApiAction.getJsonObject to 8192 bytes, so it inhibits use of the LabKey API.

Version Notes for v5.5.27

LabKey 9.1 will support Tomcat v5.5.27. However, LabKey v8.3 does not, so please use Tomcat v5.5.25 with LabKey v8.3.

LabKey v9.1 fixes two issues that appeared in earlier releases:

• "Remember Me" now saves full email addresses. Previously, it would only save half of an email address (truncated at the @) because of a change to cookie handling by Tomcat. This would have affected users.
• JSPs now complete. As of Tomcat 5.5.26, JSPs would not compile due to changes in JSP escaping handling. This would have only affected developers.

Graphviz (All)

• Component Name: Graphviz
• LabKey Development Owner: marki#at#labkey.com
• Install Instructions: http://www.graphviz.org/Download..php.
• Build instructions:
• For current version, use the INSTALL link here: http://www.graphviz.org/doc/build.html
• Alternative:
• Sun also ships an old version with Solaris (http://www.sun.com/software/solaris/freeware/), although we have not tested LabKey Server with that version.
• License: Common Public License

X!Tandem (MS2)

• Component Name: X!Tandem
• LabKey Development Owner: jeckels#at#labkey.com
• Information: CPAS uses a modified version of X! Tandem v. 2007.07.01 source (all changes have been submitted back to theGPM)
• Install Instructions: There are 2 ways to download the source files.
• Install files are available here:
• X! Tandem with k-score source (zip)
• X! Tandem with k-score source (tar.gz)
• SVN Checkout from
• https://hedgehog.fhcrc.org/tor/stedi/tags/tandem_2007-07-01
• username: cpas
• password: cpas
• Build Instructions For Windows:
• Build using VC++ 8.0
• Place tandem.exe on your server path (i.e., the path of the user running the Tomcat server process)
• Build Instructions For Linux: (Tested on Fedora Core 7):
• If you are running G++ v3.x
• Run "make" within the tandem_2007-07-01/src directory
• Place tandem_2007-07-01/bin/tandem.exe on your server path (ie the path of the user running the Tomcat server process)
• If you are running G++ v4.x … You will need to make a change to the Makefile located in tandem_2007-07-01/src
• Comment out the following line:

  CXXFLAGS = -O2 -DGCC -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPLUGGABLE_SCORING

• Uncomment the following line:

  #CXXFLAGS = -O2 -DGCC4 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPLUGGABLE_SCORING

• Run "make" within the tandem_2007-07-01/src directory
• Place tandem_2007-07-01/bin/tandem.exe on your server path (i.e., the path of the user running the Tomcat server process)
• License: Artistic License

Trans Proteomic Pipeline (MS2)

• LabKey Development Owner: jeckels#at#labkey.com
• Information on TPP: http://tools.proteomecenter.org/TPP.php
• Install Instructions:
• LabKey currently supports v3.4.2 of the Trans Proteomic Pipeline
• Download Location: http://sourceforge.net/project/showfiles.php?group_id=69281&package_id=126912
• Download v3.4.2 of the tools and unzip.
• Build the tools
• Edit trans_proteomic_pipeline/src/Makefile.incl
• Add a line with XML_ONLY=1
• Modify TPP_ROOT to point to the location you intend to install the binaries. NOTE: This location must be on your server path (ie the path of the user running the Tomcat server process).
• Run 'make configure all install' from trans_proteomic_pipeline/src
• Copy the binaries to the directory you specified in TPP_Root above.
• License: LGPL
• Notes:
• For Mac OSX, this software is only supported for Macs running on Intel CPUs.

peakaboo (MS1)

• LabKey Development Owner: jeckels#at#labkey.com
• Information on ProteoWizard: http://proteowizard.sourceforge.net/
• Windows binary is included with LabKey Server installer.
• Install Instructions:
• Download Location: http://proteowizard.sourceforge.net/download.html
• Download source or binaries as appropriate for your platform.
• If downloading source, build as appropriate for your platform.
• Copy to your pipeline tools directory.

pepmatch (MS1, MS2)

• LabKey Development Owner: jeckels#at#labkey.com
• Windows binary is included with LabKey Server installer.
• Install Instructions:
• Check out source code using Subversion. https://hedgehog.fhcrc.org/tor/stedi/trunk/tools/pepmatch
• Run "make" to build.
• Copy pepmatch binary to your pipeline tools directory.

• a caBIG™ module that contains the site and folder configuration features for caBIG™, as well as the SQL views that present the object model for caBIG™.
• a Tomcat web application named "publish" that runs on the same Tomcat server as the labkey application. The publish application accesses LabKey data through a set of views installed by the caBIG module.
• a set of demonstration and test applications that run as a separate client process and access the LabKey server via different mechanisms..

1. Download the CPAS caBIG Development Kit file from the LabKey download page in the format (zip or tar.gz) appropriate for your platform. This file contains the contents of the output directory of the caCORE SDK build process, customized for LabKey server running on localhost.
2. Extract all files into a directory.
3. Copy the webapp/publish.war file to the appBase directory of your tomcat server, which is normally <tomcat_home>/webapps.
4. Restart Tomcat. In the same directory as you put the publish.war file, you should see a directory named publish once the server start up is complete.
5. Find the file hibernate.properties in the subdirectory <tomcat_home>/webapps/publish/WEB-INF/classes. You may need to edit the database connection information in this file including the user name and password, then restart Tomcat.

Preparation Steps

If you are upgrading to a new version of Apache Tomcat, see the Supported Tomcat Versions page for important information about using different versions of Tomcat with LabKey Server.

Upgrade Options

Manual Upgrade Follow the Manual Upgrade steps if you prefer to upgrade LabKey Server manually or you need to upgrade a machine that does not run Windows.

If you are upgrading LabKey Server on Linux, you can use the upgrade.sh script to streamline the upgrade process. Type "upgrade.sh" with no parameters in a console window for help on the script's parameters.

• Download the appropriate LabKey Server archive file for your operating system from the download page. On Windows, use LabKey9.1-xxxx-bin.zip; on Unix-based systems, used LabKey9.1-xxxx-bin.tar.gz.
• Unzip or untar the archive file to a temporary directory on your computer. On Unix-based systems, the command tar xfz LabKey9.1-xxxx-bin.tar.gz will unzip and untar the archive. For a description of the files included in the distribution, see the section Install the LabKey Server System Components in the Install Required Components topic.

• Locate your LabKey Server home (<labkey-home>) directory, the directory to which you previously installed LabKey Server. For example, if you used the LabKey Server binary installer to install LabKey Server on Windows, your default <labkey-home> directory is C:\Program Files\LabKey Server.
• Find your Tomcat home directory (<tomcat-home>). If you used the LabKey Server binary installer to install an earlier version of LabKey Server on Windows, your default Tomcat directory is <labkey-home>/jakarta-tomcat-n.n.n.
• Find the existing LabKey Server files on your system for each of these three components, in preparation for replacing them with the corresponding LabKey Server files:
• lib: The existing LabKey Server libraries should be located in <tomcat-home>/common/lib.
• labkeywebapp: The directory containing the LabKey Server web application (<labkeywebapp>) may be named labkeywebapp or simply webapp. It may be in the <labkey-home> directory or may be a peer directory of the <tomcat-home> directory.
• modules: The directory containing the LabKey Server modules. This directory is found in the <labkey-home> directory.
• labkey.xml: The LabKey Server configuration file should be located in <tomcat-home>/conf/Catalina/localhost/. This file may be named labkey.xml, LABKEY.xml, or ROOT.xml.

• Shut down the Tomcat web server. If you are running LabKey Server on Windows, it may be running as a Windows service, and you should shut down the service. If you are running on a Unix-based system, you can use the shutdown script in the <tomcat-home>/bin directory. Note that you do not need to shut down the database that LabKey Server connects to.
• Create a new directory to store the a backup of your current configuration. Create the directory <labkey-home>/backup1
• NOTE: if the directory <labkey-home>/backup1 already exists, increment that directory name by 1. For example, if you already have backup directories named backup1 and backup2, then new backup directory should be named <labkey-home>/backup3
• Back up your existing labkeywebapp directory:
• Move the <labkeywebapp> directory to the backup directory
• Back up your existing modules directory:
• Move the <labkey-home>/modules directory to the backup directory
• Back up your <tomcat-home>/conf directory:
• Copy the <tomcat-home>/conf directory to the backup directory
• Create the following new directories
• <labkey-home>/labkeywebapp
• <labkey-home>/modules

• Copy the contents of the LabKey9.1-xxxx-bin/labkeywebapp directory to the new <labkey-home>/labkeywebapp directory.
• Copy the contents of the LabKey9.1-xxxx-bin/modules directory to the new <labkey-home>/modules directory.
• If you are running Windows, copy the executable files and Windows libraries in the LabKey9.1-xxxx-bin/bin directory to the <labkey-home>/bin directory. If you are running on Unix, you will need to download these components separately. See Third-Party Components and Licenses for more information.
• Copy the LabKey Server libraries from the /LabKey9.1-xxxx-bin/common-lib directory into <tomcat-home>/common/lib. Choose to overwrite any jars that are already present. Do not delete or move the other files in this folder(<tomcat-home>/common/lib), as they are required for Tomcat to run.
• Copy the LabKey Server libraries from the /LabKey9.1-xxxx-bin/server-lib directory into <tomcat-home>/server/lib. Do not delete or move the other files in this folder (<tomcat-home>/server/lib), as they are required for Tomcat to run.
• If you have customized the stylesheet for your existing LabKey Server installation, copy your modified stylesheet from the backup directory into the new <labkey-home>/labkeywebapp directory.

• If you are running Windows:
• Back up your existing bin directory: Move the <labkey-home>/bin directory to the backup directory.
• Create the directory <labkey-home>/bin
• Copy the executable files and Windows libraries in the LabKey9.1-xxxx-bin/bin directory to the <labkey-home>/bin directory.
• If you are running on Unix:
• You will need to download and upgrade these components. See Third-Party Components and Licenses for the list of required components, required versions and installation instructions.
• Ensure that the <labkey-home>/bin directory is on your system path, or on the path of the user account that will be starting Tomcat.

Copy the LabKey Server Configuration File

• Back up the existing LabKey Server configuration file (the file named labkey.xml, LABKEY.xml, or ROOT.xml)
• The file is located in <tomcat-home>/conf/Catalina/localhost/
• Copy the file to the backup directory
• Copy the new labkey.xml configuration file from the /LabKey9.1-xxxx-bin directory to <tomcat-home>/conf/Catalina/localhost/labkey.xml.
• Alternately, if your existing LabKey Server installation has been running as the root web application on Tomcat and you want to ensure that your application URLs remain identical after the upgrade, copy labkey.xml to <tomcat-home>/conf/Catalina/localhost/ROOT.xml.
• Merge any other settings you have changed in your old configuration file into the new one. Open both files in a text editor, and replace all parameters (designated as @@param@@) in the new file with the corresponding values from the old file. Note that LabKey Server 2.x added a new line to this file to tell Tomcat to use a special ClassLoader.
• Important: The name of the LabKey Server configuration file determines the URL address of your LabKey Server application. If you change this configuration file, any external links to your LabKey Server application will break. Also, since Tomcat treats URLs as case-sensitive, external links will also break if you change the case of the configuration file. For that reason, you may want to name the new configuration file to match the original one. Note that if you name the configuration file something other than labkey.xml, you will also need to edit the context path settings within the configuration file. For more information, see Modify the Configuration File.
• Note: If you are upgrading from CPAS 1.6 or previous to LabKey Server 2.2 or later, your configuration file will contain a number of additional <Environment> tags. These tags specify settings that are now saved in the database. When you upgrade, these settings will be copied to the database, so after you upgrade, you can delete them. There's no harm in leaving them either, as LabKey Server will ignore them, but you may want to clean them up to avoid confusion.
• If you are upgrading from LabKey Server 1.3 or 1.4, you only need to add one line to your LabKey Server configuration file, within the <context> tags:
• <Loader loaderClass="org.fhcrc.labkey.bootstrap.LabkeyBootstrapClassLoader"/>

• Restart the Tomcat web server. If you have any problems starting Tomcat, check the Tomcat logs in the <tomcat-home>/logs directory.
• Navigate to your LabKey Server application with a web browser using the appropriate URL address, and upgrade the LabKey Server application modules when you are prompted to do so.
• It is good practice to review the Properties on the Admin Console immediately after the upgrade to ensure they are correct.

At this point LabKey Server should be up and running. If you have problems, check the Tomcat logs, and double-check that you have properly named the LabKey Server configuration file and that its values are correct.

Upgrade PostgreSQL to Version 8.3

Postgres version 8.3 is strongly recommended when running LabKey Server 9.1. As of the release of LabKey 9.2, PostgresSQL 8.1 and 8.2 will no longer be supported.

Postgres provides instructions on how to upgrade your installation, including moving your existing data database.

If you are not familiar with your organization's LDAP servers, you will want to recruit the assistance of your network administrator for help in determining the addresses of your LDAP servers and the authentication credentials they require.

Configure LDAP

You can configure LDAP when you install LabKey. If you are installing using the Windows binary installer, choose Advanced Install and enter the settings for your LDAP server when prompted.

If you are installing manually or you want to configure LDAP after you've already installed LabKey, follow these steps to reach the LDAP configuration page:

1. Click on the Admin Console link in the left navigation pane
2. Click the authentication link.
3. On the Authentication page, click the configure link for LDAP.

On the LDAP Configuration page you can specify the URL of your LDAP server or servers, the domain of email address that should be authenticated using LDAP, and the security principal template to be used for authenticating users.

LDAP Servers: Specifies the addresses of your organization's LDAP server or servers. The form for the LDAP server address is ldap://servername.domain.org:389, where 389 is the standard port for non-secured LDAP connections. The standard port for secure LDAP (LDAP over SSL) is 636.

LDAP Domain: Specifies the email domain that will be authentication using LDAP. All users signing in with an email addresses from this domain will be authenticated against the LDAP server; all other email addresses will be authenticated against the logins table in the database. Leave blank to not use LDAP authentication (always use the database).

LDAP Principal Template: Specifies the principal authentication template required by your LDAP server. The principal authentication template is the format in which the authentication information for the security principal -- the person who is logging in -- must be passed to the LDAP server. The default value is ${email}, which is the format required by Microsoft Active Directory. Other LDAP servers require different authentication templates. Check with your network administrator to learn more about your LDAP server.

Authentication Process:

When a user log into LabKey with an email address ending in the LDAP domain, LabKey attempts an LDAP connect to the server(s) using the security principal and password the user just entered. If the connect succeeds, the user is authenticated; if the connect fails, the user is not authenticated. When configuring LabKey to use an LDAP server you are trusting that the LDAP server is both secure and reliable.

LDAP Security Principal Template:

The LDAP security principal template must be set based on the LDAP server's requirements. You can specify two properties in the string that LabKey will substitute before sending to the server:

Here are a couple sample LDAP security principal templates that worked on LDAP configurations we've tested with LabKey:

Note: Different LDAP servers and configurations have different credential requirements for user authentication. Consult the documentation for your LDAP implementation or your network administrator to determine how it authenticates users.

Testing the LDAP Configuration

To test your LDAP configuration, click on the Admin Console link in the left navigation pane, then click the Test LDAP button. Enter your server URL, the exact security principal to pass to the server (no substitution will take place), and the password. Click "Go" and an LDAP connect will be attempted. The next page will show you if the login succeeded or not, or if there were problems connecting to the server.

As discussed above, the LDAP security principal must be in the format required by your LDAP server configuration.

It may be helpful to use an LDAP client to view and test your LDAP network servers. The Softerra LDAP Browser is a freeware product that you can download to experiment with your LDAP servers.

There are 3 steps to the installation and configuration of the LabKey Server Enterprise Pipeline.

• Installation of Prerequisite software for the Enterprise Pipeline
• Install and Configure the LabKey Server to use the Enterprise Pipeline
• Using the Enterprise Pipeline
• Troubleshooting the Enterprise Pipeline

• All files (both sample files and result files from searches) will be stored on a Shared File System
• LabKey Server is running on a Windows Server
• LabKey Server will mount the Shared File System
• Conversion of RAW files to mzXML format will be included in the pipeline processing
• Conversion Server will mount the Shared File System
• MS1 and MS2 pipeline analysis tools (xtandem, tpp, msInspect, etc) will be executed on Cluster
• Cluster execution nodes will mount the Shared File System
• Instructions for SGE and PBS based clusters are available.

Missing Documentation Pages

1. Disabling the Enterprise Pipeline
2. Description of available settings for the Enterprise Pipeline
3. Debugging the Enterprise Pipeline

In order to install the LabKey Enterprise Pipeline, you will first need to have the following prerequisite software installed and configured.

1. A Working Installation of the LabKey Server
2. JMS Queue (ActiveMQ)
3. Globus GRAM Server
4. Conversion Service (convert MS2 output to mzXML )

• The Conversion Service is optional, and only required if you plan to convert files to mzXML format in your pipeline

These instructions explain how to install LabKey Enterprise Pipeline MS2 Conversion service. The Conversion service is used to convert the output of the MS2 machines to the mzXML format which is used by the LabKey Server. (Please note the Conversion Service is optional, and only required if you plan to convert files to mzXML format in your pipeline.)

Installation Requirements

1. Choose a server to run the Conversion Service
1. The server must be running Windows 2003, Windows 2000 or Windows XP
2. Install the Sun Java Runtime Environment
3. Install the Vendor Software for the Converters you will use. Currently only the following vendors are supported

• ThermoFinnigan
• Waters

1. Install mzXML converter EXEs
1. ReAdW.exe for ThermoFinnigan
2. wolf.exe for Waters
2. Test the Converter Installation

Choose a server to run the Conversion Service

• Windows Server 2003
• Windows Server 2000
• Windows XP

Install the Java Runtime Environment

1. Download the Java Runtime Environment (JRE) 1.6 from http://java.sun.com/javase/downloads/index.jsp
2. Install the JRE to the chosen directory. On Windows the default installation directory is C:\Program Files\Java.

Notes:

• The JDK includes the JRE, so if you have already installed the JDK, you don't need to also install the JRE.

Install the Vendor Software for the Supported Converters

• ThermoFinnigan
• Waters

Install mzXML converter executables

Download the converter executables from the Sashimi Project

• ReadW.exe
• Download
• Further Information
• massWolf.exe
• Download
• Further Information

Install the executables into the directory <LABKEY_HOME>\bin directory

1. Create the directory c:\labkey to be the <LABKEY_HOME> directory
2. Create the binary directory c:\labkey\bin
3. Place the directory <LABKEY_HOME>\bin directory on the PATH System Variable using the System Control Panel
4. Unzip the downloaded files and copy the executable files in <LABKEY_HOME>\bin

Test the converter installation.

1. Choose a RAW file to use for this test. For this example, the file will be called convertSample.RAW
2. Place the file in a temporary directory on the computer. For this example, we will use c:\conversion
3. Open a Command Prompt and change directory to c:\conversion
4. Attempt to convert the sample RAW file to mzXML using ReadW.exe

C:\conversion> dir
Volume in drive C has no label.
Volume Serial Number is 30As-59FG

Directory of C:\conversion

04/09/2008  12:39 PM    <DIR>          .
04/09/2008  12:39 PM    <DIR>          ..
04/09/2008  11:00 AM        82,665,342 convertSample.RAW

C:\conversion>readw.exe convertSample.RAW p
 Saving output to convertSample.mzXML
 Processing header
 Calculating sha1-sum of RAW
 Processing scans
 Writing the index
 Calculating sha1-sum of mzXML
 Inaccurate Masses: 2338
 Accurate Masses: 4755
 Charge 2: 4204
 Charge 3: 2889
 done


C:\conversion> dir
Volume in drive C has no label.
Volume Serial Number is 20AC-9682

Directory of C:\conversion

04/09/2008  12:39 PM    <DIR>           .
04/09/2008  12:39 PM    <DIR>           ..
04/09/2008  11:15 AM        112,583,326 convertSample.mzXML
04/09/2008  11:00 AM        82,665,342  convertSample.RAW

The Enterprise Pipeline requires a JMS Queue to transfer messages between the services that make up the Enterprise Pipeline . The LabKey Server currently supports the ActiveMQ JMS Queue from the Apache Software Foundation.

Installation Requirements

1. Choose a server on which to run the JMS Queue
2. Install the Java Runtime Environment
3. Install and Configure ActiveMQ
4. Test the ActiveMQ Installation

Choose a server to run the JMS Queue

Install the Java Runtime Environment

1. Download the Java Runtime Environment (JRE) 1.6 from http://java.sun.com/javase/downloads/index.jsp
2. Install the JRE to the chosen directory.
3. Create the JAVA_HOME environmental variable to point at your installation directory.

Install and Configure ActiveMQ

Download and Unpack the distribution

1. Download ActiveMQ from http://activemq.apache.org/activemq-510-release.html
2. Unpack the binary distribution from into /usr/local
1. This will create /usr/local/apache-activemq-5.1.0
3. Create the environmental variable <ACTIVEMQ_HOME> and have it point at /usr/local/apache-activemq-5.1.0

Configure logging for the ActiveMQ server

<plugins>
      <!-- lets enable detailed logging in the broker -->
      <loggingBrokerPlugin/>
</plugins>

During the installation and testing of the ActiveMQ server, you might want to show the debug output for the JMS Queue software. You can enable this by editing the file <ACTIVEMQ-HOME>/conf/log4j.properties

uncomment

#log4j.rootLogger=DEBUG, stdout, out

and comment out

log4j.rootLogger=INFO, stdout, out

Authentication, Management and Configuration

1. Configure JMX to allow us to use Jconsole and the JMS administration tools monitor the JMS Queue
2. We recommend configuring Authentication for your ActiveMQ server. There are number of ways to implement authentication. See http://activemq.apache.org/security.html
3. We recommend configuring ActiveMQ to create the required Queues at startup. This can be done by adding the following to the configuration file <ACTIVEMQ-HOME>/conf/activemq.xml

<destinations>
     <queue physicalName="job.queue" />
     <queue physicalName="status.queue" />
</destinations>

Start the server

• Logs will be written to <ACTIVEMQ_HOME>/data/activemq.log
• StdOut will be written to /usr/local/apache-activemq-5.1.0/smlog
• JMS Queue messages, status information, etc will be stored in <ACTIVEMQ_HOME>/data
• job.queue Queue and status.queue will be durable and persistant (ie messages on the queue will be saved through a restart of the process.
• We are using AMQ Message Store to store Queue messages and status information

<ACTIVEMQ_HOME>/bin/activemq-admin start xbean:<ACTIVEMQ_HOME>/conf/activemq.xml > <ACTIVEMQ_HOME>/smlog 2>&1 &

Monitoring JMS Server, Viewing JMS Queue configuration and Viewing messages on a JMS Queue.

Using the ActiveMQ management tools

<ACTIVEMQ_HOME>/bin/activemq-admin browse --amqurl tcp://localhost:61616 job.queue

<ACTIVEMQ_HOME>/bin/activemq-admin query

Using Jconsole

service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi

The LabKey Enterprise Pipeline uses the Globus WS-GRAM software to send MS2 and MS1 searches to a cluster. You can find further information on the Globus software at the Globus Alliance site

• Globus Overview
• WS-GRAM specific info at the Globus Site

Installation Requirements

1. Choose a server to run the WS-GRAM software
2. Install and Configure WS-GRAM software

• Install GridFTP
• Install Reliable File Transport (RFT)
• Install WS-GRAM

1. Enable a user to submit jobs to the WS-GRAM service
2. Test the WS-GRAM Installation

Choose a server to run the WS-GRAM software

In this first version of the LabKey Enterprise Pipeline there are some limitations to the types of clusters and the cluster configuration that we support.

• The Enterprise Pipeline only supports the use of PBS and SGE based clusters
• It is assumed that the MS2 data(files) in the pipeline(s) will be stored on a shared file system that is mountable by both the LabKey Web server and the cluster execution nodes.

• Read access to the PBS/SGE scheduler log files
• Authorization to submit jobs to scheduler

Lastly, the WS-GRAM v4.0.6 is only supported with a unix based operating system.

Install and Configure WS-GRAM software

In order to install the WS-GRAM software, the following software needs to be installed on the server

• Java 1.6 (http://java.sun.com/javase/downloads/index.jsp)
• Ant: (http://apache.inetbridge.net/ant/binaries/apache-ant-1.7.1-bin.tar.gz )

Globus Toolkit setup

• Download the Globus Toolkit software
• Setup the SimpleCA certificate authority
• This step can be skipped if your organization has a certificate authority. If your organization has it's own Certification Authority it is recommended that you use it.
• Create the globus user

• Download the Globus Toolkit installers from http://www-unix.globus.org/ftppub/gt4/4.0/4.0.6/installers/src/gt4.0.6-all-source-installer.tar.gz
• Expand the downloaded software.
• Build the software by running the following in the expanded source directory
• If using a PBS-based cluster run

./configure --enable-wsgram-pbs --prefix=/usr/local/gt4.0.6
make wsgram gridftp wsjava wsrft wstests gt4-gram-pbs install 2>&1 | tee build.logb

• If using a SGE-based cluster run

./configure --prefix=/usr/local/gt4.0.6
make wsgram gridftp wsjava wsrft wstests install 2>&1 | tee build.logb

• This will install the software in /usr/local/gt4.0.6 (if you would like the software installed anywhere else simply change the --prefix option in the configure command). For the rest of this configuration we will refer to the install location as <GLOBUS_LOCATION>
• Set environmental variables for the rest of the configuration

export GLOBUS_LOCATION=<GLOBUS_LOCATION>

Create a Certificate Authority If your organization has a certificate authority then skip this step and goto Create a Host Certificate. For these instructions, we will be installing a CA on the box using the SimpleCA toolset that is shipped with the Globus toolkit.

• Perform the following steps as the user root
• Run the setup script found at <GLOBUS_LOCATION>/setup/globus/setup-simple-ca
• You will be asked a number of questions and your answers will be used in the creation of the certificate. Below is an example of the answers used in the creation of a CA here at LabKey
• Accepted the default Subject Name which is cn=Globus Simple CA, ou=simpleCA-labkey-sample-ca.labkey.com, ou=GlobusTest, o=Grid
• email was cpas@fhcrc.org
• Number of Days for expiration = 1825 days (5 years)
• Entered a PEM Passphrase.
• you will need to remember this passphrase, as you will need it for all future administrative operations with the CA, such as signing user certs, etc. Write it down and place it with your other administrative passwords
• After the script is finished, it writes a bunch of configuration information to the screen. There is some important information that you should write down for later use
• The private key of the CA is stored in /root/.globus/simpleCA//private/cakey.pem
• The public CA certificate is stored in /root/.globus/simpleCA//cacert.pem
• Now you have to make is so that this server can request certificates from the Certificate Authority(CA) we just created.
• Run the following command <GLOBUS_LOCATION>/setup/globus_simple_ca_XXXXXXXX_setup/setup-gsi where XXXXXXX is the 8 alpha-number string that is the name of your CA

Request and Sign a Host Certificate

• Perform the following steps as the user root
• Create a certificate request by running grid-cert-request -host 'HOSTNAME'
• where HOSTNAME is fully qualified domain name of the server.
• The following files will be created:
• /etc/grid-security/hostkey.pem
• /etc/grid-security/hostcert_request.pem
• /etc/grid-security/hostcert.pem
• Sign the host certificate using the CA we created above by running grid-ca-sign -in /etc/grid-security/hostcert_request.pem -out /etc/grid-security/hostcert.pem
• you will need to use the same passphrase you used above when creating the CA.
• this command will write the newly signed certificate to both the location specified on the command line (/etc/grid-security/hostsigned.pem) and to the simpleCA certificate store located at /root/.globus/simpleCA/newcerts
• Rename both the key and the signed certificate

cp /etc/grid-security/hostkey.pem /etc/grid-security/containerkey.pem
cp /etc/grid-security/hostcert.pem /etc/grid-security/containercert.pem

Create the globus account This account will be used to run the WS-GRAM service.

• Create a user on the server named "globus"
• Set the ownership on the following to directories and their contents to be owned by the globus user.
• /etc/grid-security/containercert.*
• /usr/local/gt4.0.6
• for example you could run chown -R globus.users /usr/local/gt4.0.6
• Add the following entries to the profile for the globus user (ie .bash_profile)
• export GLOBUS_LOCATION=/usr/local/gt4.0.6
• set this to your <GLOBUS_LOCATION>
• export JAVA_HOME=/usr/lib64/jvm/java
• set this to your <JAVA_HOME>
• source $GLOBUS_LOCATION/etc/globus-user-env.sh
• export GLOBUS_OPTIONS="-server -Xmx512M -Dorg.globus.wsrf.container.persistence.dir=$GLOBUS_LOCATION/var/persistent"
• Increase the maximum heap size of the JVM and
• Changes the location of where globus stores persistent resources to <GLOBUS_LOCATION>/var/persistant

Configure the GridFTP software

auth_level 1
#Enable CAS Authorization
cas 0
# Use GSI Security on the ipc channel (connection between front-end and back-end servers,
# This is disabled. 
secure_ipc 0
# How will GSI (ie auth using certs) authentication on the ipc channel
ipc_auth_mode host
# Disable Anonymous connections
allow_anonymous 0
# Specify user for anonymous connections
anonymous_user globus
# Set the maximum connections 
connections_max 10
# Set the log level 
log_level ALL
# This will create a login /var/log/gridftp for each process or client session
log_unique /var/log/gridftp/
# Use the default port used by documentation 
port 2811

This configuration will write all logs to files in /var/log/gridftp.

• Create the directory /var/log/gridftp

• To start the server in the foreground and have all output shown on the screen execute

<GLOBUS_LOCATION>/sbin/globus-gridftp-server -port 2811

• To start the server in the background and detached from the shell, execute

<GLOBUS_LOCATION>/sbin/globus-gridftp-server -S -port 2811 > <GLOBUS_LOCATION>/var/gridftp_output.log & 2>&1

Configure the Reliable File Transport(RFT) Service

• Configure the database to log all connections. This will aid during testing and debugging.
• edit postgresql.conf in the Postgresql data directory (usually /var/lib/pgsql/data)
• set the log_connections to be on and
• uncomment the line (ie remove the "#" sign)
• Start the database server
• Create the globus database user by running the following command
• createuser globus
• Answer the questions as follows
• Shall the new role be a superuser? (y/n) n
• Shall the new role be allowed to create databases? (y/n) y
• Shall the new role be allowed to create more new roles? (y/n) n
• Add a configuration login setting for the Globus database which will be called rftDatabase
• Edit the file pg_hba.conf in the Postgresql data directory append a line similar to the following
• host rftDatabase globus xxx.xxx.xxx.xxx 255.255.255.255 trust where xxx.xxx.xxx.xxx is the IP address of the server which is running the RFT service.
• NOTE: This example uses the Trust method for authentication. It is preferable to use md5
• Create the database as the user globus
• createdb rftDatabase
• Populate the RFT database with the appropriate schemas run as the globus user:
• psql -u globus -d rftDatabase -f <GLOBUS_LOCATION>/share/globus_wsrf_rft/rft_schema.sql
• Configure the RTF web application to use this new database by editing <GLOBUS_LOCATION>/etc/globus_wsrf_rft/jndi-config.xml~~
• Change the username and password parameters under the dbConfiguration node in the xml file.
• (ie search for dbConfiguration and change the values for username and password in the next couple of lines in the file
• The last step is to allow RFT to be called locally instead of through the webservice. This will improve performance.
• Edit <GLOBUS_LOCATION>/etc/gram-service/jndi-config.xml and change the
• enableLocalInvocations parameter from false to true

Build and Configure the SGE Adapter**

• Download the software
• http://www.lesc.ic.ac.uk/projects/globusgramjobmanagersetupsge-1.1.tar.gz
• http://www.lesc.ic.ac.uk/projects/globusschedulereventgeneratorsge-1.1.tar.gz
• http://www.lesc.ic.ac.uk/projects/globusschedulereventgeneratorsgesetup-1.1.tar.gz
• http://www.lesc.ic.ac.uk/projects/globuswsrfgramservicejavasetupsge-1.1.tar.gz
• Build the SGE Adapter software
• This will be built as the globus user.
• Setup the globus user’s environment

source <SGE_HOME>/default/common/settings.sh 
export X509-USER-CERT=/etc/grid-security/containercert.pem
export X509-USER-KEY=/etc/grid-security/containerkey.pem
grid-proxy-init

gpt-build globus-gram-job-manager-setup-sge-1.1.tar.gz
gpt-build ./globus-scheduler-event-generator-sge-1.1.tar.gz gcc32dbg
gpt-build globus-scheduler-event-generator-sge-setup-1.1.tar.gz
gpt-build ./globus-wsrf-gram-service-java-setup-sge-1.1.tar.gz

• Install the software

gpt-postinstall

echo “<Scheduler xmlns="http://mds.globus.org/batchproviders/2004/09">”;

echo "</Scheduler>";

chmod +x /usr/cpas/gt4.0.6//libexec/globus-scheduler-provider-sge

Further infromation on installing the SGE Adapter can be found at http://www.globusconsortium.org/tutorial/ch8/

Configure WS-GRAM Service

• For SGE: (if you are using a SGE based cluster you should see)

<ns1:scheduler xmlns:xsd="http://www.w3.org/2001/XMLSchema" xsi:type="xsd:string">SGE</ns1:scheduler>

• For PBS: (if you are using a SGE based cluster you should see)

<ns1:scheduler xmlns:xsd="http://www.w3.org/2001/XMLSchema" xsi:type="xsd:string">PBS</ns1:scheduler>

If these lines are not in the file, then there is a problem with the installation. Send a note to the LabKey Server support boards for assistance.

Ensure that WS-GRAM server can read the Resource Manager's(Scheduler's) log file

For SGE-based clusters

1. Enable Job Log reporting for the SGE resource manager. As root run

/opt/sge/bin/lx24-x86/qconf -mconf

• This will open the editor. Change the reportingparams variable to be

reportingparams             accounting=true reporting=true 
flush_time=00:00:15 joblog=true sharelog=00:00:00

1. The job reporting output file is located at <SGE_HOME>/default/common/reporting. Ensure that the globus user is able to read the contents of this file.

1. Find the location of the server_logs log file for the PBS Server. For this documentation lets assume that is located at /var/spool/PBS/server_logs
2. Edit the PBS Adapters configuration file, <GLOBUS_LOCATION>/etc/globus-pbs.conf

• Change the variable log_path to be /var/spool/PBS/server_logs

1. Make sure that the globus user can read the contents /var/spool/PBS/server_logs

Start the WS-GRAM server

<GLOBUS_LOCATION>/bin/globus-start-container > <GLOBUS_LOCATION>/var/gram_output.log & 2>&1

This will output all log and error messages to the file <GLOBUS_LOCATION>/var/gram_output.log. In addition, the file will be overwritten during each restart.

Enable a user to submit jobs to the WS-GRAM service

1. Create a certificate request for the user
2. Have the certificate request signed by the CA
3. Add an entry to the gridmap-file to allow the user to submit a job to the cluster.

• Create an operating system account for the user named "labkey"
• Once the user is created, add the following to the labkey user's profile
• export GLOBUS_LOCATION=<GLOBUS_LOCATION>
• export JAVA_HOME=/usr/lib64/jvm/java
• Set this to whereever your JAVA home is located
• source $GLOBUS_LOCATION/etc/globus-user-env.sh
• Login as the user labkey (ie execute su - labkey )
• Request the user certificate by executing grid-cert-request
• Enter in the following information
• enter in name as "labkey user"
• PEM password (store this password away, as you will need it in the future)
• The request gets stored in ~labkey/.globus/usercert_request.pem
• Sign the user certificate request.
• To sign the key you need to perform the next tasks as root.
• execute grid-ca-sign -in ~labkey/.globus/usercert_request.pem -out ~labkey/.globus/usercert.pem
• This command will sign the certificate request created by the labkey user above and write the signed certificate into ~/home/labkey/.globus/usercert.pem and /root/.globus/simpleCA//newcerts/02.pem

• grid-proxy-init -debug -verify

User Cert File: /home/labkey/.globus/usercert.pem
User Key File: /home/labkey/.globus/userkey.pem

Trusted CA Cert Dir: /etc/grid-security/certificates

Output File: /tmp/x509up_u1002
Your identity: /O=Grid/OU=GlobusTest/OU=simpleCA-labkey-sample-ca.labkey.com/OU=labkey.com/CN=LabKey User
Enter GRID pass phrase for this identity:
Creating proxy ......++++++++++++
...++++++++++++
Done
Proxy Verify OK
Your proxy is valid until: Tue Jul  1 03:49:27 2008

The last step is to edit the gridmap-file. This file maps the certificate we created above to the operating system user who will be executing the job on the cluster (ie the user that will be executing the qsub command)

• Edit the file /etc/grid-security/grid-mapfile and append a line similar to the following

"/O=Grid/OU=GlobusTest/OU=simpleCA-labkey-sample-ca.labkey.com/OU=labkey.com/CN=LabKey User" labkey

• Note: The easiest way to create this entry is to copy the string from the output of the grid-proxy-init -debug -verify command.

Test the WS-GRAM installation

• Edit the file <GLOBUS_LOCATION>/container-log4j.properties
• uncomment the following lines

# log4j.category.org.globus.exec=DEBUG
# log4j.category.org.globus.transfer=DEBUG

• Append the following line

log4j.category.org.globus.ftp=DEBUG

• Remember to comment these lines out after your testing is complete and restart the server.

• login as the globus user
• Start the WSGRAM server and redirect all the output to the file <GLOBUS_LOCATION>/var/gram_debug.out

<GLOBUS_LOCATION>/bin/globus-start-container > <GLOBUS_LOCATION>/var/gram_debug.out & 2>&1

• Start the GridFTP server and redirect all output to the file <GLOBUS_LOCATION>/var/gridftp_output.log

<GLOBUS_LOCATION>/sbin/globus-gridftp-server -S -port 2811 > <GLOBUS_LOCATION>/var/gridftp_output.log & 2>&1

NOTE: You can find two scripts for starting and stopping the globus server that were during our testing. They are attached to the this wiki page.

Test the grid ftpserver

This test will send data to the GridFTP server

grid-proxy-init
globus-url-copy -vb gsiftp://localhost/dev/zero file:///dev/null

This will run until you hit CTRL-C to stop the transfer.

Verify that the labkey user can submit a job to the cluster.

1) Create the test script and name it qsubtest. This script, like the one above will simply run the env command on the cluster node

#!/bin/bash

date
env

2) submit the script using the qsub command

qsub -o ~labkey/globus_test/qsubtest_output.txt -e ~labkey/globus_test/qsubtest_err.txt qsubtest

This command will output

• STDOUT to the file ~labkey/globus_test/qsubtest_output.txt
• STDERR to the file ~labkey/globus_test/qsubtest_err.txt

Tue Aug 12 13:37:45 PDT 2008
MODULE_VERSION_STACK=3.1.6
LESSKEY=/etc/lesskey.bin
NNTPSERVER=news
INFODIR=/usr/local/info:/usr/share/info:/usr/info
MANPATH=/usr/local/gt4.0.6/man:/usr/local/man:/usr/share/man:/opt/mpich/man
HOSTNAME=cluster_node_name
XKEYSYMDB=/usr/share/X11/XKeysymDB
...

If the command was not successful, you can review the file for information on the failure ~labkey/globus_test/qsubtest_err.txt

Test the GRAM server: Test #1

1) Create a test script and call it gramtest. This will be a very simple script which will simply print out the environmental variables of the shell executing the job. This test script is actually a XML file written in the RSL format.

<job>
   <executable>/bin/env</executable>
   <stdout>${GLOBUS_USER_HOME}/globus_test/stdout</stdout>
   <stderr>${GLOBUS_USER_HOME}/globus_test/stderr</stderr>
</job>

This script tells the GRAM server to write the

• STDOUT to the file ~labkey/globus_test/stdout
• STDERR to the file ~labkey/globus_test/stderr

3) Submit the job to the GRAM server

globusrun-ws -submit -f gramtest

If this command is successful you should see output similar to below in the file ~labkey/globus_test/STDOUT

MODULE_VERSION_STACK=3.1.6
LESSKEY=/etc/lesskey.bin
NNTPSERVER=news
INFODIR=/usr/local/info:/usr/share/info:/usr/info
MANPATH=/usr/local/gt4.0.6/man:/usr/local/man:/usr/share/man:/opt/mpich/man
HOSTNAME=lk-globus
XKEYSYMDB=/usr/share/X11/XKeysymDB
...

If the command was not successful, you can review the two files for information on the failure

• ~labkey/globus_test/stderr
• <GLOBUS_LOCATION>/var/gram_debug.out

Test the GRAM server: Test #2

In this test we will submit a job to the PBS JobFactory. This will execute the test job out on the cluster you configured above.

1) Lets use the same test script as in Test#1. This will test if the GRAM server can successfully submit a job to the cluster. 2) Submit the job to the GRAM server

globusrun-ws -submit -f gramtest -Ft PBS

If this command is successful you should see out similar to the below in the file ~labkey/globus_test/STDOUT

MODULE_VERSION_STACK=3.1.6
LESSKEY=/etc/lesskey.bin
NNTPSERVER=news
INFODIR=/usr/local/info:/usr/share/info:/usr/info
MANPATH=/usr/local/gt4.0.6/man:/usr/local/man:/usr/share/man:/opt/mpich/man
HOSTNAME=cluster-node-name
XKEYSYMDB=/usr/share/X11/XKeysymDB
...

If the command was not successful, you can review 2 files for information on the failure

• ~labkey/globus_test/stderr
• <GLOBUS_LOCATION>/var/gram_debug.out

Enable a user to submit jobs to the WS-GRAM service

1. Create a certificate request for the user
2. Have the certificate request signed by the CA and return the signed certificate to the user
3. Add an entry to the gridmap-file to map the new certificate create in step 1 to the operating system user account

1. Create an operating system account for the user named "labkey"

• export GLOBUS_LOCATION=<GLOBUS_LOCATION>
• export JAVA_HOME=/usr/lib64/jvm/java
• Set this to whereever your JAVA home is located
• source $GLOBUS_LOCATION/etc/globus-user-env.sh

4. Request the user certificate by executing grid-cert-request

• Enter in the following information
• enter in the name as "LabKey User"
• PEM password (store this password away, as you will need it in the future)
• The request gets stored in ~labkey/.globus/usercert_request.pem

• To sign the key you need to perform the next tasks as root.
• execute grid-ca-sign -in ~labkey/.globus/usercert_request.pem -out ~labkey/.globus/usercert.pem
• This command will sign the certificate request created by the labkey user above and write the signed certificate into ~/home/labkey/.globus/usercert.pem and /root/.globus/simpleCA//newcerts/02.pem

grid-proxy-init -debug -verify

You should see output similar to the following

User Cert File: /home/labkey/.globus/usercert.pem
User Key File: /home/labkey/.globus/userkey.pem

Trusted CA Cert Dir: /etc/grid-security/certificates

Output File: /tmp/x509up_u1002
Your identity: /O=Grid/OU=GlobusTest/OU=simpleCA-labkey-sample-ca.labkey.com/OU=labkey.com/CN=LabKey User
Enter GRID pass phrase for this identity:
Creating proxy ......++++++++++++
...++++++++++++
Done
Proxy Verify OK
Your proxy is valid until: Tue Jul  1 03:49:27 2008

The last step is to edi the gridmap-file. This file maps the certificate we created above to the operating system user who will be executing the job submitted to the WS-GRAM service.

• Edit the file /etc/grid-security/grid-mapfile and append a line similar to the following
• "/O=Grid/OU=GlobusTest/OU=simpleCA-labkey-sample-ca.labkey.com/OU=labkey.com/CN=LabKey User" labkey
• Note: The easiest way to create this entry is to copy the string from the output of the grid-proxy-init -debug -verify command.

Important information

• User Cert File: /home/labkey/.globus/usercert.pem
• User Key File: /home/labkey/.globus/userkey.pem
• Pass Phrase for User Key File.

Now that the Prerequisites for the Enterprise Pipeline are installed you will need to configure the LabKey Server software to use the Enterprise Pipeline.

1. Configure the LabKey Server to use the Enterprise Pipeline
2. Using the LabKey Server Enterprise Pipeline
3. Configure the Conversion Service (this is an optional step, if you intend to use a Conversion Server)

These instructions will explain how to

1. configure the LabKey Server to use the Enterprise Pipeline and
2. Create the LabKey Tool directory (which contains the MS1 and MS2 analysis tools to be run on the cluster execution nodes)

Assumptions

• Use of a Network File System: The LabKey web server, LabKey Conversion server and the cluster nodes must be able to mount the following resources
• Pipeline directory (location where mzXML, pepXML, etc files are located)
• Pipeline Bin directory (location where third-party tools (TPP, Xtandem, etc) are located.
• MS1 and MS2 analysis tools will be run on either a PBS or SGE based cluster.
• Sun Java 1.5 or greater is installed on all cluster execution node
• You have downloaded or built from the subversion tree the following files
• LabKey Server Enterprise Edition v8.3 or greater
• Labkey Server v8.3 Enterprise Pipeline Configuration files

Verify the version of your LabKey Server.

To verify if you are running the Enterprise Edition follow the instructions below

1. Log on to your LabKey Server using a Site Admin account
2. Open the Admin Console
3. Under the Module Information section verify that the following modules are installed

• BigIron

Enable Communication with the ActiveMQ JMS Queue.

<Resource name="jms/ConnectionFactory" auth="Container"
   type="org.apache.activemq.ActiveMQConnectionFactory"
   factory="org.apache.activemq.jndi.JNDIReferenceFactory"
   description="JMS Connection Factory"
   brokerURL="tcp://@@JMSQUEUE@@:61616"
   brokerName="LocalActiveMQBroker"/>

You will need to change setting for brokerURL to point to the location of your ActiveMQ installation. (i.e. replace @@JMSQUEUE@@ with the hostname of the server running the ActiveMQ software)

Note: If this is a new installation of the LabKey server and are not an upgrade of the current installation, then the XML above will be located in the labkey.xml file, but will be commented out. Uncomment out the XML in the file instead of performing at cut and paste of the text above.

Enable Communication with the Globus GRAM server

<Resource name="services/NotificationConsumerService/home"
   type="org.globus.wsrf.impl.notification.NotificationConsumerHome"
   factory="org.globus.wsrf.jndi.BeanFactory"
   resourceClass="org.globus.wsrf.impl.NotificationConsumerCallbackManagerImpl"
   resourceKeyName="{http://www.globus.org/namespaces/2004/06/core}NotificationConsumerKey"
   resourceKeyType="java.lang.String" />
<Resource name="timer/ContainerTimer"
   type="org.globus.wsrf.impl.timer.TimerManagerImpl"
   factory="org.globus.wsrf.jndi.BeanFactory" />
<Resource name="topic/ContainerTopicExpressionEngine"
   type="org.globus.wsrf.impl.TopicExpressionEngineImpl"
   factory="org.globus.wsrf.jndi.BeanFactory" />
<Resource name="query/eval/xpath"
   type="org.globus.wsrf.impl.XPathExpressionEvaluator"
   factory="org.globus.wsrf.jndi.BeanFactory" />
<Resource name="query/ContainerQueryEngine"
   type="org.globus.wsrf.impl.QueryEngineImpl"
   factory="org.globus.wsrf.jndi.BeanFactory" />
<Resource name="topic/eval/simple"
   type="org.globus.wsrf.impl.SimpleTopicExpressionEvaluator"
   factory="org.globus.wsrf.jndi.BeanFactory" />

Note: If this is a new installation of the LabKey server and are not an upgrade of the current installation, then the XML above will be located in the labkey.xml file, but will be commented out. Uncomment out the XML in the file instead of performing at cut and paste of the text above.

Set the Enterprise Pipeline configuration directory

<Parameter name="org.labkey.api.pipeline.config" value="@@LABKEY_HOME@@/config"/>

Set this to the location of your Enterprise Pipeline configuration directory. The default setting is <LABKEY_HOME>/config. (i.e. replace @@LABKEY_HOME@@ with the full path to the LABKEY_HOME directory for your installation)

Note: If this is a new installation of the LabKey server and are not an upgrade of the current installation, then the XML above will be located in the labkey.xml file, but will be commented out. Uncomment out the XML in the file instead of performing at cut and paste of the text above.

Create the Enterprise Pipeline Configuration Files for the Web Server.

1. Unzip LabKey Server Enterprise Pipeline Configuration distribution and copy the ~~webserver configuration files to the Pipeline Configuration directory specified in the last step (ie <LABKEY_HOME>/config)
2. There are 3 configuration files.

• pipelineConfig.xml: This is used to configure the communication with the Globus WSGRAM server.
• ms2config.xml: This is used to configure
• where MS2 searches will be performed (on the cluster, on a remote server or locally)
• where the Conversion of raw files to mzXML will occur (if required)
• which analysis tools will be executed during a MS2 search
• ms1config.xml: This is used to configure
• where MS1 searches will be performed (on the cluster, on a remote server or locally)
• which analysis tools will be executed during a MS1 search

1. Edit the file pipelineConfig.xml and enter the information for your Globus WSGRAM server

• jobFactoryType is where you configure the type of Cluster Scheduler. The 2 supported options are PBS or SGE
• queue is the name of the Queue on the Cluster that you would like all Pipeline jobs to be executed in.
• javaHome is the JAVA_HOME location on the cluster execution nodes
• labKeyDir is the location of the <LABKEY_TOOLS>/labkey directory on the cluster execution nodes as described in the Create the LABKEY_TOOLS directory that will be used on the Cluster below
• globusServer is the hostname of the Globus WSGRAM server
• pathMapping allows directories on the Web Server to be mapped to directories located on the cluster nodes. This is used to map the location of the Pipeline Directories on the Web Server to their location on the cluster nodes. This is only required if you are running the LabKey Server on a Windows server.

1. Edit the file ms2Config.xml

• Documentation is under development.

1. Edit the file ms1Config.xml

• Documentation is under development.

Copy the Globus CA Certificates onto the LabKey Server.

1. Determine the home directory for the user that is running LabKey Server's Tomcat process. As of version 9.1, this is shown in the Admin Console.

• This can also be done, but placing the attached file, printenv.jsp in an available WebApp running on your tomcat server. (i.e. if you put the file into the directory <CATALINA_HOME>/webapps/ROOT, then you will be able to access it via http://localhost:8443/printenv.jsp )

1. Create the directory <USER_HOME>/.globus/certificates
2. Copy the contents of the /etc/grid-certificates/certificates directory on your Globus server to <USER_HOME>/.globus/certificates. It should contain a number of files with names like 7a1c240b.0 and globus-user-ssl.conf.7a1c240b.

1. Allow the Tomcat Server to use plain text Cipher to communicate with the Globus Server

1. Edit the file <CATALINA_HOME>/conf/server.xml
2. Add the following ciphers attribute to SSL Connector configuration in the server.xml file

ciphers="SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, 
TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, 
SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, 
SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, 
SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, 
SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_RSA_WITH_NULL_MD5"

Restart the LabKey Server.

Once the server has been restarted. You will want to ensure that the server the server started up with no errors.

1. Log on to your LabKey Server using a Site Admin account
2. Open the Admin Console by
1. Expanding the Manage Site menu on the left pane of the site
2. Click on Admin Console link
3. In the Diagnostics Section click on view all site errors
4. Check to see that no errors have occurred after the restart

Create the LABKEY_TOOLS directory that will be used on the Cluster.

• Required LabKey Software and configuration files
• TPP tools
• XTandem search engine
• msInspect
• Additional MS1 and MS2 analysis tools

Create the <LABKEY_TOOLS> directory

• This directory must be accessible from all cluster execution nodes.
• We recommend that the directory created on Shared File System which will be mounted on the cluster nodes as well as the Conversion Server.

Download the Required LabKey Software

1. Unzip the LabKey Server Enterprise Edition distribution into the directory <LABKEY_TOOLS>/labkey/dist
2. Unzip the LabKey Server Pipeline Configuration distribution into the directory <LABKEY_TOOLS>/labkey/dist/conf

Install the LabKey Software into the <LABKEY_TOOLS> directory

• The directory <LABKEY_TOOLS>/labkey/dist/LabKey<VERSION>-Enterprise-Bin/labkeywebapp
• The directory <LABKEY_TOOLS>/labkey/dist/LabKey<VERSION>-Enterprise-Bin/modules
• The directory <LABKEY_TOOLS>/labkey/dist/LabKey<VERSION>-Enterprise-Bin/pipeline-lib
• The file <LABKEY_TOOLS>/labkey/dist/LabKey<VERSION>-Enterprise-Bin/server-lib/labkeyBootstrap.jar

cd <LABKEY_TOOLS>/labkey/
java -jar labkeyBootstrap.jar

Install Enterprise Pipeline configuration files into the <LABKEY_TOOLS> directory

• All files in the directory <LABKEY_TOOLS>/labkey/dist/LabKey8.3-xxxxx-PipelineConfig/cluster

Create the Enterprise Pipeline Configuration Files for use on the Cluster.

1. There are 3 configuration files.

• Description of configuration files is under development.

1. Edit the file pipelineConfig.xml

• Documentation is under development.

1. Edit the file ms2Config.xml

• Documentation is under development.

1. Edit the file ms1Config.xml

• Documentation is under development.

1. Install the MS1 and MS2 analysis tools on the Cluster

Documentation is under development

Test the Configuration

1. Can the cluster node see the Pipeline Directory and the <LabKey_Tools> directory

• Test under development

1. Can the cluster node execute Xtandem

• Test under development

1. Can the cluster node execute the java binary

• Test under development

1. Can the cluster node execute a Xtandem search against an mzXML file located in the Pipeline Directory?

• Test under development

1. Can the cluster node execute a PeptideProphet search against the resultant pepXML file

• Test under development

1. Can the cluster node execute the Xtandem search again, but this time using the LabKey java code located on the cluster node

• Test under development

These instructions explain how to configure a Project to use the Enterprise Pipeline for MS1 and MS2 searches. For these instructions, we will create a new Project and configure a Pipeline for the new Project.

If you have not done installed the Prerequisite software for the Enterprise Pipeline and Configured the LabKey Server to use the Enterprise Pipeline please do these before performing the tasks below.

Create a new Project to test the Enterprise Pipeline

1. Log on to your LabKey Server using a Site Admin account
2. Create a new Project with the following options

• Project Name = PipelineTest
• Select MS2 Folder Type radio button

1. Choose the default settings during the Project Creation.

Configure the Project to use the Enterprise Pipeline

• Pipeline Root Directory
• Globus User Key File
• Pass Phrase for User Key File
• Globus User Cert File

NOTE: Different User Cert/Key pairs can be used for different Pipelines.

Setup the Pipeline

1. Click on the Setup button in the Data Pipeline Webpart
2. Enter in the following information

• Path to the desired Pipeline Root directory on the Web Server
• File location of the Globus User Key File
• The Pass Phrase used for the User Key File
• File location of the Globus User Cert File

1. Click on the Set button
2. Goto the MS2 Dashboard, by clicking the PipelineTest link in the upper left pane

Testing the Enterprise Pipeline

• Click on the Process and Upload Data button in the Data Pipeline Webpart
• Navigate to an mzXML file with the Pipeline Root Directory and hit the X!Tandem Pepitide Search button to the right of the filename.

These instructions explain how to configure the LabKey Server Enterprise Pipeline Conversion Service

If you have not installed Prerequisite software for the Enterprise Pipeline and Configured the LabKey Server to use the Enterprise Pipeline, please complete them before performing the tasks below.

Assumptions

• The Conversion Server can be configured to convert from native acquisition files for a number of manufacturers.
• Use of a Shared File System: The LabKey Conversion server must be able to mount the following resources
• Pipeline directory (location where mzXML, pepXML, etc files are located)
• Sun Java 1.5 or greater is installed
• You have downloaded (or built from the subversion tree) the following files
• LabKey Server Enterprise Edition v8.3 or greater
• Labkey Server v8.3 Enterprise Pipeline Configuration files

Download and Expand the LabKey Conversion Server Software

1. Create the <LABKEY_HOME> directory (LabKey recommends you use c:\LabKey )
2. Unzip the LabKey Server Enterprise Edition distribution into the directory <LABKEY_HOME>\dist
3. Unzip the LabKey Server Pipeline Configuration distribution into the directory <LABKEY_HOME>\dist\config
4. Unzip the LabKey Server Remote Service distribution into the directory <LABKEY_HOME>\dist\service

Install the LabKey Software

• The directory <LABKEY_HOME>\dist\LabKey8.3-xxxxx-Enterprise-Bin\labkeywebapp
• The directory <LABKEY_HOME>\dist\LabKey8.3-xxxxx-Enterprise-Bin\modules
• The file <LABKEY_HOME>\dist\LabKey8.3-xxxxx-Enterprise-Bin\server-lib\labkeyBootstrap.jar

• All files in the directory <LABKEY_HOME>\dist\LabKey8.3-xxxxx-PipelineConfig\remote

cd <LABKEY_HOME>
java -jar labkeyBootstrap.jar

In the System Control Panel Create the LABKEY_ROOT environment variable and set it to <LABKEY_HOME>. This should be a System Variable

Create the Tools Directory

Further Documentation is under development

Edit the Enterprise Pipeline Configuration File (pipelineConfig.xml)

Enable Communication with the JMS Queue

<bean id="activeMqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
   <constructor-arg value="tcp://@@JMSQUEUE@@:61616"/>
</bean>

Configure the WORK DIRECTORY

• tempDirectory: This is the location of the WORK DIRECTORY on the server
• lockDirectory: This setting should be commented out, unless you are installing at the FHCRC
• cleanupOnStartup: This setting tells the Conversion server to delete all files in the WORK DIRECTORY at startup. This ensures that corrupted files are not used during conversion

<property name="workDirFactory">
  <bean class="org.labkey.pipeline.api.WorkDirectoryRemote$Factory">
    <!-- <property name="lockDirectory" value="T:/tools/bin/syncp-locks"/> -->
    <property name="cleanupOnStartup" value="true" />
    <property name="tempDirectory" value="c:/TempDir" />
  </bean>
</property>

Configure the Application Properties

• toolsDirectory: This is the location where the Conversion tools (ReAdW.exe, etc) are located. For most installations this should be set to <LABKEY_HOME>\bin
• networkDrive settings: These settings specify the location of the shared network storage system. You will need to specify the approriate drive letter, UNC PATH, username and password for the Conversion Server to mount the drive at startup.

<property name="appProperties">
  <bean class="org.labkey.pipeline.api.properties.ApplicationPropertiesImpl">
    <property name="networkDriveLetter" value="t" />
    <property name="networkDrivePath" value="\\@@SERVER@@\@@SHARE@@" />
    <!-- Map the network drive manually in dev mode, or supply a user and password -->
    <property name="networkDriveUser" value="@@USER@@" />
    <property name="networkDrivePassword" value="@@PASSWORD@@" />
    <property name="toolsDirectory" value="c:/labkey/bin" />
  </bean>
</property>

Change all values in the appProperties to fix your environment

Edit the Enterprise Pipeline MS2 Configuration File (ms2Config.xml)

Documentation is under development.

Edit the Enterprise Pipeline MS1 Configuration File (ms1Config.xml)

Documentation is under development.

Install the Conversion Server as a Windows Service

Install the LabKey Remote Service

• Copy the directory <LABKEY_HOME>/dist/service to <LABKEY_HOME>/bin/service~
• Install the Windows Service by running the following from the Command Prompt

<LABKEY_HOME>binserviceinstallService.bat

How to re-install or uninstall the LabKey Remote Pipeline Service

<LABKEY_HOME>\bin\service\installService.bat

<LABKEY_HOME>\bin\service\removeService.bat

<LABKEY_HOME>\bin\service\removeService.bat

<LABKEY_HOME>\bin\service\installService.bat

NOTE: If running Windows XP, this service cannot be run as the Local System user. You will need to change the LabKey Remote Pipeline Service to log on as a different user.

This page is intended to capture information about monitoring, maintaining, and troubleshooting the Enterprise Pipeline. Due to the high level of customization that is possible, some of the information may vary from installation to installation.

 Determining What Jobs and Tasks Are Actively Running

 Each job in the pipeline is composed of one or more tasks. These tasks are assigned to run at a particular location. Locations might include the web server, cluster, remote server for RAW to mzXML conversion, etc. Each location may have one or more worker threads that runs the tasks. A typical installation might have the following locations that run the specified tasks:

When jobs are submitted, the first task in the pipeline will be added to the queue in the WAITING (SEARCH WAITING, for example) state. As soon as there is a worker thread available, it will take the job from the queue and change the state to RUNNING. When it is done, it will put the task back on the queue in the COMPLETE state. The web server should immediately advance the job to the next task and put it back in the queue in the WAITING state.

If jobs remain in a intermediate COMPLETE state for more than a few seconds, there is something wrong and the pipeline is not properly advancing the jobs.

Similarly, if there are jobs in the WAITING state for any of the locations, and no jobs in the RUNNING state for those locations, something is wrong and the pipeline is not properly running the jobs.

Overview 

Note: Due to the installation-specific nature of this feature, LabKey Corporation does not provide support for it on the free community forums.  Please contact info@labkey.com for commercial support. 

Note: The Perl-based MS2 Cluster pipeline is deprecated and will no longer be supported in future versions. As of version 8.3, use the Enterprise Pipeline instead.

This page helps you install and set up the MS2 Perl Cluster Pipeline. 

Topics:

• Install the Necessary Executables
• Set Up a Pipeline Root
• Set Up the CPAS Web Server
• Run a X! Tandem Search
• Run in Production 

Additional Topics: 

• Install the mzXML Conversion Service
• Run the MS2 Cluster Pipeline  

Install the Necessary Executables

• Choose an installation root directory that is mounted in the same location on both the scheduler node and the cluster nodes. e.g. /usr/cpas/bin

• Install the pipeline tools into this directory, using one of the following methods:

  • Extract the attached pipeline.zip to this location.
    Then give all perl scripts *.pl files executable permissions (including directories tandem and mascot).

  • Or execute the subversion command:
    svn checkout --username cpas --password cpas https://hedgehog.fhcrc.org/tor/stedi/trunk/tools/pipeline/bin bin
    This will give you the most recent revisions, and allow you to update to changes more easily.

• Install LWP and XML perl modules.
  cpan> install LWP
  cpan> install XML::DOM
  cpan> install XML::Writer

• Use a cluster node with development tools installed on it to build X!Tandem, and copy it to /tandem

• Use a cluster node with development tools installed on it to build the TPP:

• In /src/Makefile.incl add the line "XML_ONLY=1", and modify "TPP_ROOT=..." to point to /tpp/

• Run "make configure all install"

• Add viewerApp.jar for msInsptect to /bin/msInpsect.

• Review params.xml for a list of site specific configuration parameters, and set these appropriately for your system.

• Make sure your cluster supports the necessary queue name(s) for your params.xml setup. (Default: 'labkey')

• Make sure cluster job submission and status executables are on the system path. (e.g. Run 'qstat' from a command prompt, and make sure it works.)

• Run pipe.pl without arguments for a list of possible runtime parameters/overrides.

Set Up a Pipeline Root

• Choose a pipeline root location that is mounted in the same location on both the scheduler node and the cluster nodes. e.g. /home/lab/pipeline/Project

• Choose a location to store FASTA files for the entire CPAS system, again accessible to both the scheduler and the cluster nodes. e.g. /home/lab/pipeline/databases

• These locations must also be available to the CPAS web server, either by a drive mapped to a UNC path (e.g. \\server\user\pipeline\databases), if the web server is running Windows, or by mounting a shared system on a Unix box.

• First create an interactive version of a pipeline script file named "pipe.sh" in the pipeline root directory that reads something like:
  
  #!/usr/bin/bash
  
  /usr/cpas/bin/pipe.pl --v --v --i --t=30 --r=/Project /home/lab/pipeline/Project

• Again for a full list of parameters, and their usage, run "pipe.pl" at a command prompt.
• To test this script, type "./pipe.sh" from a command prompt in the pipeline root.

• The script should begin reporting "Waiting 30 seconds..." every 30 seconds.

Set Up the CPAS Web Server

• Using a web browser connected to a CPAS web server, click the Admin Console link under Manage Site.

• Click the site settings link.

• Check the Has pipeline cluster checkbox to tell CPAS to allow your cluster to run the MS2 analysis.

• Click the Save button to save the settings.

• Navigate to (or create) the project referenced in the "pipe.sh" script created above.

• Add the Data Pipeline web part to the project page, if it is not already present.

• Click the Setup button under Data Pipeline.

• Enter the path on the CPAS web server that maps to the pipeline root where you created the "pipe.sh" script. e.g. T:\lab\pipeline\Project

• Click the Set button to save this value.

• Click the Set FASTA root link, and enter the path on the CPAS web server that maps to the FASTA location you created above. e.g. T:\lab\pipeline\databases

• Click the View Status button.

Run a X! Tandem Search

• Copy or move the FASTA file to be searched to the FASTA root you have specified.

• Creat a directory for your results under the pipeline root, preferably using a directory structure you lab agrees on. e.g. /home/lab/pipeline/user/2007/04/ICAT_003

• Place mzXML (or RAW, if you have set up the ConversionQueue) files into this directory.

• In the browser showing your CPAS project, click the Process and Upload Data button.

• Click the folder icons to navigate down to the directory you created.

• The mzXML (or RAW) files will be listed with a "X!Tandem Peptide Search" button beside them.

• Click the X!Tandem Peptide Search button and proceed through the forms to start the search.

• Switch to a window showing the running "pipe.sh" script.

• Output in this window should soon show cluster jobs being submitted, and status information as the analysis moves through the pipeline.

Run in Production

• Replace the parameters "--v --v --i --t=30" in your test pipe.sh with "--t=0".
• Run crontab -e on the cluster scheduler node, and add a line like:
  
  0,10,20,30,40,50 * * * * /home/lab/pipeline/pipe.sh 2>&1 | /usr/cpas/bin/mail
  if.pl -s "Pipeline output" admin@uxyz.org >/dev/null 2>/dev/null

Installation Requirements

• Install CPAS and the MS2 cluster pipeline. (Currently the mzXML Conversion Service is only available with the MS2 cluster pipeline.)
• Choose a machine on which to run the conversion server:
  • The machine must be running Windows.
  • The server can run on the same machine as CPAS itself, if it is running Windows (or VMWare with Windows VM?)
• Install the Java 1.5 runtime and Tomcat 5 web server.
• Install vendor software for the converters you will use.  (Currently only ThermoFinnigan and Waters are supported.)
• Install mzXML converter EXEs by extracting the attached Converters.zip:
  • ReAdW.exe for ThermoFinnigan
  • wolf.exe for Waters
• Make sure these executables are on the path for the service running Tomcat.

Installing the Conversion Service

• Place the attached ConversionQueue.war in <tomcat-root>/webapps
• Place the attached ConversionQueue.xml in <tomcat-root>/conf/Catalina/localhost
• Edit the properties marked with @@ in the ConversionQueue.xml to match your system:
  • Set @@conversionQueueDocBase@@ to the directory where the WAR is exploded
  • Set @@networkDriveLetter@@ to the drive leter of your choosing.
    (NB: If you are running CPAS on a Windows server, this should be the same drive chosen for the CPAS installation.)
  • Set @@networkDrivePath@@ to the UNC path where your raw data will be placed (e.g. \\large\storage)
    (NB: If you are running CPAS on a Windows server, this should be the same path chosen for the CPAS installation.)
  • Set @@networkDriveUser@@ to the user name used to log onto this share (e.g. DOMAIN\labkey)
  • Set @@networkDrivePassword@@ to the password for the specified user
  • Set @@smtpHost@@ to the SMTP server that may be used for system errors
  • Set @@smtpUser@@ to the user name used for SMTP communication
  • Set @@smtpPort@@ to the port used by the SMTP service on the specified server
• Restart the Tomcat server.

Testing the Conversion Service

• First make sure Tomcat is now aware of the web app by pointing a browser at
  http://myserver/ConversionQueue/
  You should get a HTML page back.  Browse a couple links on the page.
• Next test your network drive by pointing to a raw data file to convert, e.g.
  http://myserver/ConversionQueue/ConvertSpectrum/submit.post?type=thermo&infile=T:\test\Test.RAW
  You should get a single line of text "Succeeded".
• Check the contents of the queue
  http://myserver/ConversionQueue/ConvertSpectrum/list.view
  You should see the request you just made.  Refresh until the job appears complete.
• Acknowledge completion of the conversion
  http://myserver/ConversionQueue/ConvertSpectrum/acknowledge.post?infile=T:\test\Test.RAW
  You should get a single line of text "Acknowledged".
• If any of the above fail, consult the Tomcat log file conversion.log in <tomcat-home>/logs.

Connecting the Cluster Pipeline

• To connect your MS2 cluster pipeline to the mzXML conversion service, edit the "params.xml" file in the directory where you installed pipe.pl.
  • Set "pipeline config, conversion server" to point to the server you just installed.
  • If your CPAS server is not running on Windows, make sure you set the "pipeline config, windows path prefix" and "pipeline config, unix path prefix" to correctly map paths between your Unix cluster and the conversion server Windows drive mapping.

Testing Conversion in the Cluster Pipeline

• You may want to set up a new pipeline root with a pipe.sh debug parameters like "--v --v --i --t=15", and run this script in a command window, rather than using an existing production pipeline that runs as a cron job.
• Place a raw data file in a folder under this debug pipeline root.
• Set up a CPAS project to point to this pipeline root directory.
• Click on the "Process and Upload Data" button, and navigate to the directory containing your raw data file.
• Initiate a simple peptide search.
• If everything is set up correctly, the pipeline should progress to completion without error.
• If you get errors, review the logs and the output in your pipeline command console.

Analysis Life Cycle

The current life cycle of CPAS-started cluster pipeline jobs:

1. CPAS writes a tandem.xml to disk, creates a dummy PipelineJob, and sets its status to "WAITING", which adds a record to the pipeline.StatusFiles table in the database for user inspection, and also writes a corresponding .status file to disk for driving the state of the pipe.pl Perl script. This job is discarded without putting it into the CPAS PipelineQueue.
2. The pipe.pl script scans the disk looking for work to perform. When it sees a tandem.xml file in a directory that lacks a pipe.log file, it makes a list of data (.RAW or .mzXML) files for which it needs to do work. Each time it does this, pipe.pl will log information about what it found and did to pipe-processing.log for the directory.
3. For each data file, pipe.pl then checks the corresponding <basename>.status file to detect the current state of processing. If the status is "WAITING" or not present ("UNKNOWN"), then pipe.pl will review the files present, and make its best guess at where to start processing. e.g. If a .xtan.xml file exists, it will skip the search, and if the .pep.xml file exists, it will upload to the CPAS web server.
4. If the only available data file is a .RAW, then pipe.pl will call the ConversionQueue web server to convert to .mzXML.
5. If the data file is .mzXML, and .pep.xml file is not present, then analysis of the data file will be scheduled on the cluster. If you log into the scheduler machine that is running pipe.pl, you can usually get more interesting information about exactly what state these scheduled jobs are in, but from the cluster pipeline's perspective, they will remain in the "PROCESSING" state until the analysis is complete, and the .pep.xml file exists. Also, you can look in the pipe-processing.log to see scheduler state for jobs in the directory, which looks like:

LOG: Checking job status sergei_digest_A_full_01\\
202136.gazoo        sergei_digest_A  edi                     0 Q xtandem\\
LOG: Checking job status sergei_digest_A_full_02\\
202137.gazoo        sergei_digest_A  edi              00:13:23 R xtandem\\
202138.gazoo        sergei_digest_A  edi                     0 H xtandem\\

1. As cluster jobs complete, the output from the script run on the cluster node will be written to a .out file in the analysis directory. These files get created by the scheduler with owner only permissions. The pipeline does its best to append all .out files to the appropriate analysis log, and remove the restricted .out files. If a job is given ERROR state (type=job failure), most of the time the specifics of what happened will have been appended to the .log file, which is accessible from the CPAS interface. If for some reason, you need to dig into a .out file, you will need to access it as the user that ran the chron job either through a Windows share, or by logging into the scheduler node.
2. Once the .pep.xml file does exist, pipe.pl may simply request a single run upload of the results, or it may consider this a "COMPLETE" fraction, waiting for all fractions in the directory to reach the "COMPLETE" status before starting analysis of the batched set, or it may do both. (See the "pipeline, data type" value in the tandem.xml.)
3. If the directory is a set of fractions, then pipe.pl will run a subsequent analysis that batches the raw fraction .pep.xml files, and then runs PeptideProphet, ProteinProphet, and any quantitation, with results written to all.pep.xml and all.prot.xml, and pipeline information in all.status and all.log.
4. When the desired .pep.xml is present pipe.pl sends a request to CPAS to upload the analyzed data. At this point, CPAS creates a new PipelineJob, sets status to "LOADING" (both in the database, and on disk), and puts the PipelineJob into the PipelineQueue.
5. When CPAS actually begins working on loading the data from the .mzXML, .pep.xml, and .prot.xml files, it changes the status to "LOADING EXPERIMENT".
6. In the meantime pipe.pl simply reports that status it finds until status returns to something it knows how to handle, assuming CPAS knows what it is doing, and that it will eventually set status to either "ERROR" or "COMPLETE". As always, pipe.pl reads only the .status file on disk, meaning changing the status in the database directly to "ERROR" or "COMPLETE" will not have the desired effect.
7. If at any time any part of the system sets the status on disk to "ERROR", pipe.pl will rename the .status file to .status.err, and automatically try again, looking at the files present to determine what to do. If the status is "ERROR", and .status.err already exists, pipe.pl will wait for human intervention. Clicking the "Retry" button in CPAS synchronizes the disk status with the status found in the database, if this is not already true, and then removes any .status.err file found.
8. When all data files for a directory have a .status file with the status "COMPLETE", pipe.pl will rename pipe-processing.log to pipe.log, and delete all *.status* files. In this state, the directory will no longer trigger any further processing.

Trouble-shooting

Job status is ERROR

• Click on the ERROR link to view the job details page.
• Click the file link for the job's .log file.
• Review the cause of the error in the log, and determine whether it was a system failure that may be sporadic or something more fixed like a code bug that requires a fix.
• If it may be sporadic, return to the job details page, and click the Retry button.

• This can happen if something goes wrong with the cluster or file system.
• After clicking the "ERROR" link and reviewing the logs for a few failed jobs. Click the "Folder" button in one of the jobs.
• On a page showing pipeline status for the folder, click the "Errors" link above the status.
• With only errors showing, click the "Select All" button at the bottom of the page.
• Click the "Retry" button.

• Look at the pipeline site administration page.
• Click the "Status Queue" button. Is the job listed as waiting?
• Look for the job in "LOADING EXPERIMENT". Is it below the job in question?
• Look at its details page.
• Check its modified time, and .log file. Does it seem to be making progress?
• If it is above, check the details and .log file for the job in question, and make sure their are no exceptions. CPAS sometimes fails while loading without setting status to "ERROR".
• If you decide it really has failed, you must delete the .status file on disk to get pipe.pl to retry.

Topics:

• Set up file transfer via FTP
• Linux Example: Configure FTP on Linux
• General Documenation: Set Up the FTP Server
• Set up R on a LabKey Server
• Linux Example: Configure R on Linux
• General Documentation: Set Up R
• Set up R graphics on a server that lacks the X Windows display system (a.k.a. a headless server)
• Linux Example: Configure the Virtual Frame Buffer on Linux
• General Documentation: See the last section on the Set Up R page.
• Install CPAS.
• Linux Example: Install CPAS on Linux

Items installed via these instructions:

• Sun Java
• Apache Tomcat
• postgres
• X!tandem
• TPP Tools
• Graphviz
• CPAS

• R (see Configure R on Linux)
• Xvfb (see Configure the Virtual Frame Buffer on Linux)
• ftpserver (see Configure FTP on Linux)

• Linux Distro: Fedora 7
• Kernel: 2.6.20-2936.fc7xen
• Processor Type: x86_64

Install Sun Java

CPAS requires the use of Sun Java and GCJ is not supported.

To install Sun Java, you will need to install two packages:

1. JDK 6 Update 3 from Sun. This is a Linux RPM self-extracting file.
2. JPackage Compatibility RPM (this RPM creates the proper links such that Sun Java is compatible with JPackage and the alternatives system)

• http://java.sun.com/javase/downloads/index.jsp

["error">root@<YourServerName> Download?]# chmod +x jdk-6u3-linux-i586-rpm.bin 
["error">root@<YourServerName> Download?]# ./jdk-6u3-linux-i586-rpm.bin 
...

This package installs both the java software and the Sun JavaDB software. You do not need the JavaDB software, so you should remove it.

["error">root@<YourServerName. Download?]# rpm --erase sun-javadb-client sun-javadb-common 
sun-javadb-core sun-javadb-demo sun-javadb-docs sun-javadb-javadoc

Now download and install the compat rpm from JPackage:

["error">root@<YourServerName. Download?]# wget 
http://mirrors.dotsrc.org/jpackage/5.0/generic/non-free/RPMS/java-1.6.0-sun-compat-1.6.0.03-1jpp.i586.rpm
["error">root@<YourServerName> Download?]# rpm --install java-1.6.0-sun-compat-1.6.0.03-1jpp.i586.rpm

Test to make sure this worked:

["error">root@<YourServerName> Download?]# alternatives --config java

Two programs provide 'java':

Selection    Command
-----------------------------------------------
   1           /usr/lib/jvm/jre-1.5.0-gcj/bin/java
*+ 2           /usr/lib/jvm/jre-1.6.0-sun/bin/java

Press "enter" to keep the current selection(+), or type a selection number:

["error">root@<YourServerName> Download?]# java -version
java version "1.6.0_03"
Java(TM) SE Runtime Environment (build 1.6.0_03-b05)
Java HotSpot(TM) Server VM (build 1.6.0_03-b05, mixed mode)
["error">root@<YourServerName> Download?]#

This shows that the installation was successful.

The last step is to make sure that the user who will be executing Tomcat has JAVA_HOME set. For the both the root user and the tomcat user you can do the following:

["error">root@<YourServerName> LabKey2.3-7771-bin?]# vi  ~/.bash_profile 
"missing" href="/Documentation/Archive/9.1/wiki-page.view?name=added">added
JAVA_HOME=/usr/lib/jvm/java-1.6.0-sun
CATALINA_HOME=/usr/local/apache-tomcat-5.5.25
CATALINA_OPTS=-Djava.awt.headless=true
export CATALINA_OPTS
export JAVA_HOME
export CATALINA_HOME

Install the Tomcat Server

Download and unpack Tomcat v5.5.25

["error">root@<YourServerName> Download?]# wget 
http://apache.mirrors.redwire.net/tomcat/tomcat-5/v5.5.25/bin/apache-tomcat-5.5.25.tar.gz
["error">root@<YourServerName> Download?]# cd /usr/local
["error">root@<YourServerName> local?]# tar xzf ~/Download/apache-tomcat-5.5.25.tar.gz 
["error">root@<YourServerName> local?]# cd apache-tomcat-5.5.25/
["error">root@<YourServerName> apache-tomcat-5.5.25?]# ls
bin  common  conf  LICENSE  logs  NOTICE  RELEASE-NOTES  RUNNING.txt  server  shared  temp  webapps  work

Create the tomcat user

["error">root@<YourServerName> ~?]# adduser -s /sbin/nologin tomcat
["error">root@<YourServerName> ~?]# su - tomcat
["error">tomcat@<YourServerName> ~?]$ vi .bashrc

JAVA_HOME=/usr/lib/jvm/java-1.6.0-sun
CATALINA_HOME=/usr/local/apache-tomcat-5.5.25
CATALINA_OPTS=-Djava.awt.headless=true
export CATALINA_OPTS
export JAVA_HOME
export CATALINA_HOME

["error">tomcat@<YourServerName> ~?]$ exit
logout

Configure the apache server

Enabled Access Logging on the server:

["error">root@<YourServerName> ~?]# vi /usr/local/apache-tomcat-5.5.25/conf/server.xml

Change:

<!--
        <Valve className="org.apache.catalina.valves.FastCommonAccessLogValve"
                 directory="logs"  prefix="localhost_access_log." suffix=".txt"
                 pattern="common" resolveHosts="false"/>
        -->

<Valve className="org.apache.catalina.valves.FastCommonAccessLogValve"
                 directory="logs"  prefix="localhost_access_log." suffix=".txt"
                 pattern="combined" resolveHosts="false"/>

Create "init" script that will be used to start and stop the tomcat server

building jsvc

["error">root@<YourServerName> ~?]# cd /usr/local/
["error">root@<YourServerName> /usr/local?]# sudo tar xzf /usr/local/apache-tomcat-5.5.25/bin/jsvc.tar.gz

Note: You need to build this package. In order to do so, you will need GCC and Autoconf. This server has both already installed.

["error">root@<YourServerName> /usr/local?]# cd /usr/local/jsvc-src
["error">root@<YourServerName> /usr/local?]# sh support/buildconf.sh
["error">root@<YourServerName> /usr/local?]# chmod +x configure
["error">root@<YourServerName> /usr/local?]# ./configure
...
["error">root@<YourServerName> /usr/local?]# make
...

We see that the compile was successful.

Create the "init" script that will use JSVC

["error">labkey@labkey jsvc-src?]$ cat vi /etc/init.d/tomcat5.sh 
    #!/bin/sh
    ##############################################################################
    #
    #   Copyright 2004 The Apache Software Foundation.
    #
    #   Licensed under the Apache License, Version 2.0 (the "License");
    #   you may not use this file except in compliance with the License.
    #   You may obtain a copy of the License at
    #
    #       http://www.apache.org/licenses/LICENSE-2.0
    #
    #   Unless required by applicable law or agreed to in writing, software
    #   distributed under the License is distributed on an "AS IS" BASIS,
    #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    #   See the License for the specific language governing permissions and
    #   limitations under the License.
    ##############################################################################
    #
    # Small shell script to show how to start/stop Tomcat using jsvc
    # If you want to have Tomcat running on port 80 please modify the server.xml
    # file:
    #
    #    <!-- Define a non-SSL HTTP/1.1 Connector on port 80 -->
    #    <Connector className="org.apache.catalina.connector.http.HttpConnector"
    #               port="80" minProcessors="5" maxProcessors="75"
    #               enableLookups="true" redirectPort="8443"
    #               acceptCount="10" debug="0" connectionTimeout="60000"/>
    #
    # That is for Tomcat-5.0.x (Apache Tomcat/5.0)
    # 
	# chkconfig: 3 98 90
    # description: Start and Stop the Tomcat Server
	#
    #Added to support labkey
    PATH=$PATH:/usr/local/labkey/bin
    export PATH
    #
    # Adapt the following lines to your configuration
    JAVA_HOME=/usr/lib/jvm/java-1.6.0-sun
    CATALINA_HOME=/usr/local/apache-tomcat-5.5.25
    DAEMON_HOME=/usr/local/jsvc-src
    TOMCAT_USER=tomcat
    
    # for multi instances adapt those lines.
    TMP_DIR=/var/tmp
    PID_FILE=/var/run/jsvc.pid
    CATALINA_BASE=/usr/local/apache-tomcat-5.5.25
    
    CATALINA_OPTS="-Djava.library.path=/home/jfclere/jakarta-tomcat-connectors/jni/native/.libs"
    CLASSPATH=
    $JAVA_HOME/lib/tools.jar:
    $CATALINA_HOME/bin/commons-daemon.jar:
    $CATALINA_HOME/bin/bootstrap.jar
    
    case "$1" in
      start)
        #
        # Start Tomcat
        #
        $DAEMON_HOME/jsvc 
        -user $TOMCAT_USER 
        -home $JAVA_HOME 
        -Dcatalina.home=$CATALINA_HOME 
        -Dcatalina.base=$CATALINA_BASE 
        -Djava.io.tmpdir=$TMP_DIR 
        -wait 10 
        -pidfile $PID_FILE 
        -outfile $CATALINA_HOME/logs/catalina.out 
        -errfile '&1' 
        $CATALINA_OPTS 
        -cp $CLASSPATH 
        org.apache.catalina.startup.Bootstrap
        #
        # To get a verbose JVM
        #-verbose 
        # To get a debug of jsvc.
        #-debug 
        exit $?
        ;;
    
      stop)
        #
        # Stop Tomcat
        #
        $DAEMON_HOME/src/native/unix/jsvc 
        -stop 
        -pidfile $PID_FILE 
        org.apache.catalina.startup.Bootstrap
        exit $?
        ;;
    
      *)
        echo "Usage Tomcat5.sh start/stop"
        exit 1;;
    esac

Use the chkconfig tool to configure the start/stop script

1. Notice the line "# chkconfig: 3 98 90" in the script. This tells the chkconfig tool how to create the links needed to start/stop the Tomcat process at each runlevel. This says that the Tomcat server should:

• Only be started if using runlevel 3. It should not be started if using any other runlevel.
• Start with a priority of 98
• Stop with a priority of 90.

1. Now run the chkconfig tool:

["error">labkey@labkey jsvc-src?]$ chkconfig --add tomcat5

Postgres Installation and Configuration

["error">root@<YourServerName> Download?]# rpm -q -a | grep postgres
postgresql-8.2.5-1.fc7
postgresql-libs-8.2.5-1.fc7
postgresql-server-8.2.5-1.fc7
postgresql-python-8.2.5-1.fc7

Here, we do not use the postgres user as the user to connect to the database. Instead, we create a new database super-user role named "tomcat." This means we need:

["error">root@<YourServerName> Download?]# su - postgres
["error">postgres@<YourServerName> ~?]# /usr/bin/createuser -P -s -e tomcat
Enter password for new role: 
Enter it again: 
CREATE ROLE "tomcat" PASSWORD 'LabKey678' SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;
CREATE ROLE

Add the PL/pgsql language support to the postgres configuration

["error">postgres@<YourServerName> ~?]# createlang -d template1 PLpgsql

Change authorization so that the Tomcat user can login.

In order to get around the lack of ident, we make "password" the authentication method for all local connections (i.e., connections coming from the localhost). See http://www.postgresql.org/docs/8.2/static/auth-methods.html for more information on authentication methods.

["error">root@<YourServerName> ~?]# vi /var/lib/pgsql/data/pg_hba.cfg

Change:

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
local   all         all                               ident sameuser
# IPv4 local connections:
host    all         all         127.0.0.1/32          ident sameuser
# IPv6 local connections:
host    all         all         ::1/128               ident sameuser

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
local   all         all                               ident sameuser
# IPv4 local connections:
host    all         all         127.0.0.1/32          password
# IPv6 local connections:
host    all         all         ::1/128               ident sameuser

Increase the join collapse limit.

# join_collapse_limit = 8

to

join_collapse_limit = 10

If you do not do this step, you may see the following error when running complex queries: org.postgresql.util.PSQLException: ERROR: failed to build any 8-way joins

Now start the postgres database

["error">root@<YourServerName> ~?]# /etc/init.d/postgresql start

Install X!Tandem

Download the X!Tandem files using subversion:

["error">root@<YourServerName> ~?]# cd Download
["error">root@<YourServerName> Download?]# mkdir svn 
["error">root@<YourServerName> Download?]# cd svn 
["error">root@<YourServerName> svn?]# svn checkout --username cpas --password cpas
 https://hedgehog.fhcrc.org/tor/stedi/tags/tandem_2007-07-01/
Error validating server certificate for 'https://hedgehog.fhcrc.org:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: hedgehog.fhcrc.org
 - Valid: from Jun 22 14:01:09 2004 GMT until Sep  8 14:01:09 2012 GMT
 - Issuer: PHS, FHCRC, Seattle, Washington, US
 - Fingerprint: d8:a6:7a:5a:e8:81:c0:a0:51:87:34:6d:d1:0d:66:ca:22:09:9e:1f
(R)eject, accept (t)emporarily or accept (p)ermanently? p
....

Now that we have the files, we need to build and install the files.

The first thing to do is check which version of G++ the server is running. If you are running G++ v4.x, you need to make a modifications to the make file before you build. Note: A bug has been submitted to make it unnecessary to make this change, but you will still need to make these changes until the fix is submitted.

["error">root@<YourServerName> snv?]# g++ --version
g++ (GCC) 4.1.2 20070925 (Red Hat 4.1.2-27)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This shows that the server is running v4.x. Now we make the change:

["error">root@<YourServerName> snv?]# cd tandem_2007-07-01/src
["error">root@<YourServerName> src?]# vi Makefile 
"missing" href="/Documentation/Archive/9.1/wiki-page.view?name=change">change
CXXFLAGS = -O2 -DGCC -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPLUGGABLE_SCORING
#CXXFLAGS = -O2 -DGCC4 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPLUGGABLE_SCORING
"missing" href="/Documentation/Archive/9.1/wiki-page.view?name=to">to
#CXXFLAGS = -O2 -DGCC -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPLUGGABLE_SCORING
CXXFLAGS = -O2 -DGCC4 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPLUGGABLE_SCORING

Now run make:

["error">root@<YourServerName> src?]# make 
....

Copy the tandem binary to the server path 
["error">root@<YourServerName> src?]# cp ../bin/tandem.exe /usr/local/labkey/bin

TPP Installation.

First, download the software:

["error">root@<YourServerName> Download?]# wget 
http://downloads.sourceforge.net/sashimi/TPP_v3.4.2_SQUALL.zip?modtime=1207909790&big_mirror=0

Next, unpack the software:

["error">root@<YourServerName> Download?]# unzip TPP_v3.4.2_SQUALL.zip
["error">root@<YourServerName> Download?]# cd trans_proteomic_pipeline/src

It is necessary to change the Makefile.incl file to specify the install path and several options. These are specified at: https://www.labkey.org/wiki/home/Documentation/page.view?name=thirdPartyCode

We choose to the install the software at /usr/local/labkey/bin/tpp:

["error">root@<YourServerName> src?]# vi Makefile.inc

TPP_ROOT=/tpp/bin/tpp/

TPP_ROOT=/usr/local/labkey/bin/tpp/

Add to the bottom of the file:

XML_ONLY=1

TPP requires libboost development packages to be installed to successfully build.

["error">root@<YourServerName> src?]# yum list available boost*
Available Packages
boost-devel-static.x86_64                1.33.1-13.fc7          fedora          
boost-doc.x86_64                         1.33.1-13.fc7          fedora          
["error">root@<YourServerName> src?]# yum install boost-devel-static.x86_64
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package boost-devel-static.x86_64 0:1.33.1-13.fc7 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size 
=============================================================================
Installing:
 boost-devel-static      x86_64     1.33.1-13.fc7    fedora            1.7 M

Transaction Summary
=============================================================================
Install      1 Package(s)         
Update       0 Package(s)         
Remove       0 Package(s)         

Total download size: 1.7 M
Is this ok "missing" href="/Documentation/Archive/9.1/wiki-page.view?name=y%2FN">y/N: y
Downloading Packages:
(1/1): boost-devel-static 100% |=========================| 1.7 MB    00:01     
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: boost-devel-static           ######################### "missing" href="/Documentation/Archive/9.1/wiki-page.view?name=1%2F1">1/1 

Installed: boost-devel-static.x86_64 0:1.33.1-13.fc7
Complete!

There is a bug in the TPP makefile for 64bit machines. Thus you need to change the make file:

["error">root@<YourServerName> src?]# vi Makefile

#
# cygwin or linux?
#
ifeq (${OS},Windows_NT)
OSFLAGS= -D__CYGWIN__
GD_LIB= /lib/libgd.a
BOOST_REGEX_LIB=  /lib/libboost_regex-gcc-mt.a
else
OSFLAGS= -D__LINUX__
GD_LIB= -lgd
BOOST_REGEX_LIB= /usr/libboost_regex/libboost_regex.a -lpthread
endif

To:

#
# cygwin or linux?
#
ifeq (${OS},Windows_NT)
OSFLAGS= -D__CYGWIN__
GD_LIB= /lib/libgd.a
BOOST_REGEX_LIB=  /lib/libboost_regex-gcc-mt.a
else
OSFLAGS= -D__LINUX__
GD_LIB= -lgd
BOOST_REGEX_LIB= /usr/lib64/libboost_regex.a -lpthread
endif

Now run the make file:

["error">root@<YourServerName> src?]# make
.....

After building successfully, the next step is to perform the install

["error">root@<YourServerName> src?]# make install
# Create Directories
mkdir -p /usr/local/labkey/bin/tpp/
mkdir -p /usr/local/labkey/bin/tpp/bin/
mkdir -p /usr/local/labkey/bin/tpp/schema/
# Copy all source executables and configuration files to their location
cp -f ASAPRatioPeptideParser /usr/local/labkey/bin/tpp/bin/
cp -f ASAPRatioProteinRatioParser /usr/local/labkey/bin/tpp/bin/
cp -f ASAPRatioPvalueParser /usr/local/labkey/bin/tpp/bin/
cp -f Comet2XML /usr/local/labkey/bin/tpp/bin/
cp -f CompactParser /usr/local/labkey/bin/tpp/bin/
cp -f DatabaseParser /usr/local/labkey/bin/tpp/bin/
cp -f EnzymeDigestionParser /usr/local/labkey/bin/tpp/bin/
cp -f InteractParser /usr/local/labkey/bin/tpp/bin/
cp -f LibraPeptideParser /usr/local/labkey/bin/tpp/bin/
cp -f LibraProteinRatioParser /usr/local/labkey/bin/tpp/bin/
cp -f Mascot2XML /usr/local/labkey/bin/tpp/bin/
cp -f PeptideProphetParser /usr/local/labkey/bin/tpp/bin/
cp -f ProteinProphet /usr/local/labkey/bin/tpp/bin/
cp -f ../perl/ProteinProphet.pl /usr/local/labkey/bin/tpp/bin/
cp -f ../perl/TPPVersionInfo.pl /usr/local/labkey/bin/tpp/bin/
cp -f ../perl/SSRCalc3.pl /usr/local/labkey/bin/tpp/bin/
cp -f ../perl/SSRCalc3.par /usr/local/labkey/bin/tpp/bin/
cp -f RefreshParser /usr/local/labkey/bin/tpp/bin/
cp -f MzXML2Search /usr/local/labkey/bin/tpp/bin/
cp -f runperl /usr/local/labkey/bin/tpp/bin/
cp -f Sequest2XML /usr/local/labkey/bin/tpp/bin/
cp -f Out2XML /usr/local/labkey/bin/tpp/bin/
cp -f Sqt2XML /usr/local/labkey/bin/tpp/bin/
cp -f CombineOut /usr/local/labkey/bin/tpp/bin/
cp -f Tandem2XML /usr/local/labkey/bin/tpp/bin/
cp -f xinteract /usr/local/labkey/bin/tpp/bin/
cp -f XPressPeptideParser /usr/local/labkey/bin/tpp/bin/
cp -f XPressProteinRatioParser /usr/local/labkey/bin/tpp/bin/
cp -f Q3ProteinRatioParser /usr/local/labkey/bin/tpp/bin/
cp -f spectrast /usr/local/labkey/bin/tpp/bin/
cp -f plotspectrast /usr/local/labkey/bin/tpp/bin/
cp -f runsearch /usr/local/labkey/bin/tpp/bin/
cp -f dtafilter /usr/local/labkey/bin/tpp/bin/
cp -f readmzXML.exe /usr/local/labkey/bin/tpp/bin/ # consider removing .exe for linux builds
cp -f dta2mzxml /usr/local/labkey/bin/tpp/bin/
cp -f out2summary /usr/local/labkey/bin/tpp/bin/ # to be retired in favor of out2xml
cp -f ../schema/msms_analysis3.dtd /usr/local/labkey/bin/tpp/schema/
cp -f ../schema/pepXML_std.xsl /usr/local/labkey/bin/tpp/schema/
cp -f ../schema/pepXML_v18.xsd /usr/local/labkey/bin/tpp/schema/
cp -f ../schema/pepXML_v9.xsd /usr/local/labkey/bin/tpp/schema/
cp -f ../schema/protXML_v1.xsd /usr/local/labkey/bin/tpp/schema/
cp -f ../schema/protXML_v3.xsd /usr/local/labkey/bin/tpp/schema/
cp -f ../schema/protXML_v4.xsd /usr/local/labkey/bin/tpp/schema/
chmod g+x /usr/local/labkey/bin/tpp/bin/*
chmod a+r /usr/local/labkey/bin/tpp/schema/*

There is a bug in the TPP make script. The bug does not copy the batchcoverage executable over to the bindir.

["error">root@<YourServerName> src?]# cd ..
["error">root@<YourServerName> trans_proteomic_pipeline?]# ls
CGI  COVERAGE  extern  HELP_DIR  HTML  images  perl  README  schema  src  TESTING  XML_sample_files.tgz
["error">root@<YourServerName> trans_proteomic_pipeline?]# cd COVERAGE/
["error">root@<YourServerName> COVERAGE?]# ls
batchcoverage             batchcoverage.dsp  batchcoverage.vcproj  Coverage.h  main.o       Protein.h
batchcoverage2003.sln     batchcoverage.dsw  constants.h           Coverage.o  Makefile     sysdepend.h
batchcoverage2003.vcproj  batchcoverage.sln  Coverage.cxx          main.cxx    Protein.cxx
["error">root@<YourServerName> COVERAGE?]# cp batchcoverage /usr/local/labkey/bin/tpp/bin/

The last step is to ensure that the TPP bindir is located on PATH env variable for the user that runs the tomcat server. In this case the user=tomcat. THIS IS A VERY IMPORTANT STEP.

["error">root@<YourServerName> COVERAGE?]# vi ~tomcat/.bashrc

PATH=$PATH:$HOME/bin

PATH=$PATH:$HOME/bin:/usr/local/labkey/bin/tpp/bin

Install the Graphviz tool

Install the LabKey CPAS server

["error">root@<YourServerName>Download?]# wget 
https://www.labkey.org/download/2.3/LabKey2.3-7771-bin.tar.gz
["error">root@<YourServerName> Download?]# tar xzf LabKey2.3-7771-bin.tar.gz
["error">root@<YourServerName> Download?]# cd LabKey2.3-7771-bin
["error">root@<YourServerName> LabKey2.3-7771-bin?]# ls
common-lib  labkeywebapp  labkey.xml  modules  README.txt  server-lib  upgrade.sh

Copy the jars in the common-lib directory the <TOMCAT_HOME>/common/lib:

["error">root@<YourServerName> LabKey2.3-7771-bin?]# cd common-lib/
["error">root@<YourServerName> common-lib?]# ls
activation.jar  jtds.jar  mail.jar  postgresql.jar
["error">root@<YourServerName> common-lib?]# cp *.jar /usr/local/apache-tomcat-5.5.25/common/lib/

Copy the jars in the server-lib directory the <TOMCAT_HOME>/server/lib

["error">root@<YourServerName> common-lib?]# cd ../server-lib/
["error">root@<YourServerName> server-lib?]# ls
labkeyBootstrap.jar
["error">root@<YourServerName> server-lib?]# cp 
labkeyBootstrap.jar /usr/local/apache-tomcat-5.5.25/server/lib/

Create the <LABKEY_HOME> directory:

["error">root@<YourServerName> server-lib?]# mkdir /usr/local/labkey

Copy the labkeywebapp and the modules directory to the <LABKEY_HOME> directory:

["error">root@<YourServerName> server-lib?]# cd ..
["error">root@<YourServerName> LabKey2.3-7771-bin?]# ls

common-lib  labkeywebapp  labkey.xml  modules  README.txt  server-lib  upgrade.sh
["error">root@<YourServerName> LabKey2.3-7771-bin?]# mkdir /usr/local/labkey/labkeywebapp
["error">root@<YourServerName> LabKey2.3-7771-bin?]# mkdir /usr/local/labkey/modules
["error">root@<YourServerName> LabKey2.3-7771-bin?]# cp -R labkeywebapp/* /usr/local/labkey/labkeywebapp/
["error">root@<YourServerName> LabKey2.3-7771-bin?]# cp 
-R modules/* /usr/local/labkey/modules/

Copy the labkey.xml file to the <TOMCAT_HOME> directory and make the necessary changes to the file:

["error">root@<YourServerName> LabKey2.3-7771-bin?]# cp labkey.xml 
/usr/local/apache-tomcat-5.5.25/conf/Catalina/localhost/
["error">root@<YourServerName> LabKey2.3-7771-bin?]# vi 
/usr/local/apache-tomcat-5.5.25/conf/Catalina/localhost/labkey.xml

The file was changed to look like this:

<Context path="/labkey" docBase="/usr/local/labkey/labkeywebapp" debug="0" 
        reloadable="true" crossContext="true">
    
    <Environment name="dbschema/--default--" value="jdbc/labkeyDataSource" 
        type="java.lang.String"/>

    <Resource name="jdbc/labkeyDataSource" auth="Container"
        type="javax.sql.DataSource"
        username="tomcat"
        password="LabKey678"
        driverClassName="org.postgresql.Driver"
        url="jdbc:postgresql://localhost/labkey"
        maxActive="20"
        maxIdle="10" accessToUnderlyingConnectionAllowed="true"/>

    <Resource name="jms/ConnectionFactory" auth="Container"
        type="org.apache.activemq.ActiveMQConnectionFactory"
        factory="org.apache.activemq.jndi.JNDIReferenceFactory"
        description="JMS Connection Factory"
        brokerURL="vm://localhost?broker.persistent=false&amp;broker.useJmx=false"
        brokerName="LocalActiveMQBroker"/>

    <Resource name="mail/Session" auth="Container"
        type="javax.mail.Session"
        mail.smtp.host="localhost"
        mail.smtp.user="tomcat"
        mail.smtp.port="25"/>

    <Loader loaderClass="org.labkey.bootstrap.LabkeyServerBootstrapClassLoader" 
        useSystemClassLoaderAsParent="false" />

<!--    <Parameter name="org.mule.webapp.classpath" value="C:mule-config"/>  -->

</Context>

The final step is to make the tomcat user the owner of all files in <TOMCAT_HOME> and <LABKEY_HOME>:

["error">root@<YourServerName> LabKey2.3-7771-bin?]# chown -R tomcat.tomcat /usr/local/labkey
["error">root@<YourServerName> LabKey2.3-7771-bin?]# chown -R tomcat.tomcat /usr/local/apache-tomcat-5.5.25

Now start the CPAS server to test it:

["error">root@<YourServerName> ~?]# /etc/init.d/tomcat5 start

You can access the CPAS server at

http://<YourServerName>:8080/labkey

Items installed via these instructions:

• Sun Java
• Xcode
• Apache Tomcat
• Postgres
• LabKey Server

• Proteomics Tools (such as X!Tandem, TPP, etc)
• R (see Configure R on Linux)
• Xvfb (see Configure the Virtual Frame Buffer on Linux)
• ftpserver (see Configure FTP on Linux)

• Mac OSX 10.5.3 (Leopard)

• These instructions assume that you will run the LabKey Flow Cytometry server as a user named "labkey".
• All downloaded files will be placed in a sub-directory of my home directory /Users/bconn/Download

Install Sun Java

Note: <YourServerName> represents the name of the server where you plan to install CPAS

<YourServerName>:~ bconn$ java -version
java version "1.5.0_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_13-b05-237)
Java HotSpot(TM) Client VM (build 1.5.0_13-119, mixed mode, sharing)

Install XCode

• Download XCode 3.0 from http://developer.apple.com/tools/download/
• Simply follow the instructions to install.
• Select all the defaults during the install

Install Apache

We will be

• Using Tomcat v5.5.26
• Installing Tomcat in the directory /usr/local/apache-tomcat-5.5.26
• Tomcat will be configured to use port 8080 (see the Configure the Tomcat Default Port section on Configure the Web Application to change the Default Port )
• Tomcat will not be configured to use SSL (see the Configure LabKey Server to Run Under SSL (Optional, Recommended) section on Configure the Web Application to configure your server to use SSL )

Download and unpack Tomcat v5.5.26

<YourServerName>:~ bconn$ cd ~/Download
<YourServerName>:Download bconn$ curl 
  http://apache.oc1.mirrors.redwire.net/tomcat/tomcat-5/v5.5.26/bin/apache-tomcat-5.5.26.tar.gz -o 
  apache-tomcat-5.5.26.tar.gz 
<YourServerName>:Download bconn$ sudo -s 
bash3.2# cd /usr/local
bash3.2# tar xzf ~/Download/apache-tomcat-5.5.26.tar.gz 
bash3.2# cd apache-tomcat-5.5.26/
bash3.2# ls
bin  common  conf  LICENSE  logs  NOTICE  RELEASE-NOTES  RUNNING.txt  server  shared  temp  webapps  work

Create the labkey user

• This user will be the user that runs the tomcat server.
• This user will have the following properties
• UID=900
• GID=900
• Home Directory= /Users/labkey
• Password: No password has been set. This means that you will not be able to login as the user labkey. This is equivalent to setting "x" in the /etc/passwd file on linux. If you want to run as the user labkey you will need to run sudo su - labkey from the command line.

bash-3.2# dseditgroup -o create -n . -r "labkey" -i 900 labkey
bash-3.2# mkdir /Users/labkey

Create the labkey user

bash-3.2# dscl . -create /Users/labkey
bash-3.2# dscl . -create /Users/labkey UserShell /bin/bash
bash-3.2# dscl . -create /Users/labkey RealName "LabKey User"
bash-3.2# dscl . -create /Users/labkey UniqueID 900
bash-3.2# dscl . -create /Users/labkey PrimaryGroupID 900
bash-3.2# dscl . -create /Users/labkey NFSHomeDirectory /Users/labkey

Now lets view the user setup

bash-3.2# dscl . -read /Users/labkey
   AppleMetaNodeLocation: /Local/Default
   GeneratedUID: A695AE43-9F54-4F76-BCE0-A90E239A9A58
   NFSHomeDirectory: /Users/labkey
   PrimaryGroupID: 900
   RealName:
    LabKey User
   RecordName: labkey
   RecordType: dsRecTypeStandard:Users
   UniqueID: 900
   UserShell: /bin/bash

Set up the users .bash_profile file

bash-3.2# vi ~labkey/.bash_profile

#Created to be used for starting up the LabKey Server
   JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
   CATALINA_HOME=/usr/local/apache-tomcat-5.5.26
   CATALINA_OPTS=-Djava.awt.headless=true
   export CATALINA_OPTS
   export JAVA_HOME
   export CATALINA_HOME
   # Append Path
   PATH=$PATH:/usr/local/pgsql/bin:/usr/local/bin:/usr/local/labkey/bin


bash-3.2# chown -R labkey.labkey /Users/labkey

Lets set the proper permissions on the Tomcat directories

bash-3.2# chown -R labkey.labkey /usr/local/apache-tomcat-5.5.26

Configure the Tomcat server

Enable Access Logging on the server(This allows you to see which URLs are accessed):

bash-3.2# vi /usr/local/apache-tomcat-5.5.26/conf/server.xml

Change:

<!--
        <Valve className="org.apache.catalina.valves.FastCommonAccessLogValve"
                 directory="logs"  prefix="localhost_access_log." suffix=".txt"
                 pattern="common" resolveHosts="false"/>
        -->

<Valve className="org.apache.catalina.valves.FastCommonAccessLogValve"
                 directory="logs"  prefix="localhost_access_log." suffix=".txt"
                 pattern="combined" resolveHosts="false"/>

Create "init" script that will be used to start and stop the tomcat server

Build JSVC Daemon

Note: You need to build this package. In order to do so, you will need GCC, Autoconf. These are installed with with the XCode package Note2: In addition, you need to make sure the JAVA_HOME environment variable is set for the user building this software

bash-3.2# cd /usr/local/
bash-3.2# tar xzf /usr/local/apache-tomcat-5.5.26/bin/jsvc.tar.gz

Before we get started, we need to modify two files in the distribution to have them compile properly on Leopard

bash-3.2# export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
bash-3.2# cd /usr/local/jsvc-src/
bash-3.2# vi native/jsvc.h

/* Definitions for booleans */
typedef enum {
false,
true
} bool;

#include <stdbool.h>

bash-3.2# vi support/apsupport.m4

CFLAGS="$CFLAGS -DOS_DARWIN -DDSO_DYLD"

CFLAGS="$CFLAGS -DOS_DARWIN -DDSO_DLFCN"

Now we can perform the build

bash-3.2# sh support/buildconf.sh
bash-3.2# sh ./configure
...
bash-3.2# make
...

You will see some warning messages produced, but it will be successful compile and the JSVC daemon will created at /usr/local/jsvc-src/jsvc

Install JSVC Daemon

bash-3.2# mkdir /usr/local/jsvc
bash-3.2# cp /usr/local/jsvc-src/jsvc /usr/local/jsvc

Configure the server to Start Tomcat using the JSVC daemon at boot-time

1. Create "start-up" script
2. Create plist file (file that launchd reads to start the Tomcat process )

bash-3.2# vi /usr/local/jsvc/Tomcat5.sh 
    #!/bin/sh
    ##############################################################################
    #
    #   Copyright 2004 The Apache Software Foundation.
    #
    #   Licensed under the Apache License, Version 2.0 (the "License");
    #   you may not use this file except in compliance with the License.
    #   You may obtain a copy of the License at
    #
    #       http://www.apache.org/licenses/LICENSE-2.0
    #
    #   Unless required by applicable law or agreed to in writing, software
    #   distributed under the License is distributed on an "AS IS" BASIS,
    #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    #   See the License for the specific language governing permissions and
    #   limitations under the License.
    ##############################################################################
    #
    # Small shell script to show how to start/stop Tomcat using jsvc
    # If you want to have Tomcat running on port 80 please modify the server.xml
    # file:
    #
    #    <!-- Define a non-SSL HTTP/1.1 Connector on port 80 -->
    #    <Connector className="org.apache.catalina.connector.http.HttpConnector"
    #               port="80" minProcessors="5" maxProcessors="75"
    #               enableLookups="true" redirectPort="8443"
    #               acceptCount="10" debug="0" connectionTimeout="60000"/>
    #
    # That is for Tomcat-5.0.x (Apache Tomcat/5.0)
    # 
	# chkconfig: 3 98 90
    # description: Start and Stop the Tomcat Server
	#
    #Added to support labkey
    PATH=$PATH:/usr/local/labkey/bin
    export PATH
    #
    # Adapt the following lines to your configuration
    JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
    CATALINA_HOME=/usr/local/apache-tomcat-5.5.26
    DAEMON_HOME=/usr/local/jsvc
    TOMCAT_USER=labkey
    
    # for multi instances adapt those lines.
    TMP_DIR=/var/tmp
    PID_FILE=/var/run/jsvc.pid
    CATALINA_BASE=/usr/local/apache-tomcat-5.5.26
    
    CATALINA_OPTS=""
    CLASSPATH=
    $JAVA_HOME/lib/tools.jar:
    $CATALINA_HOME/bin/commons-daemon.jar:
    $CATALINA_HOME/bin/bootstrap.jar
    
    case "$1" in
      start)
        #
        # Start Tomcat
        #
        $DAEMON_HOME/jsvc 
        -user $TOMCAT_USER 
        -home $JAVA_HOME 
        -Dcatalina.home=$CATALINA_HOME 
        -Dcatalina.base=$CATALINA_BASE 
        -Djava.io.tmpdir=$TMP_DIR 
        -wait 10 
        -pidfile $PID_FILE 
        -outfile $CATALINA_HOME/logs/catalina.out 
        -errfile '&1' 
        $CATALINA_OPTS 
        -cp $CLASSPATH 
        org.apache.catalina.startup.Bootstrap
        #
        # To get a verbose JVM
        #-verbose 
        # To get a debug of jsvc.
        #-debug 
        exit $?
        ;;
    
      stop)
        #
        # Stop Tomcat
        #
        $DAEMON_HOME/jsvc 
        -stop 
        -pidfile $PID_FILE 
        org.apache.catalina.startup.Bootstrap
        exit $?
        ;;
    
      *)
        echo "Usage Tomcat5.sh start/stop"
        exit 1;;
    esac

_Create the plist file_

bash-3.2$ vi /Library/LaunchDaemons/org.apache.commons.jsvc.plist
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
        <key>Disabled</key>
        <false/>
        <key>Label</key>
        <string>org.apache.commons.jsvc</string>
        <key>ProgramArguments</key>
        <array>
                <string>/usr/local/jsvc/Tomcat5.sh</string>
                <string>start</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
        <key>WorkingDirectory</key>
        <string>/usr/local/apache-tomcat-5.5.26</string>
    </dict>
    </plist>

Test Tomcat Installation

bash-3.2# export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
bash-3.2# export CATALINA_HOME=/usr/local/apache-tomcat-5.5.26
bash-3.2# export CATALINA_OPTS=-Djava.awt.headless=true
bash-3.2# /usr/local/apache-tomcat-5.5.26/bin/startup.sh

Second, lets test the "start-up" script that uses JSVC

bash-3.2# /usr/local/apache-tomcat-5.5.26/bin/shutdown.sh
bash-3.2# /usr/local/jsvc/Tomcat5.sh start

Lastly, lets test to see if the LauncherDaemon is configured properly

bash-3.2# /usr/local/jsvc/Tomcat5.sh stop
bash-3.2# launchctl load /Library/LaunchDaemons/org.apache.commons.jsvc.plist

If all the tests have passed, then the Tomcat installation was a success. Shutdown the Tomcat server at this time

bash-3.2# /usr/local/jsvc/Tomcat5.sh stop
bash-3.2# exit

Postgres Installation and Configuration

We will be

• Using Postgresql v8.2.6
• Installing Postgresql in the directory /usr/local/pgsql
• The postgres server will be run as the user postgres which will be created.
• New super-user role named labkey will be created and used by the Tomcat server to talk to postgres

Download and expand the source

<YourServerName>:Download bconn$ curl 
   http://ftp7.us.postgresql.org/pub/postgresql//source/v8.2.9/postgresql-8.2.9.tar.gz 
   -o postgresql-8.2.9.tar.gz
<YourServerName>:Download bconn$ sudo su - 
bash-3.2# cd /usr/local
bash-3.2#tar -xzf ~bconn/Download/postgresql-8.2.9.tar.gz

Build Postgres

bash-3.2# ./configure
bash-3.2# make 
...
bash-3.2# make check 
...
bash-3.2# make install
...

Create the labkey user

• This user will be the user that runs the postgres server.
• This will create a user named postgres
• This user will have the following properties
• UID=901
• GID=901
• Home Directory=/usr/local/pgsql
• Password: No password has been set. This means that you will not be able to login as the user postgres. This is equivalent to setting "x" in the /etc/passwd file on linux. If you want to run as the user postgres you will need to run sudo su - postgres from the command line.

dseditgroup -o create -n . -r "postgres" -i 901 postgres

Create the postgres user

bash-3.2# dscl . -create /Users/postgres
bash-3.2# dscl . -create /Users/postgres UserShell /bin/bash
bash-3.2# dscl . -create /Users/postgres RealName "Postgres User"
bash-3.2# dscl . -create /Users/postgres UniqueID 901
bash-3.2# dscl . -create /Users/postgres PrimaryGroupID 901
bash-3.2# dscl . -create /Users/postgres NFSHomeDirectory /usr/local/pgsql

Now lets view the user setup

bash-3.2# dscl . -read /Users/postgres
   AppleMetaNodeLocation: /Local/Default
   GeneratedUID: A695AE43-9F54-4F76-BCE0-A90E239A9A58
   NFSHomeDirectory: /usr/local/pgsql
   PrimaryGroupID: 901
   RealName:
    Postgres User
   RecordName: postgres
   RecordType: dsRecTypeStandard:Users
   UniqueID: 901
   UserShell: /bin/bash

Initialize the Postgres database

bash-3.2# mkdir /usr/local/pgsql/data
bash-3.2# mkdir /usr/local/pgsql/data/logs

bash-3.2# chown -R postgres.postgres /usr/local/pgsql/data

bash-3.2# su - postgres
<YourServerName>:pgsql postgres$ /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data

<YourServerName>:pgsql postgres$ /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l 
  /usr/local/pgsql/data/postgres.log start

Create a new database super-user role named "labkey":

<YourServerName>:pgsql postgres$ /usr/local/pgsql/bin/createuser -P -s -e labkey
Enter password for new role: 
Enter it again: 
   CREATE ROLE "labkey" PASSWORD 'LabKey678' SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;
   CREATE ROLE

Add the PL/pgsql language support to the postgres configuration

<YourServerName>:pgsql postgres$ createlang -d template1 PLpgsql

Change authorization so that the labkey user can login.

Stop the server

<YourServerName>:pgsql postgres$ /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l 
    /usr/local/pgsql/logs/logfile stop
<YourServerName>:pgsql postgres$ exit

Edit the pg_hba.cfg file

bash-3.2# vi /usr/local/pgsql/data/pg_hba.cfg

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
local   all         all                               ident sameuser
# IPv4 local connections:
host    all         all         127.0.0.1/32          ident sameuser
# IPv6 local connections:
host    all         all         ::1/128               ident sameuser

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
local   all         all                               ident sameuser
# IPv4 local connections:
host    all         all         127.0.0.1/32          password
# IPv6 local connections:
host    all         all         ::1/128               ident sameuser

Increase the join collapse limit.

bash-3.2# vi /var/lib/pgsql/data/postgresql.conf Change:

# join_collapse_limit = 8

join_collapse_limit = 10

If you do not do this step, you may see the following error when running complex queries: org.postgresql.util.PSQLException: ERROR: failed to build any 8-way joins

Start the postgres database

<YourServerName>:pgsql postgres$ /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l 
  /usr/local/pgsql/data/logs/logfile start

Create the "init" script that will start Postgres at boot-time

Create the required directories and copy of the Startup files from the source directory

bash-3.2# mkdir /Library/StartupItems/PostgreSQL/
bash-3.2# cp /usr/local/postgresql-8.2.9/contrib/start-scripts/PostgreSQL.darwin 
   /Library/StartupItems/PostgreSQL/PostgreSQL
bash-3.2# cp /usr/local/postgresql-8.2.9/contrib/start-scripts/StartupParameters.plist.darwin 
   /Library/StartupItems/PostgreSQL/StartupParameters.plist

Change the configuration of the start-up script to disable log rotation

bash-3.2# vi /Library/StartupItems/PostgreSQL/PostgreSQL

# do you want to rotate the log files, 1=true 0=false
ROTATELOGS=1

# do you want to rotate the log files, 1=true 0=false
ROTATELOGS=0

Install Graphviz

Download and expand Graphviz

<YourServerName>:Download bconn$ curl 
   http://www.graphviz.org/pub/graphviz/ARCHIVE/graphviz-2.16.1.tar.gz
   -o graphviz-2.16.1.tar.gz 
<YourServerName>:Download bconn$ sudo su - 
bash-3.2# cd /usr/local
bash-3.2#tar -xzf ~bconn/Download/graphviz-2.16.1.tar.gz

Build and install Graphviz binaries into /usr/local/bin

bash-3.2# tar xzf ~/Downloads/graphviz-2.16.1.tar.gz 
bash-3.2# /usr/local/graphviz-2.16.1
bash-3.2# ./configure
...
bash-3.2# make 
...
bash-3.2# make install
...

Install the LabKey CPAS server

Download and expand LabKey server

bash-3.2# cd /usr/local
bash-3.2# tar xzf ~bconn/Download/LabKey8.2-XXXX-bin.tar.gz
bash-3.2# cd LabKey8.2-XXXX-bin
bash-3.2# ls
common-lib  labkeywebapp  labkey.xml  modules  README.txt  server-lib  upgrade.sh

Copy the jars in the common-lib directory the <CATALINA_HOME>/common/lib:

bash-3.2# cd common-lib/
bash-3.2# ls
activation.jar  jtds.jar  mail.jar  postgresql.jar
bash-3.2# cp *.jar /usr/local/apache-tomcat-5.5.26/common/lib/

Copy the jars in the server-lib directory the <TOMCAT_HOME>/server/lib

bash-3.2# cd ../server-lib/
bash-3.2# ls
labkeyBootstrap.jar
bash-3.2# cp *.jar /usr/local/apache-tomcat-5.5.26/server/lib/

Create the <LABKEY_HOME> directory:

bash-3.2# mkdir /usr/local/labkey

Copy the labkeywebapp and the modules directory to the <LABKEY_HOME> directory:

bash-3.2# cd ..
bash-3.2# ls 
common-lib  labkeywebapp  labkey.xml  modules  README.txt  server-lib  upgrade.sh
bash-3.2# mkdir /usr/local/labkey/labkeywebapp
bash-3.2# mkdir /usr/local/labkey/modules
bash-3.2# cp -R labkeywebapp/* /usr/local/labkey/labkeywebapp/
bash-3.2# cp -R modules/* /usr/local/labkey/modules/

Copy the labkey.xml file to the <CATALINA_HOME> directory and make the necessary changes to the file:

bash-3.2# cp labkey.xml /usr/local/apache-tomcat-5.5.26/conf/Catalina/localhost/
bash-3.2# vi /usr/local/apache-tomcat-5.5.26/conf/Catalina/localhost/labkey.xml

The file was changed to look like this:

<Context path="/labkey" docBase="/usr/local/labkey/labkeywebapp" debug="0" 
        reloadable="true" crossContext="true">
    
    <Environment name="dbschema/--default--" value="jdbc/labkeyDataSource" 
        type="java.lang.String"/>

    <Resource name="jdbc/labkeyDataSource" auth="Container"
        type="javax.sql.DataSource"
        username="labkey"
        password="LabKey678"
        driverClassName="org.postgresql.Driver"
        url="jdbc:postgresql://localhost/labkey"
        maxActive="20"
        maxIdle="10" accessToUnderlyingConnectionAllowed="true"/>

    <Resource name="jms/ConnectionFactory" auth="Container"
        type="org.apache.activemq.ActiveMQConnectionFactory"
        factory="org.apache.activemq.jndi.JNDIReferenceFactory"
        description="JMS Connection Factory"
        brokerURL="vm://localhost?broker.persistent=false&amp;broker.useJmx=false"
        brokerName="LocalActiveMQBroker"/>

    <Resource name="mail/Session" auth="Container"
        type="javax.mail.Session"
        mail.smtp.host="localhost"
        mail.smtp.user="labkey"
        mail.smtp.port="25"/>

    <Loader loaderClass="org.labkey.bootstrap.LabkeyServerBootstrapClassLoader" 
        useSystemClassLoaderAsParent="false" />

<!--    <Parameter name="org.mule.webapp.classpath" value="C:mule-config"/>  -->

</Context>

The final step is to make the labkey user the owner of all files in <CATALINA_HOME> and <LABKEY_HOME>:

["error">root@<YourServerName> LabKey2.3-7771-bin?]# chown -R labkey.labkey /usr/local/labkey
["error">root@<YourServerName> LabKey2.3-7771-bin?]# chown -R labkey.labkey /usr/local/apache-tomcat-5.5.26

Now start the CPAS server to test it:

bash-3.2# /usr/local/jsvc/Tomcat5.sh start

You can access the CPAS server at

http://<YourServerName>:8080/labkey

This page provides an example of how FTP can be configured on a Linux server. Specifically, this page lists instructions for installing the Pipeline ftpserver (v2.3) on www.labkey.org. For general instructions for FTP setup, see Set Up the FTP Server.

Download and Install the Server

bconn@labkey00:~> wget https://www.labkey.org/download/2.3/pipelineftp-2.3.tar.gz 
--no-check-certificate

Next, unpack the file and move it to the proper location (/usr/local/labkey/ftpserver):

bconn@labkey00:~> tar xzf pipelineftp-2.3.tar.gz
bconn@labkey00:~> sudo cp -R ftpserver/ /usr/local/labkey/
bconn@labkey00:~> sudo chmod -R 755 /usr/local/labkey/ftpserver/
bconn@labkey00:~> ls -la /usr/local/labkey/ftpserver/
total 18
drwxr-xr-x  7 root root   216 2008-01-19 15:53 .
drwxr-xr-x 14 root root   384 2008-01-19 15:53 ..
drwxr-xr-x  2 root root   280 2008-01-19 15:53 bin
drwxr-xr-x  4 root root    96 2008-01-19 15:53 common
-rwxr-xr-x  1 root root 11558 2008-01-19 15:53 LICENSE
drwxr-xr-x  2 root root   184 2008-01-19 15:53 notes
-rwxr-xr-x  1 root root   336 2008-01-19 15:53 README
drwxr-xr-x  5 root root   184 2008-01-19 15:53 res
drwxr-xr-x 12 root root  1728 2008-01-19 15:53 site

NOTE: This is a binary distribution, so there is no need to run configure, make, etc.

Configure the Server

bconn@labkey00:~> cd /usr/local/labkey/ftpserver/res/conf
bconn@labkey00:/usr/local/labkey/ftpserver/res/conf> ls ftpd.xml

NOTE: The ftpserver configuration (ftpd.xml) is shipped with all Listener and SSL configuration information commented out. This means that the server will run with default settings

You will need to make five configuration changes:

bconn@labkey00:/usr/local/labkey/ftpserver/res/conf> sudo vi ftpd.xml

1) Uncomment the Listeners and Data-connection Configurations
Remove the "open" or "close" comments (ie <!-- or -->) from lines 26,42,45 and 73

2) Configure the Default Listener
FTP uses 2 types of connections:

• The Listener, which normally runs on port 21. All the ftp commands are sent over this connection (including the username and passwords for login)
• Data-Connection: This normally runs on port 20. All data is transfered over this connection (i.e., if you are transferring files, the files are transferred using this connection)

change

<listeners>
         <default>
             <class>org.apache.ftpserver.listener.mina.MinaListener</class>
             <address>localhost</address>
             <port>21</port>

<listeners>
         <default>
             <class>org.apache.ftpserver.listener.mina.MinaListener</class>
             <!-- <address>localhost</address> -->
             <port>21</port>

This configuration tells the FTPServer to bind the listener to all available IP addresses. If you need to bind to just a single IP address, enter the IP address in the <address> node above.

3) Configure the Data-Connection Settings for this Listener
Comment out the <local-address>, <address> and <external-address> nodes.

change

<data-connection>
          <class>org.apache.ftpserver.DefaultDataConnectionConfig</class>
          <idle-time>10</idle-time>
          <active>
              <enable>true</enable>
              <local-address>localhost</local-address>
              <local-port>20</local-port>
              <ip-check>false</ip-check>
          </active>
          <passive>
              <address>localhost</address>
              <ports>0</ports>
              <external-address>192.1.2.3</external-address>
          </passive>

<data-connection>
         <class>org.apache.ftpserver.DefaultDataConnectionConfig</class>
         <idle-time>10</idle-time>
         <active>
             <enable>true</enable>
             <!-- <local-address>localhost</local-address> -->
             <local-port>20</local-port>
             <ip-check>false</ip-check>
         </active>
         <passive>
             <!-- <address>localhost</address> -->
             <ports>0</ports>
             <!-- <external-address>192.1.2.3</external-address> -->
         </passive>

Please note that there are 2 types of Modes in which a FTP Server can be run: Active or Passive (see http://www.slacksite.com/other/ftp.html for more information on the difference). The changes made below tell the FTP Server to do the following:

• For Active Data-Connections, bind to port 20 on all IP addresses.
• For Passive Data-Connections, bind use any port that is larger than 1024 on the IP address used by the Listener connection.

<data-connection>
        <class>org.apache.ftpserver.DefaultDataConnectionConfig</class>
        <idle-time>10</idle-time>
        <active>
            <enable>true</enable>
            <!-- <local-address>localhost</local-address> -->
            <local-port>20</local-port>
            <ip-check>false</ip-check>
        </active>
        <passive>
            <!-- <address>localhost</address> -->
            <ports>0</ports>
            <!-- <external-address>192.1.2.3</external-address> -->
        </passive>
        <!-- <ssl>
            <class>org.apache.ftpserver.ssl.DefaultSsl</class>
            <keystore-file>/usr/local/tomcat/tomcat.keystore</keystore-file>
            <keystore-password>changeit</keystore-password>
            <keystore-type>JKS</keystore-type>
            <keystore-algorithm>SunX509</keystore-algorithm>
            <ssl-protocol>TLS</ssl-protocol>
            <client-authentication>false</client-authentication>
            <key-password></key-password>
        </ssl> -->
    </data-connection>

See Set Up the FTP Server for information on configuring SSL for the ftpserver

5) Lastly, Change the LabKey User Manager Configuration Block.
The <labkey-url> node contains the URL that is used by the FTP Server to communicate with the CPAS server.

Here is an example of how to set this configuration:

• If your CPAS server is located at http://www.institutionname.edu:8080/labkey then the URL for this setting should be <labkey-url>http://localhost:8080/labkey/ftp </labkey-url>
• If your CPAS server is located at https://www.institutionname.edu/labkey then the URL for this setting should be <labkey-url>https://localhost/labkey/ftp </labkey-url>

change

<!-- LabKey user manager configuration block -->
<user-manager>
    <class>org.labkey.pipelineftp.UserManager</class>
    <labkey-url>http://localhost:8080/labkey/ftp </labkey-url>
</user-manager>

<!-- LabKey user manager configuration block -->
<user-manager>
    <class>org.labkey.pipelineftp.UserManager</class>
    <labkey-url>https://localhost/labkey/ftp </labkey-url>
</user-manager>

In most cases, this setting should use localhost as the hostname in the URL because the FTP server currently must be run on the same host as the CPAS server.

Add the JAVA_HOME Variable and Other Changes into the Start-Up Script

bconn@labkey00:/usr/local/labkey/ftpserver/res/conf> cd /usr/local/labkey/ftpserver/bin
bconn@labkey00:/usr/local/labkey/ftpserver/bin> sudo vi ftpd.sh

# Added by bconn on 1/21/2008. Needed as GCJ is currently installed 
	# and set as default java implementation
	export JAVA_HOME=/usr/local/java

Next, edit the ftpd.sh script in order for the process to be placed in the background and release stdout and stderr. This is needed to have the server started properly during boot up:

change

#
	# Execute command
	#
	CURR_DIR=`pwd`
	cd $FTPD_HOME
	MAIN_CLASS=org.apache.ftpserver.commandline.CommandLine
	"$JAVACMD" -classpath "$FTPD_CLASSPATH" $MAIN_CLASS $@
	RESULT=$?
	cd $CURR_DIR
	exit $RESULT

#
	# Execute command
	#

	#Add Date and Time of Startup into the ftpserver.out log 
	# This log file contains stdout/stderr from the ftpserver
	# process
	FTPSERVER_OUT=/usr/local/labkey/ftpserver/res/log/ftpserver.out
	echo "" >> $FTPSERVER_OUT
	echo "" >> $FTPSERVER_OUT
	echo "FTP Server Start up Time: `date`" >> $FTPSERVER_OUT

	CURR_DIR=`pwd`
	cd $FTPD_HOME
	MAIN_CLASS=org.apache.ftpserver.commandline.CommandLine
	"$JAVACMD" -classpath "$FTPD_CLASSPATH" $MAIN_CLASS $@ >> "$FTPSERVER_OUT" 2>&1&
	RESULT=$?
	cd $CURR_DIR
	exit $RESULT

Start up the Server

# /usr/local/labkey/ftpserver/bin/ftpd.sh -xml /usr/local/labkey/ftpserver/res/conf/ftpd.xml

Testing shows that this works smashingly. We are almost done.

The last change is to set things up so that the ftpserver is restarted at boot time. The way to do this is to add the following line to /etc/init.d/rc.local

/usr/local/labkey/ftpserver/bin/ftpd.sh -xml /usr/local/labkey/ftpserver/res/conf/ftpd.xml

Stop the Server

To kill the process, issue the following command to determine the PID of the FTP Server process:

bconn@labkey00:/usr/local/labkey/ftpserver> ps aux | grep ftp
root      7865  2.2  0.4 262052 20040 pts/0    Sl   16:34   0:00 
  /usr/local/java/bin/java -classpath 
  :/usr/local/labkey/ftpserver/bin/../common/classes
  :/usr/local/labkey/ftpserver/bin/../common/lib/backport-util-concurrent-2.2.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib/commons-codec-1.3.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib/commons-httpclient-3.0.1.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib/commons-logging-1.1.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib
  /ftplet-api-1.0-incubator-SNAPSHOT.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib
  /ftpserver-admin-gui-1.0-incubator-20070611.111048-1.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib
  /ftpserver-core-1.0-incubator-SNAPSHOT.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib/log4j-1.2.13.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib/mina-core-1.0.2.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib/mina-filter-ssl-1.0.2.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib/pipelineftp2.3.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib/slf4j-api-1.3.0.jar
  :/usr/local/labkey/ftpserver/bin/../common/lib/slf4j-log4j12-1.3.0.jar 
  org.apache.ftpserver.commandline.CommandLine 
  -xml /usr/local/labkey/ftpserver/res/conf/ftpd.xml

In the example above, the PID of the FTP Server is 7865. Thus you would issue the following command to kill the process:

bconn@labkey00:/usr/local/labkey/ftpserver> sudo kill 7865

Steps

If <YourServerName> represents the name of your server, these are the steps for building:

["error">root@<YourServerName> Download?]# wget http://cran.r-project.org/src/base/R-2/R-2.6.2.tar.gz 
["error">root@<YourServerName> Download?]# tar xzf R-2.6.2.tar.gz 
["error">root@<YourServerName> Download?]# cd R-2.6.2
["error">root@<YourServerName> R-2.6.2?]# ./configure 
...
["error">root@<YourServerName> R-2.6.2?]# make 
...
["error">root@<YourServerName> R-2.6.2?]# make install 
...

Additional Notes

• These instructions install R under /usr/local (with the executable installed at /usr/local/bin/R
• Support for the X11 device (including png() and jpeg()) is compiled in R by default.
• In order to use the X11, png and jpeg devices, an Xdisplay must be available. Thus you may still need to Configure the Virtual Frame Buffer on Linux.

Example Configuration

• Linux Distro: Fedora 7
• Kernel: 2.6.20-2936.fc7xen
• Processor Type: x86_64

Install R

Install Xvfb

["error">root@<YourServerName> R-2.6.1?]# yum update xorg-x11-server-Xorg 

["error">root@<YourServerName> R-2.6.1?]# yum install xorg-x11-server-Xvfb.x86_64

Start and Test Xvfb

["error">root@<YourServerName> R-2.6.1?]# /usr/bin/Xvfb :2 -nolisten tcp -shmem

This starts a Display on servernumber = 2 and screen number = 0.

To test whether the X11, PNG and JPEG devices are available in R:

["error">root@<YourServerName> R-2.6.1?]# export DISPLAY=:2.0 

["error">root@<YourServerName> R-2.6.1?]# bin/R

You will see many lines of output. At the ">" prompt, run the capabilities() command. It will tell you whether the X11, JPEG and PNG devices are functioning. The following example output shows success:

> capabilities() 

    jpeg png tcltk X11 http/ftp sockets libxml fifo 

    TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE 

  cledit iconv NLS profmem 

    TRUE TRUE TRUE FALSE

Make configuration changes to ensure that Xvfb is started at boot-time

The script:

["error">root@<YourServerName> R-2.6.1?]# cd /etc/init.d 

["error">root@<YourServerName> init.d?]# vi xvfb 

    #!/bin/bash 

    # 

    # /etc/rc.d/init.d/xvfb 

    # 

    # Author: Brian Connolly (LabKey.org) 

    # 

    # chkconfig: 345 98 90 

    # description: Starts Virtual Framebuffer process to enable the  

    # LabKey server to use R. 

    # 

    # 



    XVFB_OUTPUT=/usr/local/labkey/Xvfb.out 

    XVFB=/usr/bin/Xvfb 

    XVFB_OPTIONS=":2 -nolisten tcp -shmem" 



    # Source function library. 

    . /etc/init.d/functions 





    start() { 

            echo -n "Starting : X Virtual Frame Buffer " 

            $XVFB $XVFB_OPTIONS >>$XVFB_OUTPUT 2>&1& 

            RETVAL=$? 

            echo 

            return $RETVAL 

    } 



    stop() { 

            echo -n "Shutting down : X Virtual Frame Buffer" 

            echo 

            killproc Xvfb 

            echo 

            return 0 

    } 



    case "$1" in 

        start) 

            start 

            ;; 

        stop) 

            stop 

            ;; 

        *) 

            echo "Usage: xvfb {start|stop}" 

            exit 1 

            ;; 

    esac 

    exit $?

Now test the script with the standard:

["error">root@<YourServerName> etc?]# /etc/init.d/xvfb start 

["error">root@<YourServerName> etc?]# /etc/init.d/xvfb stop 

["error">root@<YourServerName> etc?]# /etc/init.d/xvfb

Note: Any error messages produced by Xvfb will be sent to the file set in

$XVFB_OUTPUT.

The last thing to do is to run chkconfig to finish off the configuration. This creates the appropriate start and kills links in the rc#.d directories. The script above contains a line in the header comments that says "# chkconfig: 345 98 90". This tells the chkconfig tool that xvfb script should be executed at runlevels 3,4,5. It also specifies the start and stop priority (98 for start and 90 for stop). You should change these appropriately.

["error">root@<YourServerName> init.d?]# chkconfig --add xvfb

["error">root@<YourServerName> init.d?]# chkconfig --list xvfb 

xvfb 0:off 1:off 2:off 3:on 4:on 5:on 6:off

Verify that the appropriate soft links have been created:

["error">root@<YourServerName> init.d?]# ls -la /etc/rc5.d/ | grep xvfb 

lrwxrwxrwx 1 root root 14 2008-01-22 18:05 S98xvfb -> ../init.d/xvfb

Start the Xvfb Process and Setup the DISPLAY Env Variable

["error">root@<YourServerName> init.d?]# /etc/init.d/xvfb start

Now you will need to the set the DISPLAY env variable for the user. This is the DISPLAY variable that is used to run the TOMCAT server. Add the following the .bash_profile for this user. On this serer, the TOMCAT process is run by the user tomcat

["error">root@<YourServerName> ~?]# vi ~tomcat/.bash_profile 

"missing" href="/Documentation/Archive/9.1/wiki-page.view?name=added">added 

# Set DISPLAY variable for using LabKey and R. 

DISPLAY=:2.0 

export DISPLAY

Restart the LabKey Server or it will not have the DISPLAY variable set

["error">root@<YourServerName> ~?]# /etc/init.d/tomcat restart

Test the configuration

The last step is to test that when R is run inside of the LabKey server, the X11,JPEG and PNG devices are available

Example:

The following steps enable R in a folder configured to track Issues:

1. Log into the Labkey Server with an account with Administrator privs
2. In any Project, create a new SubFolder
3. Choose a "Custom"-type folder
4. Uncheck all boxes on the right side of the screen except "Issues."
5. Hit Next
6. Click on the button "Views" and a drop-down will appear
7. Select "Create R View"
8. In the text box, enter "capabilities()" and hit the "Execute Script" button.

jpeg png tcltk X11 http/ftp sockets libxml fifo 

   TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE 

  cledit iconv NLS profmem 

   FALSE TRUE TRUE FALSE 

> proc.time() 

   user system elapsed 

  0.600 0.040 0.631

The important thing to see here is that X11, png and jpeg all say "TRUE." If the do not, something is wrong.

Administrator Setup for R

1. Install R
2. Configure the Study Module to Work with R
3. Extend the Tomcat Session-Timeout Duration
4. Install & Load Additional R Packages
5. Optional: Install the X Virtual Frame Buffer (Headless Unix Servers Only)

Install R

Tips:

• You don’t need to download the “contrib” folder on the Install site. It’s easy to obtain additional R packages individually from within R.
• Details of R installation/admin can be found here.

• Linux. An example of installing R on Linux is included on the Configure R on Linux page.
• Windows. On Windows, install R in a directory whose path does not include a space character. The R FAQ warns to avoid spaces if you are building packages from sources.

Steps to Configure the Study Module to Work with R

It is only necessary for the Admin to configure a Study Module to work with R once.

Navigate to the "Views and Scripting Configuration" page

1. Sign in to your LabKey Server
2. Select "Enable Admin" on the left Nav
3. Expand the "Manage Site" drop-down on the left Nav
4. Click on "Admin Console"
5. Click on “[views and scripting]” under the Configuration suite of options. All further instructions in this section address this page.

If an R engine has not yet been added, click on the "Add" button and select "New R Engine" from the drop-down menu. If an R engine already exists and needs to be configured, select the R engine and then click the "Edit" button instead.

You will then fill in the fields necessary to configure the R scripting engine in the popup dialog box. The final state of the box for the LabKey.org server appears in the screen capture below:

Name. Choose a name for this engine. For example, we call this engine the "R Scripting Engine" on LabKey.org.

Language. Choose "R".

File extensions. These extensions will be associate with this scripting engine. Choose "R,r" to associate the R engine with both uppercase (.R) and lowercase (.r) extensions.

Program Path

Specify the absolute path of the R instance on your LabKey Server. The R Program will be named "R.exe" on Windows, but "R" on Unix and Mac machines.

Program Command

Typically, you will use the default command: "CMD BATCH --slave". The R command is the command used by the LabKey server to execute scripts created in an R view. The default command is sufficient for most cases and usually would not need to be modified.

Output File Name

You can either

• Specify a folder location
• Use the system temporary folder

Enabled

Please click this checkbox to enable the R engine.

Submit

Click "Submit" to change your changes. You will see the following when you have finished:

Permissions

Refer to How Permissions Work for information on how to adjust the permissions necessary to create and edit R Views. Note that only users who are part of the "Developers" site group or have Site Admin permissions can edit R Views.

Note: Batch Mode

Scripts are executed in batch mode, so a new instance of R is started up each time a script is executed. The instance of R is run using the same privileges as the LabKey server, so care must be taken to ensure that security settings (see above) are set accordingly. Packages must be re-loaded at the start of every script because each script is run in a new instance of R.

Increase the Session-Timeout Duration

The Problem

Tomcat’s short session-timeout setting causes problems for users of LabKey R. Users can lose their scripts when sessions time out. If a user edits a script in the Script Builder window for longer than the session-timeout duration, the browser produces a 401 Error when the users presses “Execute Script.” It is not possible to navigate back to the script after renewing login credentials.

The Solution

To increase the session-timeout setting, go to the web.xml file in your Server's Tomcat installation. You’ll need to change the session-timeout variable from 30 (minutes) to a more reasonable working time (e.g., 120 or longer).

<session-config>
  <session-timeout>30</session-timeout> 
</session-config>

You may also wish to warn your users about the timeout settings you select.

Install & Load Additional R Packages

You will likely need additional packages to flesh out functionality that basic install does not include. Additional details on CRAN packages are available here. Packages only need to be installed once on your LabKey Server. However, they will need to be loaded at the start of every script when running in batch mode. (Note: When using RServe instead of batch mode-- still a highly experimental option-- you only need to load packages at the start of the first script you run during a session.)

How to Install

Use the R command line or a script (including a LabKey R script) to install packages. For example, use the following to install two useful packages, "GDD" and "Cairo":

install.packages(c("GDD", "Cairo"), repos="http://cran.r-project.org" )

You can also use the R GUI (Packages->Install Packages) to select and install packages.

How to Load

Each package needs to be installed AND loaded. If the installed package is not set up as part of your native R environment (check ‘R_HOME/site-library’), it needs to be loaded every time you start an R session.

To load an installed package (e.g., Cairo), call:

library(Cairo)

Which Packages You Need

GDD &/or Cairo: If R runs on a headless Unix server, you will likely need at least one extra graphics package. When LabKey R runs on a headless Unix server, it may not have access to the X11 device drivers (and thus fonts) required by the basic graphics functions jpeg() and png(). Installing the Cairo and/or GDD packages will allow your users to output .jpeg and .png formats without using the jpeg() and png() functions. More details on these packages are provided on the Determine Available Graphing Functions page.

You can avoid the use of Cairo and/or GDD by installing a display buffer for your headless server (see below for more info).

Lattice: Optional. This package is the commonly used, sophisticated graphing package for R. It is particularly useful for creating Participant Charts.

Headless Unix Servers Only: Install the X Virtual Frame Buffer

On Unix servers, the png() and jpg() functions use the device drivers provided by the X-windows display system to do rendering. This is a problem on a headless server where there is generally no display running at all.

As a workaround, you can install the X Virtual Frame Buffer. This allows applications to connect to an X Windows server that renders to memory rather than a display.

For instructions on how to install and configure the X Virtual Frame Buffer on Linux, see Configure the Virtual Frame Buffer on Linux.

If you do not install the X Virtual Frame Buffer, your users may need to use graphics packages such as GDD or Cairo to replace the png() and jpeg() functions. See Determine Available Graphing Functions for further details.

Introduction

Please see Single Sign-On Overview for a description of the goals and benefits of Single Sign-On.

LabKey Server can be configured to delegate authentication to OpenSSO. OpenSSO is an open-source project that implements multiple Single Sign-On (SSO) authentication solutions. The high-level goal of SSO is to let users authenticate only once and still gain access to multiple web sites across multiple organizations. As an example, we've used OpenSSO to "federate" authentication between a LabKey Server installation and a web site in a different organization running Microsoft SharePoint -- using OpenSSO, the LabKey Server will accept a user who is logged into SharePoint without requiring another login.

This specific solution used the WS-Federation protocol (used by SharePoint), but OpenSSO implements a variety of other protocols including SAML1.1, SAML 2.0, ID-FF 1.2, and OpenID. LabKey Server communicates with OpenSSO using a standard mechanism. Administrators then configure OpenSSO with appropriate settings and trust relationships. LabKey Server is thereby insulated from the details of the specific protocols or authentication configurations.

The OpenSSO project was created when Sun Microsystems decided to open source two commercial products, Java System Access Manager and Java System Federation Manager. The commercial products are still being sold and supported, but development is happening in the open-source project. The project is new and not well documented. The site has a bewildering number of downloads and broken links. Terminology is inconsistent and confusing. The software is quirky and hard to configure. This guide is an attempt to reduce the clutter to a simple set of steps to get you up and running quickly.

It appears that Sun is merging Access Manager (aka OpenSSO) and Federation Manager (aka OpenFM) into a single product for the upcoming 8.0 release. Looking through the site and documentation you will encounter many product names, but for our purposes, Access Manager, OpenSSO, Federation Manager, and OpenFM are interchangeable terms. Going forward, we'll use "OpenFM" to describe the component we will install and configure.

Install OpenFM

These steps will get OpenFM installed, configured, and talking to LabKey Server. We're assuming Tomcat is installed in c:\tomcat and configured for http://localhost:8080/. If your configuration is different then adapt the instructions below appropriately.

• Install Apache Tomcat 5.5. We've tested this with 5.5.16, but other versions probably work.
• Make sure Tomcat is stopped.
• Download openfm.war. Save it to your c:\tomcat\webapps directory. (This is the 9/28/07 stable release, the latest version from OpenSSO that doesn't crash horribly. Of course, this version can't be found on their web site any more. Also, this WAR file includes two additional classes that work around problems with some ADFS configurations.)
• The Tomcat section of the Release Notes lists the following important steps
• Copy webservices-api.jar (attached to these instructions) to c:/tomcat/common/endorsed directory.
• Increase JVM heapsize by editing catalina.sh. For example, add the following VM option: -Xms256m -Xmx512m
• Increase JVM PermGen setting with a VM option such as: -XX:MaxPermSize=256m
• If you're using IntelliJ to start Tomcat you'll need to put these VM options in the Run/Debug configuration. If you're running OpenFM and LabKey Server on the same Tomcat instance you may want to increase memory further. If you're planning to debug and redeploy you may want to increase PermGen size further.
• Make sure Tomcat will start up pointed at the proper endorsed directory. E.g., -Djava.endorsed.dirs="C:/tomcat/common/endorsed"
• Start Tomcat

Configure OpenFM

• Browse to the Federated Access Manager: http://localhost:8080/openfm
• Click the Enter only the password link under "Simple"
• Enter a password (twice) for the amAdmin user and click "Configure"
• OpenFM will crank for a while. It's creating a bunch of files and directories in your home directory (e.g., c:\Documents and Settings\<username>).
• When it's done, click the Login to the administration console link and login using amAdmin and your password

Configure LabKey Server

• Make sure you have the OpenSSO module installed. This is included in the standard dev build and the dist_chavi build. You can also grab a built version of the module and plunk it in your externalModules directory.
• Visit the Admin Console
• Verify that OpenSSO appears in the list of modules
• Click [authentication]
• Click [configure] next to OpenSSO
• Click Update
• Most of the settings on this page are ignored (blank them out or just leave the comments in place). The ones that seem to be required are:

• Click "Submit". You may need to restart LabKey Server after changing these values; the OpenFM library inside LabKey appears to store these properties in statics, so they can't be changed dynamically.
• On the OpenSSO configure page, click the "Pick a link and logos..." link
• Click "Browse..." and select a page header logo. Do the same for the login page logo. These can be the same or you can customize the logo to the specific purpose (e.g., include text instructions within the image on the login page). You can also omit one or both of these logos. A couple sample logos are attached to this page.
• For URL, enter: http://localhost:8080/openfm/UI/Login?service=adminconsoleservice&goto=%returnURL%
• Using %returnURL% is important. LabKey Server replaces this with a URL to the login page including the current page as a redirect parameter. After OpenFM authenticates the user it will redirect to the login page. Before displaying the login page the login action will verify the OpenFM credentials and immediately redirect to the requested page if valid. Only the login page will check for OpenFM credentials.
• Save
• Done
• Click [enable] next to OpenSSO to enable this authentication provider

Configure OpenFM for Simple Authentication Test

• Create a test user in OpenFM:
• Click the opensso link (under Realm Name heading)
• Click the Subjects tab (far right)
• Click New...
• Enter an email address for user id (e.g., test@opensso.com), fill in names, and enter a password (twice)
• Click OK

• Sign out of LabKey Server and you should see your icon next the Sign In link
• Click the new icon. You should see the Federated Access Manager login page
• Type your test user email address (e.g., test@opensso.com) and password
• You should be redirected back to LabKey Server, logged in as this user. If the user existed in the LabKey user list, you'll be back on the page where you started. If the user didn't exist, you'll be on the update profile page for the newly added user.

Configure OpenFM for WS-Federation

• The OpenFM Tomcat server must be running and accessible using SSL.
• In your home directory, open AMConfig.properties in a text editor and add this line: com.sun.identity.plugin.datastore.class.wsfederation=org.labkey.opensso.LabKeyPassThroughDataStore
• Restart the server (OpenFM will not dynamically reload the properties file)
• Login to the Access Manager as amAdmin
• Configure dynamic profile creation
• Click "Configuration" tab
• Click "Core" link in the Authentication / Service Name list
• Scroll down to "Realm Attributes" and change "User Profile" setting to "Dynamic"
• Scroll down to the bottom or up to the top -> Save
• Click "Back to Configuration" button
• Create and Configure the "Circle of Trust"
• Click "Federation" tab
• Click "New..." button under Circle of Trust
• Name: cot1
• OK
• Click "Import Entity..." under Entity Providers
• Click "Browse..." next to the Standard Metadata Configuration box.
• Navigate to "adfsaccount.xml" using their horrible file browser. Once you get to the file do not double-click it (that will produce an error). Instead, single click it and click "Choose File".
• Click "Browse..." next to the Extended Metadata Configuration box and choose "adfsaccountx.xml". Or, better yet, just copy the path from the first box, paste into the second box, and add an "x".
• Click OK
• Click "Import Entity..." again
• Repeat the import steps for "wsfedsp.xml" and "wsfedspx.xml"
• Click cot1
• Click "Add All >>" to add all the entity providers to this circle of trust
• Save
• Configure LabKey Server for WS-Federation
• Add OpenSSO icons that corresponds to the ADFS server
• Configure the OpenSSO URL to something like: https://dhcp155191.fhcrc.org:8443/openfm/WSFederationServlet/metaAlias/wsfedsp?wreply=%returnURL%
• Enable the provider

Reconfiguring WS-Federation

If you need to reconfigure your OpenFM WS-Federation setup (e.g., when iterating to get the initial configuration correct or when updating an expired token signing certificate) then follow these steps:

• Prepare the new configuration files (adfsaccount.xml, adfsaccountx.xml, wsfedsp.xml, wsfedspx.xml)
• Login to the Access Manager as amAdmin
• Click the "Federation" tab
• Click on your Circle of Trust (e.g., cot1)
• Click "<< Remove All" to remove all the entity providers
• Click Save
• Click Back
• Select the checkbox next to both entity providers
• Click Delete to delete both entity providers
• Click "Import Entity..." and import "adfsaccount.xml" and "adfsaccountx.xml" as discussed above
• Click OK
• Click "Import Entity..." and import "wsfedsp.xml" and "wsfedspx.xml" as discussed above
• Click OK
• Click on your Circle of Trust (e.g., cot1)
• Click "Add All >>" to add both entity providers into the circle of trust
• Save
• Restart Tomcat to update OpenFM with the new configuration

Troubleshooting

• Make sure the OpenSSO configuration has a correct link back to your server (e.g., production vs. test server)
• When testing an auth logo from labkey server, make sure you're starting from an SSL page and the base server URL is set for https://

Inserting a Certificate into adfsaccount.xml

The ADFS token signing certificate must be inserted into adfsaccount.xml in base-64 encoded X.509 ASCII format (also called PEM format). Use one of these steps to convert a .cer binary file into this format:

On Windows XP

• Open the certificate (no need to install it)
• Click the "Details" tab
• Click "Copy to File..." to start the certificate export wizard
• Click "Next >"
• Select "Base-64 encoded X.509"
• Click "Next >"
• Enter a filename (e.g., c:mycert). Note that the wizard insists on adding .cer to the end of your filename.
• Click "Next >"
• Click "Finish"

openssl x509 -in mycert.cer -inform DER -out mycert.pem -outform PEM

Open the converted/exported certificate and copy everything between (but not including) "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" to the <ns2:X509Certificate> tag in adfsaccount.xml.

Configure a Referrer URL Prefix (optional)

Configuring a Referrer URL Prefix, combined with coordinated permissions settings, can make federated login with a partner site nearly transparent to the user. Say, for example, that you're operating an instance of LabKey Server and you have a partner site that runs Microsoft SharePoint authenticating against ADFS. Configuring a Referrer URL Prefix lets your partner include links to protected content on your LabKey Server that their authenticated users can access without explicitly logging in.

Your partner site must have a URL prefix that indicates the user is logged in (e.g., http://protected.foo.org or http://foo.org/protected). Specify this URL prefix on the Referrer URL Prefix settings page. When a SharePoint user who is not logged into LabKey clicks a link to protected content on LabKey Server, LabKey checks the referring URL, see that it starts with the specified prefix, and automatically redirects the user to the OpenFM URL. This should cause ADFS to pass the credentials to OpenFM and OpenFM to pass credentials to LabKey, resulting in transparent authentication.

Authentication is, of course, not sufficient to view protected content -- the user must have the appropriate permissions in the destination page. In any federated authentication environment it's important to coordinate permissions so (in this example) the SharePoint users have appropriate permissions on LabKey before they attempt to visit the protected content. If permissions are not set ahead of time then users will receive "User does not have permission" error messages when they attempt to follow the links from the partner site.

1. Visit the OpenSSO site at https://opensso.dev.java.net/
2. Click the "Downloads" link in the middle of the page (not the Downloads link on the left)
3. Download the Stable Build: "OpenSSO V1 Build 1 Zip" dated 9/28/07. There are more recent builds (e.g., under "Periodic Builds") but these have crashed spectacularly when tried; stick with the stable build. This is a large (181MB) file and their server is slow, so go get a cup of coffee while it downloads.

• Customize site and/or project "Look and Feel" settings
• Establish Terms of Use for Project

• Create a Project or Folder
• Customize a Project or Folder. Select Applications, Modules and Tabs, plus set the Portal Page.
• Set Permissions
• Manage Project Members
• Navigate Folder Hierarchy
• Move/Rename/Delete/Hide Folders
• Access Module Services
• Add Web Parts to add tools to the Portal Page.
• Manage Web Parts
• Look & Feel Settings. These include style sheet settings that can be applied at the project and/or site level.

In general, a project corresponds to an area of work. For example, you might create a project for each study laboratory that is collaborating on your research. You can structure your project however you wish.

These tailored workspaces leverage LabKey's rich suite of tools and services. Admins can set up folders to present their teams with precisely the subset of LabKey tools needed-- no more, no less. Any project or folder can include any LabKey module and any web parts. For example, when properly customized, any folder can display a Wiki page, a message board and the results of an MS2 run.

Projects and their folders show up in the left navigation page. You can click on their links in the navigation pane or in the breadcrumb links at the top of the page to move up or down in the hierarchy (See Navigate Folder Hierarchy).

The LabKey security model allows you to secure projects and folders and strictly control which users can access which parts.

The Home Project

The Home project is a special project on the LabKey site. You can add folders to it, but it can't be deleted, moved, or renamed. The Home project is always visible, regardless of which other project you are working in.

When you first install LabKey, the Portal page for the Home project includes a Wiki module. The wiki page that's displayed as part of the Portal page includes some welcome text. You can keep this text, modify or delete the wiki page, or delete the wiki web part altogether.

By default, the Home project's Portal page can be viewed by users who are not logged in. To change this, modify the security settings for the Home project.

The Portal Page

By default, projects and folders have an associated Portal page which is loaded when you click on the name of a project or folder in the navigation hierarchy. The Portal page is designated by the "Portal" tab that appears on the top of the page; when this tab is selected, you know that you are working in the portal.

You can choose not to display the Portal page by selecting Manage Project->Customize Folder, setting the folder type to Custom, and clearing the Portal check box. You can also choose to make a different page the default page for the project or folder.

The Portal page displays web parts for any modules that you add to the project or folder. Web parts are portal components -- that is, they are ready-made components that you can add to the Portal page to display data that's stored in a module. Web parts only appear in the Portal page. See Add Web Parts for details on adding Web Parts.

If you want to add text to the Portal page for a project or folder -- for example, to describe the project to users -- you should add a wiki if the project or folder doesn't already contain one.

Module Tabs

Depending on what type of project or folder you create, you'll see different tabs in the tab navigation area. Folders of type Collaboration, MS2, and Study show those names in the tab area.

If you create a custom folder, you'll see a tab for each module that you've elected to display for that folder. When you add a web part to the Portal page for a project or folder, the module that's represented by that web part is now available for you to use in that project or folder. The web part shows module data in the Portal page; you can also work directly with the module by clicking on the tab that's associated with it at the top of the page, or by clicking a link in the web part that takes you to the module.

You can add tabs by Customizing a Folder.

Create a New Project

Enter a name for the new project. If you wish to hide the new project or folder from non-admins, see Hidden Folders for naming conventions.

Specify which Type of project you want to create. You can choose "Custom" as the type or select a type that corresponds to one of the LabKey Applications. Your choices:

• Collaboration. Build a web site for publishing and exchanging information. Your tools include Message Boards, Issue Trackers and Wikis. Share information within your own group, across groups or with the public by configuring user permissions.
• Flow. Perform statistical analysis and create graphs for high-volume, highly standardized flow experiments. Organize, archive and track statistics and keywords for FlowJo experiments.
• MS1. Combine MS1 quantitation results with MS2 data.
• MS2. Manage tandem mass spectrometry analyses using a variety of popular search engines, including Mascot, Sequest, and X-Tandem. Use existing analytic tools like PeptideProphet and ProteinProphet.
• Study. Manage human and animal studies involving long-term observations at distributed sites. Use specimen tracking for samples. Design and manage specialized assays. Analyze, visualize and share results.
• Custom. Create a tab for each LabKey module you select. Used in older LabKey installations. Note that any LabKey module can also be used from any folder type via Customize Folder. For further details, see Reasons to Choose a "Custom"-Type Folder.

Create a New Folder

To add a folder to a project, select the project in the left navigation pane and click Manage Folders beneath Manage Project in the left navigation pane. Then click Create Subfolder to create a new folder. You can also rename, move, and delete projects and folders from this page (for more information, see Move/Rename/Delete/Hide).

You will have the same options for folder types as you did for project types. See the bullets in the previous section for your options.

Set Permissions

Default, admin-only permissions have one exception. In the case where the folder admin is not a project or site admin, permissions are inherited from the parent project/folder. This avoids locking the folder creator out of his/her own new folder. Make sure to check that these permissions are appropriate. If they are not, Set Permissions.

Customize a Project or Folder to Add Modules

For example, if an admin creates a separate folder to hold source data displayed in multiple end-user folders, the admin may wish to hide this source data folder. The material (e.g., a list) in a hidden folder is then only visible to users in the folders where it is used.

Create a Hidden Folder.

Folders whose names begin with "." or "_" are automatically hidden from non-admins in the navigation tree.

Note that the folder will still be visible in the navigation tree if it has non-hidden subfolders (i.e., folders where the user has read permissions). If an admin wishes to hide subfolders of a hidden folder, he/she can prefix the names of these subfolders with a dot or underscore as well.

Hiding a folder only affects its visibility in the navigation tree, not permissions to the folder. So if user is linked to the folder or enters the URL directly, the user will be able to see and use the folder.

View Hidden Folders.

You can use the "Show Admin" / "Hide Admin" toggle to show the effect of hiding folders from the perspective of a non-admin.

Steps:

1. Select "Manage Project" and then "Customize Folder" from the left navigation panel. Note that you must Enable Admin to do so.
2. On the "Customize Folder" page, change any of the following items (all detailed in later sections):
1. Folder Type
2. Module (Tab) Checkboxes
3. Portal Page
3. When you are done customizing the folder, click "Update Folder."

Folder Type

• A LabKey Application. If you choose of the LabKey Applications (Collaboration, Flow, MS1, MS2 or Study), a suite of modules is selected for you. You can still add more modules to your folder using the module checkboxes. Only checked modules make their web parts available for inclusion on your portal page. To see which modules provide which web parts, see the Application & Module Inventory.
• A "Custom" Application. If you choose "Custom," all modules are automatically included. Checkboxes allow you to select which modules appear on tabs in the UI. A "Custom"-type folder makes all web parts are available for inclusion on your portal page regardless of which modules are selected for display as tabs.

Module (Tab) Checkboxes

For Custom-Type Folders. If your folder type is "Custom," the module checkboxes let you choose which modules display as tabs in your folder. Checkboxes do not affect module availability within the folder. Unlike Application-Type folders, all web parts are always available for inclusion on the portal page of a custom portal. The selection of modules to display as tabs does not influence the availability of web parts. See also Reasons to Choose a "Custom"-Type Folder.

Change the portal page

A "Custom"-type folder automatically has access to all LabKey Web Parts. Thus, you do not need to know which modules provide which web parts (and add these modules to your folder before desired web parts become available). In Custom-type folders, all possible web parts are always available in the Add Web Part drop-down menus. Other folder types (Study, MS2, Flow and Collab) provide access to only a subset of all web parts.

2) You want tabs for each module to be displayed in your folder.

You may desire a Custom Folder if you wish to set up separate tabs for different modules. Typically, however, LabKey encourages you to prefer web parts over tabs when possible. Some functionality is only available through tabs, but will become available through web parts in the future. You can have a custom folder (and thus access to the full suite of web parts) without adding a bunch of tabs.

After you click Create New Project, you'll be directed to the Permissions page, where you can create groups of users and assign permissions to them. You'll need to Set Permissions. For additional information on users and groups Security.

Permission Management

By setting Permissions on a Project or Folder, you secure the Project or Folder against unauthorized access.

To set permissions for a project or folder, select the project or folder in the left navigation page, then click "Permissions" under "Manage Project" in the left navigation page. From the "Permissions" page you can assign users to groups and then set permissions for each group for the selected project or folder. You can also create new custom security groups if you like. For more information, see Project Groups.

By default, only site administrators have admin privileges on a project. To grant admin privileges on a project to a user who is not a site administrator, you can do one of two things. You can add the user to the Administrators group for that project, then make sure that the permissions for the Administrators group are set to Admin (All Permissions), as it is by default. This is a straightforward way to grant administrative access to one or a few users. Alternately, you can set the permissions for another group to Admin (All Permissions), so that all members of that group will have administrative permissions. For more information, see Configuring Permissions.

You should also consider whether anonymous users (or Guests) should have access your project or folder, and set permissions for the Guests group accordingly.

Project Member Management for Project Administrators

The "Project Members" page allows project administrators to manage project members at the project level without access to site-level user pages.

Site admins can manage users across the site via the "Site Users" page. For this option, see: Manage Users

Project Member List

On the Manage Project -> Project Members page, project admins can view and export a list of all project members, plus view the full user event history for all project members. The project members page looks and works like Manage Site->Site Users, which is described on the Manage Users page.

A project member is defined as any user who is a member of any group within the project. Note that there may be users who have permissions to a project but are not project members (e.g., site admins or users who have permissions because of a site group). Likewise, a project member may not actually have any permissions within a project (e.g., the group they belong to has not been granted any permissions).

View/Edit Project Member Details

On the Manage Project -> Project Members page, project admins can view (but not modify) each project member's details: profile, user event history, permissions tree within the project, and group events within the project.

Impersonate Project Members

Project admins can impersonate project members within the project, allowing the admin to view the project just as the member sees it. While impersonating, the admin can not navigate to any other project (including the Home project). Impersonation is available on the "Project Members" and "Permissions" pages that can be reached via the "Manage Project" menu in the left-hand navigation bar.

When working in a project, you will see the project listed at the top of the folder list in the left navigation pane titled "Project Folders." Folders within this project that are visible to you are listed beneath it. The folder where you are working is highlighted in bold in the list of folders.

To switch to a new folder, click on the name of the folder in the list.

Expand/Collapse SubFolder Hierarchies

Some folder or sub-folder hierarchies may be displayed as collapsed, as indicated by a "+" sign to the left such a folder hierarchy. Expand such a folder hierarchy by clicking on the "+" sign to see all the folders it contains. Collapse an expanded hierarchy by clicking on the "-" that appears next to it.

Navigate Between Projects

Other projects on your LabKey Server are not visible in the top left pane. They are visible in the second pane on this left-hand side bar.

To switch projects, select the name of the desired project from the "Project" list in the second pane of the left navigation bar. All projects available to you appear in this list.